If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Clinical Evaluation of the Automatic Coronary Artery Disease Reporting and Data System (CAD-RADS) in Coronary Computed Tomography Angiography Using Convolutional Neural Networks
The coronary artery disease reporting and data system (CAD-RADS™) was recently introduced to standardise reporting. We aimed to evaluate the utility of an automatic postprocessing and reporting system based on CAD-RADS™ in suspected coronary artery disease (CAD) patients.
Materials and Methods
Clinical evaluation was performed in 346 patients who underwent coronary computed tomography angiography (CCTA). We compared deep learning (DL)-based CCTA with human readers for evaluation of CAD-RADS™ with commercially-available automated segmentation and manual postprocessing in a retrospective validation cohort.
Results
Compared with invasive coronary angiography, the sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the DL model for diagnosis of CAD were 79.02%, 86.52%, 89.50%, 73.94%, and 82.08%, respectively. There was no significant difference between the DL-based and the reader-based CAD-RADS™ grading of CCTA results. Consistency testing showed that the Kappa value between the model and the readers was 0.775 (95% confidence interval [CI]: 0.728–0.823, p < 0.001), 0.802 (95% CI: 0.756–0.847, p < 0.001), and 0.796 (95% CI: 0.750–0.843, p < 0.001), respectively. This system reduces the time taken from 14.97 ± 1.80 min to 5.02 ± 0.8 min (p < 0.001).
Conclusion
The standardised reporting of DL-based CAD-RADS™ in CCTA can accurately and rapidly evaluate suspected CAD patients, and has good consistency with grading by radiologists.
Rivaroxaban with or without aspirin in patients with stable coronary artery disease: an international, randomised, double-blind, placebo-controlled trial.
). Coronary computed tomography angiography (CCTA) is increasingly being used as a noninvasive evaluation method to assess patients with chest pain because it has high sensitivity and specificity for the detection of CAD (
). Moreover, unlike functional stress testing, CCTA provides an anatomical assessment of the coronary arteries, allowing for the detection and quantification of obstructive and non-obstructive atherosclerotic plaques (
CAD-RADS(TM) Coronary Artery Disease - Reporting and Data System. An expert consensus document of the Society of Cardiovascular Computed Tomography (SCCT), the American College of Radiology (ACR) and the North American Society for Cardiovascular Imaging (NASCI). Endorsed by the American College of Cardiology.
). Unlike prior classifications of CAD that were based on the number of affected vessels, CAD-RADS™ classifies patients based on the highest grade of coronary artery stenosis. Xie et al. recently reported that CAD-RADS™ effectively identified patients at risk of myocardial infarctions and all-cause mortality to a similar degree as the CAD extent classification and the Duke CAD Index (
The Coronary Artery Disease-Reporting and Data System (CAD-RADS): prognostic and clinical implications associated with standardized coronary computed tomography angiography reporting.
). Deep convolutional neural networks (CNN) are a newly-emerging form of computer-aided diagnostic analysis that allows the automatic extraction of features and supervised learning of large amounts of data to make quantitative decisions (
). Accumulated evidence suggests that deep learning (DL) analysis might be a potential alternative to conventional hand-crafted methodologies for solving pattern-recognition and imaging-classification problems (
). To date, there has been no research focusing on the combination of CNN and CCTA to solve lesion detection and classification problems based on CAD-RADS™.
Thus, we aimed to assess the utility of an automatic post-processing and reporting system based on CAD-RADS™ in CAD or suspected CAD patients. We also sought to investigate the performance of DL-based CCTA to assist radiologists in their daily work and establish a time-saving work process.
MATERIALS AND METHODS
The present study was approved by the local institutional review board and was conducted in compliance with the Health Insurance Portability and Accountability Act of 1996.
Patients
We retrospectively searched a database of CCTA data from patients who were examined between July 2017 and December 2019. The study population consisted of 346 consecutive patients who were confirmed or suspected to have CAD (Fig 1). Scans affected by stair-step artifacts, respiratory motion artifacts, excessive image noise, or artifacts caused by arrhythmia, or poor quality images were excluded from analysis. We also excluded patients without complete records and those who previously underwent percutaneous coronary intervention or coronary artery bypass grafting or other cardio surgery from analysis.
Figure 1Flowchart illustrating exclusion criteria and final study population.
Multidetector row computed tomography (CT) imaging was performed with a dual-source CT scanner (Somatom Definition, Siemens Medical Solutions, Forchheim, Germany) and a 256-slice CT scanner (Brilliance iCT; Philips Healthcare, Cleveland, OH). Heart rate (HR) control (≥ 65 beats/min) was performed with beta-blockers before the scan. For contrast enhancement, 60–80 mL of iopromide (370 mgI/mL, Bayer Schering Pharma, Leverkusen, Germany) followed by 30–40 mL of pure saline was administered with a flow rate of 4–5 mL/s. The iodine contrast agent was automatically triggered into the descending aorta of 100 Hounsfield units threshold units. Then the scanning was performed during an inspiratory breath hold of 8–14 s after a delay of 2 s. The reconstructed images were automatically sent to a workstation (CoronaryDoc, Shukun technology, Beijing, China) equipped with a coronary analysis software tool (Computer Aided Diagnosis of Coronary Artery, Version 1.8, Shukun technology).
Deep Learning (DL)
The patients in the test set were examined by invasive coronary angiography (ICA) with an interval of fewer than 30 days after the CCTA procedure. We have previously reported the validation of our DL system (
The correlation of deep learning-based CAD-RADS evaluated by coronary computed tomography angiography with breast arterial calcification on mammography.
). Before training, the aorta, coronary artery and plaques were labeled on each image using a multi-layer manual annotation system consisting of multiple layers of trained graders. The first layer of graders consisted of radiologists who had knowledge of medical imaging and coronary anatomy. The second layer of graders comprised radiologists with more than 3 years of work experience in radiology, who performed a preliminary inspection of the accuracy of the label. The third and final layer of graders was made up of experienced experts with over 5 years of work experience who verified the correctness of the label of each image. In this study, we adopted an improved three-dimensional (3D) U-Net architecture and added a bottle-neck model for segmentation of coronary arteries and aorta, then a growing iterative prediction network model was developed to solve the problem of vascular segmentation fracture, and finally full coronary tree segmentation was obtained (
). Based on coronary tree segmentation, multiple planar reformat (MPR), curve planar reformat (CPR), maximum intensity projection, and volume rendering images were reconstructed. To detect stenosis, we developed a 3D segmentation neural network and a one-dimensional sequence checking hybrid technique (
Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 2015.
). Firstly, a 3D segmentation neural network was applied to MPR and CPR images to detect stenosis, and then a one-dimensional sequence-checking algorithm was used to reduce false-positive results (Fig 2).
Figure 2Flowchart of the process of the model; convolutional neural networks (CNN) and deep learning (DL)-based coronary computed tomography angiography (CCTA) assistant diagnosis system. (a) Diagram showing complete workflow of the aorta, coronary artery and plaque labelling and segmentation network. Each blue box corresponds to a convolutional layer, with matrix size and depth indicated at each layer. Colored arrows indicate different operations: yellow = convolution and rectified linear unit (ReLU), orange = copy and concatenate from down-sampling path to up-sampling path, green = maximum pooling, blue = up-convolution and purple = convolution. The input was images from CCTA acquisition, whereas the output was the aorta, coronary artery and plaque contours if these are detected on the image. (b) CNN networks for coronary artery disease reporting and data system (CAD-RADS™) classification. Plaque segmentation was applied by a 3D UNET network on a curve planar reformat (CPR) image and a straightened image respectively, and then the output results were modified and classified by a one-dimensional back-detection model. Finally, the CAD-RADS™ classification output was determined according to the degree of stenosis. C&D, DL-based CCTA assistant diagnostic system. (Color version of figure is available online.)
Three readers with 5 years of experience in cardiac CT imaging diagnosis recorded the CAD-RADS™ classification based on the degree of coronary stenosis (CAD-RADS 0: 0%, CAD-RADS 1: 1–24%, CAD-RADS 2: 25–49%, CAD-RADS 3: 50–69%, CAD-RADS 4A: 70–99%, CAD-RADS 4B: Left main > 50% or 3-vessel disease, 70–99%, CAD-RADS 5: 100%) according to the CAD-RADS™ consensus document (
CAD-RADS(TM) Coronary Artery Disease - Reporting and Data System. An expert consensus document of the Society of Cardiovascular Computed Tomography (SCCT), the American College of Radiology (ACR) and the North American Society for Cardiovascular Imaging (NASCI). Endorsed by the American College of Cardiology.
). A structured report including CAD-RADS™ category was created based on the model independently. All the readers and the model provided CAD-RADS™ scores independently, which remained hidden to each. Coronary artery lesions were evaluated for stenosis on a diameter-based approach. The readers, who were blinded to the results of each other and to the model, manually measured the relevant lesions independently. One hundred patients were randomly selected for time consumption (including postprocessing, report-writing, typesetting and print) analysis. CAD was defined as stenosis > 50% in a coronary artery segment ≥ 2mm in diameter.
Statistical Analysis
Continuous variables are presented as mean ± SD. Categorical variables are presented as percentages or absolute values. We used either the chi-square test, or Fisher's exact test as appropriate for categorical variables. The diagnostic performance of CAD with DL-based CCTA was determined with regard to sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy by comparison with ICA as the standard of reference. The agreement of the CAD-RADS™ categories was compared between readers and the model using the linear weighted kappa identity test. p < 0.05 was considered to represent statistical significance. All statistical analysis was performed using SPSS version 18 (SPSS, Inc., Chicago, IL, USA) and MedCalc Statistical Software version 16.8.4.0 (Ostend, Belgium).
RESULTS
Table 1 shows the baseline characteristics of the 346 study participants (mean age: 62.9 years; range from 33 to 85 years). Of the 346 patients, 181 (52.3%), 181 (52.3%), 180 (52.0%), and 180 (52.0%) were diagnosed with CAD by the DL model, Reader 1, Reader 2, and Reader 3, respectively. There were no significant differences between readers and the DL model in the detection of CAD. The PPV, NPV, sensitivity, specificity, and accuracy of the DL model for diagnosis of CAD were 79.02%, 86.52%, 89.50%, 73.94%, and 82.08%, respectively when using ICA as the reference standard (Table 2). The area under the receiver operating characteristic curve (AUC) of the DL model for CAD diagnosis was 0.83, which was similar to those of Reader 1 (0.84), Reader 2 (0.82) and Reader 3 (0.84) (all p > 0.05, Fig 3). In addition, the diagnostic performance of CAD between readers was similar (all p > 0.05, Fig 3).
Table 1Basic Characteristics of the Included Patients
Characteristics
Total (N = 346)
Age
62.9 ± 9.1
Male, no. (%)
191 (55)
BMI (kg/m2)
25.2 ± 7.3
Hypertension
210 (60.7)
DM
93 (26.9)
CKD
37 (10.7)
TG
2.0 ± 1.7
TC
4.7 ± 1.1
HDL-C
1.2 ± 0.3
LDL-C
2.8 ± 0.9
SCR
71.4 ± 38
BUN
5.6 ± 1.9
EF (%)
59.6 ± 5.8
BMI, body mass index; BUN, blood urea nitrogen; CKD, chronic kidney disease; LDL-C, low density lipoprotein-C; DM, diabetes mellitus; EF, ejection fraction; HDL-C, high density lipoprotein-C; SCR, serum creatinine; TC, total cholesterol; TG, triglyceride.
Figure 3Receiver operating characteristic (ROC) curves for coronary artery disease (CAD) diagnosis by DL and Readers. (Color version of figure is available online.)
The distribution classification of CAD-RADS™ assessed by the DL model was 38 as CAD-RADS 0, 19 as CAD-RADS 1, 108 as CAD-RADS 2, 90 as CAD-RADS 3, 58 as CAD-RADS 4A, 24 as CAD-RADS 4B and 9 as CAD-RADS 5 (Fig 4). As shown in Table 3, the agreement between the DL model and Reader 1 for CAD-RADS™ classification was good (Kappa = 0.775; 95% confidence interval [CI]: 0.782–0.823). The agreement between the DL model and Reader 2 working in consensus was excellent (Kappa = 0.802; 95% CI: 0.756–0.847). The agreement between the DL model and Reader 3 was also good (Kappa = 0.796; 95% CI: 0.750–0.843). Moreover, the agreement between readers for CAD-RADS™ showed excellent results (Supplementary file; Table S1).
Figure 4The distribution classification of CAD-RADS by DL and Readers. (Color version of figure is available online.)
In comparison with ICA, a total of 62 patients were misclassified by the DL model. The DL model underestimated 43 patients with CAD including five patients with a DL-based CAD-RADS 1 classification and 38 patients with a DL-based CAD-RADS 2 classification. In contrast, the DL model overestimated 19 patients without CAD who were classified as CAD-RADS 2. Moreover, the clinical workup would be changed in 60 (17.3%), 46 (13.3%), and 49 (14.2%) patients if the DL model alone was used when compared with Reader 1, Reader 2, And Reader 3, respectively. In addition, the clinical workup of 62 (17.9%), 58 (16.8%), 65 (18.8%), and 59 (17.1%) patients would be changed if using the DL model, Reader 1, Reader 2, And Reader 3, respectively when compared with ICA.
The time taken (including postprocessing, report-writing, typesetting, and printing) of the three readers was 14.51 ± 1.92 min, 15.28 ± 1.64 min and 15.11 ± 1.83 min, respectively. In comparison, the average time taken by DL was reduced from 14.97 ± 1.80 min to 5.02 ± 0.8 min (p < 0.001, Fig 5). We also found that the time taken differed between manually-recognised CAD patients and non-CAD patients, while there was no significant difference when DL was used.
Figure 5The comparison of consumed time between the DL and Readers.
In this study, we first developed a novel CNN model using CCTA images and evaluated its performance in categorization of coronary artery stenosis based on CAD-RADS™. Then, the utilisation of an automatic post-processing and reporting system based on CAD-RADS was assessed in suspected CAD patients.
As a standardised reporting system of CCTA, the primary aim of CAD-RADS™ is to facilitate consistent assessment among physicians, including recommendations for further investigations and management (
CAD-RADS(TM) Coronary Artery Disease - Reporting and Data System. An expert consensus document of the Society of Cardiovascular Computed Tomography (SCCT), the American College of Radiology (ACR) and the North American Society for Cardiovascular Imaging (NASCI). Endorsed by the American College of Cardiology.
). Moreover, the common language of coronary artery stenosis provided by this reporting system increases the clarity between radiologists and clinicians in the diagnosis and treatment planning of suspected CAD patients. In addition, the achieved standardisation of reporting will be of benefit in education, research, peer review, and quality assurance, ultimately resulting in improved quality of care (
Cancer imaging standardised reporting systems, such as Lung-RADS and BI-RADS have been widely accepted by offering reliable and consistent assessment categories to radiologists and physicians who can then supply appropriate management recommendations for specific patients. Structured reporting tools have been applied in cardiac imaging to improve imaging data integrity and then establish standard databases for education, patient care and research purposes (
). However, no machine learning or DL models have been used in these studies.
Recently, promising applications of machine learning or artificial intelligence (AI) have been used in cardiovascular imaging and cardiology to process several algorithms that are already in routine clinical use (
). These algorithms are likely to provide a process to improve the quality and speed of acquisition, reduce manual postprocessing, measurement and manual report writing time, and allow prompt diagnoses, which in turn would improve workflow and patient care. Moreover, these could facilitate screening programs and promote establishment of more effective referral mechanisms in medicine, particularly in remote or low-resource areas, leading to a broad, positive clinical, and public health impact (
). In addition, they encourage researchers to improve the performance of future models and help drive the field forward.
Machine learning algorithms have been extensively used for the optimisation of information extraction from CCTA. A two-step support vector machine based on learning with CCTA was established by Kang et al. (
) for detecting nonobstructive and obstructive CAD, and resulted in an accuracy of 94% and an AUC of 0.94 and thus had a higher accuracy than obtained in the present study. However, only two classifications were used in the study and obstructive CAD was defined as lesion with stenosis ≥25%. However in our study the CAD-RADS classification system was applied and we used the widely-accepted definition of CAD (lesion with stenosis ≥50%). Moreover, a small CCTA dataset with only 42 patients were included in the study by Kang et al., while our study included a relatively larger sample size of 346 patients. In addition, our DL platform was trained and validated using the Digital Imaging and Communications in Medicine (DICOM) standards, which makes CCTA images from different manufacturers (dual-source CT and Philips ICT) reasonably consistent. Finally, our study developed a DL model which was able to automatically extract coronary arteries and detect stenosis.
Accumulated researches revealed good automatic detection and quantification with CCTA images. Schuhbaeck et al. evaluated the interscan reproducibility of quantitative measurements of coronary plaque volumes using a standardised automated method (
). A recent study showed that CCTA-derived plaque markers combined with a DL-based CT-FFR had a good predictive value in identifying lesion-specific ischaemia when compared to CCTA stenosis grading alone (
Coronary CT angiography-derived plaque quantification with artificial intelligence CT fractional flow reserve for the identification of lesion-specific ischemia.
). A previous study demonstrated that a machine learning algorithm-based CT-FFR value was able to assess the functional significance of CAD and make a therapeutic decision. This algorithm showed excellent performance (sensitivity of 97%, specificity of 100%, PPV 100%, NPV of 97%, and accuracy of 99%) of ML-based CT-FFR with CCTA in determining an appropriate treatment strategy (
). However, few studies have focussed on DL-based CCTA using CAD-RADS™ for stratifying the risk of suspected CAD patients. The present study was designed to evaluate the feasibility of CAD-RADS™ classifications by using DL based on CCTA images (Fig 6).
Figure 6DL-based CAD-RADS™ classification. (a and b) CAD-RADS 0; (c) CAD-RADS 1; (d) CAD-RADS 2; (e) CAD-RADS 3; (f) CAD-RADS 4A; (g) CAD-RADS 4B; (h) CAD-RADS 5. (Color version of figure is available online.)
Some limitations of this study should be addressed. First, this was a retrospective analysis of a relatively small sample size from a single centre. Prospective multicentre studies of large samples will enhance the application of CAD-RADS™ among cardiologists and radiologists. Second, in order to restrict our analysis, we did not include CAD-RADS™ modifiers to describe patients with stents (modifier S), vulnerable plaque features (modifier V), or grafts (modifier G). Further studies should include more patients in the training set and testing set of DL to improve the CAD-RADS™ classification scheme. Third, the DL model slightly underestimated CAD-RADS 4A and CAD-RADS 5 patients compared to the readers as shown in Figure 4. As a result, this would impact clinical decision-making as the consideration of ICA would be reduced in a portion of patients that might need them. It should be noted that the sensitivity of the DL model for diagnosing CAD is 79%, which is lower than previous studies with a sensitivity of 84–94% using CCTA (
). Further improvement of this model is still needed to reduce the possibility of underdiagnosing CAD. In addition, visual evaluation by ICA may overestimate stenosis grade, as has been demonstrated in previous research (
Comparison of clinical interpretation with visual assessment and quantitative coronary angiography in patients undergoing percutaneous coronary intervention in contemporary practice: the Assessing Angiography (A2) project.
The standardised reporting of DL-based CAD-RADS™ in CCTA images can accurately evaluate suspected CAD patients while saving time, and has excellent consistency with radiologists.
ACKNOWLEDGMENTS
None.
FUNDING
Not applicable.
ETHICS APPROVAL AND CONSENT TO PARTICIPATE
The research was approved by the Institutional Review Board of the Clinical Research Institute at The Central Hospital of Wuhan.
Rivaroxaban with or without aspirin in patients with stable coronary artery disease: an international, randomised, double-blind, placebo-controlled trial.
CAD-RADS(TM) Coronary Artery Disease - Reporting and Data System. An expert consensus document of the Society of Cardiovascular Computed Tomography (SCCT), the American College of Radiology (ACR) and the North American Society for Cardiovascular Imaging (NASCI). Endorsed by the American College of Cardiology.
The Coronary Artery Disease-Reporting and Data System (CAD-RADS): prognostic and clinical implications associated with standardized coronary computed tomography angiography reporting.
The correlation of deep learning-based CAD-RADS evaluated by coronary computed tomography angiography with breast arterial calcification on mammography.
Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 2015.
Coronary CT angiography-derived plaque quantification with artificial intelligence CT fractional flow reserve for the identification of lesion-specific ischemia.
Comparison of clinical interpretation with visual assessment and quantitative coronary angiography in patients undergoing percutaneous coronary intervention in contemporary practice: the Assessing Angiography (A2) project.
DL vs. Reader 1 (p = 0.960).
00318-X&id=gr1.jpg){kind=link}
00318-X&id=gr2.jpg){kind=link}
00318-X&id=gr3.jpg){kind=link}
00318-X&id=gr4.jpg){kind=link}
00318-X&id=gr5.jpg){kind=link}
00318-X&id=gr6.jpg){kind=link}