Advertisement

Multiparametric Data-driven Imaging Markers: Guidelines for Development, Application and Reporting of Model Outputs in Radiomics

Published:November 18, 2022DOI:https://doi.org/10.1016/j.acra.2022.10.001
      This paper is the fifth in a five-part series on statistical methodology for performance assessment of multi-parametric quantitative imaging biomarkers (mpQIBs) for radiomic analysis. Radiomics is the process of extracting visually imperceptible features from radiographic medical images using data-driven algorithms. We refer to the radiomic features as data-driven imaging markers (DIMs), which are quantitative measures discovered under a data-driven framework from images beyond visual recognition but evident as patterns of disease processes irrespective of whether or not ground truth exists for the true value of the DIM. This paper aims to set guidelines on how to build machine learning models using DIMs in radiomics and to apply and report them appropriately. We provide a list of recommendations, named RANDAM (an abbreviation of “Radiomic ANalysis and DAta Modeling”), for analysis, modeling, and reporting in a radiomic study to make machine learning analyses in radiomics more reproducible. RANDAM contains five main components to use in reporting radiomics studies: design, data preparation, data analysis and modeling, reporting, and material availability. Real case studies in lung cancer research are presented along with simulation studies to compare different feature selection methods and several validation strategies.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Sullivan DC
        • Obuchowski NA
        • Kessler LG
        • et al.
        Metrology standards for quantitative imaging biomarkers.
        Radiology. 2015; 277: 813-825https://doi.org/10.1148/radiol.2015142202
        • Raunig DL
        • McShane LM
        • Pennello G
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment.
        Stat Methods Med Res. 2015; 24: 27-67https://doi.org/10.1177/0962280214537344
        • Obuchowski NA
        • Reeves AP
        • Huang EP
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons.
        Stat Methods Med Res. 2015; 24: 68-106https://doi.org/10.1177/0962280214537390
        • Kessler LG
        • Barnhart HX
        • Buckler AJ
        • et al.
        The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions.
        Stat Methods Med Res. 2015; 24: 9-26https://doi.org/10.1177/0962280214537333
        • Gillies RJ
        • Kinahan PE
        • Hricak H.
        Radiomics: images are more than pictures, they are data.
        Radiology. 2016; 278: 563-577https://doi.org/10.1148/radiol.2015151169
        • Aerts HJ.
        The potential of radiomic-based phenotyping in precision medicine: a review.
        JAMA oncology. 2016; 2: 1636-1642
        • Lambin P
        • Leijenaar RTH
        • Deist TM
        • et al.
        Radiomics: the bridge between medical imaging and personalized medicine.
        Nat Rev Clin Oncol. 2017; 14: 749-762https://doi.org/10.1038/nrclinonc.2017.141
        • Lambin P
        • Rios-Velazquez E
        • Leijenaar R
        • et al.
        Radiomics: extracting more information from medical images using advanced feature analysis.
        Eur J Cancer. 2012; 48: 441-446
        • Alahmari SS
        • Cherezov D
        • Goldgof D
        • et al.
        Delta radiomics improves pulmonary nodule malignancy prediction in Lung Cancer screening.
        IEEE Access. 2018; 6: 77796-77806https://doi.org/10.1109/ACCESS.2018.2884126
        • Tixier F
        • Le Rest CC
        • Hatt M
        • et al.
        Intratumor heterogeneity characterized by textural features on baseline 18F-FDG PET images predicts response to concomitant radiochemotherapy in esophageal cancer.
        J Nucl Med. 2011; 52: 369-378https://doi.org/10.2967/jnumed.110.082404
        • Sun R
        • Limkin EJ
        • Vakalopoulou M
        • et al.
        A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study.
        Lancet Oncol. 2018; 19: 1180-1191https://doi.org/10.1016/S1470-2045(18)30413-3
        • Aerts HJ
        • Velazquez ER
        • Leijenaar RT
        • et al.
        Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
        Nat Commun. 2014; 5: 4006https://doi.org/10.1038/ncomms5006
        • Drabycz S
        • Roldan G
        • de Robles P
        • et al.
        An analysis of image texture, tumor location, and MGMT promoter methylation in glioblastoma using magnetic resonance imaging.
        Neuroimage. 2010; 49: 1398-1405https://doi.org/10.1016/j.neuroimage.2009.09.049
        • Gutman DA
        • Dunn Jr., WD
        • Grossmann P
        • et al.
        Somatic mutations associated with MRI-derived volumetric features in glioblastoma.
        Neuroradiology. 2015; 57: 1227-1237https://doi.org/10.1007/s00234-015-1576-7
        • Kurhanewicz J
        • Vigneron D
        • Carroll P
        • et al.
        Multiparametric magnetic resonance imaging in prostate cancer: present and future.
        Curr Opin Urol. 2008; 18: 71-77https://doi.org/10.1097/MOU.0b013e3282f19d01
        • Parekh VS
        • Jacobs MA
        Multiparametric radiomics methods for breast cancer tissue characterization using radiological imaging.
        Breast Cancer Res Treat. 2020; 180: 407-421https://doi.org/10.1007/s10549-020-05533-5
        • Zhang B
        • Tian J
        • Dong D
        • et al.
        Radiomics features of multiparametric MRI as novel prognostic factors in advanced nasopharyngeal carcinoma.
        Clin Cancer Res. 2017; 23: 4259-4269
        • Ng S-H
        • Lin C-Y
        • Chan S-C
        • et al.
        Clinical utility of multimodality imaging with dynamic contrast-enhanced MRI, diffusion-weighted MRI, and 18F-FDG PET/CT for the prediction of neck control in oropharyngeal or hypopharyngeal squamous cell carcinoma treated with chemoradiation.
        PLoS One. 2014; 9e115933
        • Balagurunathan Y
        • Kumar V
        • Gu Y
        • et al.
        Test-retest reproducibility analysis of lung CT image features.
        J Digit Imaging. 2014; 27: 805-823https://doi.org/10.1007/s10278-014-9716-x
        • Traverso A
        • Wee L
        • Dekker A
        • et al.
        Repeatability and reproducibility of radiomic features: a systematic review.
        Int J Radiat Oncol Biol Phys. 2018; 102: 1143-1158https://doi.org/10.1016/j.ijrobp.2018.05.053
        • Zwanenburg A
        • Vallieres M
        • Abdalah MA
        • et al.
        The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping.
        Radiology. 2020; 295: 328-338https://doi.org/10.1148/radiol.2020191145
        • Zwanenburg A
        • Stefan L
        • Vallieres M
        • et al.
        Image biomarker standardisation initiative reference manual.
        arXiv preprint. 2016; : 1-168
        • Sanduleanu S
        • Woodruff HC
        • De Jong EEC
        • et al.
        Tracking tumor biology with radiomics: a systematic review utilizing a radiomics quality score.
        Radiother Oncol. 2018; 127: 349-360
        • Sollini M
        • Cozzi L
        • Antunovic L
        • et al.
        PET radiomics in NSCLC: state of the art and a proposal for harmonization of methodology.
        Scientific Reports. 2017; 7: 1-15
        • Collins GS
        • Reitsma JB
        • Altman DG
        • et al.
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.
        Ann Intern Med. 2015; 162: 55-63https://doi.org/10.7326/M14-0697
        • Moons KG
        • Altman DG
        • Reitsma JB
        • et al.
        Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration.
        Ann Intern Med. 2015; 162: W1-73https://doi.org/10.7326/M14-0698
        • Yusuf M
        • Atal I
        • Li J
        • et al.
        Reporting quality of studies using machine learning models for medical diagnosis: a systematic review.
        BMJ open. 2020; 10e034568
        • Obuchowski NA
        • Barnhart HX
        • Buckler AJ
        • et al.
        Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example.
        Stat Methods Med Res. 2015; 24: 107-140https://doi.org/10.1177/0962280214537392
        • Huang EP
        • Wang XF
        • Choudhury KR
        • et al.
        Meta-analysis of the technical performance of an imaging procedure: guidelines and statistical methodology.
        Stat Methods Med Res. 2015; 24: 141-174https://doi.org/10.1177/0962280214537394
        • Cui Z
        • Gong G.
        The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features.
        Neuroimage. 2018; 178: 622-637https://doi.org/10.1016/j.neuroimage.2018.06.001
        • Baum E
        • Haussler D.
        What size net gives valid generalization?.
        Adv Neural Inf Process Syst. 1988; 1: 81-90
        • Harrell FE
        • Lee KL
        • Califf RM
        • et al.
        Regression modelling strategies for improved prognostic prediction.
        Stat med. 1984; 3: 143-152
        • Peduzzi P
        • Concato J
        • Kemper E
        • et al.
        A simulation study of the number of events per variable in logistic regression analysis.
        J Clin Epidemiol. 1996; 49: 1373-1379
        • Vittinghoff E
        • McCulloch CE.
        Relaxing the rule of ten events per variable in logistic and Cox regression.
        Am J Epidemiol. 2007; 165: 710-718
        • Harrell FE.
        Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis.
        2nd ed. Springer, New York2015
        • Riley RD
        • Snell KIE
        • Ensor J
        • et al.
        Minimum sample size for developing a multivariable prediction model: part I–Continuous outcomes.
        Stat Med. 2019; 38: 1262-1275
        • Riley RD
        • Snell KI
        • Ensor J
        • et al.
        Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes.
        Stat Med. 2019; 38: 1276-1296https://doi.org/10.1002/sim.7992
        • Nagelkerke NJD.
        A note on a general definition of the coefficient of determination.
        Biometrika. 1991; 78: 691-692
        • Mukherjee S
        • Tamayo P
        • Rogers S
        • et al.
        Estimating dataset size requirements for classifying DNA microarray data.
        J Comput Biol. 2003; 10: 119-142https://doi.org/10.1089/106652703321825928
        • Buckler AJ
        • Bresolin L
        • Dunnick NR
        • et al.
        A collaborative enterprise for multi-stakeholder participation in the advancement of quantitative imaging.
        Radiology. 2011; 258: 906-914https://doi.org/10.1148/radiol.10100799
        • Shukla-Dave A
        • Obuchowski NA
        • Chenevert TL
        • et al.
        Quantitative imaging biomarkers alliance (QIBA) recommendations for improved precision of DWI and DCE-MRI derived biomarkers in multicenter oncology trials.
        J Magn Reson Imaging. 2019; 49: e101-e121https://doi.org/10.1002/jmri.26518
        • Hesamian MH
        • Jia W
        • He X
        • et al.
        Deep learning techniques for medical image segmentation: achievements and challenges.
        J Dig Imag. 2019; 32: 582-596
        • Minaee S
        • Boykov YY
        • Porikli F
        • Plaza A
        • Kehtarnavaz N
        • Terzopoulos D
        Image segmentation using deep learning: a survey.
        IEEE T Pattern Anal Machine Intelligence. 2021; 44: 3523-3542
        • van Timmeren JE
        • Cester D
        • Tanadini-Lang S
        • et al.
        Radiomics in medical imaging-"how-to" guide and critical reflection.
        Insights Imaging. 2020; 11: 91https://doi.org/10.1186/s13244-020-00887-2
        • Parmar C
        • Barry JD
        • Hosny A
        • et al.
        Data analysis strategies in medical imaging.
        Clin Cancer Res. 2018; 24: 3492-3499https://doi.org/10.1158/1078-0432.CCR-18-0385
        • Vial A
        • Stirling D
        • Field M
        • et al.
        The role of deep learning and radiomic feature extraction in cancer-specific predictive modelling: a review.
        Transl Cancer Res. 2018; 7: 803-816
        • Johnson WE
        • Li C
        • Rabinovic A.
        Adjusting batch effects in microarray expression data using empirical Bayes methods.
        Biostatistics. 2007; 8: 118-127
        • Orlhac F
        • Frouin F
        • Nioche C
        • et al.
        Validation of A Method to Compensate Multicenter Effects Affecting CT Radiomics.
        Radiology. 2019; 291: 53-59https://doi.org/10.1148/radiol.2019182023
        • Težak Ž
        • Ranamukhaarachchi D
        • Russek-Cohen E
        • et al.
        FDA perspectives on potential microarray-based clinical diagnostics.
        Human Genomics. 2006; 2: 1-8
        • Stekhoven DJ
        • Buhlmann P.
        MissForest–non-parametric missing value imputation for mixed-type data.
        Bioinformatics. 2012; 28: 112-118https://doi.org/10.1093/bioinformatics/btr597
        • Sv Buuren
        • mice Groothuis-Oudshoorn K.
        Multivariate imputation by chained equations in R.
        J Stat Soft. 2010; 45: 1-68
        • Shah AD
        • Bartlett JW
        • Carpenter J
        • et al.
        Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study.
        Am J Epidemiol. 2014; 179: 764-774https://doi.org/10.1093/aje/kwt312
        • Benjamini Y
        • Hochberg Y.
        Controlling the false discovery rate: a practical and powerful approach to multiple testing.
        J R stat soc series B Method. 1995; 57: 289-300
        • Holm S.
        A simple sequentially rejective multiple test procedure.
        Scandinavian J Stat. 1979; : 65-70
        • Kuhn M
        • Johnson K
        Applied predictive modeling.
        Springer, New York, USA2013
        • Tibshirani R.
        Regression shrinkage and selection via the lasso.
        J R Stat Soc. 1996; 58: 267-288
        • Genuer R
        • Poggi J-M
        • Tuleau-Malot C
        Variable selection using random forests.
        Pattern Recognit Lett. 2010; 31: 2225-2236
        • Hofner B
        • Hothorn T
        • Kneib T
        • et al.
        A framework for unbiased model selection based on boosting.
        J Comput Graph Stat. 2011; 20: 956-971
        • Fan J
        • Lv J
        A selective overview of variable selection in high dimensional feature space.
        Statistica Sinica. 2010; 20: 101-148
        • Huang J
        • Breheny P
        • Ma S.
        A selective review of group selection in high-dimensional models.
        Statistical Science. 2012; 27: 481-499
        • Hesterberg T
        • Choi NH
        • Meier L
        • et al.
        Least angle and ℓ1 penalized regression: a review.
        Statistics Surveys. 2008; 2: 61-93
        • Zou H.
        The adaptive lasso and its oracle properties.
        J Am stat assoc. 2006; 101: 1418-1429
        • Zou H
        • Hastie T.
        Regularization and variable selection via the elastic net.
        J R Stat Soc Series B Stat Methodol. 2005; 67: 301-320
        • Fan JQ
        • Li RZ.
        Variable selection via nonconcave penalized likelihood and its oracle properties.
        J Am Stat Assoc. 2001; 96: 1348-1360https://doi.org/10.1198/016214501753382273
        • Zhang C-H.
        Nearly unbiased variable selection under minimax concave penalty.
        Ann stat. 2010; 38: 894-942
        • Wu C
        • Ma S.
        A selective review of robust variable selection with applications in bioinformatics.
        Brie Bioinform. 2015; 16: 873-883
        • Heinze G
        • Wallisch C
        • Dunkler D.
        Variable selection–a review and recommendations for the practicing statistician.
        Biom J. 2018; 60: 431-449
        • Bach FR.
        Bolasso: model consistent lasso estimation through the bootstrap.
        in: Proceedings of the 25th international conference on Machine learning. 2008: 33-40
        • Meinshausen N
        • Bühlmann P.
        Stability selection.
        J R Stat Soc Series B Stat Methodol. 2010; 72: 417-473
        • Barber RF
        • Candès EJ.
        Controlling the false discovery rate via knockoffs.
        Ann Stat. 2015; 43: 2055-2085
        • Candes E
        • Fan Y
        • Janson L
        • et al.
        Panning for gold:‘model-X'knockoffs for high dimensional controlled variable selection.
        J R Stat Soc Series B Stat Methodol. 2018; 80: 551-577
        • Dy JG
        • Brodley CE.
        Feature selection for unsupervised learning.
        J Mach Learn Res. 2004; 5: 845-889
        • Fop M
        • Murphy TB.
        Variable selection methods for model-based clustering.
        Statistics Surveys. 2018; 12: 18-65
        • Efron B
        • Tibshirani R.
        Improvements on cross-validation: the 632+ bootstrap method.
        J Am Stat Assoc. 1997; 92: 548-560
        • Harrell Jr, FE
        • Lee KL
        • Mark DB
        Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.
        Stat Med. 1996; 15: 361-387
        • Zhang B
        • Lian Z
        • Zhong L
        • et al.
        Machine-learning based MRI radiomics models for early detection of radiation-induced brain injury in nasopharyngeal carcinoma.
        BMC cancer. 2020; 20: 1-9
        • Cawley GC
        • Talbot NLC.
        On over-fitting in model selection and subsequent selection bias in performance evaluation.
        J Mach Learn Res. 2010; 11: 2079-2107
        • Varma S
        • Simon R.
        Bias in error estimation when using cross-validation for model selection.
        BMC bioinformatics. 2006; 7: 1-8
        • Ambroise C
        • McLachlan GJ.
        Selection bias in gene extraction on the basis of microarray gene-expression data.
        Proceed Natl Acad Sci. 2002; 99: 6562-6566
        • Simon R
        • Radmacher MD
        • Dobbin K
        • et al.
        Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification.
        J Natl Cancer Inst. 2003; 95: 14-18
        • Cook NR.
        Use and misuse of the receiver operating characteristic curve in risk prediction.
        Circulation. 2007; 115: 928-935
        • Moons KGM
        • Harrell FE.
        Sensitivity and specificity should be de-emphasized in diagnostic accuracy studies.
        Academic radiology. 2003; 10: 670-672
        • Kattan MW
        • Vickers AJ.
        Statistical analysis and reporting guidelines for CHEST.
        Chest. 2020; 158: S3-S11https://doi.org/10.1016/j.chest.2019.10.064
        • Lin LI.
        A concordance correlation coefficient to evaluate reproducibility.
        Biometrics. 1989; 45: 255-268
        • Barnhart HX
        • Haber MJ
        • Lin LI.
        An overview on assessing agreement with continuous measurements.
        J Biopharm Stat. 2007; 17: 529-569
        • Heil BJ
        • Hoffman MM
        • Markowetz F
        • et al.
        Reproducibility standards for machine learning in the life sciences.
        Nature Methods. 2021; 18: 1132-1135
        • Walsh I
        • Fishman D
        • Garcia-Gasulla D
        • et al.
        DOME: recommendations for supervised machine learning validation in biology.
        Nature methods. 2021; 18: 1122-1127
        • De A
        • Meier K
        • Tang R
        • et al.
        Evaluation of heart failure biomarker tests: a survey of statistical considerations.
        J Cardiovasc Transl Res. 2013; 6: 449-457
        • Siegel RL
        • Miller KD
        • Fuchs HE
        • et al.
        Cancer statistics, 2021.
        Ca Cancer J Clin. 2021; 71: 7-33
        • Aberle DR
        • Adams AM
        • et al.
        • The National Lung Screening Trial Research Team
        Reduced lung-cancer mortality with low-dose computed tomographic screening.
        N Engl J Med. 2011; 365: 395-409https://doi.org/10.1056/NEJMoa1102873
        • Hawkins S
        • Wang H
        • Liu Y
        • et al.
        Predicting malignant nodules from screening CT scans.
        J Thorac Oncol. 2016; 11: 2120-2128
        • Baatz M
        • Zimmermann J
        • Blackmore CG.
        Automated analysis and detailed quantification of biomedical images using definiens cognition network technology.
        Comb Chem High Throughput Screen. 2009; 12: 908-916
        • Steyerberg EW
        • Vickers AJ
        • Cook NR
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138https://doi.org/10.1097/EDE.0b013e3181c30fb2
        • Balagurunathan Y
        • Gu Y
        • Wang H
        • et al.
        Reproducibility and prognosis of quantitative features extracted from CT images.
        Transl Oncol. 2014; 7: 72-87https://doi.org/10.1593/tlo.13844
        • Aerts H
        • Velazquez ER
        • Leijenaar RTH
        • et al.
        Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
        Nature communications. 2014; 5: 1-9
        • Cohen JF
        • Korevaar DA
        • Altman DG
        • et al.
        STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration.
        BMJ open. 2016; 6e012799
        • McShane LM
        • Cavenagh MM
        • Lively TG
        • et al.
        Criteria for the use of omics-based predictors in clinical trials.
        Nature. 2013; 502: 317-320