Multiparametric Quantitative Imaging in Risk Prediction: Recommendations for Data Acquisition, Technical Performance Assessment, and Model Development and Validation

Published:October 20, 2022DOI:
      Combinations of multiple quantitative imaging biomarkers (QIBs) are often able to predict the likelihood of an event of interest such as death or disease recurrence more effectively than single imaging measurements can alone. The development of such multiparametric quantitative imaging and evaluation of its fitness of use differs from the analogous processes for individual QIBs in several key aspects. A computational procedure to combine the QIB values into a model output must be specified. The output must also be reproducible and be shown to have reasonably strong ability to predict the risk of an event of interest. Attention must be paid to statistical issues not often encountered in the single QIB scenario, including overfitting and bias in the estimates of model performance. This is the fourth in a five-part series on statistical methodology for assessing the technical performance of multiparametric quantitative imaging. Considerations for data acquisition are discussed and recommendations from the literature on methodology to construct and evaluate QIB-based models for risk prediction are summarized. The findings in the literature upon which these recommendations are based are demonstrated through simulation studies. The concepts in this manuscript are applied to a real-life example involving prediction of major adverse cardiac events using automated plaque analysis.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Sullivan DC
        • Obuchowski NA
        • Kessler LG
        • et al.
        Metrology standards for quantitative imaging biomarkers.
        Radiology. 2015; 277: 813-825
        • Kessler LG
        • Barnhart HX
        • Buckler AJ
        • et al.
        The emerging science of quantitative imaging biomarkers: Terminology and definitions for scientific studies and regulatory submissions.
        Stat Methods in Med Res. 2015; 24: 9-26
        • Raunig DL
        • McShane LM
        • Pennello G
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment.
        Stat Methods in Med Res. 2015; 24: 27-67
        • Obuchowski NA
        • Reeves AP
        • Huang EP
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons.
        Stat Methods in Med Res. 2015; 24: 68-106
        • Huang EP
        • Wang XF
        • Roy Chowdhury K
        • et al.
        Meta-analysis of the technical performance of an imaging procedure: guidelines and statistical methodology.
        Stat Methods in Med Res. 2015; 24: 141-174
        • Obuchowski NA
        • Barnhart HX
        • Buckler AJ
        • et al.
        Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary node module volume as an example.
        Stat Methods in Med Res. 2015; 24: 107-140
        • Li W
        • Newitt DC
        • Wilmes LJ
        • et al.
        Additive value of diffusion weighted MRI in the I-SPY 2 TRIAL.
        J Magnetic Resonance Imaging. 2019; 50: 1742-1753
        • Tadayyon H
        • Sannachi L
        • Gangeh MJ
        • et al.
        A priori prediction of neoadjuvant chemotherapy response and survival in breast cancer patients using quantitative ultrasound.
        Natr Sci Rep. 2017; 745733
        • Obuchowski NA
        • Huang EP
        • deSouza NM
        • et al.
        A framework for evaluating the technical performance of multiparameter quantitative imaging biomarkers (mp-QIBs).
        Acad Radiol. 2022; (In press)
        • Raunig DL
        • Pennello GA
        • Delfino JG
        • et al.
        Multiparametric quantitative imaging biomarker as a multivariate descriptor of health.
        Acad Radiol. 2022; (Submitted for publication)
        • Delfino JG
        • Pennello GA
        • Barnhart HX
        • et al.
        Multiparametric quantitative imaging biomarkers for phenotype classification: a framework for development and validation.
        Acad Radiol. 2022; (In press)
        • Wang X
        • Pennello GA
        • deSouza NM
        • et al.
        Multiparametric data-driven markers: guidelines for development, application, and reporting of model outputs in radiomics.
        Acad Radiol. 2022; (In press)
        • Song J
        • Chen Z
        • Huang D
        • et al.
        Nomogram predicting overall survival of resected locally advanced rectal cancer patients with neoadjuvant chemoradiotherapy.
        Cancer Manag Res. 2020; 12: 7375-7382
        • Xiong Z
        • Deng G
        • Huang X
        • et al.
        Score for the survival probability in metastatic breast cancer: a nomogram-based risk assessment model.
        Cancer Res Treatment. 2018; 50: 1260-1269
        • Zhang S
        • Wang X
        • Li Z
        • et al.
        Score for the overall survival probability of patients with first-diagnosed distantly metastatic cervical cancer: a novel nomogram-based risk assessment system.
        Front Oncol. 2019; 9: 1106
        • de Mestier L
        • Muller M
        • Cros J
        • et al.
        Appropriateness of pancreatic resection in high-risk individuals for familial pancreatic ductal adenocarcinoma: A patient-level meta-analysis and proposition of the Beaujon score.
        United Eur Gastroenterol J. 2019; 7: 358-368
        • Alabi RO
        • Mäkitie AA
        • Pirenen M
        • et al.
        Comparison of nomogram with machine learning techniques for prediction of overall survival in patients with tongue cancer.
        Int J Med Infor. 2021; 145104313
        • Augustin T
        • Schwarz R
        Cox’s proportional hazards model under covariate measurement error.
        in: van Huffel S Lemmering P Total Least Squares and Errors-in-Variables Modeling. Springer, Dordrecht, NL2002: 179-188
        • Griliches Z
        • Ringstad V
        Error-in-the-variables bias in nonlinear contexts.
        Econometrica. 1970; 38: 368-370
        • Stefanski LA
        • Carroll RJ
        Covariate measurement error in logistic regression.
        Ann Stat. 1985; 13: 1335-1351
      1. Profiles. QIBA Wiki. Available at: Accessed January 7, 2022.

        • Carroll RJ
        • Ruppert D
        The use and misuse of orthogonal regression in linear errors-in-variables models.
        The Am Stat. 1996; 50: 1-6
        • Access the Data
        The cancer imaging archive.
        2022 (Accessed January 7)
        • Simon RM
        • Paik S
        • Hayes DF
        Use of archived specimens in evaluation of prognostic and predictive biomarkers.
        J Natl Cancer Institute. 2009; 101: 1446-1452
        • Rutjes AWS
        • Reitsma JB
        • Vandenbroucke JP
        • et al.
        Case-control and two-gate designs in diagnostic accuracy studies.
        Clin Chem. 2005; 51: 1335-1341
        • Pepe MS
        • Feng Z
        • Janes H
        • et al.
        Pivotal evaluation of the accuracy of a biomarker used for classification or prediction: standards for study design.
        J Natl Cancer Institute. 2008; 100: 1432-1438
        • Ogundimu EO
        • Altman DG
        • Collins GS
        Adequate sample size for developing prediction models is not simply related to events per variable.
        J Clin Epidemiol. 2016; 76: 175-182
        • Harrell FE
        • Lee KL
        • Mark DB
        Tutorial in biostatistics: multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.
        Stat in Med. 1996; 15: 361-387
        • Peduzzi P
        • Concato J
        • Feinstein AR
        • et al.
        Importance of events per independent variable in proportional hazards regression analysis. II: accuracy and precision of regression estimates.
        J Clin Epidemiol. 1995; 48: 1503-1510
        • Peduzzi P
        • Concato J
        • Kemper E
        • et al.
        A simulation study of the number of events per variable in logistic regression analysis.
        J Clin Epidemiol. 1996; 49: 1373-1379
        • Vittinghoff E
        • McCulloch CE.
        Relaxing the rule of ten events per variable in logistic and Cox regression.
        Am J Epidemiol. 2007; 165: 710-718
        • Riley RD
        • Snell KI
        • Ensor J
        • et al.
        Minimum sample size for developing a multivariable prediction model: Part II – Binary and time-to-event outcomes.
        Stat Med. 2019; 38: 1276-1296
        • Luo J
        • Schumacher M
        • Scherer A
        • et al.
        A comparison of batch effect removal methods for enhancement of prediction performance using MACQ-II microarray gene expression data.
        The Pharmacogenomics J. 2010; 10: 278-291
        • Orlhac F
        • Boughdad S
        • Philippe C
        • et al.
        A post-reconstruction harmonization method for multicenter radiomic studies in PET.
        J Nucl Med. 2018; 59: 1321-1328
        • Johnson WE
        • Li C
        • Rabinovic A
        Adjusting batch effects in microarray expression data using empirical Bayes methods.
        Biostatistics. 2007; 8: 118-127
        • Haller B
        • Schmidt G
        • Ulm K
        Applying competing risks regression models: an overview.
        Lifetime Data Analysis. 2013; 19: 33-58
        • Hastie T
        • Tibshirani R
        • Friedman J
        The Elements of Statistical Learning: Data Mining, Inference and Prediction. Second Edition. Springer-Verlag, Berlin, DE2009
        • Heinze G
        • Wallisch C
        • Dunkler D
        Variable selection – a review and recommendations for the practicing statistician.
        Biometr J. 2018; 60: 431-449
        • Zou H
        • Hastie T
        Regularization and variable selection via the elastic net.
        J Royal Stat Soc – Series B (Methodological). 2005; 67: 301-320
        • Tibshirani R.
        Regression shrinkage and selection via the LASSO.
        J Royal Stat Soc—Series B (Methodological). 1996; 58: 267-288
        • Haykin S
        Neural Networks: A Comprehensive Foundation. Prentice Hall, Upper Saddle River, NJ1998
        • Breitenbach M
        • Nielsen R
        • Grudic GZ
        Probabilistic random forests: Predicting data point specific misclassification probabilities.
        University of Colorado at Boulder, 2003 (Technical report CU-CS-954-03)
        • Ishwaran H
        • Kogalur UB
        • Blackstone EH
        • et al.
        Random survival forests.
        The Ann Appl Stat. 2008; 2: 841-860
        • Biganzoli E
        • Boracchi P
        • Mariani L
        • et al.
        Feed-forward neural networks for the analysis of censored survival data: a partial logistic regression approach.
        Stat Med. 1998; 17: 1169-1186
        • Dudoit S
        • Fridlyand J
        • Speed TP.
        Comparison of discrimination methods for the classification of tumors using gene expression data.
        J Am Stat Assoc. 2002; 97: 77-87
        • Ben-Dor A
        • Bruhn L
        • Friedman N
        • et al.
        Tissue classification with gene expression profiles.
        J Comput Biol. 2000; 7: 559-583
        • Friedman JH
        On bias, variance, 0/1 loss, and the curse of dimensionality.
        Data Mining and Knowl Discovery. 1997; 1: 55-77
        • Hosmer DW
        • Lemeshow S
        Goodness of fit tests for the multiple logistic regression model.
        Commun in Stat – Theory and Methods. 1980; 9: 1043-1069
        • Lemeshow S
        • Hosmer D
        A review of goodness of fit statistics for use in the development of logistic regression model.
        Am J Epidemiol. 1982; 115: 92-106
        • Cox DR
        Two further applications of a model for binary regression.
        Biometrika. 1958; 45: 562-565
        • van Calster B
        • Steyerberg EW
        Wiley StatsRef: Statistics Reference Online. John Wiley and Sons, Ltd, Hoboken, NJ2018
        • Pfeiffer RM
        • Gail MH
        Absolute Risk: Methods and Applications in Clinical Management and Public Health. First Edition. Chapman and Hall/CRC, London, UK2017
        • D'Agostino RB
        • Nam B
        Evaluation of the performance of survival analysis models: discrimination and calibration measures.
        Handbook of Statistics. 2004; 23: 1-25
        • Brier GW
        Verification of forecasts expressed in terms of probability.
        Monthly Weather Rev. 1950; 78: 1-3
        • Graf E
        • Schmoor C
        • Sauerbrei W
        • et al.
        Assessment and comparison of prognostic classification schemes for survival data.
        Stat Med. 1999; 18: 2529-2545
        • Sadatsafavi M
        • Saha-Chaudhuri P
        • Petkau J
        Model-based ROC curve: examining the effect of case mix and model calibration on the ROC plot.
        Med Decision Making. 2021; (In press)
        • Uno H
        • Cai T
        • Pencina MJ
        • et al.
        On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data.
        Stat Med. 2011; 30: 1105-1117
        • Pencina MJ
        • D'Agostino RB
        Overall c as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation.
        Stat Med. 2004; 23: 2109-2123
        • Hsu MJ
        • Chang YC
        • Hsueh HM
        Biomarker selection for medical diagnosis using the partial area under the ROC curve.
        Biomed Central Res Notes. 2014; 725
        • Molinaro AM
        • Simon R
        • Pfeffer RM
        Prediction error estimation: a comparison of resampling methods.
        Bioinformatics. 2005; 21: 3301-3307
        • Sachs MC
        • McShane LM
        Issues in developing multivariable molecular signatures for guiding clinical care decisions.
        J Biopharma Stat. 2016; 26: 1098-1110
        • McLachlan GJ
        Estimation of Error Rates.
        in: McLachlan GJ Discriminant Analysis and Statistical Pattern Recognition. John Wiley and Sons, Inc, Hoboken, NJ2002: 337-378
        • Altman DG
        • Royston P
        What do we mean by validating a prognostic model?.
        Stat Med. 2000; 19: 453-473
        • Efron B
        • Tibshirani R
        Improvements on cross-validation: the 0.632+ bootstrap method.
        J Am Stat Assoc. 1997; 92: 548-560
        • Varma S
        • Simon R
        Bias in error estimation when using cross-validation for model selection.
        Biomed Central Bioinformatics. 2006; 791
        • Dobbin KK
        • Simon RM
        Optimally Splitting cases for training and testing high dimensional classifiers.
        Biomed Central Med Genomics. 2011; 431
        • CLSI
        Expression of measurement uncertainty in laboratory medicine: approved guideline.
        Clinical and Laboratory Standards Institute, 2012 (CLSI document EP29-A)
        • CLSI
        Estimation of total analytical error for clinical laboratory methods: approved guideline.
        second edition. Clinical and Laboratory Standards Institute, 2016 (CLSI Document EP21-A2)
        • ISO
        Guidance for the use of repeatability, reproducibility and trueness estimates in measurement uncertainty estimation.
        second edition. International Organization for Standardization, 2017 (ISO 217482017-04)
        • ISO
        Medical laboratories – Practical guidance for the estimation of measurement uncertainty.
        first edition. International Organization for Standardization, 2019 (ISO 209142019-07)
        • ISO
        Statistical methods – Guidelines for the evaluation of conformity with specified requirements – Part 1: general principles.
        International Organization for Standardization, 2003 (ISO 10576-1:2003(E) 2003-03-01)
        • ISO/ASTM
        Guide for estimation of measurement uncertainty in dosimetry for radiation processing.
        third edition. International Organization for Standardization, 2015 (ISO/ASTM517072015-03-15)
        • ISO/TR
        Three statistical approaches for the assessment and interpretation of measurement uncertainty.
        first edition. International Organization for Standardization, 2012 (ISO/TR 135872012-07-15)
        • ISO/TS
        Measurement uncertainty for metrological applications – repeated measurements and nested experiments.
        International Organization for Standardization, 2005 (ISO/TS 21749:2005(E), Corrected Version 2017-07-15)
        Quantifying uncertainty in analytical measurement.
        second edition. 2012 (Accessed January 17)
        • NIST
        Guidelines for evaluating and expressing the uncertainty of NIST measurement results: Technical note 1297.
        1994 edition. 2012 (Accessed January 17)
        • Kristiansen J
        Description of a generally applicable model for the evaluation of uncertainty of measurement in clinical chemistry.
        Clin Chem and Lab Med. 2001; 39: 920-931
        • Kristiansen J
        The guide to expression of uncertainty in measurement approach for estimating uncertainty: an appraisal.
        Clin Chem. 2003; 49: 1822-1829
        • Kenny D
        • Fraser CG
        • Hyltoft Petersen P
        • et al.
        Consensus agreement: conference on strategies to set global analytical quality specifications in laboratory medicine.
        Scandinavian J Clin Laboratory Investigation. 1999; 59: 585
        • Janson S
        • Vigelius J
        On generalizations of the G index and the phi coefficient to nominal scales.
        Multivariate Behav Res. 1979; 14: 255-269
        • van Essen M
        • Varga-Szemes A
        • Schoepf UJ
        • et al.
        Automated plaque analysis for the prognostication of major adverse cardiac events.
        Eur J Radiol. 2019; 116: 76-83
      2. CT angiography committee.
        QIBA Wiki, 2022 (Accessed January 7)
        • Sheahan M
        • Ma X
        • Paik D
        • et al.
        Atherosclerotic plaque tissue: Noninvasive quantitative assessment of characteristics with software-aided measurements from conventional CT angiography.
        Radiology. 2018; 286: 622-631
        • Huang EP
        • Lin FI
        • Shankar LK
        Beyond correlations, sensitivities, and specificities: A roadmap for demonstrating utility of advanced imaging in oncology treatment and clinical trial design.
        Acad Radiol. 2017; 24: 1036-1049
        • Bossuyt PM
        • Lijmer JG
        • Mol BW
        Randomised comparisons of medical tests: sometimes invalid, not always efficient.
        Lancet. 2000; 356: 1844-1847
        • Simon R
        Clinical trial designs for evaluating the medical utility of prognostic and predictive biomarkers in oncology.
        Personalized Med. 2010; 7: 33-47
        • Subramanian J
        • Simon R
        What should physicians look for in evaluating prognostic gene expression signatures?.
        Natr Rev Clin Oncol. 2010; 7: 327-334