A Framework for Evaluating the Technical Performance of Multiparameter Quantitative Imaging Biomarkers (mp-QIBs)

Published:September 27, 2022DOI:
      Multiparameter quantitative imaging incorporates anatomical, functional, and/or behavioral biomarkers to characterize tissue, detect disease, identify phenotypes, define longitudinal change, or predict outcome. Multiple imaging parameters are sometimes considered separately but ideally are evaluated collectively. Often, they are transformed as Likert interpretations, ignoring the correlations of quantitative properties that may result in better reproducibility or outcome prediction. In this paper we present three use cases of multiparameter quantitative imaging: i) multidimensional descriptor, ii) phenotype classification, and iii) risk prediction. A fourth application based on data-driven markers from radiomics is also presented. We describe the technical performance characteristics and their metrics common to all use cases, and provide a structure for the development, estimation, and testing of multiparameter quantitative imaging. This paper serves as an overview for a series of individual articles on the four applications, providing the statistical framework for multiparameter imaging applications in medicine.

      Key Words


      QIBA (Quantitative Imaging Biomarker Alliance), QIB (Quantitative Imaging Biomarker)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Kessler LG
        • Barnhart HX
        • Buckler AJ
        • et al.
        The Emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions.
        SMMR. 2015; 24: 9-f26
        • Raunig DL
        • McShane LM
        • Pennello G
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment.
        SMMR. 2015; 24: 27-67
        • Obuchowski NA
        • Reeves AP
        • Huang EP
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons.
        SMMR. 2015; 24: 68-106
        • Huang EP
        • Wang XF
        • Choudhury KR
        • et al.
        Meta-analysis of the technical performance of an imaging procedure: guidelines and statistical methodology.
        SMMR. 2015; 24: 141-174
      1. Raunig et al. Multidimensional quantitative imaging biomarkers as a multivariate descriptor of health. Acad Radiol.

      2. Delfino et al. Multiparametric quantitative imaging biomarkers in phenotype classification. Acad Radiol.

      3. Huang et al. Multiparametric quantitative imaging biomarkers in risk prediction: recommendations for data acquisition, technical performance assessment, and model development and validation. Acad Radiol.

      4. Wang et al. Multiparametric data-driven imaging markers: guidelines for development, application and reporting of model outputs in radiomics. Acad Radiol.

        • Hall TJ
        • Zhu Y
        • Spalding CS
        In vivo real-time freehand palpation imaging.
        Ultrason Med Biol. 2003; 29: 427-435
        • LeCarpentier GL
        • Roubidoux MA
        • Fowlkes JB
        • et al.
        Suspicious breast lesions: assessment of 3D Doppler US indexes for classification in a test population and fourfold cross-validation scheme.
        Radiology. 2008; 249: 463-470
        • d'Astous FT
        • Foster FS
        Frequency dependence of ultrasound attenuation and backscatter in breast tissue.
        Ultrason Med Biol. 1986; 12: 795-808
        • Nam K
        • Zagzebski JA
        • TJ Hall
        Quantitative assessment of in vivo breast masses using ultrasound attenuation and backscatter.
        Ultrason Imaging. 2013; 35: 146-161
        • Rosado-Mendez IM.
        Advanced spectral analysis methods for quantification of coherent ultrasound scattering: Applications in the breast.
        The University of Wisconsin-Madison;, 2014 (Doctoral dissertation)
        • Choi H
        • Charnsangavej C
        • Faria SC
        • et al.
        Correlation of computed tomography and positron emission tomography in patients with metastatic gastrointestinal stromal tumor treated at a single institution with imatinib mesylate: proposal of new computed tomography response criteria.
        J Clin Oncol. 2007; 25: 1753-1759
        • Sheahan M
        • Ma X
        • Paik D
        • et al.
        Atherosclerotic plaque tissue: noninvasive quantitative assessment of characteristics with software-aided measurements from conventional CT angiography.
        Radiology. 2018; 286: 622-631
        • Chrencik MT
        • Khan AA
        • Luther L
        • et al.
        Quantitative assessment of carotid plaque morphology (geometry and tissue composition) using computed tomography angiography.
        J Vasc Surg. 2019; 70: 858-868
        • Institute of Medicine of the National Academies
        Evolution of translational omics: lessons learned and the path forward.
        The National Academies Press, Washington, DC:2012 (Report Brief)
        • Hoehndorf R
        • Schofield PN
        • Gkoutos GV
        Analysis of the human diseasome using phenotype similarity between common, genetic, and infectious diseases.
        Nature Scientific Reports. 2015; 5: 1-15
        • Fusco R
        • Di Marzo M
        • Sansone C
        • et al.
        Breast DCE-MRI: lesion classification using dynamic and morphological features by means of a multiple classifier system.
        Eur Radiol Exp. 2017; 1: 10
        • Barr RG
        • Ferraioli G
        • Palmeri ML
        • et al.
        Elastography assessment of liver fibrosis: society of radiologists in ultrasound consensus conference statement.
        Radiology. 2015; 276: 845-861
        • McDonald N
        • Eddowes PJ
        • Hodson J
        • et al.
        Multiparameter magnetic resonance imaging for quantitation of liver disease: a two-centre cross-sectional observational study.
        Sci Rep. 2018; 15: 8
        • FDA-NIH Biomarker Working Group
        BEST (Biomarkers, EndpointS, and other Tools) Resource.
        Food and Drug Administration (US), Silver Spring, MD2016 (Available at:) (Accessed September 22, 2022)
        • Zakaria R
        • Chen YJ
        • Hughes DM
        • et al.
        Does the application of diffusion weighted imaging improve the prediction of survival in patients with resected brain metastases? A retrospective multicenter study.
        Cancer Imaging. 2020; 20: 16
        • Gillies RJ
        • Kinahan PE
        • Hricak H
        Radiomics: images are more than pictures, they are data.
        Radiology. 2016; 278: 563-577
        • Fournier L
        • Costaridou L
        • Bidaut L
        • et al.
        Incorporating radiomics into clinical trials: expert consensus on considerations for data-driven compared to biologically-driven quantitative biomarkers.
        Eur Radiology. 2020; 31: 6001-6012
        • Sun R
        • Limkin EJ
        • Vakalopoulou M
        • et al.
        A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study.
        Lancet Oncol. 2018; 19: 1180-1191
        • Fan M
        • Li H
        • Wang S
        • et al.
        Radiomic analysis reveals DCE-MRI features for prediction of molecular subtypes of breast cancer.
        PLoS One. 2017; 12
        • Hassani C
        • Varghese BA
        • Nieva J
        • et al.
        Radiomics in pulmonary lesion imaging.
        Am J Roentgenol. 2019; 212: 497-504
        • Hackstadt AJ
        • Hess AM
        Filtering for increased power for microarray data analysis.
        BMC Bioinformatics. 2009; 10: 1471-2105
        • Park JE
        • Park SY
        • Kim HJ
        • et al.
        Reproducibility and generalizability in radiomics modeling: possible strategies in radiologic and statistical perspectives.
        Korean J Radiol. 2019; 20: 1124-1137
        • Su JQ
        • Liu JS
        Linear combinations of multiple diagnostic markers.
        JASA. 1993; 88: 1350-1355
        • McIntosh MW
        • Pepe MS
        Combining several screening tests: optimality of the risk score.
        Biometrics. 2002; 58: 657-664
        • Liu A
        • Schisterman EF
        • Zhu Y
        On linear combinations of biomarkers to improve diagnostic accuracy.
        Stat Med. 2005; 24: 37-47
        • Pepe MS
        • Cai T
        • Longton G
        Combing predictors for classification using the area under the receiver operating characteristic curve.
        Biometrics. 2006; 62: 221-229
        • van Calster B
        • Steyerberg EW
        Calibration of prognostic risk score.
        Wiley StatsRef: Statistics Reference Online. 2018; : 1-10
        • Pepe M
        • Feng Z
        • Huang Y
        • et al.
        Integrating the predictiveness of a marker with its performance as a classifier.
        Am J Epidemiol. 2008; 167: 362
        • Jin H
        • Lu Y
        The optimal linear combination of multiple predictors under the generalized linear models.
        Stat Probab Lett. 2009; 79: 2321-2327
        • Kang L
        • Liu A
        • Tian L
        Linear combination methods to improve diagnostic/prognostic accuracy on future observations.
        SMMR. 2013; 25: 1359-1380
        • Hsu MJ
        • Hsueh HM
        The linear combinations of biomarkers which maximize the partial area under the ROC curves.
        Comput Stat. 2013; 28: 647-666
        • Ma H
        • Halabi S
        • Liu A
        On the use of min-max combination of biomarkers to maximize the partial area under the ROC curve.
        Probab Stat. 2019; 2019: 1-13
        • Pepe MS
        • Janes H
        Methods for evaluating prediction performance of biomarkers and tests. Risk assessment and evaluation of predictions.
        Lecture Notes in Statistics. Springer Science + Business Media, New York2013
        • Pepe M
        • Kerr K
        • Longton G
        • et al.
        Testing for improvement in prediction model performance.
        Stat Med. 2013; 32: 1467-1482
        • Pencina MJ
        • D'Agostino Sr, RB
        • D'Agostin Jr, o RB
        • et al.
        Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.
        Stat Med. 2008; 27 (With comments by Pepe M, Feng Z, Gu J): 157-172
        • Pencina MJ
        • D'Agostino RB
        • Sr Steyerberg EW
        Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers.
        Stat Med. 2011; 30: 11-21
        • Huang Y
        • Pepe M
        A parameter ROC model-based approach for evaluating the predictiveness of continuous markers in case-control studies.
        Biometrics. 2009; 65: 1133-1144
        • EP29-A., C.A.G.C.d
        Expression of Measurement Uncertainty in Laboratory Medicine.
        Clinical and Laboratory Standards Institute Wayne, PA:2012
      5. Standardization, I.O.f., Guidance for the use of repeatability, reproducibility and trueness estimates in measurement uncertainty estimation, in (ISO Standard No 21748:2017(E)). Geneva, Switzerland; 2017.

      6. Institute of Medicine, Board on Heath Care Services, Board on Health Sciences Policy, Committee on the review of omics-based tests for predicting patient outcomes in clinical trials. Evolution of Translational Omics: Lessons learned and the path forward. National Academies Press, Washington DC: 2012.

        • Pepe MS
        • Etzioni R
        • Feng Z
        • et al.
        Phases of biomarker development for early detection of cancer.
        J Natl Cancer Inst. 2001; 93: 1054-1061
        • Pepe MS
        • Feng Z
        • Janes H
        • et al.
        Pivotal evaluation of the accuracy of a biomarker used for classification or prediction: standards for study design.
        J Natl Cancer Inst. 2008; 100: 1432-1438
        • Hastie T
        • Tibshirani R
        • Friedman J
        The Elements of Statistical Learning.
        2nd edition. Springer Science+Business Media, New York2008
        • Fryback DG
        • Thornbury JR
        The efficacy of diagnostic imaging.
        Med Decis Making. 1991; 11: 88-94
        • O'Connor JPB
        • Aboagye EO
        • Waterton JC
        Imaging biomarker roadmap for cancer studies.
        Nat Rev Clin Onc. 2017; 14: 169-186
      7. Context of Use. Available at: Accessed February 4, 2022.

        • Criner GJ
        • Delage A
        • Voelker K
        • et al.
        improving lung function in severe heterogenous emphysema with the spiration valve system (EMPROVE). A multicenter, open-label randomized controlled clinical trial.
        Am J of Respir Crit Care Med. 2019; 200: 1354-1362
        • Simon RM
        • Paik S
        • DF Hayes
        Use of archived specimens in evaluation of prognostic and predictive biomarkers.
        JNCI. 2009; 101: 1446-1452
        • Kuhn M
        • Johnson K
        Applied predictive modeling.
        Springer, New York2013
      8. Uncertainty of measurements – Part 6: Developing and using measurement models.
        International Electrotechnical Commission (IEC), Switzerland2020 (ISO/IEC FDGuide 98-6:2020Available at:)
      9. Dudoit S, Fridlyand J, Speed TP. Comparison of discrimination methods for the classification of tumors using gene expression data. JASA 2002; 97:77–87.

        • Rudin C
        Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.
        Nat Mach Intell. 2019; 1: 206-215
        • Altman DG
        • Royston P
        What do we mean by validating a prognostic model?.
        Statist Med. 2000; 19: 453-473
        • Bossuyt PMM
        • Lijmer JG
        • Mol BWJ
        Randomised comparisons of medical tests: sometimes invalid, not always efficient.
        Lancet. 2000; 356: 1844-1847
        • Huang EP
        • Lin FI
        • Shankar L.
        Beyond correlations, sensitivities, and specificities: a roadmap for demonstrating utility of advanced imaging in oncology treatment and clinical trial design.
        Acad Radiol. 2017; 24: 1036-1049