Multiparametric Quantitative Imaging Biomarker as a Multivariate Descriptor of Health: A Roadmap

  • David L. Raunig
    Address correspondence to: D. R.
    Department of Statistical and Quantitative Sciences, Data Science Institute, Takeda Pharmaceuticals, Cambridge, Massachusetts
    Search for articles by this author
  • Gene A. Pennello
    Center for Devices and Radiological Health, US Food and Drug Administration Division of Imaging, Diagnostic and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland
    Search for articles by this author
  • Jana G. Delfino
    Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland
    Search for articles by this author
  • Andrew J. Buckler
    Elucid Bioimaging, Inc., Boston, Massachusetts
    Search for articles by this author
  • Timothy J. Hall
    Department of Medical Physics, University of Wisconsin, Madison, Wisconsin
    Search for articles by this author
  • Alexander R. Guimaraes
    Department of Diagnostic Radiology, Oregon Health & Sciences University, Portland, Oregon
    Search for articles by this author
  • Xiaofeng Wang
    Department of Quantitative Health Sciences, Lerner Research Institute, Cleveland, Ohio
    Search for articles by this author
  • Erich P. Huang
    Biometric Research Program, Division of Cancer Treatment and Diagnosis – National Cancer Institute, National Institutes of Health, Bethesda, MD
    Search for articles by this author
  • Huiman X. Barnhart
    Department of Biostatistics and Bioinformatics, Duke University, Durham, North Carolina
    Search for articles by this author
  • Nandita deSouza
    Division of Radiotherapy and Imaging, the Insitute of Cancer Research and Royal Marsden NHS Foundation Trust, London, United Kingdom
    Search for articles by this author
  • Author Footnotes
    # For the Alzheimer's Disease Neuroimaging Initiative: Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database ( As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at:
    Nancy Obuchowski
    # For the Alzheimer's Disease Neuroimaging Initiative: Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database ( As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at:
    Department of Quantitative Health Sciences, Lerner Research Institute Cleveland Clinic Foundation, Cleveland, Ohio
    Search for articles by this author
  • Author Footnotes
    # For the Alzheimer's Disease Neuroimaging Initiative: Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database ( As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at:
Published:December 01, 2022DOI:
      Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described.

      Key Words


      AD (Alzheimer's disease), ADNI (Alzheimer's Disease Neuroimaging Initiative), AI (Artificial intelligence), BLOQ (Below the lower limit of quantification), CCC (Concordance correlation coefficient), CFA (Confirmatory factor analysis), CN (Cognitively normal), CT (Computed tomography), DCE-MRI (Dynamic contrast enhanced MRI), DECT (Dual energy computed tomography), DFn (Kurtosis corrected degrees of freedom), DM (Mahalanobis distance), DM2 (Mahalanobis distance squared), DSC (Dynamic susceptibility contrast imaging), DTI (Diffusion tensor imaging), DX (X2-X1), E[X] (The expected value of X), EFA (Exploratory factor analysis), EMCI (Early mild cognitive impairment), FDA (Food and Drug Administration), FDG (Flurodeoxyglucose), GFI (Goodness of fit index), LLOQ (Lower limit of quantification), LMCI (Late mild cognitive impairment), LOA (Limits of Agreement), LOQ (Limit of quantification), MANOVA (Multivariate analysis of variance), MANOVA-RM (MANOVA Repeated measures), MAR (Missing at random), MCAR (Missing completely at random), MCCC (Multivariate CCC), MCD (Minimum covariance determinant), MCI (Mild cognitive impairment), mp-QIB (Multiparametric QIB), MRI (Magnetic resonance imaging), PET (Positron emission tomography), QIB (Quantitative Imaging Biomarker), QIBA (Quantitative Imaging Biomarker Alliance), RC (Repeatability coefficient), RCmp (Multiparametric RC), RDCmp (Multiparametric reproducibility coefficient), RMSEA (Root mean square error of approximation), S (Sample variance estimate), SAS® (Statistical analysis software)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Kessler L
        • Barnhart H
        • Buckler A
        • et al.
        QIBA Terminology Working Group. The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions.
        Stat Methods Med Res. 2015; 24: 9-26
        • QIBA
        Quantitative Imaging Biomarkers Alliance (QIBA).
        2021 (Available at) (Accessed July 6, 2021)
        • Raunig DL
        • McShane LM
        • Pennello G
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment.
        Stat Methods Med Res. 2015; 24: 27-67
        • Sullivan DC
        • Obuchowski NA
        • Kessler LG
        • et al.
        Metrology standards for quantitative imaging biomarkers.
        Radiology. 2015; 277: 813-825
        • Obuchowski NA
        • Reeves AP
        • Huang EP
        • et al.
        Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons.
        Stat Methods Med Res. 2015; 24: 68-106
        • Huang EP
        • Wang X-F
        • Choudhury KR
        • et al.
        Meta-analysis of the technical performance of an imaging procedure: guidelines and statistical methodology.
        Stat Methods Med Res. 2015; 24: 141-174
      1. Committee QM. Accessed June 22, 2022

        • Huang EP
        • Pennello G
        • deSouza NM
        • et al.
        Multiparametric quantitative imaging in risk prediction: recommendations for data acquisition, technical performance assessment, and model development and validation.
        Acad Radiol. 2022;
        • Delfino JG
        • Pennello GA
        • Barnhart HX
        • et al.
        Multiparametric quantitative imaging biomarkers for phenotype classification: a framework for development and validation.
        Acad Radiol. 2022;
        • Obuchowski NA
        • Huang E
        • deSouza NM
        • et al.
        A framework for evaluating the technical performance of multiparameter quantitative imaging biomarkers (mp-QIBs).
        Acad Radiol. 2022;
        • Wang X
        • Pennello G
        • deSouza NM
        • et al.
        Multiparametric data-driven imaging markers: guidelines for development, application reporting of model outputs in radiomics.
        Acad Radiol. 2022;
        • Atkinson Jr, AJ
        • Colburn WA
        • et al.
        • Biomarkers Definitions Working Group
        Biomarkers and surrogate endpoints: preferred definitions and conceptual framework.
        Clin Pharmacol Ther. 2001; 69: 89-95
        • O'Brien PC.
        Procedures for comparing samples with multiple endpoints.
        Biometrics. 1984; : 1079-1087
        • Adams LC
        • Bressem KK
        • Scheibl S
        • et al.
        Multiparametric assessment of changes in renal tissue after kidney transplantation with quantitative mr relaxometry and diffusion-tensor imaging at 3 T.
        J Clin Med. 2020; (9 2020/05/28)
        • Vamvakas A
        • Williams SC
        • Theodorou K
        • et al.
        Imaging biomarker analysis of advanced multiparametric MRI for glioma grading.
        Phys Med. 2019; 60 (2019/03/27): 188-198
        • Sankoh AJ
        • D'Agostino Sr., RB
        • Huque MF
        Efficacy endpoint selection and multiplicity adjustment methods in clinical trials with inherent multiple endpoint issues.
        Stat Med. 2003; 22: 3133-3150
        • Cheson BD
        • Fisher RI
        • Barrington SF
        • et al.
        Recommendations for initial evaluation, staging, and response assessment of Hodgkin and non-Hodgkin lymphoma: the Lugano classification.
        J Clin Oncol. 2014; 32: 3059
        • Galbraith SM
        • Rustin GJ
        • Lodge MA
        • et al.
        Effects of 5, 6-dimethylxanthenone-4-acetic acid on human tumor microcirculation assessed by dynamic contrast-enhanced magnetic resonance imaging.
        J Clin Oncol. 2002; 20: 3826-3840
        • Nanni C
        • Cottereau AS
        • Lopci E
        • et al.
        Report of the 6th International Workshop on PET in lymphoma.
        Leuk Lymphoma. 2017; 58: 2298-2303
        • Padhani AR
        • Miles KA.
        Multiparametric imaging of tumor response to therapy.
        Radiology. 2010; 256: 348-364
      2. Bosca RJ. Methodological development of a multi-parametric quantitative imaging biomarker framework for assessing treatment response with MRI. 2014.

        • Heggemann F
        • Grotz H
        • Welzel G
        • et al.
        Cardiac function after multimodal breast cancer therapy assessed with functional magnetic resonance imaging and echocardiography imaging.
        Int J Radiat Oncol* Biol* Phys. 2015; 93: 836-844
        • Eslami P
        • Parmar C
        • Foldyna B
        • et al.
        Radiomics of coronary artery calcium in the framingham heart study.
        Radiol: Cardiothor Imag. 2020; 2e190119
        • Biering-Sørensen T
        • Solomon SD
        Assessing contractile function when ejection fraction is normal: a case for strain imaging.
        Am Heart Assoc. 2015; : 1-3
        • Selmeryd J
        • Henriksen E
        • Dalen H
        • et al.
        Derivation and evaluation of age-specific multivariate reference regions to aid in identification of abnormal filling patterns: the HUNT and VaMIS studies.
        JACC: Cardiovasc Imag. 2018; 11: 400-408
        • Bots ML
        • Evans GW
        • Riley WA
        • et al.
        Carotid intima-media thickness measurements in intervention studies: design options, progression rates, and sample size considerations: a point of view.
        Stroke. 2003; 34: 2985-2994
        • Wan T
        • Madabhushi A
        • Phinikaridou A
        • et al.
        Spatio-temporal texture (SpTeT) for distinguishing vulnerable from stable atherosclerotic plaque on dynamic contrast enhancement (DCE) MRI in a rabbit model.
        Med Phys. 2014; 41042303
        • Pierre SS
        • Siegelman J
        • Obuchowski NA
        • et al.
        Measurement accuracy of atherosclerotic plaque structure on CT using phantoms to establish ground truth.
        Acad Radiol. 2017; 24: 1203-1215
        • Nir TM
        • Jahanshad N
        • Villalon-Reina JE
        • et al.
        Effectiveness of regional DTI measures in distinguishing Alzheimer's disease, MCI, and normal aging.
        NeuroImage: clin. 2013; 3: 180-195
        • Watson C
        • Busovaca E
        • Foley JM
        • et al.
        White matter hyperintensities correlate to cognition and fiber tract integrity in older adults with HIV.
        J Neurovirol. 2017; 23: 422-429
        • Choi H
        • Charnsangavej C
        • Faria SC
        • et al.
        Correlation of computed tomography and positron emission tomography in patients with metastatic gastrointestinal stromal tumor treated at a single institution with imatinib mesylate: proposal of new computed tomography response criteria.
        J Clin Oncol. 2007; 25: 1753-1759
        • Veale D
        • Reece R
        • Parsons W
        • et al.
        Intra-articular primatised anti-CD4: efficacy in resistant rheumatoid knees. a study of combined arthroscopy, magnetic resonance imaging, and histology.
        Ann Rheum Dis. 1999; 58: 342-349
        • Nachimuthu DS
        • Baladhandapani A
        Multidimensional texture characterization: on analysis for brain tumor tissues using MRS and MRI.
        J Digit Imaging. 2014; 27: 496-506
        • Murgia A
        • Balestrieri A
        • Francone M
        • et al.
        Plaque imaging volume analysis: technique and application.
        Cardiovasc Diagnos Ther. 2020; 10: 1032
        • Sheahan M
        • Ma X
        • Paik D
        • et al.
        Atherosclerotic plaque tissue: noninvasive quantitative assessment of characteristics with software-aided measurements from conventional CT angiography.
        Radiology. 2018; 286: 622-631
        • Barrington SF
        • Mikhaeel NG
        • Kostakoglu L
        • et al.
        Role of imaging in the staging and response assessment of lymphoma: consensus of the International Conference on Malignant Lymphomas Imaging Working Group.
        J Clin Oncol. 2014; 32 (2014/08/11): 3048-3058
        • Nishiyama KK
        • Shane E.
        Clinical imaging of bone microarchitecture with HR-pQCT.
        Curr osteoporos rep. 2013; 11: 147-155
        • Si Y
        • Merz SF
        • Jansen P
        • et al.
        Multidimensional imaging provides evidence for down-regulation of T cell effector function by MDSC in human cancer tissue.
        Sci immunol. 2019; 4 (p.eaaw9159)
        • US Food and Drug Administration
        Multiple endpoints in clinical trials guidance for industry draft guidance.
        Washington, DC, FDA Maryland2017
        • Fleming TR
        • Powers JH.
        Biomarkers and surrogate endpoints in clinical trials.
        Stat Med. 2012; 31: 2973-2984
        • Zemans RL
        • Jacobson S
        • Keene J
        • et al.
        Multiple biomarkers predict disease severity, progression and mortality in COPD.
        Respir Res. 2017; 18: 117
        • Offen W
        • Chuang-Stein C
        • Dmitrienko A
        • et al.
        Multiple co-primary endpoints: medical and statistical solutions a report from the multiple endpoints expert team of the pharmaceutical research and manufacturers of America.
        Drug Inf J. 2007; 41: 31
      3. QIBA Wiki. Available at: Accessed January 7, 2022.

        • Offen WW
        • Helterbrand JD.
        Multiple comparison adjustments when two or more co-primary endpoints must all be statistically significant.
        in: Proceedings of the Annual Meeting of the American Statistical Association, Chicago, IL. 1986
        • Chuang-Stein C
        • Stryszak P
        • Dmitrienko A
        • et al.
        Challenge of multiple co-primary endpoints: a new approach.
        Stat Med. 2007; 26: 1181-1192
        • Chuang-stein C
        • Dmitrienko A
        • Offen W
        Discussion of “some controversial multiple testing problems in regulatory applications.
        J Biopharm Stat. 2009; 19: 14-21
        • Offen W
        • Chuang-Stein C
        • Dmitrienko A
        • et al.
        Multiple co-primary endpoints: medical and statistical solutions: a report from the multiple endpoints expert team of the Pharmaceutical Research and Manufacturers of America.
        Drug Inf J. 2007; 41: 31-46
        • Dmitrienko A
        • Tamhane AC.
        Gatekeeping procedures with clinical trial applications.
        Pharmaceut Stat. 2007; 6: 171-180
        • Pocock SJ
        • Geller NL
        • Tsiatis AA.
        The analysis of multiple endpoints in clinical trials.
        Biometrics. 1987; : 487-498
        • Cordoba G
        • Schwartz L
        • Woloshin S
        • et al.
        Definition, reporting, and interpretation of composite outcomes in clinical trials: systematic review.
        BMJ. 2010; 18: 341
        • Boers M
        • Tugwell P
        • Felson D
        • et al.
        World Health Organization and International League of Associations for Rheumatology core endpoints for symptom modifying antirheumatic drugs in rheumatoid arthritis clinical trials.
        J Rheumatol Suppl. 1994; 41: 86-89
        • Stevens SR
        • Ke MS
        • Parry EJ
        • et al.
        Quantifying skin disease burden in mycosis fungoides–type cutaneous T-cell lymphomas: the Severity-Weighted Assessment Tool (SWAT).
        Arch Dermatol. 2002; 138: 42-48
        • Olsen EA
        • Kim YH
        • Kuzel TM
        • et al.
        Phase IIb multicenter trial of vorinostat in patients with persistent, progressive, or treatment refractory cutaneous T-cell lymphoma.
        J Clin Oncol. 2007; 25: 3109-3115
        • Armstrong PW
        • Westerhout CM
        Composite end points in clinical research: a time for reappraisal.
        Circulation. 2017; 135: 2299-2307
        • Buyse M
        • Molenberghs G
        • Paoletti X
        • et al.
        Statistical evaluation of surrogate endpoints with examples from cancer clinical trials.
        Biomet J. 2016; 58: 104-132
        • Evans SR
        • Rubin D
        • Follmann D
        • et al.
        Desirability of outcome ranking (DOOR) and response adjusted for duration of antibiotic risk (RADAR).
        Clin Infect Dis. 2015; 61: 800-806
        • Finkelstein DM
        • Schoenfeld DA.
        Combining mortality and longitudinal measures in clinical trials.
        Stat Med. 1999; 18: 1341-1354
        • Buyse M.
        Generalized pairwise comparisons of prioritized outcomes in the two-sample problem.
        Stat Med. 2010; 29: 3245-3257
        • Pocock SJ
        • Ariti CA
        • Collier TJ
        • et al.
        The win ratio: a new approach to the analysis of composite endpoints in clinical trials based on clinical priorities.
        Eur Heart J. 2012; 33: 176-182
        • Sun H
        • Davison BA
        • Cotter G
        • et al.
        Evaluating treatment efficacy by multiple end points in phase II acute heart failure clinical trials: analyzing data using a global method.
        Circul: Heart Fail. 2012; 5: 742-749
        • Berry JD
        • Miller R
        • Moore DH
        • et al.
        The Combined Assessment of Function and Survival (CAFS): a new endpoint for ALS clinical trials.
        Amyo lat scleros front deg. 2013; 14: 162-168
        • Doernberg SB
        • Tran TTT
        • Tong SY
        • et al.
        Good studies evaluate the disease while great studies evaluate the patient: development and application of a desirability of outcome ranking endpoint for Staphylococcus aureus bloodstream infection.
        Clin Infect Dis. 2019; 68: 1691-1698
        • Phillips PPJ
        • Morris TP
        • Walker AS.
        DOOR/RADAR: A gateway into the unknown?.
        Clin Infect Dis. 2016; 62: 814-815
        • Follmann D
        • Fay MP
        • Hamasaki T
        • et al.
        Analysis of ordered composite endpoints.
        Stat Med. 2020; 39: 602-616
        • Johnson RA
        • Wichern DW
        Applied multivariate statistical analysis.
        Pearson London, UK2014
        • Wiemker R
        • Bergtholdt M
        • Dharaiya E
        • et al.
        Agreement of CAD features with expert observer ratings for characterization of pulmonary nodules in CT using the LIDC-IDRI database.
        Medical imaging 2009: computer-aided diagnosis. H. International Society for Optics and Photonics, Bellingham, WA2009: 72600
        • Jiang J
        • Wang M
        • Alberts I
        • et al.
        Using radiomics-based modelling to predict individual progression from mild cognitive impairment to Alzheimer's disease.
        Eur J Nucl Med Mol Imaging. 2022; 15: 1-11
      4. Little R, D'Agostino R, Dickersin K, et al. The prevention and treatment of missing data in clinical trials. pannel on handling missing data in clinical trials. 2010.

        • Borman PJ
        • Chatfield MJ.
        Avoid the perils of using rounded data.
        J Pharm Biomed Anal. 2015; 115: 502-508
        • Box GE
        • Cox DR.
        An analysis of transformations.
        J Royal Stat Soc: Series B (Methodol). 1964; 26: 211-243
        • Wayne D.
        CLSI Evaluation of detection capability for clinical laboratory measurement procedures; approved guideline. CLSI document EP17-A2.
        Clinical and Laboratory Standards Institute, Wayne, PA2012
        • Liao SG
        • Lin Y
        • Kang DD
        • et al.
        Missing value imputation in high-dimensional phenomic data: imputable or not, and how?.
        BMC Bioinf. 2014; 15 (2014/11/06): 346
        • Schafer JL.
        Multiple imputation: a primer.
        Stat Methods Med Res. 1999; 8: 3-15
        • Little RJ
        • Rubin DB
        Statistical analysis with missing data.
        John Wiley & Sons, Hoboken, NJ2019
      5. Wang Z, Akande O, Poulos J, et al. Are deep learning models superior for missing data imputation in large surveys? Evidence from an empirical comparison. arXiv preprint arXiv:210309316 2021.

        • Jiang X
        • Hintenlang D
        • White RD
        Lower limit of iron quantification using dual-energy CT—a phantom study.
        J Applied Clin Med Phys. 2021; 22: 299-307
        • Kremkau FW.
        Sonography principles and instruments E-Book.
        Elsevier Health Sciences, St. Louis, MO2019
        • Baba T.
        Evaluation of post wall filter for doppler ultrasound systems. acoustical imaging.
        Springer, Dordrecht, Netherlands2008: 133-138
        • Guo Y
        • Harel O
        • Little RJ.
        How well quantified is the limit of quantification?.
        Epidemiology. 2010; : S10-S16
        • Lyles RH
        • Fan D
        • Chuachoowong R
        Correlation coefficient estimation involving a left censored laboratory assay variable.
        Stat Med. 2001; 20: 2921-2933
        • Barnett HY
        • Geys H
        • Jacobs T
        • et al.
        Methods for non-compartmental pharmacokinetic analysis with observations below the limit of quantification.
        Stat Biopharmaceut Res. 2021; 13: 59-70
        • Zhou H
        • Hartford A
        • Tsai K.
        A bayesian approach for PK/PD modeling with PD data below limit of quantification.
        J Biopharm Stat. 2012; 22: 1220-1243
        • Senn S
        • Holford N
        • Hockey H.
        The ghosts of departed quantities: approaches to dealing with observations below the limit of quantitation.
        Stat Med. 2012; 31: 4280-4295
        • Herbers J
        • Miller R
        • Walther A
        • et al.
        How to deal with non-detectable and outlying values in biomarker research: Best practices and recommendations for univariate imputation approaches.
        Comprehens Psychoneuroendocrinol. 2021; 7100052
        • Harel O
        • Perkins N
        • Schisterman EF.
        The use of multiple imputation for data subject to limits of detection.
        Sri Lankan J Appl Stat. 2014; 5 (2014/12/15): 227-246
      6. Nassiri V, Barnett H, Geys H, et al. BLOQ: impute and analyze data with observations below the limit of quantification, Accessed June 22, 2022

        • Williams JR
        • Kim H-W
        • Crespi CM.
        Modeling observations with a detection limit using a truncated normal distribution with censoring.
        BMC Med Res Method. 2020; 20: 170
        • US Food and Drug Administration
        Technical performance assessment of quantitative imaging in radiological device premarket submissions guidance for industry and food and drug administration staff.
        Center for devices and radiological health. Silver Spring, MD, FDA2022 (June 16)
        • Taouli B
        • Ehman RL
        • Reeder SB.
        Advanced MRI methods for assessment of chronic liver disease.
        Am J Roentgenol. 2009; 193: 14-27
        • Harrell FE.
        Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis.
        Springer, New York, NY2015
      7. Madukaife MS. Use of the theory of Euclidean distance in testing for multivariate normality with application to breast cancer diagnostic data. 2020.

        • Wang F
        • Xiang X
        • Cheng J
        • et al.
        Normface: L2 hypersphere embedding for face verification.
        in: Proceedings of the 25th ACM international conference on Multimedia. 2017: 1041-1049
        • Kullback S
        • Leibler RA.
        On information and sufficiency.
        Ann Mathemat Stat. 1951; 22 (78): 79-86
        • Rao CR.
        Tests of significance in multivariate analysis.
        Biometrika. 1948; 35: 58-79
        • Huberty CJ.
        Mahalanobis distance.
        Wiley Stats Ref: Stat Ref Online. 2014; : 1110-1111
        • Johnson NL
        • Kotz S
        • Johnson NL
        Cont univariat distribut. 1994; 1: 451
        • Knuth DE.
        Art of computer programming, volume 2: Seminumerical algorithms.
        Addison-Wesley Professional, Boston, MA2014
        • Davatzikos C
        • Rathore S
        • Bakas S
        • et al.
        Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome.
        J Med imag. 2018; 5011018
        • Ahmad OF
        • Mori Y
        • Misawa M
        • et al.
        Establishing key research questions for the implementation of artificial intelligence in colonoscopy: a modified Delphi method.
        Endoscopy. 2021; 53: 893-901
        • Katragadda C
        • Finnane A
        • Soyer HP
        • et al.
        Technique standards for skin lesion imaging: a delphi consensus statement.
        JAMA Dermatol. 2017; 153: 207-213
        • Scheltema M
        • Tay K
        • Postema A
        • et al.
        Utilization of multiparametric prostate magnetic resonance imaging in clinical practice and focal therapy: report from a Delphi consensus project.
        World J Urol. 2017; 35: 695-701
        • DeVellis RF.
        Scale development: theory and applications.
        Sage publications, Thousand Oaks, CA2016
        • Joliffe IT
        • Morgan B
        Principal component analysis and exploratory factor analysis.
        Stat Methods Med Res. 1992; 1: 69-95
        • Hatcher L
        • O'Rourke N.
        A step-by-step approach to using SAS for factor analysis and structural equation modeling.
        Sas Institute, Cary, NC2013
        • Anderson JC
        • Gerbing DW.
        Structural equation modeling in practice: a review and recommended two-step approach.
        Psychol Bull. 1988; 103: 411
        • Lin J
        • Bentler PM.
        A third moment adjusted test statistic for small sample factor analysis.
        Multi Behav Res. 2012; 47: 448-462
        • Browne MW.
        Asymptotically distribution-free methods for the analysis of covariance structures.
        Br J Math Stat Psychol. 1984; 37: 62-83
        • Wiens BL
        • Dmitrienko A.
        On selecting a multiple comparison procedure for analysis of a clinical trial: fallback, fixed sequence, and related procedures.
        Stat Biopharmaceut Res. 2010; 2: 22-32
        • Hung HJ
        • Wang S-J
        Some controversial multiple testing problems in regulatory applications.
        J Biopharm Stat. 2009; 19: 1-11
        • Hollander M
        • Wolfe DA
        • Chicken E
        Nonparametric statistical methods.
        John Wiley & Sons, New York, NY2013
        • Petersen RC
        • Aisen PS
        • Beckett LA
        • et al.
        Alzheimer's Disease Neuroimaging Initiative (ADNI): clinical characterization.
        Neurology. 2010; 74 (2009/12/30): 201-209
        • MacCallum RC
        • Widaman KF
        • Zhang S
        • et al.
        Sample size in factor analysis.
        Psychol Methods. 1999; 4: 84
      8. Kohl M, Kolampally S. mpe: Multiple Primary Endpoints. 2017.

        • Yang S
        • Moerbeek M
        • Taljaard M
        • Li F
        Power analysis for cluster randomized trials with continuous co-primary endpoints.
        Biometrics. 2022; : 1-13
        • Rencher AC.
        A review of “methods of multivariate analysis”.
        Taylor & Francis, New York, NY2005: 558-559
        • Hubert M
        • Debruyne M
        • Rousseeuw PJ.
        Minimum covariance determinant and extensions.
        Wiley Interdiscip Rev Comput Stat. 2018; 10: e1421
        • Rao CR.
        Generalized inverse of a matrix and its applications. 1. University of California Press, Berkley, CA1972: 601-620 (Theory of Statistics)
        • Ben-Israel A
        • Greville TN
        Generalized inverses: theory and applications.
        Springer Science & Business Media, New York, NY2003
        • Rousseeuw PJ
        • Driessen KV.
        A fast algorithm for the minimum covariance determinant estimator.
        Technometrics. 1999; 41: 212-223
        • Lin H
        • Sardana M
        • Zhang Y
        • et al.
        Association of habitual physical activity with cardiovascular disease risk.
        Circ Res. 2020; : 1253-1260
        • Hiriote S
        • Chinchilli VM.
        Matrix-based concordance correlation coefficient for repeated measures.
        Biometrics. 2011; 67: 1007-1016
        • Brown MB
        • Forsythe AB.
        Robust tests for the equality of variances.
        J Am Statist Assoc. 1974; 69: 364-367
        • O'Brien PC.
        Robust procedures for testing equality of covariance matrices.
        Biometrics. 1992; 1: 819-827
        • Tiku M
        • Balakrishnan N.
        Testing the equality of variance-covariance matrices the robust way.
        Commun Theory Methods. 1985; 14: 3033-3051
        • BIPM I
        • IFCC ILAC
        • IUPAC IUPAP
        • et al.
        3rd Edition. The international vocabulary of metrology—basic and general concepts and associated terms (VIM. 200. JCGM, 2012: 2012
        • Schott JR.
        Some tests for the equality of covariance matrices.
        J Stat Plan Inference. 2001; 94: 25-36
        • Garcia C.
        A simple procedure for the comparison of covariance matrices.
        BMC Evol Biol. 2012; 12: 222
        • Cho E
        • Cho MJ.
        Variance of sample variance.
        Sec Survey Res Methods–JSM. 2008; 2: 1291-1293
        • O'Neill B.
        Some useful moment results in sampling problems.
        Am Stat. 2014; 68: 282-296
        • Guerrero-Gonzalez JM
        • Yeske B
        • Kirk GR
        • et al.
        Mahalanobis distance tractometry (MaD-Tract)–a framework for personalized white matter anomaly detection applied to TBI.
        Neuroimage. 2022; 260119475
        • Kang K
        • Pan D
        • Song X.
        A joint model for multivariate longitudinal and survival data to discover the conversion to Alzheimer's disease.
        Stat Med. 2022; 41: 356-373
        • Lin J
        • Li K
        • Luo S
        Functional survival forests for multivariate longitudinal outcomes: dynamic prediction of Alzheimer's disease progression.
        Stat Methods Med Res. 2021; 30: 99-111
        • Sur C
        • Kost J
        • Scott D
        • et al.
        BACE inhibition causes rapid, regional, and non-progressive volume reduction in Alzheimer's disease brain.
        Brain. 2020; 143: 3816-3826
        • Schwarz AJ
        • Sundell KL
        • Charil A
        • et al.
        Magnetic resonance imaging measures of brain atrophy from the EXPEDITION3 trial in mild Alzheimer's disease.
        Alzheimer's Dement: Translat Res Clin Intervent. 2019; 5: 328-337
        • Jack Jr., CR
        • Knopman DS
        • Jagust WJ
        • et al.
        Tracking pathophysiological processes in Alzheimer's disease: an updated hypothetical model of dynamic biomarkers.
        lancet neurol. 2013; 12: 207-216
      9. Imaging & Data Archive at the Laboratory of Neuroimging (LONI). ADNI study data: ADNIMERGE, Available at: (Accessed June 23, 2022).