Academic Radiology
Volume 14, Issue 7 , Pages 814-829 , July 2007

Reliable and Computationally Efficient Maximum-Likelihood Estimation of “Proper” Binormal ROC Curves

Received 1 July 2006 ,Accepted 23 March 2007.

References 

  1. Roe CA, Metz CE. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 1997;4:587–600
  2. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis (Generalization to the population of readers and patients with the jackknife method). Invest Radiol. 1992;27:723–731
  3. Obuchowski NA, Rockette HE. Hypothesis-testing of diagnostic-accuracy for multiple readers and multiple tests—An ANOVA approach with dependent observations. Commun Stat Simulation Comput. 1995;24:285–308
  4. Beiden SV, Wagner RF, Campbell G. Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol. 2000;7:341–349
  5. Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computer-assist systems: Lessons from recent experience. Acad Radiol. 2002;9:1264–1277
  6. Toledano AY, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 1996;15:1807–1826
  7. Gallas BD. One-shot estimate of MRMC variance: AUC. Acad Radiol. 2006;13:353–362
  8. Dorfman DD, Alf E. Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data. J Math Psychol. 1969;6:487–496
  9. Hanley JA. Receiver operating characteristic (ROC) methodology: The state of the art. Crit Rev Diagn Imaging. 1989;29:307–335
  10. Swets JA. Form of empirical ROCs in discrimination and diagnostic tasks: Implications for theory and measurement of performance. Psychol Bull. 1986;99:181–198
  11. Metz CE, Herman BA, Shen JH. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Stat Med. 1998;17:1033–1053
  12. Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–245
  13. Hanley JA. The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis Making. 1988;8:197–203
  14. Obuchowski NA, Beiden SV, Berbaum KS, et al. Multireader, multicase receiver operating characteristic analysis: An empirical comparison of five methods. Acad Radiol. 2004;11:980–995
  15. Metz CE, Pan X. “Proper” binormal ROC curves: Theory and maximum-likelihood estimation. J Math Psychol. 1999;43:1–33
  16. Pan X, Metz CE. The “proper” binormal model: Parametric receiver operating characteristic curve estimation with degenerate data. Acad Radiol. 1997;4:380–389
  17. Dorfman DD, Berbaum KS. A contaminated binormal model for ROC data: Part III (Initial evaluation with detection ROC data). Acad Radiol. 2000;7:438–447
  18. Bandos AI, Rockette HE, Gur D. Incorporating utility-weights when comparing two diagnostic systems: A preliminary assessment. Acad Radiol. 2005;12:1293–1300
  19. Obuchowski NA. Estimating and comparing diagnostic tests’ accuracy when the gold standard is not binary. Acad Radiol. 2005;12:1198–1204
  20. Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imaging. 2004;23:891–895
  21. Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720–733
  22. Flannery BP, Teukolsky SA, Press WH, Vetterling WT. Numerical recipes in FORTRAN: The art of scientific computing. Cambridge, England/New York: Cambridge University Press; 1992;
  23. Gay DM. Subroutines for unconstrained minimization using a model trust-region approach. ACM Trans Math Software. 1983;9:503–524
  24. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics. 1988;44:837–845
  25. Efron B, Tibshirani R. An introduction to the bootstrap. New York: Chapman & Hall; 1993;
  26. Rao CR. Linear statistical inference and its applications. New York: Wiley; 1965;
  27. Papoulis A. Probability, random variables, and stochastic processes. New York: McGraw-Hill; 1984;
  28. Campbell SL, Meyer CD. Generalized inverses of linear transformations. New York: Dover Publications; 1991;
  29. Rao CR, Mitra SK. Generalized inverse of matrices and its applications. New York: Wiley; 1971;
  30. McClish DK. Analyzing a portion of the ROC curve. Med Decis Making. 1989;9:190–195
  31. Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996;201:745–750
  32. Metz C, Kronman H. Statistical significance tests for binormal ROC curves. J Math Psychol. 1980;22:218–243
  33. Kupinski MA, Edwards DC, Giger ML, Metz CE. Ideal observer approximation using Bayesian classification neural networks. IEEE Trans Med Imaging. 2001;20:886–899
  34. Glantz SA. Primer of biostatistics. New York: McGraw-Hill Medical Publishing Division; 2002;
  35. Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–298
  36. Shiraishi J, Katsuragawa S, Ikezoe J, et al. Development of a digital image database for chest radiographs with and without a lung nodule: Receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. AJR Am J Roentgenol. 2000;174:71–74
  37. Bamber D. Area above ordinal dominance graph and area below receiver operating characteristic graph. J Math Psychol. 1975;12:387–415

1 This work was supported by National Institutes of Health grant R01 EB000863 (Kevin S. Berbaum, Principal Investigator) through a University of Chicago contract with the University of Iowa.

PII: S1076-6332(07)00177-8

doi: 10.1016/j.acra.2007.03.012

Academic Radiology
Volume 14, Issue 7 , Pages 814-829 , July 2007

Article Tools

* Image Usage & Resolution