Academic Radiology
Volume 14, Issue 7 , Pages 871-876 , July 2007

“Binary” and “Non-Binary” Detection Tasks: Are Current Performance Measures Optimal?

  • David Gur, ScD

      Affiliations

    • Department of Radiology, School of Medicine, University of Pittsburgh, Suite 4200, Magee-Womens Hospital, Pittsburgh, PA 15213-3180
    • Corresponding Author InformationAddress correspondence to: D.G.
    • ,
  • Howard E. Rockette, PhD

      Affiliations

    • Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA.
    • ,
  • Andriy I. Bandos, PhD

      Affiliations

    • Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA.

Received 1 February 2007 ,Accepted 26 March 2007.

References 

  1. Hadjiiski L, Chan HP, Sahiner B, et al. Improvement in radiologists’ characterization of malignant and benign breast masses on serial mammograms with computer-aided diagnosis: an ROC study. Radiology. 2004;233:255–265
  2. Shiraishi J, Abe H, Li F, et al. Computer-aided diagnosis for the detection and classification of lung cancers on chest radiographs ROC analysis of radiologists’ performance. Acad Radiol. 2006;13:995–1003
  3. Horsch K, Giger ML, Vyborny CJ, et al. Classification of breast lesions with multimodality computer-aided diagnosis: observer study results on an independent clinical data set. Radiology. 2006;240:357–368
  4. Herron JM, Bender T, Campbell WL, et al. Effects of luminance and resolution on observer performance. Radiology. 2000;215:169–174
  5. Gur D, Rockette HE, Armfield DR, et al. Prevalence effect in a laboratory environment. Radiology. 2003;228:10–14
  6. Egan JP, Schulman Al, Greenberg GZ. Operating characteristics determined by binary decisions and by ratings. J Acoust Soc Am. 1959;31:768–773
  7. Rockette HE, Gur D, Metz CE. The use of continuous and discrete confidence judgments in ROC studies. Invest Radiol. 1992;27:169–172
  8. Beiden SV, Wagner RF, Doi K, et al. Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol. 2002;9:1036–1043
  9. Jiang Y, Nishikawa RM, Schmidt RA, Metz CE. Comparison of independent double readings and computer-aided diagnosis (CAD) for the diagnosis of breast calcifications. Acad Radiol. 2006;13:84–94
  10. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic analyses. Med Phys. 1989;16:561–568
  11. Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis and validation. Med Phys. 2004;31:2313–2330
  12. Chakraborty DP. A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol. 2006;51:3449–3462
  13. Gur D, King JL, Rockette HE, et al. Practical issues of experimental ROC analysis (Selection of controls). Invest Radiol. 1990;25:583–586
  14. Dorfman DD, Berbaum KS, Brandser EA. A contaminated binormal model for ROC data: part I (Some interesting examples of binormal degeneracy). Acad Radiol. 2000;7:420–426
  15. Dorfman DD, Berbaum KS. A contaminated binormal model for ROC data: part II (A formal model). Acad Radiol. 2000;7:427–437
  16. Dorfman DD, Berbaum KS. A contaminated binormal model for ROC data: part III (Initial evaluation with detection ROC data). Acad Radiol. 2000;7:438–447
  17. Wagner RF, Beiden SV, Metz CE. Continuous versus categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 2001;8:328–334
  18. Coffin M, Sukhatme S. Receiver operating characteristic studies and measurement error. Biometrics. 1997;53:823–837
  19. Faraggi D. The effect of random measurement error on receiver operating characteristic (ROC) curves. Stat Med. 2000;19:61–70
  20. Wagner RF. US Food and Drug Administration. Computer-aided diagnosis and the general bioinformatics problem. Presented at the SPIE Medical Imaging meeting San Diego (February 2007). Proc SPIE 6514:S3.

 Supported in part by Grants EB001694, EB002106, and EB003503 (to the University of Pittsburgh) from the National Institute for Biomedical Imaging and Bioengineering (NIBIB), National Institute of Health.

PII: S1076-6332(07)00179-1

doi: 10.1016/j.acra.2007.03.014

Academic Radiology
Volume 14, Issue 7 , Pages 871-876 , July 2007

Article Tools