Rationale and Objectives
Computer-aided diagnostic (CAD) schemes have been developed for assisting radiologists
in the detection of various lesions in medical images. The reliable evaluation of
CAD schemes is an important task in the field of CAD research.
Materials and Methods
Many evaluation approaches have been proposed for evaluating the performance of various
CAD schemes in the past. However, some important issues in the evaluation of CAD schemes
have not been systematically analyzed. The first important issue is the analysis and
comparison of various evaluation methods in terms of certain characteristics. The
second includes the analysis of pitfalls in the incorrect use of various evaluation
methods and the effective approaches to the reduction of the bias and variance caused
by these pitfalls. We attempt to address the first important issue in details in this
article by conducting Monte Carlo simulation experiments, and to discuss the second
issue in the Discussion section.
Results
No single evaluation method is universally superior to the others; different situations
of CAD applications require different evaluation methods, as recommended in this article.
Bias and variance in the estimated performance levels caused by various pitfalls can
be reduced considerably by the correct use of good evaluation methods.
Conclusions
This article would be useful to researchers in the field of CAD research for selecting
appropriate evaluation methods and for improving the reliability of the estimated
performance of their CAD schemes.
Key Words
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Academic RadiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Improvement in radiologists’ detection of clustered microcalcifications on mammograms: the potential of computer-aided diagnosis.Invest Radiol. 1990; 25: 1102-1110
- Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs.Radiology. 1996; 199: 843-848
- Effects of sample size on classifier design.IEEE Trans Pattern Anal Mach Intell. 1989; 11: 873-885
- Estimation of classifier performance.IEEE Trans Pattern Anal Mach Intell. 1989; 11: 1087-1101
- Finite-sample effects and resampling plans: applications to linear classifiers in computer-aided diagnosis.Proc SPIE Conf Medical Imaging. 1997; 3034: 467-477
- Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers.Med Phys. 1999; 26: 2654-2668
- Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size.Med Phys. 2000; 27: 1509-1522
- On the repeated use of database for testing incremental improvement of computer-aided detection schemes.Acad Radiol. 2004; 11 (R.): 103-105
- The reduction of bias and variance for the evaluation of computer-aided diagnostic scheme.Med Phys. 2006; 33: 868-875
- An Antroduction to bootstrap. Chapman and Hall, New York1993
- Automated computerized scheme for distinction between benign and malignant solitary pulmonary nodules on chest images.Med Phys. 2002; 29: 701-708
- Computerized scheme for determination of the likelihood measure of malignancy for pulmonary nodules on low-dose CT images.Med Phys. 2003; 30: 387-394
- Bayesian methods for adaptive models. 1992 (Available online at: http://wol.ra.phy.cam.ac.uk/mackay/thesis.pdf)
- Ideal observer approximation using Bayesian classification neural networks.IEEE Trans Med Imaging. 2001; 20: 886-899
- Computer-aided diagnosis of mammographic microcalcification clusters.Med Phys. 2004; 31: 314-326
- Computerized analysis of mammographic parenchymal patterns for assessing breast cancer risk: effect of ROI size and location.Med Phys. 2004; 31: 549-555
- Computerized characterization of breast masses on three-dimensional ultrasound volumes.Med Phys. 2004; 31: 744-754
- Computer-aided diagnosis scheme for histological classification of clustered microcalcifications on magnification mammograms.Med Phys. 2004; 31: 789-799
- Computerized detection of colorectal masses in CT colonography based on fuzzy merging and wall-thickening analysis.Med Phys. 2004; 31: 860-872
- Incorporation of an iterative, linear segmentation routine into a mammographic mass CAD system.Med Phys. 2004; 31: 1512-1520
- Reduction of false positives on the rectal tube in computer-aided detection for CT colonography.Med Phys. 2004; 31: 2855-2862
- A computer-aided diagnosis for locating abnormalities in bone scintigraphy by a fuzzy system with a three step minimization approach.IEEE Trans Med Imag. 2004; 23: 639-654
- Surface normal overlap: a computer-aided detection algorithm with application to colonic polyps and lung nodules in helical CT.IEEE Trans Med Imag. 2004; 23: 661-675
- Computer-aided diagnosis of solid breast nodules: use of an artificial neural network based on multiple sonographic features.IEEE Trans Med Imaging. 2004; 23: 1292-1300
Article info
Publication history
Accepted:
April 29,
2007
Received:
December 9,
2006
Footnotes
1This work was supported by USPHS grants CA62625, CA64370, and CA113820. Q. Li is a consultant to Riverain Medical Group, Miamisburg, OH. CAD technologies developed at the Kurt Rossmann Laboratories for Radiologic Image Research, the University of Chicago, have been licensed to companies including R2 Technologies, Riverain Medical Group, Deus Technologies, Median Technology, Mitsubishi Space Software Co., General Electric Corporation, and Toshiba Corporation. It is the policy of the University of Chicago that investigators disclose publicly actual or potential significant financial interests that may appear to be affected by research activities.
Identification
Copyright
© 2007 AUR. Published by Elsevier Inc. All rights reserved.
