Advertisement

Assessment of Radiologist Performance in the Detection of Lung Nodules

Dependence on the Definition of “Truth”

      Rationale and Objectives

      Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this study was to analyze (
      • Wormanns D.
      • Ludwig K.
      • Beyer F.
      • et al.
      Detection of pulmonary nodules at multirow-detector CT: effectiveness of double reading to improve sensitivity at standard-dose and low-dose chest CT.
      ) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (
      • Leader J.K.
      • Warfel T.E.
      • Fuhrman C.R.
      • et al.
      Pulmonary nodule detection with low-dose CT of the lung: agreement among radiologists.
      ) variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the context of lung nodule detection in computed tomographic (CT) scans.

      Materials and Methods

      Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel “truth” sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel “truth” sets.

      Results

      The number of “true” nodules in the different panel “truth” sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel “truth” sets for different panel “truth” conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case.

      Conclusions

      Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of “truth” on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Wormanns D.
        • Ludwig K.
        • Beyer F.
        • et al.
        Detection of pulmonary nodules at multirow-detector CT: effectiveness of double reading to improve sensitivity at standard-dose and low-dose chest CT.
        Eur Radiol. 2005; 15: 14-22
        • Leader J.K.
        • Warfel T.E.
        • Fuhrman C.R.
        • et al.
        Pulmonary nodule detection with low-dose CT of the lung: agreement among radiologists.
        Am J Roentgenol. 2005; 185: 973-978
        • Novak C.L.
        • Qian J.
        • Fan L.
        • et al.
        Inter-observer variations on interpretation of multi-slice CT lung cancer screening studies, and the implications for computer-aided diagnosis.
        SPIE Proc. 2002; 4684: 68-79
        • Ochs R.
        • Kim H.J.
        • Angel E.
        • et al.
        Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance.
        SPIE Proc. 2007; 6514
        • Paquerault S.
        • Petrick N.
        • Myers K.J.
        • et al.
        Impact of a computer-aided detection (CAD) system on reader performance: assessment based on a truthing panel compared to the true gold standard.
        Radiology. 2007; 245: 546-547
        • Petrick N.
        • Gallas B.D.
        • Samuelson F.W.
        • et al.
        Influence of panel size and expert skill on truth panel performance when combining expert ratings.
        SPIE Proc. 2005; 5749: 49-57
        • Armato III, S.G.
        • McLennan G.
        • McNitt-Gray M.F.
        • et al.
        Lung Image Database Consortium: developing a resource for the medical imaging research community.
        Radiology. 2004; 232: 739-748
        • Armato III, S.G.
        • McNitt-Gray M.F.
        • Reeves A.P.
        • et al.
        The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans.
        Acad Radiol. 2007; 14: 1409-1421
        • Dodd L.E.
        • Wagner R.F.
        • Armato III, S.G.
        • et al.
        Assessment methodologies and statistical issues for computer-aided diagnosis of lung nodules in computed tomography: contemporary research topics relevant to the Lung Image Database Consortium.
        Acad Radiol. 2004; 11: 462-475
        • McNitt-Gray M.F.
        • Armato III, S.G.
        • Meyer C.R.
        • et al.
        The Lung Image Database Consortium (LIDC) data collection process for nodule detection and annotation.
        Acad Radiol. 2007; 14: 1464-1474
        • Miller D.P.
        • O'Shaughnessy K.F.
        • Wood S.A.
        • et al.
        Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions.
        SPIE Proc. 2004; 5372: 173-184
        • Revesz G.
        • Kundel H.L.
        • Bonitatibus M.
        The effect of verification on the assessment of imaging techniques.
        Invest Radiol. 1983; 18: 194-198
        • Jiang Y.
        A Monte Carlo simulation method to understand expert-panel consensus truth and double readings.
        The University of Iowa, Iowa City, IA2007
        • Clarke L.P.
        • Croft B.Y.
        • Staab E.
        • et al.
        National Cancer Institute initiative: lung image database resource for imaging research.
        Acad Radiol. 2001; 8: 447-450
        • Department of Health and Human Services
        Standards for privacy of individually identifiable health information: final rules.
        Fed Register. 2002; 67: 53182-53272
        • Kundel H.L.
        • Nodine C.F.
        • Carmody D.
        Visual scanning, pattern recognition and decision-making in pulmonary nodule detection.
        Invest Radiol. 1978; 13: 175-181
        • Meyer C.R.
        • Johnson T.D.
        • McLennan G.
        • et al.
        Evaluation of lung MDCT nodule annotation across radiologists and methods.
        Acad Radiol. 2006; 13: 1254-1265
        • Reeves A.P.
        • Biancardi A.M.
        • Apanasovich T.V.
        • et al.
        The Lung Image Database Consortium (LIDC): a comparison of different size metrics for pulmonary nodule measurements.
        Acad Radiol. 2007; 14: 1475-1485
        • Schwartz L.H.
        • Ginsberg M.S.
        • DeCorato D.
        • et al.
        Evaluation of tumor measurements in oncology: Use of film-based and electronic techniques.
        J Clin Oncol. 2000; 18: 2179-2184
        • Ross J.C.
        • Miller J.V.
        • Turner W.D.
        • et al.
        An analysis of early studies released by the Lung Imaging Database Consortium (LIDC).
        Acad Radiol. 2007; 14: 1382-1388