Assessment of Radiologist Performance in the Detection of Lung Nodules: Dependence on the Definition of “Truth”

Armato, Samuel G.; Roberts, Rachael Y.; Kocherginsky, Masha; Aberle, Denise R.; Kazerooni, Ella A.; MacMahon, Heber; van Beek, Edwin J.R.; Yankelevitz, David; McLennan, Geoffrey; McNitt-Gray, Michael F.; Meyer, Charles R.; Reeves, Anthony P.; Caligiuri, Philip; Quint, Leslie E.; Sundaram, Baskaran; Croft, Barbara Y.; Clarke, Laurence P.

doi:10.1016/j.acra.2008.05.022

« Previous Next »Academic Radiology
Volume 16, Issue 1 , Pages 28-38, January 2009

Assessment of Radiologist Performance in the Detection of Lung Nodules:1

Dependence on the Definition of “Truth”

Samuel G. Armato III, PhD
- Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
- Address correspondence to: S.G.A.
Rachael Y. Roberts, MD
- Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
Masha Kocherginsky, PhD
- Department of Health Studies, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
Denise R. Aberle, MD
- Department of Radiological Sciences, University of California-Los Angeles, Los Angeles, CA
Ella A. Kazerooni, MD, MS
- Department of Radiology, University of Michigan, Ann Arbor, MI
Heber MacMahon, MD
- Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
Edwin J.R. van Beek, MD, PhD
- Department of Radiology, University of Iowa, Iowa City, IA
David Yankelevitz, MD
- Weill Medical College, Cornell University, Ithaca, NY
Geoffrey McLennan, MD, PhD
- Departments of Medicine, Radiology, and Biomedical Engineering, University of Iowa, Iowa City, IA
Michael F. McNitt-Gray, PhD
- Department of Radiological Sciences, University of California-Los Angeles, Los Angeles, CA
Charles R. Meyer, PhD
- Department of Radiology, University of Michigan, Ann Arbor, MI
Anthony P. Reeves, PhD
- Department of Electrical and Computer Engineering, Cornell University, Ithaca, NY
Philip Caligiuri, MD
- Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
Leslie E. Quint, MD
- Department of Radiology, University of Michigan, Ann Arbor, MI
Baskaran Sundaram, MD
- Department of Radiology, University of Michigan, Ann Arbor, MI
Barbara Y. Croft, PhD
- Cancer Imaging Program, National Cancer Institute, Rockville, MD
Laurence P. Clarke, PhD
- Cancer Imaging Program, National Cancer Institute, Rockville, MD

Received 28 February 2008; accepted 19 May 2008.

Rationale and Objectives

Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this study was to analyze (1) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the context of lung nodule detection in computed tomographic (CT) scans.

Materials and Methods

Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥3 mm in maximum diameter. Panel “truth” sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel “truth” sets.

Results

The number of “true” nodules in the different panel “truth” sets ranged from 15 to 89 (mean 49.8 ± 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel “truth” sets for different panel “truth” conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case.

Conclusions

Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of “truth” on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.

Key Words: Lung nodule, computed tomography (CT), thoracic imaging, interobserver variability, computer-aided diagnosis (CAD)