Academic Radiology
Volume 16, Issue 1 , Pages 28-38 , January 2009

Assessment of Radiologist Performance in the Detection of Lung Nodules: Dependence on the Definition of “Truth”

  • Samuel G. Armato III, PhD

      Affiliations

    • Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
    • Corresponding Author InformationAddress correspondence to: S.G.A.
    • ,
  • Rachael Y. Roberts, MD

      Affiliations

    • Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
    • ,
  • Masha Kocherginsky, PhD

      Affiliations

    • Department of Health Studies, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
    • ,
  • Denise R. Aberle, MD

      Affiliations

    • Department of Radiological Sciences, University of California-Los Angeles, Los Angeles, CA
    • ,
  • Ella A. Kazerooni, MD, MS

      Affiliations

    • Department of Radiology, University of Michigan, Ann Arbor, MI
    • ,
  • Heber MacMahon, MD

      Affiliations

    • Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
    • ,
  • Edwin J.R. van Beek, MD, PhD

      Affiliations

    • Department of Radiology, University of Iowa, Iowa City, IA
    • ,
  • David Yankelevitz, MD

      Affiliations

    • Weill Medical College, Cornell University, Ithaca, NY
    • ,
  • Geoffrey McLennan, MD, PhD

      Affiliations

    • Departments of Medicine, Radiology, and Biomedical Engineering, University of Iowa, Iowa City, IA
    • ,
  • Michael F. McNitt-Gray, PhD

      Affiliations

    • Department of Radiological Sciences, University of California-Los Angeles, Los Angeles, CA
    • ,
  • Charles R. Meyer, PhD

      Affiliations

    • Department of Radiology, University of Michigan, Ann Arbor, MI
    • ,
  • Anthony P. Reeves, PhD

      Affiliations

    • Department of Electrical and Computer Engineering, Cornell University, Ithaca, NY
    • ,
  • Philip Caligiuri, MD

      Affiliations

    • Department of Radiology, University of Chicago, 5841 S. Maryland Ave, MC 2026, Chicago, IL 60637
    • ,
  • Leslie E. Quint, MD

      Affiliations

    • Department of Radiology, University of Michigan, Ann Arbor, MI
    • ,
  • Baskaran Sundaram, MD

      Affiliations

    • Department of Radiology, University of Michigan, Ann Arbor, MI
    • ,
  • Barbara Y. Croft, PhD

      Affiliations

    • Cancer Imaging Program, National Cancer Institute, Rockville, MD
    • ,
  • Laurence P. Clarke, PhD

      Affiliations

    • Cancer Imaging Program, National Cancer Institute, Rockville, MD

Received 28 February 2008 ,Accepted 19 May 2008.

References 

  1. Wormanns D, Ludwig K, Beyer F, et al. Detection of pulmonary nodules at multirow-detector CT: effectiveness of double reading to improve sensitivity at standard-dose and low-dose chest CT. Eur Radiol. 2005;15:14–22
  2. Leader JK, Warfel TE, Fuhrman CR, et al. Pulmonary nodule detection with low-dose CT of the lung: agreement among radiologists. Am J Roentgenol. 2005;185:973–978
  3. Novak CL, Qian J, Fan L, et al. Inter-observer variations on interpretation of multi-slice CT lung cancer screening studies, and the implications for computer-aided diagnosis. SPIE Proc. 2002;4684:68–79
  4. Ochs R, Kim HJ, Angel E, et al. Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance. SPIE Proc. 2007;6514:
  5. Paquerault S, Petrick N, Myers KJ, et al. Impact of a computer-aided detection (CAD) system on reader performance: assessment based on a truthing panel compared to the true gold standard. Radiology. 2007;245:546–547
  6. Petrick N, Gallas BD, Samuelson FW, et al. Influence of panel size and expert skill on truth panel performance when combining expert ratings. SPIE Proc. 2005;5749:49–57
  7. Armato SG, McLennan G, McNitt-Gray MF, et al. Lung Image Database Consortium: developing a resource for the medical imaging research community. Radiology. 2004;232:739–748
  8. Armato SG, McNitt-Gray MF, Reeves AP, et al. The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans. Acad Radiol. 2007;14:1409–1421
  9. Dodd LE, Wagner RF, Armato SG, et al. Assessment methodologies and statistical issues for computer-aided diagnosis of lung nodules in computed tomography: contemporary research topics relevant to the Lung Image Database Consortium. Acad Radiol. 2004;11:462–475
  10. McNitt-Gray MF, Armato SG, Meyer CR, et al. The Lung Image Database Consortium (LIDC) data collection process for nodule detection and annotation. Acad Radiol. 2007;14:1464–1474
  11. Miller DP, O'Shaughnessy KF, Wood SA, et al. Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. SPIE Proc. 2004;5372:173–184
  12. Revesz G, Kundel HL, Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol. 1983;18:194–198
  13. Jiang Y. A Monte Carlo simulation method to understand expert-panel consensus truth and double readings (Medical Image Perception Conference XII. 2007). Iowa City, IA: The University of Iowa; 2007;
  14. Clarke LP, Croft BY, Staab E, et al. National Cancer Institute initiative: lung image database resource for imaging research. Acad Radiol. 2001;8:447–450
  15. Department of Health and Human Services. Standards for privacy of individually identifiable health information: final rules. Fed Register. 2002;67:53182–53272
  16. Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Invest Radiol. 1978;13:175–181
  17. Meyer CR, Johnson TD, McLennan G, et al. Evaluation of lung MDCT nodule annotation across radiologists and methods. Acad Radiol. 2006;13:1254–1265
  18. Reeves AP, Biancardi AM, Apanasovich TV, et al. The Lung Image Database Consortium (LIDC): a comparison of different size metrics for pulmonary nodule measurements. Acad Radiol. 2007;14:1475–1485
  19. Schwartz LH, Ginsberg MS, DeCorato D, et al. Evaluation of tumor measurements in oncology: Use of film-based and electronic techniques. J Clin Oncol. 2000;18:2179–2184
  20. Ross JC, Miller JV, Turner WD, et al. An analysis of early studies released by the Lung Imaging Database Consortium (LIDC). Acad Radiol. 2007;14:1382–1388

1 Supported in part by USPHS Grants U01CA091085, U01CA091090, U01CA091099, U01CA091100, and U01CA091103.

PII: S1076-6332(08)00359-0

doi: 10.1016/j.acra.2008.05.022

Academic Radiology
Volume 16, Issue 1 , Pages 28-38 , January 2009

Article Tools

* Image Usage & Resolution