Management of Incidental Thyroid Nodules on Chest CT: Using Natural Language Processing to Assess White Paper Adherence and Track Patient Outcomes

Published:March 20, 2021DOI:


      The purpose of this study was to develop a natural language processing (NLP) pipeline to identify incidental thyroid nodules (ITNs) meeting criteria for sonographic follow-up and to assess both adherence rates to white paper recommendations and downstream outcomes related to these incidental findings.


      21583 non-contrast chest CT reports from 2017 and 2018 were retrospectively evaluated to identify reports which included either an explicit recommendation for thyroid ultrasound, a description of a nodule ≥ 1.5 cm, or description of a nodule with suspicious features. Reports from 2018 were used to train an NLP algorithm called fastText for automated identification of such reports. Algorithm performance was then evaluated on the 2017 reports. Next, any patient from 2017 with a report meeting criteria for ultrasound follow-up was further evaluated with manual chart review to determine follow-up adherence rates and nodule-related outcomes.


      NLP identified reports with ITNs meeting criteria for sonographic follow-up with an accuracy of 96.5% (95% CI 96.2-96.7) and sensitivity of 92.1% (95% CI 89.8-94.3). In 10006 chest CTs from 2017, ITN follow-up ultrasound was indicated according to white paper criteria in 81 patients (0.8%), explicitly recommended in 46.9% (38/81) of patients, and obtained in less than half of patients in which it was appropriately recommended (17/35, 48.6%).


      NLP accurately identified chest CT reports meeting criteria for ITN ultrasound follow-up. Radiologist adherence to white paper guidelines and subsequent referrer adherence to radiologist recommendations showed room for improvement.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ahmed S
        • Horton KM
        • Brooke Jeffrey R
        • et al.
        Incidental thyroid nodules on chest ct: review of the literature and management suggestions [internet].
        Am J Roentgenol. 2010; 195: 1066-1071
        • Mortensen JD
        • Woolner LB
        • Bennett WA.
        Gross and microscopic findings in clinically normal thyroid glands* [Internet].
        J Clin Endocrinol Metab. 1955; 15: 1270-1280
        • Hoang JK
        • Langer JE
        • Middleton WD
        • et al.
        Managing incidental thyroid nodules detected on imaging: white paper of the ACR incidental thyroid findings committee [internet].
        J Am Coll Radiol. 2015; 12: 143-150
        • Short RG
        • Befera NT
        • Hoang JK
        • et al.
        A normal thyroid by any other name: linguistic analysis of statements describing a normal thyroid gland from noncontrast chest CT reports.
        J Am Coll Radiol. 2018; 15: 1642-1647
        • Carrodeguas E
        • Lacson R
        • Swanson W
        • et al.
        Use of machine learning to identify follow-up recommendations in radiology reports.
        J Am Coll Radiol. 2019; : 336-343
        • Pons E
        • Braun LMM
        • Myriam Hunink MG
        • et al.
        Natural language processing in radiology: a systematic review.
        [internet] Radiology. 2016; 279: 329-343
        • Dutta S
        • Long WJ
        • Brown DFM
        • et al.
        Automated detection using natural language processing of radiologists recommendations for additional imaging of incidental findings [internet].
        Ann Emerg Med. 2013; 62: 162-169
        • Farjah F
        • Halgrim S
        • Buist DSM
        • et al.
        An automated method for identifying individuals with a lung nodule can be feasibly implemented across health systems [internet].
        eGEMs. 2016; 4: 15
        • Kang SK
        • Garry K
        • Chung R
        • et al.
        Natural language processing for identification of incidental pulmonary nodules in radiology reports.
        J Am Coll Radiol. 2019; (accessed 10/14/20)
        • Xu Y
        • Tsujii J
        • Chang EIC.
        Named entity recognition of follow-up and time information in 20 000 radiology reports.
        J Am Med Inform Assoc. 2012; ([Internet]. Available from:) (accessed 10/14/20)
        • Short RG
        • Bralich J
        • Bogaty D
        • et al.
        Comprehensive word-level classification of screening mammography reports using a neural network sequence labeling approach.
        J Digit Imaging. 2019; ([Internet]) (accessed 10/14/20)
        • Chen MC
        • Ball RL
        • Yang L
        • et al.
        Deep learning to classify radiology free-text reports.
        Radiology. 2018; 286: 845-852
        • Bird S
        • Klein E
        • Loper E.
        Natural language processing with Python: analyzing text with the natural language toolkit.
        O'Reilly Media, Inc., Valencia, Spain, 2009: 504
        • Joulin A
        • Grave E
        • Bojanowski P
        • et al.
        Bag of Tricks for Efficient Text Classification.
        in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2. 2017 (Short Papers [Internet])
        • Chollet F.
        2015 (accessed 10/14/20)
        • Kingma DP
        • Adam BJ.
        A Method for Stochastic Optimization [Internet]. arXiv [cs.LG].
        2014 (accessed 10/14/20)
        • Abadi M
        • Barham P
        • Chen J
        • et al.
        Tensorflow: A system for large-scale machine learning. on operating systems … [Internet].
        2016 (accessed 10/14/20)
        • Oliphant TE.
        A Bayesian perspective on estimating mean, variance, and standard-deviation from data.
        2006 (accessed 10/14/20)
        • Development Core Team RR
        R: A language and environment for statistical computing.
        2011 (accessed 10/14/20)
        • Virtanen P
        • Gommers R
        • Oliphant TE
        • et al.
        SciPy 1.0: fundamental algorithms for scientific computing in Python.
        Nat Methods. 2020; 17: 261-272
        • Wandtke B
        • Gallagher S.
        Reducing delay in diagnosis: multistage recommendation tracking.
        AJR Am J Roentgenol. 2017; 209: 970-975
        • Berlin L.
        Communicating findings of radiologic examinations: whither goest the radiologist's duty?.
        AJR Am J Roentgenol. 2002; 178: 809-815
        • Whang JS
        • Baker SR
        • Patel R
        • et al.
        The causes of medical malpractice suits against radiologists in the United States.
        Radiology. 2013; 266: 548-554
        • Singh H
        • Hirani K
        • Kadiyala H
        • et al.
        Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record-based study.
        J Clin Oncol. 2010; 28: 3307-3315
        • Wiener RS
        • Gould MK
        • Slatore CG.
        Resource use and guideline concordance in evaluation of pulmonary nodules for cancer: too much and too little care.
        JAMA Intern Med [Internet]. 2014; (accessed 10/14/20)
        • Mabotuwana T
        • Hombal V
        • Dalal S
        • et al.
        Determining adherence to follow-up imaging recommendations.
        J Am Coll Radiol. 2018; 15: 422-428
        • Mabotuwana T
        • Hall CS
        • Hombal V
        • et al.
        Automated tracking of follow-up imaging recommendations.
        AJR Am J Roentgenol. 2019; 212: 1-8
        • Blagev DP
        • Lloyd JF
        • Conner K
        • et al.
        Follow-up of incidental pulmonary nodules and the radiology report.
        J Am Coll Radiol. 2016; 13: R18-R24
        • McDonald JS
        • Koo CW
        • White D
        • et al.
        Addition of the Fleischner Society Guidelines to Chest CT examination interpretive reports improves adherence to recommended follow-up care for incidental pulmonary nodules.
        Acad Radiol. 2017; 24: 337-344
        • Neal RD
        • Tharmanathan P
        • France B
        • et al.
        Is increased time to diagnosis and treatment in symptomatic cancer associated with poorer outcomes? Systematic review.
        Br J Cancer. 2015; 112: S92-S107
        • Gildea TR
        • DaCosta Byfield S
        • Hogarth DK
        • et al.
        A retrospective analysis of delays in the diagnosis of lung cancer and associated costs.
        Clinicoecon Outcomes Res. 2017; 9: 261-269
        • Murphy DR
        • Singh H
        • Berlin L.
        Communication breakdowns and diagnostic errors: a radiology perspective.
        Diagnosis (Berl). 2014; 1: 253-261