Utility of Artificial Intelligence Tool as a Prospective Radiology Peer Reviewer — Detection of Unreported Intracranial Hemorrhage

Published:February 24, 2020DOI:

      Rationale and Objectives

      Misdiagnosis of intracranial hemorrhage (ICH) can adversely impact patient outcomes. The increasing workload on the radiologists may increase the chance of error and compromise the quality of care provided by the radiologists.

      Materials and Methods

      We used an FDA approved artificial intelligence (AI) solution based on a convolutional neural network to assess the prevalence of ICH in scans, which were reported as negative for ICH. We retrospectively applied the AI solution to all consecutive noncontrast computed tomography (CT) head scans performed at eight imaging sites affiliated to our institution.


      In the 6565 noncontrast CT head scans, which met the inclusion criteria, 5585 scans were reported to have no ICH (“negative-by-report” cases). We applied AI solution to these “negative-by-report” cases. AI solution suggested there were ICH in 28 of these scans (“negative-by-report” and “positive-by-AI solution”). After consensus review by three neuroradiologists, 16 of these scans were found to have ICH, which was not reported (missed diagnosis by radiologists), with a false-negative rate of radiologists for ICH detection at 1.6%. Most commonly missed ICH was overlying the cerebral convexity and in the parafalcine regions.


      Our study demonstrates that an AI solution can help radiologists to diagnose ICH and thus decrease the error rate. AI solution can serve as a prospective peer review tool for non-contrast head CT scans to identify ICH and thus minimize false negatives.

      Key Words


      ICH (Intracranial Hemorrhage), CNN (Convolutional Neural Hemorrhage), AI (Artificial Intelligence), NLP (Natural Language Processing)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • van Asch C.J.J.
        • Luitse M.J.A.
        • Rinkel G.J.E.
        • et al.
        Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis.
        Lancet Neurol. 2010; 9: 167-176
        • Heit J.J.
        • Iv M.
        • Wintermark M.
        Imaging of intracranial hemorrhage.
        J Stroke. 2017; 19: 11-27
        • McDonald R.J.
        • Schwartz K.M.
        • Eckel L.J.
        • et al.
        The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload.
        Acad Radiol. 2015; 22: 1191-1198
        • Sokolovskaya E.
        • Shinde T.
        • Ruchman R.B.
        • et al.
        The effect of faster reporting speed for imaging studies on the number of misses and interpretation errors: a pilot study.
        J Am Coll Radiol. 2015; 12: 683-688
        • Jenny C.
        • Hymel K.P.
        • Ritzen A.
        • et al.
        Analysis of missed cases of abusive head trauma.
        JAMA. 1999; 281: 621-626
        • Pow R.E.
        • Mello-Thoms C.
        • Brennan P.
        Evaluation of the effect of double reporting on test accuracy in screening and diagnostic imaging studies: a review of the evidence.
        J Med Imaging Radiat Oncol. 2016; 60: 306-314
        • Lauritzen P.M.
        • Hurlen P.
        • Sandbæk G.
        • et al.
        Double reading rates and quality assurance practices in Norwegian hospital radiology departments: two parallel national surveys.
        Acta Radiol. 2015; 56: 78-86
        • Geijer H.
        • Geijer M.
        Added value of double reading in diagnostic radiology, a systematic review.
        Insights Imaging. 2018; 9: 287-301
        • Chilamkurthy S.
        • Ghosh R.
        • Tanamala S.
        Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study.
        Lancet. 2018; 392: 2388-2396
        • Arbabshirani M.R.
        • Fornwalt B.K.
        • Mongelluzzo G.J.
        • et al.
        Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration.
        npj Digit Med. 2018; 1: 9
        • Prevedello L.M.
        • Erdal B.S.
        • Ryu J.L.
        • et al.
        Automated critical test findings identification and online notification system using artificial intelligence in imaging.
        Radiology. 2017; 285: 923-931
        • Ojeda P.
        • Zawaideh M.
        • Mossa-Basha M.
        • et al.
        The utility of deep learning: evaluation of a convolutional neural network for detection of intracranial bleeds on non-contrast head computed tomography studies.
        SPIE. 2019; (10949, doi:10.1117/12.2513167)
        • Goldberg-Stein S.
        • Frigini L.A.
        • Long S.
        • et al.
        ACR RADPEER Committee White Paper with 2016 Updates: Revised Scoring System, New Classifications, Self-Review, and Subspecialized Reports.
        J Am Coll Radiol. 2017; 14: 1080-1086
        • Medicine Io. To Err Is Human
        Kohn L.T. Corrigan J.M. Donaldson M.S. Building a Safer Health System. The National Academies Press, Washington, DC2000: 312 (p)
        • Brady A.P.
        Error and discrepancy in radiology: inevitable or avoidable?.
        Insights Imaging. 2016; 8: 171-182
        • Kelleher Jr., M.S.
        • Forman H.P.
        • Goodman T.R.
        • et al.
        Proctoring of new emergency radiologists to promote clinical excellence and ensure quality of care.
        J Am Coll Radiol. 2016; 13: 967-972
        • Bruno M.A.
        • Walker E.A.
        • Abujudeh H.H.
        Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction.
        Radiographics. 2015; 35: 1668-1676
        • Wu M.Z.
        • McInnes M.D.F.
        • Blair Macdonald D.
        • et al.
        CT in adults: systematic review and meta-analysis of interpretation discrepancy rates.
        Radiology. 2013; 270: 717-735
        • Babiarz L.S.
        • Yousem DM.
        Quality control in neuroradiology: discrepancies in image interpretation among academic neuroradiologists.
        Am J Neuroradiol. 2012; 33: 37
        • Waite S.
        • Scott J.
        • Gale B.
        • et al.
        Interpretive error in radiology.
        Am J Roentgenol. 2016; 208: 739-749
        • Hanna T.N.
        • Lamoureux C.
        • Krupinski E.A.
        • et al.
        Effect of shift, schedule, and volume on interpretive accuracy: a retrospective analysis of 2.9 million radiologic examinations.
        Radiology. 2017; 287: 205-212
        • Ruutiainen A.T.
        • Durand D.J.
        • Scanlon M.H.
        • et al.
        Increased error rates in preliminary reports issued by radiology residents working more than 10 consecutive hours overnight.
        Acad Radiol. 2013; 20: 305-311
        • Miyakoshi A.
        • Nguyen Q.T.
        • Cohen W.A.
        • et al.
        Accuracy of preliminary interpretation of neurologic CT examinations by on-call radiology residents and assessment of patient outcomes at a level I trauma center.
        J Am Coll Radiol. 2009; 6: 864-870
        • Goddard P.
        • Leslie A.
        • Jones A.
        • et al.
        Error in radiology.
        Br J Radiol. 2001; 74: 949-951
        • Briggs G.M.
        • Flynn P.A.
        • Worthington M.
        • et al.
        The role of specialist neuroradiology second opinion reporting: is there added value?.
        Clin Radiol. 2008; 63: 791-795
        • Chang P.D.
        • Kuoy E.
        • Grinband J.
        • et al.
        Hybrid 3D/2D convolutional neural network for hemorrhage evaluation on head CT.
        Am J Neuroradiol. 2018; 39: 1609
        • Li Y.-.H.
        • Zhang L.
        • Hu Q.-.M.
        • et al.
        Automatic subarachnoid space segmentation and hemorrhage detection in clinical head CT scans.
        Int J Comput Assist Radiol Surg. 2012; 7: 507-516
        • Yuh E.L.
        • Gean A.D.
        • Manley G.T.
        • et al.
        Computer-aided assessment of head computed tomography (CT) studies in patients with suspected traumatic brain injury.
        J Neurotrauma. 2008; 25: 1163-1172
        • Paiva O.A.
        • Prevedello LM.
        The potential impact of artificial intelligence in radiology.
        Radiol Bras. 2017; 50 (V-VI)
        • Mahgerefteh S.
        • Kruskal J.B.
        • Yam C.S.
        • et al.
        Peer review in diagnostic radiology: current state and a vision for the future.
        Radiographics. 2009; 29: 1221-1231
        • Borgstede J.P.
        • Lewis R.S.
        • Bhargavan M.
        • et al.
        RADPEER quality assurance program: a multifacility study of interpretive disagreement rates.
        J Am Coll Radiol. 2004; 1: 59-65
        • Kaewlai R.
        • Abujudeh H.
        Peer review in clinical radiology practice.
        Am J Roentgenol. 2012; 199: W158-WW62
        • Davis M A R.B.
        • Cedeno P.
        • Saha A.
        • et al.
        Utilizing Machine Learning to Improve ED and In-Patient Throughput in Cases of Acute Intracranial Hemorrhage by Non-Contrast Head CT.
        in: 105th Scientific Assembly and Annual Meeting of the RSNA, Chicago, IL2019