Advertisement

Automated Endotracheal Tube Placement Check Using Semantically Embedded Deep Neural Networks

  • Matthew S. Brown
    Correspondence
    Address correspondence to: M.S.B.
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Koon-Pong Wong
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Liza Shrestha
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Muhammad Wahi-Anwar
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Morgan Daly
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • George Foster
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Fereidoun Abtin
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Kathleen L. Ruchalski
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Jonathan G. Goldin
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
  • Dieter Enzmann
    Affiliations
    Department of Radiological Sciences, Center for Computer Vision and Imaging Biomarkers, David Geffen School of Medicine at UCLA, 924 Westwood Blvd., Suite 615, Los Angeles, CA 90024
    Search for articles by this author
Open AccessPublished:May 27, 2022DOI:https://doi.org/10.1016/j.acra.2022.04.022

      Rationale and Objectives

      To develop artificial intelligence (AI) system that assists in checking endotracheal tube (ETT) placement on chest X-rays (CXRs) and evaluate whether it can move into clinical validation as a quality improvement tool.

      Materials and Methods

      A retrospective data set including 2000 de-identified images from intensive care unit patients was split into 1488 for training and 512 for testing. AI was developed to automatically identify the ETT, trachea, and carina using semantically embedded neural networks that combine a declarative knowledge base with deep neural networks. To check the ETT tip placement, a “safe zone” was computed as the region inside the trachea and 3–7 cm above the carina. Two AI outputs were evaluated: (
      • Laroia AT
      • Donnelly EF
      • Henry TS
      • et al.
      ACR appropriateness criteria: Intensive care unit patients.
      ) ETT overlay, (
      • Brunel W
      • Coleman DL
      • Schwartz DE
      • et al.
      Assessment of routine chest roentgenograms and the physical examination to confirm endotracheal tube position.
      ) ETT misplacement alert messages. Clinically relevant performance metrics were compared against prespecified thresholds of >85% overlay accuracy and positive predictive value (PPV) > 30% and negative predictive value NPV > 95% for alerts to move into clinical validation.

      Results

      An ETT was present in 285 of 512 test cases. The AI detected 95% (271/285) of ETTs, 233 (86%) of these with accurate tip localization. The system (correctly) did not generate an ETT overlay in 221/227 CXRs where the tube was absent for an overall overlay accuracy of 89% (454/512). The alert messages indicating that either the ETT was misplaced or not detected had a PPV of 83% (265/320) and NPV of 98% (188/192).

      Conclusion

      The chest X-ray AI met prespecified performance thresholds to move into clinical validation.

      Key Words

      INTRODUCTION

      Chest radiographs (CXR) are used in the intensive care unit (ICU) to examine and monitor critically ill patients on life-supporting devices. Endotracheal tubes (ETTs) are used to maintain airway patency and lung ventilation. The American College of Radiology recommends that a CXR is performed following intubation to check tube placement (
      • Laroia AT
      • Donnelly EF
      • Henry TS
      • et al.
      ACR appropriateness criteria: Intensive care unit patients.
      ), with repositioning being required in about 15% of patients (
      • Brunel W
      • Coleman DL
      • Schwartz DE
      • et al.
      Assessment of routine chest roentgenograms and the physical examination to confirm endotracheal tube position.
      ,
      • Gary P
      • Sullivan G
      • Ostryzniuk P
      • et al.
      Value of postprocedural chest radiographs in the adult intensive care unit.
      ,
      • Marik PE
      • Janower ML
      The impact of routine chest radiography on ICU management decisions: an observational study.
      ,
      • Trotman-Dickenson BM
      Radiology in the intensive care unit (Part I).
      ,
      • Godoy MCB
      • Leitman BS
      • de Groot PM
      • et al.
      Chest radiography in the ICU: part 1, evaluation of airway, enteric, and pleural tubes.
      ). The desired ETT tip position is within the mid trachea, approximately 5 ± 2 cm above the carina (
      • Goodman LR
      • Conrardy PA
      • Laing F
      • et al.
      Radiographic evaluation of endotracheal tube position.
      ), with the head of the patient in a neutral position, and at least 2 cm above the carina (
      • Roberts JR
      • Spadafora M
      • Cone DC.
      Proper depth of placement of oral endotracheal tubes in adults prior to radiographic confirmation.
      ). If the ETT is too high, there is a risk of inefficient ventilation or inadvertent extubation and may even cause vocal cord injury (
      • Wiener MD
      • Garay SM
      • Leitman BS
      • et al.
      Imaging of the intensive care unit patient.
      ). On the contrary, selective bronchial intubation may occur if the ETT tip is too low (e.g., close to or lower than carina), causing severe complications, such as partial or complete collapse of the nonventilated lung, hyperinflation of the ipsilateral lung, pneumothorax and even death (
      • Brunel W
      • Coleman DL
      • Schwartz DE
      • et al.
      Assessment of routine chest roentgenograms and the physical examination to confirm endotracheal tube position.
      ). Based on this literature, a goal of our system will be to issue alerts to physicians if the ETT tip is < 2 cm or > 8 cm above the carina with high sensitivity, i.e., if the ETT tip is outside of the desired range by 1 cm or more.
      Tertiary ICUs can generate hundreds of CXRs each day to check for tube placement. Given the high volume of cases and urgent need for intervention if the ETT is misplaced, it is not always practical to wait for radiology reads so ICU physicians may take a preliminary look at the CXR and adjust for the misplaced tube immediately. Due to the low conspicuity of tubes, superposition of anatomy and medical devices, and image quality issues, assessment of tube placement can be challenging without high quality monitors and honed radiologic interpretation skills.
      We propose an artificial intelligence (AI) system that provides two forms of decision support: ETT detection assistance and position check alerts. Our evaluation will focus on supporting the ICU physician at the point of care. The AI system may also benefit radiologists to expedite their reads and improve accuracy.
      Our goal is to perform analytic validation of the AI and determine whether it can move into clinical evaluation as a quality improvement tool. Clinically relevant performance metrics will be compared against pre-specified thresholds.

      MATERIALS AND METHODS

      Image Data Set

      This retrospective study was approved by our institutional review board (IRB #11-000126). Our study cohort included consecutive patients from medical and surgical ICUs across our hospital system between April 2018 and September 2019. CXRs were bedside anteroposterior images from adult patients (aged ≥ 18 years) with or without ETT. For each patient, CXR images from up to separate three examinations were included. Data were de-identified to comply with Health Insurance Portability and Accountability Act and IRB requirements.
      Using an in-house developed imaging research platform (
      • Brown MS
      • Shah SK
      • Pais R
      • et al.
      Database design and implementation for quantitative image analysis research.
      ), CXR images were manually annotated by trained image analysts, with supervision by three cardiothoracic radiologists who have been in practice for 10 to over 25 years. Following training by the radiologists, image analysts reviewed cases with radiologists if uncertain about the location of the ETT or anatomic landmarks. The trachea was outlined, the carina point marked, and the ETT traced.
      Two thousand CXR images were annotated and split into 1488 for training and 512 for testing (approx. 3:1). Training and test sets had similar proportions of cases with and without ETT, approximately 60:40.

      Algorithm Development

      The AI automatically identifies the ETT, trachea, and carina using a “semantically embedded neural network” that combines a declarative knowledge base with deep convolutional networks (DCNNs). Each node of the semantic network represents an entity (anatomy or ETT) and defines a segmentation method (DCNN) and expected size and shape features. The connections of the semantic network reflect expected spatial relationships between the entities. Nodes were created for the ETT, ETT tip, trachea, carina, and safe zone. Based on the relationships defined in the semantic network, segmentation proceeds in a hierarchical fashion, with the segmentation of the trachea spatially guiding the CNNs detecting the carina and ETT. Spatial relationships are used directly to define the ETT tip and safe zone regions from these segmentation results. As described in (
      • Brown MS
      • McNitt-Gray MF
      • Mankovich NJ
      • et al.
      Method for segmenting chest CT image data using an anatomical model: preliminary results.
      ), the segmentation runs for each node to generate candidate image regions. The candidate that best satisfies spatial, size, and shape constraints is then matched to that node, excluding extraneous fragments generated by the DCNN. The spatial constraints are also used to define a search area for the node, based on previously identified nodes (e.g., the detected trachea guides image cropping for DCNN segmentation of the ETT).
      Automatic trachea segmentation is performed using U-Net DCNN with five encoder and five decoder blocks. CXRs were rescaled to 512 × 512 pixels and intensity normalized. The DCNN was trained using Adam optimization with Dice coefficient loss and a learning rate of 0.00005. Two DCNNs were trained for ETT segmentation, one as for the trachea and another using a cropped image around the trachea with contrast limited adaptive histogram equalization and their outputs were combined.
      Carina point localization is performed using a regression predictive model based on the VGGNet architecture with five blocks and 16 weight layers. The DCNN output is a 2 × 1 array of points representing the coordinates of the carina.
      The “safe zone” is the region where the ETT tip should be located if correctly placed. It is defined using spatial relationships that describe a region inside the trachea and between 3 and 7 cm above the carina. This range is based on desired position of an ETT tip 5 ± 2 cm above the carina in a neutral neck position (
      • Goodman LR
      • Conrardy PA
      • Laing F
      • et al.
      Radiographic evaluation of endotracheal tube position.
      ). The system will issue an alert if the ETT tip is outside of this safe zone, so the AI is designed to be highly (overly) sensitive to misplacement of tubes, given that our goal is to issue alerts for ETT tips <2 cm or >8 cm above the carina.

      AI System Outputs

      The AI generated CXR overlays showing the ETT path and measurement in cm from the ETT tip to the carina. It also displayed one of three possible ETT messages: (1) “Found” (ETT tip was determined to be in the safe zone), (2) “Position Alert” (ETT tip was not in the safe zone or the AI could not determine the safe zone), (3) “Not Found.”
      Messages 2 and 3 are intended to prompt a check by the radiologist or physician.

      Evaluation

      The two forms of AI decision support were evaluated: (1) the ETT overlay, (2) the alert messages. Evaluations were performed over the entire set of test cases (N = 512) and the subgroup with ETT present (N = 285). In the ICU, where alerts will be presented to the physician at the point of care, we expect a high proportion of cases will have ETTs present, therefore, we prespecified evaluation metric targets for the ETT subgroup to confirm the system is ready to move into clinical evaluation.
      The ETT overlay was evaluated in terms of percentage of cases where the overlay was accurate, i.e., the ETT was detected and the tip coordinates were within 10 mm of reference (if the ETT was present), or no ETT was detected (if the ETT was not present). The prespecified performance requirement on this metric was >85% accurate to move into clinical evaluation.
      For alert messages, we define a positive output (alert) as either a “Position Alert” message (misplaced ETT) or a “Not Found” message (missing ETT), and a negative output (no alert) as a “Found” message (ETT present and correctly placed). When the AI output is positive, a true positive (TP) requires (ETT tip outside trachea) or (tip < 2 cm above carina) or (tip > 8 cm above carina) or (ETT is missing), otherwise it is a false positive (FP). When the output is negative, a false negative (FN) requires (ETT tip point outside trachea) or (tip < 2 cm above carina) or (tip > 8 cm above carina), otherwise it is a true negative (TN). Positive predictive value (PPV = TP/[TP + FP]) and negative predictive value (NPV=TN/(TN+FN)) metrics were computed to give a sense of trustworthiness from a physician perspective. Pre-specified thresholds for the ETT subgroup were PPV > 30% and NPV > 95%, respectively, to move into clinical evaluation. The PPV threshold considers the deliberate design of the system to be overly sensitive to misplaced tubes and is based on the premise that an ICU physician will tolerate 1 to 2 FPs for each TP alert. To be safe to move into clinical evaluation, the NPV must be high, i.e., avoid missed alerts when tubes are misplaced.
      We also performed a sub-analysis of sensitivity to misplaced tubes. The primary analysis included both misplaced and absent ETTs as positives. Misplaced ETTs were identified in the test set based on the reference annotations as follows: (ETT tip outside trachea) or (tip < 2 cm above carina) or (tip > 8 cm above carina). For ETT detected, the accuracy of tip location determined by the AI was computed as the mean absolute difference and standard deviation from the reference annotated manually.

      RESULTS

      The composition of the test set is shown in Figure 1. It provides a breakdown of cases with and without ETT and whether the ETT was correctly or incorrectly positioned, based on the reference annotations according to the definitions above.
      Figure 1
      Figure 1CXR test set composition. CXR, chest X-ray. (Color version of figure is available online).
      Overlay (detection assist) accuracy was 89% in the test set, i.e., in (233 + 221)/512 test cases. Two hundred seventy-one of 285 tubes were detected in the image, having a mean absolute difference of 0.72 ± 1.05 cm in tip position from the reference annotation, and 233 of these had the tip within 10 mm of the reference, i.e., 233 overlays were accurate when the ETT was present. The system (correctly) did not generate an ETT overlay in 221/227 CXRs where the tube was absent.
      The system PPVs/NPVs are shown in Table 1. The prespecified PPV and NPV thresholds for the ETT subgroup were met. False positive was defined as correctly placed ETT classified by the AI as misplaced ETT or missing ETT (i.e., not detected). Of the 55 FPs, 14 were due to missed ETT detections and 41 due to unnecessary alerts for misplacement according to the reference.
      Table 1Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of Chest X-Ray AI Alerts
      All Test Cases (N = 512)ETT Present Subgroup (N = 285)
      PPV83% (TP = 265, FP = 55)42% (TP = 40, FP = 55)
      NPV98% (TN = 188, FN = 4)99% (TN = 188, FN = 2)
      The system sensitivity to misplaced tubes was 95%. There were 42 misplaced ETTs in the test set, of which 40 were positive alerts by the system. The two FNs were due to incorrect localization of the ETT tip and/or carina.
      Three examples are shown in Figures 24, including the original CXR and the AI output image automatically generated by the AI. The output image shows the ETT overlay and alert message on the contrast enhanced image generated by the system for input to the DCNN. The example images show an ETT detected by the system with correct placement (Fig 2), and two with ETT alerts, one with the ETT tip too low relative to the carina (Fig 3) and one too high (Fig 4). Figure 5 shows a false positive case flagged by the AI, which incorrectly identified the ETT tip at the point where the projection of NG tube crossed it, resulting in an unnecessary alert message that the ETT tip was too high, when in fact the ETT extended downward to a safe distance from the carina.
      Figure 2
      Figure 2Original CXR and the AI output image for a correctly placed ETT. AI, artificial intelligence; CXR, chest X-ray; ETT, endotracheal tube. (Color version of figure is available online).
      Figure 3
      Figure 3Original CXR and the AI output image for an ETT with tip placed too low. AI, artificial intelligence; CXR, chest X-ray; ETT, endotracheal tube (Color version of figure is available online).
      Figure 4
      Figure 4Original CXR and the AI output image for an ETT with tip placed too high. AI, artificial intelligence; CXR, chest X-ray; ETT, endotracheal tube. (Color version of figure is available online).
      Figure 5
      Figure 5Original CXR and the AI output image for a false-positive case in which the ETT tip position was flagged by the AI as being too high even though the ETT was placed properly. AI, artificial intelligence; CXR, chest X-ray; ETT, endotracheal tube. (Color version of figure is available online).

      DISCUSSION

      Deep learning (DL) has been widely used in image processing since the successful application of CNNs to image classification and object recognition (
      • Krizhevsky A
      • Sutskever I
      • Hinton GE
      ImageNet classification with deep convolutional neural networks.
      ). Various DL approaches have been applied to many different aspects of CXR, ranging from image quality control (

      Zhang M, Nye K, Avinash G, et al. Leveraging deep learning artificial intelligence to conduct quality control on chest X-ray images. American Association of Physicists in Medicine (AAPM) Annual Meeting 2018. Available at: http://amos3.aapm.org/abstracts/pdf/134-38613-437584-135062-1245313483.pdf. Accessed November 16, 2021.

      ) and detection of the orientation of CXR images (

      Younis K, Soni R, Zhang M, et al. Leveraging deep learning artificial intelligence in detecting the orientation of chest X-ray images. Society for Imaging Informatics in Medicine (SIIM), Conference on Machine Intelligence in Medical Imaging (CMIMI) 2019. Available at: https://cdn.ymaws.com/siim.org/resource/resmgr/mimi19/oral5/Leveraging_Deep_Learning_Kha.pdf. Accessed November 16, 2021.

      ,
      • Younis K
      • Dalal P
      • Vera G
      • et al.
      Leveraging deep learning for orientation detection and correction of X-ray images.
      ) to more complicated applications, such as anatomic segmentation (
      • Ginneken BV
      • Stegmann MB
      • Loog M
      Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database.
      ,

      Frid-Adar M, Ben-Cohen A, Amer R, et al. Improving the segmentation of anatomical structures in chest radiographs using U-net with an ImageNet pre-trained encoder. In: Stoyanov D, et al. (eds) RAMBO/BIA/TIA-2018. Lecture notes in computer science, Springer, 2018;11040:159-168. doi: 10.1007/978-3-030-00946-5_17.

      ), disease classification (
      • Rajpurkar P
      • Irvin J
      • Ball RL
      • et al.
      Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists.
      ,
      • Taylor AG
      • Mielke C
      • Mongan J
      Automated detection of moderate and large pneumothorax on frontal chest X-rays using deep convolutional neural networks: a retrospective study.
      ,
      • Wang H
      • Gu H
      • Qin P
      • Wang J
      CheXLocNet: automatic localization of pneumothorax in chest radiographs using deep convolutional neural networks.
      ,
      • Abbas A
      • Abdelsamea MM
      • Gaber MM
      Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network.
      ,
      • Yao S
      • Chen Y
      • Tian X
      • et al.
      Pneumonia detection using an improved algorithm based on faster R-CNN.
      ), and ETT detection (
      • Lakhani P
      Deep convolutional neural networks for endotracheal tube position and X-ray image classification: challenges and opportunities.
      ,
      • Lakhani P
      • Flanders A
      • Gorniak R
      Endotracheal tube position assessment on chest radiographs using deep learning.
      ,
      • Kara S
      • Akers JY
      • Chang PD
      Identification and localization of endotracheal tube on chest radiographs using a cascaded convolutional neural network approach.
      ,
      • Harris RJ
      • Baginski SG
      • Bronstein Y
      • et al.
      Measurement of endotracheal tube positioning on chest X-Ray using object detection.
      ,
      • Kao EF
      • Jaw TS
      • Li CW
      • et al.
      Automated detection of endotracheal tubes in paediatric chest radiographs.
      ). Several reports have been published for automated detection of the ETT on CXR. Computer-aided detection methods have yielded a true-positive rate of 0.93 using template matching and region growing segmentation (
      • Ramakrishna B
      • Brown M
      • Goldin J
      • et al.
      An improved automatic computer aided tube detection and labeling system on chest radiographs.
      ) and an area under the receiver operating characteristic curve (AUC) of 0.94 using image enhancement and nonlinear spatial filtering (
      • Chen S
      • Zhang M
      • Yao L
      • et al.
      Endotracheal tubes positioning detection in adult portable chest radiography for intensive care unit.
      ). Using a DL approach with a U-Net-like architecture, Frid-dar et al. (

      Frid-Adar M, Amer R, Greenspan H. Endotracheal tube detection and segmentation in chest radiographs using synthetic data. In: Shen D, et al. (eds) Medical image computing and computer assisted intervention – MICCAI 2019. Lecture notes in computer science, Springer, 2019;11769:784-792. doi:10.1007/978-3-030-32226-7_87.

      ) achieved an AUC of 0.99 in classifying CXRs having an ETT or not. They also proposed an approach for synthesizing ET tubes to generate ground-truth data for ETT detection and segmentation. These systems focused on ETT detection only, without placement checking.
      Several studies have included assessment the ETT tip position relative to the carina in their evaluation. Using a GoogLeNet CNN and 300 labeled images, Lakhani (
      • Lakhani P
      Deep convolutional neural networks for endotracheal tube position and X-ray image classification: challenges and opportunities.
      ) achieved an AUC of 0.94 on classifying presence vs absence of the ETT, but a more modest accuracy of 0.81 in differentiating low versus normal positioning of the ETT. Lakhani et al. (
      • Lakhani P
      • Flanders A
      • Gorniak R
      Endotracheal tube position assessment on chest radiographs using deep learning.
      ) evaluated an ETT-carina distance estimation using the Inception V3 deep CNN architecture and demonstrated excellent interrater agreement (intraclass correlation coefficient >0.8) between AI and radiologists. The sensitivity was 90.1% and specificity 92.4% in detecting ETT tip located in <2 cm above the carina. However, the sensitivity was only 66.5% and specificity 99.2% when assessing ETT-carina distance ≥7 cm. Using the open-source MIMIC Chest X-Ray dataset and 3 CNN algorithms, Kara et al. (
      • Kara S
      • Akers JY
      • Chang PD
      Identification and localization of endotracheal tube on chest radiographs using a cascaded convolutional neural network approach.
      ) reported the accuracy, sensitivity, specificity, PPV, NPV, and AUC of 97.14%, 97.37%, 96.89%, 97.12%, 97.15%, and 99.58% respectively, upon a five-fold cross-validation to classify the presence or absence of the ETT, and the model-predicted locations of the carina and the distal tip of ETT were estimated having a median error of 0.46 cm and 0.60 cm from manual ground-truth annotations, respectively. Harris et al. (
      • Harris RJ
      • Baginski SG
      • Bronstein Y
      • et al.
      Measurement of endotracheal tube positioning on chest X-Ray using object detection.
      ) trained a bounding box-based CNN for localization of the ET tube and the carina, with a mean difference of 0.7 cm between the model-predicted ETT-carina distance and the ground-truth measurement agreed by two experienced radiologists in 200 CXR images.
      Our study was not designed to compare against other systems, but the metrics are favorable to the prior literature in both classification problems, presence/absence of ETT and assessment of ETT tip position, and in terms of the number of CXRs used for testing. Performing a complete segmentation of the ETT using fully annotated data gives our system the additional capability of generating an overlay to assist the physician with their visual interpretation, and allows explicit (explainable and adjustable) rules for checking the ETT tip position, as opposed to a black box approach of classifying ETTs as correctly or incorrectly placed.
      The combination of neural and semantic networks in our system was beneficial in this application. The deep neural networks allow learning of complex or subtle patterns (e.g., relating to the gradient profiles and trajectories of the ETT), that are difficult to describe as explicit features, to be learned directly from the data. The semantic networks are complementary in allowing direct expression of placement check rules. This is advantageous in this setting because there are relatively few misplaced ETT examples to train on for a data driven approach. There is also the benefit of explainability of the decision when a tube is declared as misplaced which aids in user confidence and adoption. The semantic network also allowed constraints, such as continuity and length of the ETT to be applied and contributed to the reliability of the detection and segmentation.
      The current implementation of the AI system is that alert messages are provided for cases with ETT misplacement and ETT missing, which is appropriate for our initial clinical application to cases that are ordered for checking ETT placement in the ICU, and as such our data analysis was focused on the subset with ETT present. If the system were deployed for all chest X-rays, where a greater proportion would have no ETT, then issuing alerts for ETT missing may not be appropriate and the system analysis and operating points revised accordingly.
      Our prespecified metric thresholds were defined to allow the AI to move into a clinical evaluation phase. With ongoing system refinement, this overlay accuracy metric should eventually be above 90%, but >85% was considered sufficient at this stage (89% was achieved). The prespecified PPV target was lower than the NPV because it is crucial to alert on misplaced ETTs and we are willing to accept false positives. The system is being designed to detect misplaced ETTs outside of the range 2–8 cm above the carina, but internally the system issues an alert if the ETT tip is outside of a 3–7 cm range, i.e., we are designing it to be overly sensitive and as such we were willing to accept a PPV above 30% to move into clinical evaluation (42% was achieved). While false positives can be inefficient and over time cause alert fatigue, they do not pose a direct risk to patient management. We will ultimately strive for a low false positive rate, but this target was considered sufficient to permit moving into the initial clinical evaluation where physician acceptance can be further studied. Our prespecified NPV threshold was >95% because the biggest safety concern is a misplaced ETT, i.e., there is a very low tolerance for false negatives. We achieved an NPV of 99% in this small data set of 285 ETT cases with 42 misplaced tubes and the NPV will need to be maintained very close to 100% in larger studies before being cleared for clinical use. It should be noted that the 15% (42/285) of cases with misplaced ETT by reference in our data set is consistent with the incidences of misplaced tubes reported in the literature in clinical practice.
      There are some limitations to this study. First, training and test samples were obtained from ICU facilities of a single institution where acquisition protocols and equipment are usually well standardized. The performance of AI might be affected if the trained models were applied to CXR images obtained from other institutions. Another potential limitation is the heterogeneity of samples as they were collected from both medical and surgical ICUs. In comparison to typical medical ICU patients, surgical ICU patients will generally have more life-supporting lines and tubes that could represent a different level of complexity and challenge to most AI models and algorithms. Our prespecified decision support metric benchmarks were met, but this is not a clinical validation of our system. A simple analysis was performed to make a practical decision about moving into the next phase of clinical evaluation. We believe it is important for radiology AI evaluations to have prespecified targets for clinically relevant metrics if we are to advance such technology into clinical practice. The intended use in our next phase of clinical evaluation is for healthcare quality improvement providing ETT detection assist overlays and position check alerts. The system outputs will be made available to both ICU physicians at the point of care and reporting radiologists and the impact in each setting will be evaluated. The physician remains responsible for the final diagnosis and patient management decisions. Our conclusion for this phase of our development is that the AI decision support performance is sufficient to move into clinical implementation and further evaluation.

      REFERENCES

        • Laroia AT
        • Donnelly EF
        • Henry TS
        • et al.
        ACR appropriateness criteria: Intensive care unit patients.
        J Am Coll Radiol. 2021; 18: S62-S72https://doi.org/10.1016/j.jacr.2021.01.017
        • Brunel W
        • Coleman DL
        • Schwartz DE
        • et al.
        Assessment of routine chest roentgenograms and the physical examination to confirm endotracheal tube position.
        Chest. 1989; 96: 1043-1045https://doi.org/10.1378/chest.96.5.1043
        • Gary P
        • Sullivan G
        • Ostryzniuk P
        • et al.
        Value of postprocedural chest radiographs in the adult intensive care unit.
        Crit Care Med. 1992; 20: 1513-1518https://doi.org/10.1097/00003246-199211000-00006
        • Marik PE
        • Janower ML
        The impact of routine chest radiography on ICU management decisions: an observational study.
        Am J Crit Care. 1997; 6: 95-98https://doi.org/10.4037/ajcc1997.6.2.95
        • Trotman-Dickenson BM
        Radiology in the intensive care unit (Part I).
        J Intensive Care Med. 2003; 18: 198-210https://doi.org/10.1177/0885066603251897
        • Godoy MCB
        • Leitman BS
        • de Groot PM
        • et al.
        Chest radiography in the ICU: part 1, evaluation of airway, enteric, and pleural tubes.
        Am J Roentgenol. 2012; 198: 563-571https://doi.org/10.2214/AJR.10.7226
        • Goodman LR
        • Conrardy PA
        • Laing F
        • et al.
        Radiographic evaluation of endotracheal tube position.
        Am J Roentgenol. 1976; 127: 433-434https://doi.org/10.2214/ajr.127.3.433
        • Roberts JR
        • Spadafora M
        • Cone DC.
        Proper depth of placement of oral endotracheal tubes in adults prior to radiographic confirmation.
        Acad Emerg Med. 1995; 2: 20-24https://doi.org/10.1111/j.1553-2712.1995.tb03073.x
        • Wiener MD
        • Garay SM
        • Leitman BS
        • et al.
        Imaging of the intensive care unit patient.
        Clin Chest Med. 1991; 12: 169-198https://doi.org/10.1016/S0272-5231(21)00732-2
        • Brown MS
        • Shah SK
        • Pais R
        • et al.
        Database design and implementation for quantitative image analysis research.
        IEEE Trans Inf Technol Biomed. 2005; 9: 99-108https://doi.org/10.1109/titb.2004.837854
        • Brown MS
        • McNitt-Gray MF
        • Mankovich NJ
        • et al.
        Method for segmenting chest CT image data using an anatomical model: preliminary results.
        IEEE Trans Med Imaging. 1997; 16: 828-839https://doi.org/10.1109/42.650879
        • Krizhevsky A
        • Sutskever I
        • Hinton GE
        ImageNet classification with deep convolutional neural networks.
        in: Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS 2012). 1. 2012: 1097-1105https://doi.org/10.5555/2999134.2999257
      1. Zhang M, Nye K, Avinash G, et al. Leveraging deep learning artificial intelligence to conduct quality control on chest X-ray images. American Association of Physicists in Medicine (AAPM) Annual Meeting 2018. Available at: http://amos3.aapm.org/abstracts/pdf/134-38613-437584-135062-1245313483.pdf. Accessed November 16, 2021.

      2. Younis K, Soni R, Zhang M, et al. Leveraging deep learning artificial intelligence in detecting the orientation of chest X-ray images. Society for Imaging Informatics in Medicine (SIIM), Conference on Machine Intelligence in Medical Imaging (CMIMI) 2019. Available at: https://cdn.ymaws.com/siim.org/resource/resmgr/mimi19/oral5/Leveraging_Deep_Learning_Kha.pdf. Accessed November 16, 2021.

        • Younis K
        • Dalal P
        • Vera G
        • et al.
        Leveraging deep learning for orientation detection and correction of X-ray images.
        in: Society for Imaging Informatics in Medicine (SIIM), Conference on Machine Intelligence in Medical Imaging (CMIMI). 2020 (Available at:) (Accessed November 16, 2021)
        • Ginneken BV
        • Stegmann MB
        • Loog M
        Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database.
        Med Image Anal. 2006; 10: 19-40https://doi.org/10.1016/j.media.2005.02.002
      3. Frid-Adar M, Ben-Cohen A, Amer R, et al. Improving the segmentation of anatomical structures in chest radiographs using U-net with an ImageNet pre-trained encoder. In: Stoyanov D, et al. (eds) RAMBO/BIA/TIA-2018. Lecture notes in computer science, Springer, 2018;11040:159-168. doi: 10.1007/978-3-030-00946-5_17.

        • Rajpurkar P
        • Irvin J
        • Ball RL
        • et al.
        Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists.
        PLoS Med. 2018; 15 (Published November 20, 2018Accessed November 16, 2021)e1002686https://doi.org/10.1371/journal.pmed.1002686
        • Taylor AG
        • Mielke C
        • Mongan J
        Automated detection of moderate and large pneumothorax on frontal chest X-rays using deep convolutional neural networks: a retrospective study.
        PLOS Med. 2018; 15 (Published: November 20, 2018Accessed November 16, 2021)e1002697https://doi.org/10.1371/journal.pmed.1002697
        • Wang H
        • Gu H
        • Qin P
        • Wang J
        CheXLocNet: automatic localization of pneumothorax in chest radiographs using deep convolutional neural networks.
        PLOS One. 2020; 15 (Published: November 9, 2020Accessed November 16, 2021)e0242013https://doi.org/10.1371/journal.pone.0242013
        • Abbas A
        • Abdelsamea MM
        • Gaber MM
        Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network.
        Applied Intelligence. 2021; 51: 854-864https://doi.org/10.1007/s10489-020-01829-7
        • Yao S
        • Chen Y
        • Tian X
        • et al.
        Pneumonia detection using an improved algorithm based on faster R-CNN.
        Comput Math Methods Med. 2021; 2021 (Published: April 21, 2021Accessed November 16, 2021)8854892https://doi.org/10.1155/2021/8854892
        • Lakhani P
        Deep convolutional neural networks for endotracheal tube position and X-ray image classification: challenges and opportunities.
        J Digit Imaging. 2017; 30: 460-468https://doi.org/10.1007/s10278-017-9980-7
        • Lakhani P
        • Flanders A
        • Gorniak R
        Endotracheal tube position assessment on chest radiographs using deep learning.
        Radiology. 2021; 3 (Published: November 18, 2020Accessed November 16, 2021)e200026https://doi.org/10.1148/ryai.2020200026
        • Kara S
        • Akers JY
        • Chang PD
        Identification and localization of endotracheal tube on chest radiographs using a cascaded convolutional neural network approach.
        J Digit Imaging. 2021; 34: 898-904https://doi.org/10.1007/s10278-021-00463-0
        • Harris RJ
        • Baginski SG
        • Bronstein Y
        • et al.
        Measurement of endotracheal tube positioning on chest X-Ray using object detection.
        J Digit Imaging. 2021; 34: 846-852https://doi.org/10.1007/s10278-021-00495-6
        • Kao EF
        • Jaw TS
        • Li CW
        • et al.
        Automated detection of endotracheal tubes in paediatric chest radiographs.
        Comput Methods Programs Biomed. 2015; 118: 1-10
        • Ramakrishna B
        • Brown M
        • Goldin J
        • et al.
        An improved automatic computer aided tube detection and labeling system on chest radiographs.
        in: Proc. SPIE 8315, Medical Imaging 2012: Computer-Aided Diagnosis, 83150R. 2012https://doi.org/10.1117/12.911839 (Published:Accessed November 16, 2021)
        • Chen S
        • Zhang M
        • Yao L
        • et al.
        Endotracheal tubes positioning detection in adult portable chest radiography for intensive care unit.
        Int J Comput Assist Radiol Surg. 2016; 11: 2049-2057https://doi.org/10.1007/s11548-016-1430-3
      4. Frid-Adar M, Amer R, Greenspan H. Endotracheal tube detection and segmentation in chest radiographs using synthetic data. In: Shen D, et al. (eds) Medical image computing and computer assisted intervention – MICCAI 2019. Lecture notes in computer science, Springer, 2019;11769:784-792. doi:10.1007/978-3-030-32226-7_87.