Advertisement

FDA-regulated AI Algorithms: Trends, Strengths, and Gaps of Validation Studies

Published:December 27, 2021DOI:https://doi.org/10.1016/j.acra.2021.09.002

      Rationale and Objectives

      To assess key trends, strengths, and gaps in validation studies of the Food and Drug Administration (FDA)-regulated imaging-based artificial intelligence/machine learning (AI/ML) algorithms.

      Materials and Methods

      We audited publicly available details of regulated AI/ML algorithms in imaging from 2008 until April 2021. We reviewed 127 regulated software (118 AI/ML) to classify information related to their parent company, subspecialty, body area and specific anatomy type, imaging modality, date of FDA clearance, indications for use, target pathology (such as trauma) and findings (such as fracture), technique (CAD triage, CAD detection and/or characterization, CAD acquisition or improvement, and image processing/quantification), product performance, presence, type, strength and availability of clinical validation data. Pertaining to validation data, where available, we recorded the number of patients or studies included, sensitivity, specificity, accuracy, and/or receiver operating characteristic area under the curve, along with information on ground-truthing of use-cases. Data were analyzed with pivot tables and charts for descriptive statistics and trends.

      Results

      We noted an increasing number of FDA-regulated AI/ML from 2008 to 2021. Seventeen (17/118) regulated AI/ML algorithms posted no validation claims or data. Just 9/118 reviewed AI/ML algorithms had a validation dataset sizes of over 1000 patients. The most common type of AI/ML included image processing/quantification (IPQ; n = 59/118), and triage (CADt; n = 27/118). Brain, breast, and lungs dominated the targeted body regions of interest.

      Conclusion

      Insufficient public information on validation datasets in several FDA-regulated AI/ML algorithms makes it difficult to justify clinical applications since their generalizability and presence of bias cannot be inferred.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

      1. Benjamens S, Dhunnoo P, Meskó B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ digital medicine. 2020;3:1-8.

      2. Chacon, A, Plasencia, JT, Avila, G, et al. (2019). A deep learning model to aid in detection of pneumothorax via CXR: a retrospective cohort analysis of the NIH-based CXR dataset. Chest, 156(4), A917-A918.

      3. Gawlitza, J, Sturm, T, Spohrer, K, et al. (2019). Predicting pulmonary function testing from quantified computed tomography using machine learning algorithms in patients with COPD. Diagnostics, 9(1), 33.

      4. Uthoff, J, Stephens, MJ, Newell Jr, JD, et al. (2019). Machine learning approach for distinguishing malignant and benign lung nodules utilizing standardized perinodular parenchymal features from CT. Med Phys, 46(7), 3207-3216.

      5. Ji, Y, Li, H, Edwards, AV, et al. (2019). Independent validation of machine learning in diagnosing breast Cancer on magnetic resonance imaging within a single institution. Cancer Imaging, 19(1), 1-11.

      6. Mushtaq, J, Pennella, R, Lavalle, S, et al. (2021). Initial chest radiographs and artificial intelligence (AI) predict clinical outcomes in COVID-19 patients: analysis of 697 Italian patients. Eur Radiol, 31(3), 1770-1779.

      7. Sharma, A, Rani, S, & Gupta, D (2020). Artificial intelligence-based classification of chest X-ray images into COVID-19 and other infectious diseases. Int J Biomed Imaging, 2020;2020:8889023. Published 2020 Oct 6. https://doi.org/10.1155/2020/8889023

      8. Belfiore, MP, Urraro, F, Grassi, R, et al. (2020). Artificial intelligence to codify lung CT in Covid-19 patients. La radiologia medica, 125(5), 500-504.

      9. Li, L, Qin, L, Xu, Z, et al. (2020). Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: evaluation of the diagnostic accuracy. Radiology, 296(2), E65-E71.

      10. Harmon, SA, Sanford, TH, Xu, S, et al. (2020). Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat Commun, 11(1), 1-7.

      11. Roberts M, Driggs D, Thorpe M, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat Mach Intell. 2021;3:199-217.

      12. FDA Cleared AI Algorithms. American College of Radiology Data Science Institute. Available from: https://models.acrdsi.org/ [Accessed May 10, 2021]

      13. Available from: accessed at https://ncihub.org/groups/eedapstudies/wiki/DeviceAdviceAIMLImaging/File:J.A.Segui.ACR.Informatics.2019.Slides.FINAL.pdf on August 10, 2021

      14. Available from: https://www.fda.gov/media/135712/download May 10, 2021

      15. Medical Futurist. Available from: https://medicalfuturist.com/fda-approved-ai-based-algorithms/ [Accessed May 10, 2021]

      16. DeGrave AJ, Janizek JD, Lee SI. AI for radiographic COVID-19 detection selects shortcuts over signal. Preprint. medRxiv. 2020;2020.09.13.20193565. Published 2020 Oct 7. https://doi.org/10.1101/2020.09.13.20193565

      17. Wu E, Wu K, Daneshjou R, et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021;27:582-4.

      18. van Leeuwen KG, Schalekamp S, Rutten MJ, et al. Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol. 2021 Jun;31(6):3797-804.

      19. Dreyer KJ, Allen B, Wald C. Real-World Surveillance of FDA-cleared AI models: Rationale and Logistics. JACR (in press)