AI-RADS was well-received amongst trainees at our institution. From our metrics of quality, trainees overwhelmingly feel that the content depth of the AI-RADS lecture series is ideal, and the examples used are helpful vehicles to understand key concepts in artificial intelligence. Exit surveys demonstrated a high degree of learner satisfaction, with an aggregate rating of 9.8/10. Resident interest in artificial intelligence has remained at a stable high, suggesting that this course has not deterred learners from the field.
With the exception of Lecture 7, resident confidence in their ability to read an artificial intelligence related journal article in radiology statistically increased after each lecture (Fig. 3
). Only five residents were able to attend Lecture 7, likely contributing to its borderline results. Parametric tests, such as Students t
Test, yielded significant p
-values in all lectures, however, given the low sample size, Wilcoxon Sign Rank was felt to be more accurate, though less statistically powerful. Anecdotally, discussions during these sessions have been robust and the questions residents ask suggest a deeper understanding of the underlying computational methods.
With the content-related questions, learners’ perceived ability to explain these concepts showed a marked increase from before to after each didactic session (Fig. 4
). In lieu of formal assessment, these results demonstrate an increased understanding of core principles of artificial intelligence. This, in conjunction with perceived confidence in reading new articles related to the algorithm discussed, strongly imply an increased sense of comfort when dealing with techniques in artificial intelligence. Future iterations of this course may entail anonymous content-specific multiple-choice quizzes to further evaluate concept mastery.
Journal club was initially successful, but demonstrated a progressive decline in learner preparation, which was evident in later discussions. Residents indicated that added clinical responsibilities and lack of free time were main contributing factors to lack of engagement. In written commentary, residents stated that the journal club was very helpful in solidifying the concepts presented in lecture, and despite limited preparation, found the conversations illuminating. Many proposals have been regarding how journal clubs will be managed when this course is redeployed, including resident-led paper presentations with faculty support, take-home assignments focusing on specific articles, or integration of paper discussion into the didactic session.
Limitations of the course include heterogeneity of learner attendance, usually as a result of the clinical duties of certain rotations, scheduling, and resident burnout. It is important to note the likely contribution that lecture scheduling had in resident attendance. For the final three lectures, there was no availability in the normal didactic timeslot, requiring residents to be dismissed early from clinical duties to attend an end-of-day session. This was suboptimal and likely contributed to the relatively lower turnout. Additionally, this may have influenced overall comprehension, as the last three lectures are conceptually highly interrelated. Transitioning to an online platform would ideally circumvent these issues, as learners would be able to complete modules at their own pace. However, this may come at the cost of large group discussion opportunities. An interactive learn to code session was proposed, with preexisting datasets and skeleton code available such that learners may try to implement the algorithms they learned. While surveyed resident interest was high, many voiced concerns over time constraints with an already heavy schedule. As such, these sessions were felt to be more appropriate for future iterations once the basic curricula was completely incorporated. Lastly, while analysis by trainee demographics (such as sex, PGY, etc.) would add increased resolution to specific learner needs and satisfaction, the size of our program would effectively eliminate feedback anonymity for some learners. For this, demographic information was not obtained to encourage trainee candor in their response.
To ensure longevity and sustainability of this course as a unique hallmark of our institution's residency training program, the department is working on establishing online infrastructure to permanently house this resource. Future plans entail publishing all materials and freely sharing this educational series for all interested learners. These videos will be uploaded to the following YouTube channel as they become available: https://rb.gy/ychu2k
As artificial intelligence continues to reshape the world of medicine, it will become imperative that physicians are familiar with fundamental algorithms and techniques in artificial intelligence. This will become an essential skill for interpreting medical literature, assessing potential clinical software augmentations, formulating research questions, and purchasing equipment. By having an intuitive foundation of machine learning based around fundamental algorithms, learners will likely be better equipped to understand strengths and weaknesses of various techniques and be empowered to make more informed decisions.
In summary, residency programs are only beginning to employ basic computing concepts in their training, a skill that will become essential for the radiologists of tomorrow; proficiency in artificial intelligence will be a required skill in the near future of imaging services. We present our institution's efforts to address this problem as a model of a successful introductory curriculum into the applications of artificial intelligence on radiology.
Appendix A. Journal Club Articles
Do, B. H., Langlotz, C., & Beaulieu, C. F. (2017). Bone Tumor Diagnosis Using a Naïve Bayesian Model of Demographic and Radiographic Features. Journal of Digital Imaging, 30
(5), 640–647. https://doi.org/10.1007/s10278-017-0001-7
Emblem, K. E., Pinho, M. C., Zöllner, F. G., Due-Tonnessen, P., Hald, J. K., Schad, L. R., Meling, T. R., Rapalino, O., & Bjornerud, A. (2015). A Generic Support Vector Machine Model for Preoperative Glioma Survival Associations. Radiology, 275
(1), 228–234. https://doi.org/10.1148/radiol.14140770
Jensen, C., Carl, J., Boesen, L., Langkilde, N. C., & Østergaard, L. R. (2019). Assessment of prostate cancer prognostic Gleason grade group using zonal-specific features extracted from biparametric MRI using a KNN classifier. Journal of Applied Clinical Medical Physics, 20
(2), 146–153. https://doi.org/10.1002/acm2.12542
Srinivasan, A., Galbán, C. J., Johnson, T. D., Chenevert, T. L., Ross, B. D., & Mukherji, S. K. (2010). Utility of the K-Means Clustering Algorithm in Differentiating Apparent Diffusion Coefficient Values of Benign and Malignant Neck Pathologies. American Journal of Neuroradiology, 31
(4), 736–740. https://doi.org/10.3174/ajnr.A1901
Zhang, B., Tian, J., Pei, S., Chen, Y., He, X., Dong, Y., Zhang, L., Mo, X., Huang, W., Cong, S., & Zhang, S. (2019). Machine Learning–Assisted System for Thyroid Nodule Diagnosis. Thyroid, 29
(6), 858–867. https://doi.org/10.1089/thy.2018.0380
Published online: October 16, 2020
Received in revised form:
Funding sources: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
© 2020 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.