Rationale and Objectives
Medical schools were upended by the COVID-19 pandemic, resulting in suspension of all in-person educational activities, and leaving clinical clerkships on hold indefinitely. A virtual curriculum and novel teaching methods were needed to fulfill curricular requirements. We developed a comprehensive virtual radiology clerkship and evaluated the efficacy of this novel method of teaching.
Materials and Methods
A 4-week virtual radiology clerkship was designed to accommodate medical students who had not yet completed the required clerkship. The design included online flipped classroom modules, large group didactic lectures, and small group homeroom activities. Student performance was assessed via a standardized online final exam. Feedback from students was collected using online surveys. Student performance was compared to the in-person radiology clerkship.
Results
One hundred and eleven medical students were enrolled in the virtual radiology clerkship. Final exam scores were similar to the in-person clerkship. Students indicated that small group homeroom activities had the highest overall satisfaction. Students recognized enthusiastic teachers regardless of class format. Exceptional course content and organization were also noted. Course weaknesses included didactic lecture content which was repetitive or too advanced, the limited opportunity to build personal connections with faculty, and scheduling conflicts with other competing school activities.
Conclusion
A completely virtual radiology core clerkship can be a successful educational experience for medical students during a time when remote learning is required. A small group learning environment is most successful for student engagement. Personal connections between faculty and students can be challenging in a virtual course.
Keywords
INTRODUCTION
The COVID-19 pandemic has created a worldwide health care crisis with a range of unprecedented challenges, including a lasting and profound impact on the delivery of medical education (
1
). The COVID-19 pandemic swept through hospital-based medical education like a tsunami, with little time for advanced planning and no clear endpoint. In early winter, Coronavirus was a distant threat, affecting mainly international travel and conferences, but by mid-March the virus had invaded our daily lives with dramatic changes to our home, work, and educational environments, with no clear end in sight (2
).Fears of disease exposure, lack of personal protective equipment, and the need for social distancing abruptly exiled medical students from their hospital-based clerkships. On March 17, 2020, the Association of American Medical Colleges provided guidelines suggesting that medical schools pause clinical rotations for medical students (
3
). Students were quarantined and dispersed to remote locations, and an immediate and dramatic pivot to an alternative educational model was required.Whelan A, Prescott J, Young G, et al. Guidance on medical students’ clinical participation: effective immediately, 2020, https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak .
At our institution, the Radiology Clerkship Committee was called upon to provide a comprehensive 4-week core radiology clerkship for all students enrolled in the year-long Principal Clinical Experience course who had not yet completed the required course. In 2 weeks, a unique Virtual Radiology Core Clerkship was designed, utilizing a variety of learning experiences including online flipped classroom modules, large group didactic lectures, and small group homeroom activities. In this article we describe the design and the logistical challenges involved in structuring a virtual radiology core clerkship for medical students and assess the efficacy of such a novel course.
MATERIALS AND METHODS
Virtual Radiology Course Description
Harvard Medical School utilizes three major teaching hospitals to provide clinical educational experiences for its students. A required 4-week radiology clerkship is a component of the core clerkship year. During the embedded month of dedicated radiology training, students participate in small group lectures, simulation sessions, clinical observations, and assessments. Typically, 4–6 students are enrolled in the radiology clerkship each month at each hospital site. Our radiology clerkship curriculum is well-established, and historically, feedback from faculty and students has been excellent. At the onset of the COVID-19 pandemic, approximately one half of the students in the clinical year had yet to complete the radiology clerkship. With an immediate need to transition to online learning, the radiology core clerkship became the disaster plan for students enrolled in their required clinical year.
With 2 weeks of preparation, a Virtual Radiology Core Clerkship was designed to educate all clerkship level medical students who had not yet completed their radiology clerkship. One hundred and eleven students were enrolled in the Virtual Radiology Core Clerkship, which was taught over a 4-week block in April 2020. The group of students was heterogeneous, including students midway through their clinical year, students for whom this was their first clerkship, and MD/PhD students whose laboratory work was put on immediate hold. The challenge to educate a diverse student population via online teaching methods with very short preparation time is presented.
The construct of the Virtual Radiology Core Clerkship was designed by the clerkship directors from the three hospitals. The learning objectives of the Alliance of Medical Student Educators in Radiology (AMSER) were the basis for the 4-week clerkship. Nineteen online “Aquifer Modules” served as a framework for the clinical material (Aquifer, Inc.). These modules emphasize a patient-centered approach to clinical scenarios, imaging, proper imaging utilization, and patient safety. These resources had been the foundation of our pre-existing in-person core radiology clerkship, which was critical in allowing a quick pivot to virtual learning.
Each night, Aquifer modules were assigned as prereading for the large group didactic lecture on the topic of the day. These lectures, lasting approximately 90 minutes, were given to the entire class via Zoom. A diverse group of lecturers was chosen from the three hospitals to highlight the wealth of expertise at our medical school. Internationally renowned experts in their field gave presentations alongside established educational leaders, retired faculty, and talented junior attendings. Anticipating that the large group didactic format would pose challenges to learner engagement, a moderator was engaged in most lectures (usually one of the course directors). To promote clarification of material, the “chat” function on Zoom was enabled during the lectures, and students were encouraged to submit questions as they arose. The moderator monitored the chat function during the lectures, answering questions directly via the chat function or collecting several questions to be reviewed in real-time with the speaker or during a pause in the presentation. Intermittent question and answer breaks during the lecture helped maintain learner engagement in an otherwise lengthy presentation and allowed the speaker to interact with the audience. Some of the lecturers employed “PollEverywhere” for additional means of learner engagement.
One of the highlights of our traditional in-person radiology clerkship course has been the opportunity to work and learn in a small group setting. Representative past student experiences have included interactive case-based tutorials, unknown case panels, reading room observations, and gamified presentations. Modifying these types of experiences for a large group of students via an exclusively virtual platform posed a particular challenge. In order to provide a comparable experience to our traditional clerkship, the concept of a “virtual homeroom” model was created, allowing for small group interactive learning to occur. A total of 12 homerooms were formed, across the three hospitals, with 8–10 students assigned to each homeroom. Two homeroom teaching sessions were planned for each day, for a total of 24 homeroom sessions taught per day across all hospital sites, and a total of 432 homeroom sessions over the entire course. The first homeroom session each day was an interactive workshop on the topic of the day, with the teaching material provided by Aquifer, and presented via Zoom. Because the students had prepared the material the night before, the workshop employed the “flipped classroom” model.
The second homeroom session each day was an unknown case conference, related to the topic of the day and also presented via Zoom. In this session, students experienced a more typical radiology readout session. The students took turns describing imaging findings using appropriate radiologic terminology and generating differential diagnoses. Our hope with these sessions was to emulate the learning which typically occurs during reading room observations while reinforcing the content presented the same day in the didactic lecture and the workshop. Both daily homeroom sessions included prepared speaker notes so that the material was uniformly covered across homerooms. The readily accessible teaching material was helpful for faculty recruitment. Little preparation was necessary to teach, and even junior residents could run a homeroom session, gaining experience and comfort in teaching.
To provide continuity and mentorship in each homeroom, one or two leaders or “captains” were assigned to each homeroom for the duration of the course. These instructors were radiology residents who were handpicked by the clerkship directors because of their enthusiasm and demonstrated effectiveness as medical student teachers. The homeroom captains were tasked with providing a supportive learning environment and sense of community for the students throughout the month, teaching unknown case sessions, serving as the contact person for concerns, and providing assessments on student participation and engagement. In total, 17 homeroom “captains” were recruited across the three hospital sites to lead the 12 homerooms.
Students were assessed during the course in a similar fashion to our traditional in-person clerkship. An ungraded online multiple choice AMSER exam was completed at the midpoint of the course, in order for students to self-identify areas of strengths and weaknesses which could guide students’ study plans for the remainder of the course. The online multiple choice Standardized AMSER Final Exam was administered on the last day of the course. This multiple choice exam is used in medical schools across the country as a means of assessing competency in radiology clerkships and has been used as the final exam in our in-person clerkship for several years. The students were required to earn a passing score on this exam in order to pass the clerkship. In addition, attendance at didactic lectures and completion of all of the online Aquifer modules were required to pass. These requirements were made to formally monitor student participation in a completely virtual course and also allowed course directors to assess student professionalism.
Standard medical school course evaluation forms were sent to students at the completion of the virtual radiology clerkship. The course survey included 32 questions asking the students to assess facets of the course using a rating system of excellent/good/average/fair/poor (1 = excellent, 5 = poor). Course evaluations were anonymized and were submitted by students before they received their final grade.
In order to comply with LCME standards, the course learning objectives and requirement checklists from our traditional core clerkship were reviewed, and no modifications were needed as we transitioned to a virtual platform. Self-reported student duty hours were tracked in accordance with LCME requirements. Students received online feedback at both the midpoint of the clerkship and after its completion. Optional one-on-one meetings were also available at the end of the course for a student to meet virtually with one of the course directors, to obtain additional feedback or address any concerns the student might have had. Because students routinely complete course evaluations as part of an ongoing quality assurance program, this study was considered to be a quality improvement study, and IRB approval was waived.
RESULTS
Assessment Results
The online, closed book AMSER Standardized Final Exam was administered to all students simultaneously in real-time, with a 3-hour time limit. All students passed the exam, with a mean test score of 85% (range 64%–95%). These scores were similar to final exam scores achieved by students enrolled in our in-person clerkship over the past 5 years, where the mean score was 78% (range 62%–95%).
The midterm multiple choice exam was completed online as an independent formative exercise, without time limitations. This format is identical to what has been used in our in-person clerkship. As the test served as a self-assessment, results are not reported.
Student Experience
Fifty-six students completed the course evaluation survey (50% of enrolled students). Survey results are reported in Table 1. Eighty four percent of responding students rated the course as excellent, while 16% rated the course as good. No students rated the course as average, fair or poor. More than 95% of responding students rated the overall structural organization of the clerkship (including delineation of course objectives and expectations) as either excellent or good. The quality of teaching in the course was rated as excellent or good by 98% of responding students. Ninety-five percent of students felt that the content in the clerkship enabled them to master core knowledge related to radiology, and all students reported that the clerkship promoted an integrated view of the material. The clerkship fostered self-directed learning in 91% of students and enhanced an interest in radiology in 87% of students.
Table 1Course Evaluation Results
| Survey Question | Mean (1 = Excellent to 5 = Poor), Virtual Clerkship | % Students Rating Excellent/Good, Virtual Clerkship |
|---|---|---|
| Overall rating for this clerkship | 1.16 | 100% |
| How well were the clerkship objectives stated at the outset? | 1.23 | 96% |
| Were the criteria used in evaluating your performance explained to you at the beginning of the clerkship? | 1.16 | 96% |
| Rate the organization of the clerkship | 1.25 | 96% |
| How accessible were the faculty | 1.14 | 98% |
| Please rate the overall quality of the teaching in the clerkship | 1.27 | 98% |
| How much did this clerkship enhance your interest in the subject matter | 1.57 | 87% |
| How much did this clerkship foster your self-directed learning? | 1.32 | 91% |
| How well did the content in this clerkship enable you to master core knowledge related to this subject? | 1.38 | 95% |
| How well did this clerkship promote an integrated view of the material? | 1.20 | 100% |
Despite an exclusively virtual platform, 98% of students reported that faculty were very accessible. Direction and constructive feedback were “always” or “very often” given to 86% of students. On average, students received feedback approximately three times during the month from a variety of teachers, including clerkship directors, tutors, house staff, preceptors, and others.
Students were also asked to rate the learning environment of our virtual radiology course (Table 2). Hundred percent of students reported that throughout the course faculty and staff “always” or “almost always” resolved conflicts in ways that respected the dignity of all involved, respected diversity, was respectful of other health professions, showed respectful interactions with students and showed empathy and compassion. No students experienced mistreatment during the clerkship. Ninety-one percent of students reported that professional language was “always” or “almost always” used.
Table 2Student Evaluation of Learning Environment
| Survey Question | Always/Very Often |
|---|---|
| Used professional language/avoiding derogatory language | 91% |
| Resolving conflicts in ways that respect the dignity of all involved | 100% |
| Respecting diversity | 100% |
| Being respectful of other health professions | 100% |
| Being respectful of other specialties | 96% |
| Showing respectful interaction with students | 100% |
| Showing empathy and compassion | 100% |
| Providing direction and constructive feedback | 86% |
As part of the formal course evaluation, students were asked to list three major strengths of the virtual radiology clerkship. The strength most frequently noted by the students was the small group homeroom sessions. Students felt that the homeroom format was interactive and fostered an engaging learning environment. Students identified the enthusiastic and dedicated teachers as a course strength, particularly those who led the homeroom sessions. The students felt that the course content was a major strength, noting a cohesive curriculum, the clinical relevance of the material, and targeting the material to the right audience level. In addition, the students identified the course organization as another strength, appreciating clear communication, an organized schedule, and readily available resources for class and homework.
As part of the formal course evaluation, students were also asked to identify three major weaknesses. The most frequently cited weakness was the repetition of material in the didactic lectures. At times, didactic content overlapped. Students also noted that some of the didactic lectures were too advanced for a medical student audience or were too long in duration. Some students felt the virtual format limited the opportunity to build personal connections with faculty and to obtain feedback on image interpretation. Scheduling conflicts were also experienced by a small number of students while juggling course meetings with other medical student events which were not suspended during the pandemic.
Students were asked to provide suggestions for improvements to the virtual radiology course. The most frequent responses included limiting the didactic lectures to 1 hour and adding audience response questions to every didactic lecture. Students suggested informing didactic lecturers of content previously covered to avoid repetition. A few students requested more imaging interpretation activities and others suggested that more emphasis be placed on developing longitudinal relationships with faculty.
DISCUSSION
At the onset of the COVID-19 pandemic, medical schools across the United States were tasked with balancing medical student education with safety and diversion of resources to clinical care (
4
). To address this unprecedented challenge, most medical schools suspended student participation in direct patient care. Medical students in the midst of clinical clerkships were sent home on short notice with little understanding of how they could continue their education during the pandemic. Medical schools had to think quickly and creatively in order to provide virtual educational experiences for students which were of high educational quality and still met educational program objectives and graduation requirements. Previous studies have validated the effective use of online teaching in medical education (Whelan A.Updated interim guidance for medical students’ participation in patient care during the coronavirus (COVID-19) outbreak, 2020, https://www.aamc.org/news-insights/press-releases/updated-interim-guidance-medical-students-participation-patient-care-during-coronavirus-covid-19.
5
, 6
, 7
, 8
, 9
, 10
). These established teaching tools would be needed as medical schools quickly transitioned to a completely online format.Because of the unique nature of the field of radiology, including its reliance on electronic transmission of imaging and information, radiology educators were in a unique position to quickly respond to the need for virtual education for a large number of students as the COVID-19 pandemic emerged. At our institution, using a combination of readily available radiologic teaching resources (i.e., Aquifer modules) and original material (didactic lectures and unknown case conferences), we were able to modify the traditional in-person core radiology clerkship to an online platform in a very short period of time. The online platform enabled us to enroll a much larger number of students (111 students/class vs. 4–6 students/class in the in-person clerkship). Despite this, small group learning was preserved through the use of virtual homerooms, allowing for learner engagement, clarification of material, and student assessment, similar to opportunities afforded in an in-person clerkship. In order to meet LCME standards, learning objectives, requirement checklists, and assessment activities remained the same for the virtual radiology clerkship.
Overall, the virtual radiology core clerkship was successful. A large number of students were able to complete a clinical course required for graduation, exclusively via an online platform. Overall, learning objectives were met, and student performance on the AMSER standardized final exam was excellent. The majority of students rated the overall course as excellent. Highlights of the course included the virtual homeroom experiences, dedicated faculty, a cohesive course curriculum, and course organization. One reason for the success of the virtual clerkship is consistent with the adult learning theory of “blending learning” in which presenting material in a variety of formats maximizes educational benefits for a variety of learners (
11
). Presenting the course content in a variety of learning formats including large group didactic lectures, small group learning, and flipped classroom models created an effective blended learning environment and were well received.Given the rapid move to virtual learning, with little time allotted for planning, one would expect that didactic lectures would have played the most prominent role in such our course. Theoretically, faculty could be recruited on short notice to give previously prepared lectures to the class, and course material could be covered in an efficient manner. Most of the didactic lectures presented during the course were rated as excellent or outstanding. Students rated lectures with appropriately targeted content and level and with lively, engaged speakers the highest. Praise was also given to lectures with some form of audience response system as this was felt by the students to enhance engagement with the speaker and the material. These results are similar to results from a recent study that reported on patterns of narrative student feedback that are associated with relatively higher and lower evaluation scores (
12
). The Zoom chat function was also well received and created an ongoing dialogue during the lectures to enhance student engagement and understanding.The disadvantage of the didactic sessions of the course was the potential lack of continuity throughout the course due to the multiple speakers. Students expressed frustration when content was repeated in more than one lecture or if content was too advanced for the expectations of the course. Students also noted that attention waned after 1 hour. The traditional didactic approach is passive and does not foster critical thinking and complex reasoning skills that are at the core of medical education (
13
).Providing students with interactive activities such as problem solving, discussions, and debates during class time has been found to improve learning outcomes, motivation, and attitudes (
14
). Creating an interactive learning environment in our virtual radiology course, however, was particularly challenging given the large class size. In addition, we wanted to provide a “reading room observation” type experience for students enrolled in the virtual radiology clerkship, similar to the reading room observation experience students have in an in-person clerkship. To meet these goals, virtual homerooms were created.Overall, the homeroom experience was rated as the highest and most rewarding component of the virtual course. This is not surprising as the homerooms were designed as an active learning environment to enhance the students’ learning experience. In the homerooms, case-based learning was emphasized, and students had an opportunity to work through cases independently, ask questions, and clarify concepts. Active learning in the homeroom facilitated higher order cognitive skills as students could connect new knowledge and skills to existing knowledge (
15
, 16
). Faculty were able to monitor student progress through the course and identify as well as address any potential knowledge gaps. Resident and fellow teachers had an opportunity to teach and provide mentorship.While the homeroom experience was an integral and essential part of the course, the disadvantage of a virtual homeroom was the demand it made on departmental resources and faculty and radiology trainee time. The extensive resources needed to operate a homeroom-based curriculum are similar to those required in a fully problem-based in-person medical school curriculum (
17
, 18
). For such a large course, 12 homerooms were required in order to achieve the small group size desired. Because two homeroom activities were scheduled per day, 432 unique homeroom sessions met over the 4-week course. This required a large number of interested faculty and resident trainees to run these sessions. Because our in-person radiology clerkship has historically been a core rotation at our medical school, the radiology departments at our teaching hospitals have traditionally been very supportive of medical student education and are committed to providing teaching resources and personnel to the clerkship. Faculty and trainees routinely teach in the clerkship and are very familiar with the student’ level of education and the curriculum. These institution-based factors made recruitment of faculty and trainees easier. Teacher recruitment was also made easier because many faculty and trainees had reduced or reconfigured clinical responsibilities during the COVID-19 pandemic and were available to teach. Schedule flexibility was also required by both the medical students and the teaching faculty in order to support both groups’ nonclerkship responsibilities, although students had few additional academic commitments during his time period. Daily homeroom sessions could be scheduled around a resident teacher's clinical responsibilities or around educational resident conferences. Administrative support was handled by a dedicated team at the medical school as well as by individual clerkship coordinators on site. With these factors in mind, if a repetition of this completely virtual course was desired in nonpandemic times, when clinical volume are robust and clinical responsibilities are more taxing, we anticipate it would be difficult to recruit the necessary number of faculty and trainees to lead the small group sessions. Given how important and valued these homeroom sessions were in the overall success of the program, it is likely that the virtual clerkship experience would be diminished if these sessions were limited.While we were able to meet all of our learning objectives for the radiology core clerkship through a virtual platform, there were some areas of concern. In our course feedback, students missed the in-person interactions that would have occurred in the reading rooms in the traditional curriculum. Students had less of a chance to build long-term relationships with faculty and staff. Because the students were not in the reading room, they did not observe a radiologist in action – what a typical day is like, how a radiologist “reads” a case, or how a radiologist consults with referring clinicians. The remote nature of this course left no opportunity to see how “live” cases unfold in the reading room. These reading room experiences have been shown to be extremely impactful on students and could potentially influence specialty choice (
19
). In order to recruit enthusiastic medical students into radiology, either virtual or in-person reading room observations should be included in a radiology clerkship. While the large number of students enrolled in the course made it nearly impossible to provide these types of personalized experiences to students, it would be important to find ways in the future to incorporate real-time reading room experiences into a virtual radiology course. Video conferencing has recently been applied to virtual read-outs to improve the educational experience of radiology trainees in the setting of distance learning (20
). As virtual read-outs with trainees become more routine in the post COVID-19 workflow, medical students could be incorporated into these virtual read-outs. This is an area for further investigation.CONCLUSION
The Virtual Radiology Core Clerkship was a successful educational experience for medical students despite the background of chaos and anxiety created by the COVID-19 pandemic.
Students were able learn the fundamentals of radiology while respecting social distancing requirements. Students enjoyed the sense of community and engagement afforded by small group homerooms. Personal connections between faculty and students, however, were challenging in this virtual course.
References
- Closure of universities due to coronavirus disease 2019 (COVID-19): impact on education and mental health of students and academic staff.Cureus. 2020; 12: e7541
- The socio-economic implications of the coronavirus and COVID-19 pandemic: a review.Int J Surg. 2020; 78: 185-193
Whelan A, Prescott J, Young G, et al. Guidance on medical students’ clinical participation: effective immediately, 2020, https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak .
Whelan A.Updated interim guidance for medical students’ participation in patient care during the coronavirus (COVID-19) outbreak, 2020, https://www.aamc.org/news-insights/press-releases/updated-interim-guidance-medical-students-participation-patient-care-during-coronavirus-covid-19.
- Effect of virtual versus traditional education on theoretical knowledge and reporting skills of dental students in radiographic interpretation of bony lesions of the jaw.BMC Med Educ. 2019; 19: 233
- Case-based asynchronous interactive modules in undergraduate medical education.Acad Pathol. 2019; 62374289519884711115
- Learning clinical reasoning; how virtual patient case format and prior knowledge interact.BMC Med Educ. 2020; 20: 73
- Introduction of a virtual workstation into radiology medical student education.AJR. 2015; 204: W289-W292
- Adaptive tutorials versus web-based resources in radiology: a mixed methods analysis of efficacy and engagement in senior medical students.Acad Radiol. 2019; 26: 1421-1431
- Medical student performance after a vertically integrated radiology clerkship.J Am Coll Radiol. 2016; 13: 67-71
- Blended learning in medical education: use of an integrated approach with web-based small group modules and didactic instruction for teaching radiologic anatomy.Acad Radiol. 2004; 11: 1059-1070
- Lecture evaluations by medical students: concepts that correlate with scores.J Am Coll Radiol. 2016; 13: 72-76
- Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010.Acad Med. 2010; 85: 220-227
- Flipping radiology education right side up.Acad Radiol. 2016; 23: 810-822
- The integration of active learning teaching strategies into a radiology rotation for medical students improves radiological interpretation sills and attitudes toward radiology.Curr Probl Diagn Radiol. 2019; (S0363-0188(19)30125-2)
- Express team-based learning (eTBL): a time-efficient TBL approach in neuroradiology.Acad Radiol. 2020; 27: 284-290
- Effectiveness of problem-based learning curricula: research and theory.Acad Med. 2000; 75: 259-266
- Undergraduate medical education: comparison of problem-based learning and conventional teaching.Hong Kong Med J. 2000; 6: 301-306
- Factors influencing choice of radiology and relationship to resident job satisfaction.Curr Probl Diagn Radiol. 2019; 48: 333-341
- Virtual read-out: radiology education for the 21st century during the COVID-19 pandemic.Acad Radiol. 2020; 27: 872-881
Article info
Publication history
Published online: July 24, 2020
Accepted:
July 17,
2020
Received in revised form:
July 14,
2020
Received:
June 1,
2020
Identification
Copyright
© 2020 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.
