Advertisement

Adapting to the Era of Virtual Recruitment: Radiology Departmental Website Response to COVID-19 and Portrayal of the Resident Experience

      Rationale and Objectives

      The COVID-19 pandemic has transformed radiology recruitment into a virtual affair and placed an even stronger emphasis on the importance of departmental websites. In this study, we evaluate residency websites in detailing the response to COVID-19 as well as initiatives which help describe the resident experience.

      Materials and Methods

      Program websites for diagnostic radiology residencies listed in the 2022 Electronic Residency Application Service (ERAS) program list were evaluated for 31 criteria related to departmental response to COVID-19, online outreach, and resident wellness.

      Results

      Of 184 programs, 182 had functioning websites for review. One program was excluded from analysis as the website was almost entirely video-based. In response to COVID-19, ≤1% described resident redeployment, vaccination information, departmental response to ABR Core Exam changes, or regular administration updates. Six (3.3%) described revised read-out protocols, four (2.2%) mentioned supplementary non-clinical education, and 14 (7.7%) indicated changes to educational conferences. The majority of websites (122, 67.4%) offered an informational or tour video, while 44 (24.3%) described expectations for virtual interviewing, and 20 (11.0%) had virtual “open-houses.” Departmental social media, primarily Twitter, was linked for 60 (33.1%) programs. A total of 134 (74.0%) websites described community highlights.
      More than a quarter mentioned meal stipends (72, 39.8%), paid sick time (54, 29.8%) and healthcare resources (57, 31.5%). Although social activities were described by 44 (24.3%) programs, some specifically indicating changes to COVID-19, formal resident mentoring (25, 13.8%) and wellness committees (28, 15.5%) were less common. These criteria were found more commonly at the largest third of residency programs (chi square, p < 0.00625).

      Conclusion

      Programs rarely described work flow changes to COVID-19, and websites could improve in virtual outreach. Compared with prior literature, departmental websites have improved in describing wellness initiatives and related measures.

      Keywords

      INTRODUCTION

      The COVID-19 pandemic has brought unprecedented challenges for radiology residency programs. Unique hurdles included the delay of the 2020 ABR Core Exam, which caused significant disruption to PGY4 residents preparing for the exam (
      • Shi J
      • Miskin N
      • Dabiri BE
      • et al.
      Quantifying impact of disruption to radiology education during the COVID-19 pandemic and implications for future training.
      ), as well as clinical redeployment, which heavily impacted junior residents (
      • Poyiadji N
      • Klochko C
      • LaForce J
      • et al.
      COVID-19 and radiology resident imaging volumes–differential impact by resident training year and imaging modality.
      ). To address COVID-19 challenges, programs adopted social distancing, virtual curricula, and remote-read out techniques (
      • Matalon SA
      • Souza DAT
      • Gaviola GC
      • et al.
      Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
      ,
      • Larocque N
      • Shenoy-Bhangle A
      • Brook A
      • et al.
      Resident experiences with virtual radiology learning during the COVID-19 pandemic.
      ,
      • Recht MP
      • Fefferman NR
      • Bittman ME
      • et al.
      Preserving radiology resident education during the COVID-19 pandemic: the simulated daily readout.
      ). Despite these changes, a recent survey of program directors still suggests an overall negative impact on educational activities and decreased resident morale (
      • Robbins JB
      • Shubeck SP
      • Kanters AE
      • et al.
      Lactation policy and resources for trainees in the department of radiology.
      ). Given COVID-19 remains an active concern globally and with uncertain times still ahead, prospective residents are also stakeholders in how a program has responded to COVID-19.
      In response to the COVID-19 pandemic, the 2020–2021 residency application cycle was performed completely virtually per recommendations from the Accreditation Council for Graduate Medical Education (

      Accreditation Council for Graduate Medical Education. Recommendations for Away Rotations and Interviews for Fellowship Applicants in 2020-2021. ACGME 2020. Available at: https://acgme.org/Newsroom/Newsroom-Details/ArticleID/10421/Recommendations-for-Away-Rotations-and-Interviews-for-Fellowship-Applicants-in-2020-2021. Accessed June 16, 2020.

      ). In June of 2021, the Association of Program Directors in Radiology issued a statement recommending continued virtual interviews for the 2021–2022 residency application cycle (

      Association of Program Directors in Radiology. APDR/APDIR Position Statement Regarding Residency Recruitment in the 2021-2022 Cycle. APDR 2021. Available at: https://www.apdr.org/-/media/Files/APDR/New-and-Events/APDR_APDIR-Position-Statement-2021-22-Residency-Recruitment.ashx?la=en&hash=E066261A83096672666517EE5B485FDDDD6A3688. Accessed October 1, 2021.

      ). In this virtual era, studies within and outside of radiology have reported generally positive feedback among faculty and applicants to the electronic interview structure (
      • Yee JM
      • Moran S
      • Chapman T.
      From beginning to end: a single radiology residency program's experience with web-based resident recruitment during COVID-19 and a review of the literature.
      ,
      • Hill MV
      • Ross EA
      • Crawford D
      • et al.
      Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic.
      ,
      • Donahue LM
      • Morgan HK
      • Peterson WJ
      • et al.
      The carbon footprint of residency interview travel.
      ,

      Gallo K, Becker R, Borin J, et al. Virtual residency interviews reduce cost and carbon emissions. J Urol WoltersKluwer; 0 (0):10.1097/JU.0000000000002197. doi: 10.1097/JU.0000000000002197.

      ).
      One of the main challenges to virtual interviewing is the limited exposure that applicants have to each program along with its culture and geographic location. In place of in-person interactions, program websites and other online resources have increased in importance in describing the resident experience to applicants and their decision making for program ranking (
      • Yee JM
      • Moran S
      • Chapman T.
      From beginning to end: a single radiology residency program's experience with web-based resident recruitment during COVID-19 and a review of the literature.
      ,
      • Czawlytko C
      • Smith E
      • Awan O
      • et al.
      The effect of virtual interviews and social media on applicant decision-making during the 2020-2021 resident match cycle.
      ). Departmental websites vary widely in the level of information they provide (
      • Novin S
      • Yi PH
      • Vanderplas T
      • et al.
      Integrated interventional radiology residency program websites: a development in progress.
      ,
      • Hansberry DR
      • Bornstein J
      • Agarwal N
      • et al.
      An assessment of radiology residency program websites.
      ,
      • Wong TY
      • Huang JJ
      • Hoffmann JC
      • et al.
      Resident wellness in radiology as portrayed by departmental websites.
      ,
      • Makary MS
      • Niedermeier M
      • Grewal BK
      • et al.
      Nationwide analysis of integrated and independent interventional radiology residency websites during the COVID-19 pandemic.
      ). Prompt website updates may be particularly important at this juncture to highlight a program's response to these unprecedented times to applicants (
      • Poyiadji N
      • Klochko C
      • LaForce J
      • et al.
      COVID-19 and radiology resident imaging volumes–differential impact by resident training year and imaging modality.
      ,
      • Matalon SA
      • Souza DAT
      • Gaviola GC
      • et al.
      Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
      ,
      • Larocque N
      • Shenoy-Bhangle A
      • Brook A
      • et al.
      Resident experiences with virtual radiology learning during the COVID-19 pandemic.
      ,
      • Recht MP
      • Fefferman NR
      • Bittman ME
      • et al.
      Preserving radiology resident education during the COVID-19 pandemic: the simulated daily readout.
      ,
      • Robbins JB
      • Shubeck SP
      • Kanters AE
      • et al.
      Lactation policy and resources for trainees in the department of radiology.
      ,
      • Chong A
      • Kagetsu NJ
      • Yen A
      • et al.
      Radiology residency preparedness and response to the COVID-19 pandemic.
      ,
      • England E
      • Kanfi A
      • Flink C
      • et al.
      Radiology residency program management in the COVID Era – strategy and reality.
      ,
      • Nguyen JK
      • Shah N
      • Heitkamp DE
      • et al.
      COVID-19 and the radiology match: a residency program's survival guide to the virtual interview season.
      ).
      Literature within and outside of radiology suggest that virtual interviews may persist beyond the era of heightened COVID-19 precautions (
      • Yee JM
      • Moran S
      • Chapman T.
      From beginning to end: a single radiology residency program's experience with web-based resident recruitment during COVID-19 and a review of the literature.
      ,
      • Hill MV
      • Ross EA
      • Crawford D
      • et al.
      Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic.
      ,
      • Wright AS.
      Virtual interviews for fellowship and residency applications are effective replacements for in-person interviews and should continue post-COVID.
      ). While it is too early to evaluate the long-term impacts from virtual interviewing, there are a number of immediate benefits which may argue for maintaining their use even if not necessitated by a pandemic. Cost and time savings can be tangible, with estimates of average interview travel expenses costing $4,000–$4,500, often requiring additional loans or compelling applicants to forego a portion of interviews (
      • Heitkamp NM
      • Snyder AN
      • Ramu A
      • et al.
      Lessons learned: applicant equity and the 2020-2021 virtual interview season.
      ,
      • Fogel HA
      • Liskutin TE
      • Wu K
      • et al.
      The economic burden of residency interviews on applicants.
      ,
      • Joshi A
      • Bloom DA
      • Spencer A
      • et al.
      Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond.
      ). Cost savings have helped increase interview attendance (
      • Czawlytko C
      • Smith E
      • Awan O
      • et al.
      The effect of virtual interviews and social media on applicant decision-making during the 2020-2021 resident match cycle.
      ) and likely improved equitable access (
      • Heitkamp NM
      • Snyder AN
      • Ramu A
      • et al.
      Lessons learned: applicant equity and the 2020-2021 virtual interview season.
      ). Environmental cost savings are also substantial, with literature estimating the carbon footprint of travel at 0.21-0.49 metric ton of CO2 per interview (
      • Donahue LM
      • Morgan HK
      • Peterson WJ
      • et al.
      The carbon footprint of residency interview travel.
      ,

      Gallo K, Becker R, Borin J, et al. Virtual residency interviews reduce cost and carbon emissions. J Urol WoltersKluwer; 0 (0):10.1097/JU.0000000000002197. doi: 10.1097/JU.0000000000002197.

      ). If extrapolated across the number of diagnostic radiology ranks per US MD/DO applicant (

      National Resident Matching Program. Charting Outcomes in the Match: Senior Students of US DO Medical Schools. 2020. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2020/07/Charting-Outcomes-in-the-Match-2020_DO-Senior_final.pdf. Accessed September 30, 2021.

      ,

      National Resident Matching Program. Charting Outcomes in the Match: Senior Students of US MD Medical Schools. 2020. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2020/07/Charting-Outcomes-in-the-Match-2020_MD-Senior_final.pdf. Accessed September 30, 2021.

      ), the carbon footprint in 2020 amounted up to 5,092 metric tons of CO2. This carbon offset is equivalent to powering 925 homes for 1 year (

      United States Environmental Protection Agency. Greenhouse Gas Equivalencies Calculator. 2021. Available at: https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator. Accessed October 6, 2021.

      ). With unknown perpetuity of virtual interviewing, programs should consider continuing to improve their online presence and efforts in virtual recruiting.
      The purpose for this study was to evaluate available radiology residency websites for the presence of criteria pertaining to a department's COVID-19 response and its virtual outreach, as well as factors which may reflect the resident experience at a program. This data may help guide programs in improving the transparency and completeness of their online presence.

      METHODS

      A list of diagnostic radiology residency programs within the United States was obtained from the 2021–2022 Electronic Residency Application Service (ERAS). If departmental websites were incorrectly hyperlinked, the residency program was found via Google search. Two authors tabulated the presence or absence of 31 criteria on program websites during September of 2021. Criteria were developed by the authors, ranging from resident trainees to program director and faculty with a variety of experience in educational leadership, both before and during the COVID-19 pandemic. A subset of criteria were also selected based upon prior literature. Table 1 lists the surveyed criteria. To reduce interobserver variability, the two website reviewers revisited tabulated data after the initial 20 websites were assessed to maintain consensus. Criteria were also marked present if program websites directly linked to a pertinent webpage in a single hyperlink. Institutional Review Board approval was not needed as data was publicly available online.
      Table 1Categories and Criteria Evaluated
      CategoryCriteriaDescription/Example
      COVID-19 ResponseRedeploymentRadiology residents working outside of their home department related to COVID-19 needs
      Vaccination InformationCOVID-19 vaccine provision, requirements
      Read-Out ChangesTiming, frequency, or social distancing changes due to COVID-19
      Remote Clinical WorkAvailability/use of off-campus clinical work stations
      Remote Non-Clinical EducationAsynchronous education not directly impacting patient care
      ABR Core Exam ChangesTesting or study accommodations
      Educational Conference ChangesTiming, frequency, or social distancing changes due to COVID-19
      Regular Administrative UpdatesDescription of regular program or departmental communications to trainees
      Virtual OutreachVirtual Open HouseAdvertisement or mention of online events to provide program information to prospective applicants
      Virtual Interview DescriptionInterview day timeline, expectations, virtual software utilized, as well as pre/post interview events
      Program VideoInformational videos, either GME or specific to the radiology department
      Social MediaOnline media platforms and accounts specific to the radiology department
      Program InformationMeal StipendFunds provided to residents to purchase hospital food
      Free FoodFree meals or snacks provided to residents on a regular basis, catered conferences
      Book FundDiscretionary education funds over four years
      AIRP FundFunds provided for AIRP in addition to tuition
      Provided TechnologyProvided tablet or computer devices
      PGY2/R1 CallPGY2 after-hours clinical responsibilities
      Personal WellnessPaid Sick/Wellness TimePaid time off per year for illness or wellness
      GymAvailable fitness facilities for trainees
      Wellness/Non-clinical CurriculaNon-interpretive curriculum such as wellness, resiliency, financial literacy, business practice
      Healthcare/Mental Wellness ResourcesPrimary care or mental health support for trainees
      Partner/Childcare ResourcesSupport resources for significant others or children
      Work-Life WellnessResident SpaceDedicated non-clinical space for trainees, either GME or specific to the radiology department
      Community HighlightsIntroducing social activities or city highlights outside of the hospital institution
      RetreatsDesignated social activities at various time points in the year: resident welcomes, holiday parties
      SocialsRegular department-sponsored social outings
      Resident Wellness/Social CommitteeFormal workgroups tasked with promoting trainee wellness or social activities
      Wellness/Social WebpageOnline information specifically for resident wellness or social activities
      Pictures of Social GatheringsMedia of residents in non-clinical, social activities
      Resident MentoringFormal “buddy” or “family” systems to provide resident support
      AIRP, American Institute for Radiologic Pathology; GME, graduate medical education.
      The number of residents at each program was determined on program websites and was inclusive of integrated interventional radiology residents within PGY2–PGY5 years. Program size was grouped into equal thirds (small, ≤18 residents; medium, 19–32 residents; large, ≥33 residents) for chi-square analysis. Chi-square analysis was performed within Microsoft Excel to determine if criteria varied in prevalence according to program size. Similar to prior literature (
      • Wong TY
      • Huang JJ
      • Hoffmann JC
      • et al.
      Resident wellness in radiology as portrayed by departmental websites.
      ), eight criteria related to work-life wellness were analyzed with chi-square analysis. Data without program size information was censored from chi-square analysis. Due to the number of criteria included within the chi-square, Bonferroni correction for multiple comparisons was applied to analysis (significant p = 0.00625).
      Educational book funds and stipends beyond tuition for the American Institute for Radiologic Pathology (AIRP) were quantified, if available. Descriptive statistics and 95% confidence intervals were provided to describe variance and to provide a comparison with prior literature. The presence of criteria was not evaluated within program videos, due to the wide variability in the length and structure of videos.

      RESULTS

      Website Availability and Program Size

      Out of 184 programs participating in the 2021–2022 ERAS, 182 had websites available for review. Two programs lacked websites, and one program was excluded from analysis as the entire website was primarily comprised of a multitude of videos. Therefore, criteria data was available for 181 programs. Program size was determined for 180 programs, with a mean of 26.4 residents.

      COVID-19 Response

      In response to COVID-19, only select programs described resident redeployment(1, 0.6%), vaccination information (2, 1.1%), or regular updates from departmental administration (2, 1.1%). No programs described the departmental response to the delay in the 2020 ABR Core Exam. No programs mentioned any clinical work performed remotely off campus. Six (3.3%) described revised read-out protocols, which consisted of descriptions for virtual read-outs using telecommunication software as well as two accounts of socially distanced in-person read-outs. Four (2.2%) mentioned remote non-clinical education: two were online virtual curricula accessible from home and two described a simulation imaging work list. A total of 14 (7.7%) indicated changes to educational conferences, primarily describing lectures broadcast electronically for at least a portion of the resident classes.

      Virtual Outreach

      Virtual outreach initiatives are summarized in Figure 1. The majority of websites (122, 67.4%) offered a program video. These varied in length and content, involvement of residents and faculty, as well as in overall style. For example, a number centered on tours, while others focused on resident and faculty testimonials. Out of the 122 websites with a program video, 26 (21.3%) displayed a generic graduate medical education (GME) video and did not have media specific to the radiology residency.
      Figure 1
      Figure 1Prevalence of criteria related to program virtual outreach among departmental websites.
      Forty-four programs (24.3%) mentioned virtual interviewing procedures, and 20 (11.0%) had information about a virtual “open-house.” Departmental social media was provided for 60 programs (33.1%). The most common social media outlet was Twitter (52/60, 86.7%), followed by Instagram (34/60, 56.7%), and YouTube (22/60, 35.7%).

      Program Information

      Seventy-two out of 181 programs (39.8%) mentioned meal stipends on their websites. Most described stipends per call shift, while some described a set amount per meal or per year. A free food criterion, which encompasses a broad spectrum across sponsored snacks or drinks in a resident lounge to regularly catered lunches for conferences, was described on 26 websites (14.4%). Educational book funds were present for 100 programs (55.2%), of which 73 programs provided a quantitative amount averaging $3782 (SD $2261, 95% CI ± $519) over 4 years. AIRP funds were described in 93 programs (51.4%), of which 47 programs provided quantitative data averaging $2347 (SD $1054, 95% CI ± $301) over 4 years. Department sponsored technological tools, most commonly described as an iPad or tablet, were present for 31 programs (17.1%). The criteria that may impact financial considerations in the resident experience are summarized in Figure 2.
      Figure 2
      Figure 2Prevalence of criteria with program details which may impact financial considerations for residents among departmental websites.
      PGY2 resident after-hours call responsibility was described on 89 program websites (49.2%), but only explicitly defined in 68 of these, with responsibilities starting in the fall of PGY2, spring of PGY2, or the beginning of PGY3 in fairly equal proportion (Figure 3).
      Figure 3
      Figure 3Prevalence of mentioning junior resident after-hour call responsibilities and, if available, the percentage of overall programs describing when call duties start.

      Personal Wellness

      Paid sick days or wellness days were explicitly mentioned on 54 department websites (29.8%). Sixteen programs (8.8%) specifically mentioned wellness days at the discretion of trainees, averaging 3.1 days per year. A wellness or non-clinical curriculum, which encompasses non-interpretive skills such as building of resiliency, learning about personal finance, and/or practice administration was present for 41 websites (22.7%). At least one quarter of programs mentioned fitness facilities or healthcare and mental health resources, with fewer describing partner or childcare support. The prevalence of personal wellness criteria online is displayed in Figure 4.
      Figure 4
      Figure 4Prevalence of personal wellness resources on departmental websites.

      Work-Life Wellness

      A dedicated resident lounge space was mentioned on 40 websites (22.1%). This criterion did not differentiate between a radiology specific or house staff specific space. Community highlights and available activities were posted for 134 programs (74.0%).
      Criteria pertaining to resident wellness and social interactions were variably present. The most commonly available item was media demonstrating resident social activities (67, 37.0%), and the least common was description of a resident “family” or mentoring group (25, 13.8%). Criteria analyzed by chi-square are described in Figure 5  (observed data provided in Appendix Table A1). Formal retreats (p = 0.0162) and a wellness or non-clinical curriculum (p = 0.00780) were observed more often than expected within programs in the top third of residency size, but this was not statistically significant after Bonferroni correction.
      Figure 5
      Figure 5Prevalence of wellness criteria on radiology residency websites. *Found at larger programs (≥33 residents) and at ^mid-sized programs  (19–32 residents) more often than expected after Bonferroni correction for multiple comparisons  (chi-square, P<0.00625).
      Table A1Observed Data Evaluated by Chi-Square Analysis to Determine if Criteria Appeared More Often Than Expected Depending on Program Size Grouped Into Equal Thirds  (Small, ≤18 Residents; Medium, 19–32 Residents; Large, ≥33 Residents). Shaded Boxes Indicate Statistical Significance After Bonferonni Correction for Multiple Comparisons (p < 0.00625).
      RetreatsSocialsPictures of Social GatheringsWellness/Non-clinical CurriculumResident MentoringResident Wellness/Social CommitteeWellness/Social PageResident Space
      Program SizeSmall6310802118
      Medium141222101082114
      Large2124352315172117
      Sum Observations4139674125275339
      Expected Observations per Program Size if Null13.71322.313.78.33917.713
      Chi-square p-Value  (statistically significant p = 0.00625)0.01620.0001960.0009120.00780.0009120.001780.1520.199

      DISCUSSION

      COVID-19 Response

      Departmental responses to COVID-19 are in various stages of implementation depending on local institutional policies and COVID-19 prevalence. Websites may offer information to prospective residency applicants interested in evaluating a program's level of response to COVID-19 challenges. Transparency and close communication with program administration can be critical for resident wellness (
      • England E
      • Kanfi A
      • Flink C
      • et al.
      Radiology residency program management in the COVID Era – strategy and reality.
      ). Although there is a plethora of radiology education literature regarding COVID-19 and anecdotally at least some changes among all programs, we find a marked disconnect with the paucity of information available online, specifically on residency program websites.

      Virtual Outreach

      Without in-person interactions, prospective residents rely on virtual resources to learn about programs. Virtual open houses have been described as an effective method for applicants to learn about a program culture, its educational offerings, and what to expect in virtual interviews (
      • Tanaka ME
      • Brideau HR
      • An TJ
      • et al.
      Utilization of a virtual information session to increase engagement with prospective applicants in the setting of COVID-19.
      ). A recent survey indicated that 57% of a program's applicants who attended a virtual open house said the event impacted their application decision (
      • Czawlytko C
      • Smith E
      • Awan O
      • et al.
      The effect of virtual interviews and social media on applicant decision-making during the 2020-2021 resident match cycle.
      ). Despite this importance, virtual open houses were advertised on only 11.0% of residency program websites. Candidates may learn of open houses on social media (
      • Tanaka ME
      • Brideau HR
      • An TJ
      • et al.
      Utilization of a virtual information session to increase engagement with prospective applicants in the setting of COVID-19.
      ), but we find only one-third of programs linked their social media accounts on their websites. Although open houses may be most impactful at the start of an ERAS application season, improving applicant exposure to current residents and faculty as well as bolstering avenues to connect on social media may still impact rank list decisions (
      • Czawlytko C
      • Smith E
      • Awan O
      • et al.
      The effect of virtual interviews and social media on applicant decision-making during the 2020-2021 resident match cycle.
      ).
      Program videos, virtual tours, and testimonies likely play an important role in introducing applicants to a program's culture and its resident experience (
      • Yee JM
      • Moran S
      • Chapman T.
      From beginning to end: a single radiology residency program's experience with web-based resident recruitment during COVID-19 and a review of the literature.
      ,
      • Nguyen JK
      • Shah N
      • Heitkamp DE
      • et al.
      COVID-19 and the radiology match: a residency program's survival guide to the virtual interview season.
      ). Out of the programs that displayed a video on their website, 21.3% (14.4% of all residency websites) had only a generic video from the institution's GME. These videos may provide a general sense of the overarching institution and location, but generic house-staff videos are unlikely to adequately convey factors unique to radiology residency and its trainees. If budgeting allows, departments should consider developing a program specific video.

      The Resident Experience

      Department websites are great avenues to communicate clinical curricula and educational programs, and these have been described in prior literature (
      • Novin S
      • Yi PH
      • Vanderplas T
      • et al.
      Integrated interventional radiology residency program websites: a development in progress.
      ,
      • Hansberry DR
      • Bornstein J
      • Agarwal N
      • et al.
      An assessment of radiology residency program websites.
      ,
      • Makary MS
      • Niedermeier M
      • Grewal BK
      • et al.
      Nationwide analysis of integrated and independent interventional radiology residency websites during the COVID-19 pandemic.
      ). We also aimed to evaluate criteria describing program information, personal wellness, and work-life wellness which may impact a resident's overarching experience at a program. Multiple elements factor into resident welfare, but we specifically selected a subset of criteria previously evaluated in April of 2020, and subsequently published (
      • Wong TY
      • Huang JJ
      • Hoffmann JC
      • et al.
      Resident wellness in radiology as portrayed by departmental websites.
      ), to investigate how program websites have responded to COVID-19 and the 2020–2021 virtual interview season. All criteria compared with previous data have increased in prevalence on departmental websites.
      Both availability of meal stipends and book funds showed increases greater than 10% (14.4% and 13.6% respectively). No significant difference was observed between mean book funds ($3782, 95% CI ± $519, versus $3762) or mean AIRP funds ($2347, 95% CI ± $301, versus $2204). With AIRP presented entirely virtually until August of 2022, and at least a portion of virtual options extending to November of 2023 (
      ACR Institute for Radiologic Pathology
      Four Week Radiologic Pathology Correlation Course.
      ), book and AIRP funds may be subject to change in the coming years.
      Junior resident call responsibilities were mentioned in slightly greater frequency than previously reported (49.2% versus 47.6%). There continues to be similar proportion of programs describing call duties starting in the fall of PGY2, spring of PGY2, or the beginning of PGY3. In the initial COVID-19 surge, PGY2 residents were described as having the largest decrease in imaging interpretation volume, likely due to clinical redeployment. This may have downstream effects on radiology resident call preparedness (
      • Poyiadji N
      • Klochko C
      • LaForce J
      • et al.
      COVID-19 and radiology resident imaging volumes–differential impact by resident training year and imaging modality.
      ). The timing of commencement of after-hour call responsibilities, which impacts work-life wellness, may be susceptible to changes with COVID-19 surges and imaging volume.
      The criteria pertaining to personal wellness with the greatest improvements in prevalence on departmental websites were: healthcare and mental health resources (31.5% versus 13.5%), as well as wellness and non-clinical curricula (22.7% versus 8.1%), followed by sick and wellness days (29.8% versus 20.0%). The increase in website information for healthcare and mental health resources is a positive development, and likely reflects a recent report that program directors felt there has been adequate access for residents (
      • Robbins JB
      • Shubeck SP
      • Kanters AE
      • et al.
      Lactation policy and resources for trainees in the department of radiology.
      ). Days specifically allotted for wellness were identified at fewer programs than before (8.8% versus 10.8%), but this is likely attributable to the current study's technique of evaluating departmental websites only (and not institutional GME sites). Overall, increased visibility and availability of the assessed criteria on program websites may reflect heightened program initiatives to focus on resident wellness during the pandemic.
      Work-life wellness continues to play an important role in training and is even more important with added COVID-19 stressors; therefore, some have suggested innovative activities to implement in the era of social distancing (
      • Chong A
      • Kagetsu NJ
      • Yen A
      • et al.
      Radiology residency preparedness and response to the COVID-19 pandemic.
      ,
      • England E
      • Kanfi A
      • Flink C
      • et al.
      Radiology residency program management in the COVID Era – strategy and reality.
      ). In addition to the increase in program websites describing work-life wellness criteria, some described innovative techniques such as: virtual happy hours, virtual painting activities, movie nights, and work-out challenges. Chi-square analysis of eight criteria previously evaluated shows that department-funded socials and pictures of social activities remain more prevalent than expected at the largest third of programs, in addition to resident mentoring as well as wellness and social committees. On the other hand, the other criteria (formal retreats, wellness and non-clinical curricula, and dedicated wellness or social pages) are no longer statistically significantly associated with the largest programs, though still increased in online representation. These changes suggest that there has been shifts in website information and possibly work-life wellness offerings among programs. Programs should continue their progress in improving the thoroughness of their websites and reflecting on the support they provide to their residents.

      Limitations

      A true resident experience can be difficult to glean from online resources. However, utilizing the presence or absence of website criteria as a proxy offers a program-to-program comparison accessible to prospective applicants. Program videos, especially thorough ones, may offer great insight into a program. A limitation of this study is that we did not specifically include the content of the videos in evaluating criteria, though acknowledge that there may be information provided in video form and not otherwise documented on a website. In the same vein, we did not attend any virtual open houses, which is likely another effective avenue to distribute information to applicants.
      Although we noted the number of trainees at a program, we did not record information about resident diversity. Subjectively, some programs did include diversity, equity, and inclusion statements on webpages, but further study is necessary to evaluate program offerings.
      Intra- and inter-observer variability is also a possible limitation. We attempted to mitigate this by performing a systematic and exhaustive search of all available webpages and links on program websites, as well as assessing consensus early in the review period.
      Programs vary in frequency and thoroughness in updating their websites. On our review, there were websites which were clearly outdated, for example with graduated resident classes listed as current residents, or interview instructions describing local restaurant expectations in pre-pandemic interview socials. These websites were included in review to offer a broad evaluation of program websites. Hopefully this data provides encouragement for programs to be mindful of the accuracy of their online resources.

      CONCLUSIONS AND FUTURE DIRECTIONS

      This study evaluated radiology residency program websites for information related to changes from COVID-19 and program initiatives in virtual recruitment, as well as portrayal of the resident experience. Despite substantial impacts from COVID-19, programs rarely offered online information about their workflow changes. Virtual recruitment initiatives could be upgraded, and this should be a strong consideration given the uncertainty of whether virtual interviews will persist on account of time, cost, and carbon footprint savings. In summary, between April 2020 and September 2021, program websites have improved in describing wellness initiatives and the overall resident experience. However, opportunities exist to make additional improvements and updates.
      Until more data becomes available and given lack of standardized guidelines, programs should consider soliciting feedback from their residency class starting radiology in July of 2022 regarding the content and effectiveness of their websites. These trainees were first to experience virtual recruitment and may have crucial insight for website improvement. Programs can also consider utilizing their websites to provide updated information and offerings for their current trainees, which would then also be visible to prospective applicants, serving a dual purpose.

      REFERENCES

        • Shi J
        • Miskin N
        • Dabiri BE
        • et al.
        Quantifying impact of disruption to radiology education during the COVID-19 pandemic and implications for future training.
        Curr Probl Diagn Radiol. 2020; https://doi.org/10.1067/j.cpradiol.2020.07.008
        • Poyiadji N
        • Klochko C
        • LaForce J
        • et al.
        COVID-19 and radiology resident imaging volumes–differential impact by resident training year and imaging modality.
        Acad Radiol. 2021; 28: 106-111https://doi.org/10.1016/j.acra.2020.09.010
        • Matalon SA
        • Souza DAT
        • Gaviola GC
        • et al.
        Trainee and attending perspectives on remote radiology readouts in the era of the COVID-19 pandemic.
        Acad Radiol. 2020; 27: 1147-1153https://doi.org/10.1016/j.acra.2020.05.019
        • Larocque N
        • Shenoy-Bhangle A
        • Brook A
        • et al.
        Resident experiences with virtual radiology learning during the COVID-19 pandemic.
        Acad Radiol. 2021; 28: 704-710https://doi.org/10.1016/j.acra.2021.02.006
        • Recht MP
        • Fefferman NR
        • Bittman ME
        • et al.
        Preserving radiology resident education during the COVID-19 pandemic: the simulated daily readout.
        Acad Radiol. 2020; 27: 1154-1161https://doi.org/10.1016/j.acra.2020.05.021
        • Robbins JB
        • Shubeck SP
        • Kanters AE
        • et al.
        Lactation policy and resources for trainees in the department of radiology.
        J Am Coll Radiol. 2019; 16: 365-368https://doi.org/10.1016/j.jacr.2018.09.003
      1. Accreditation Council for Graduate Medical Education. Recommendations for Away Rotations and Interviews for Fellowship Applicants in 2020-2021. ACGME 2020. Available at: https://acgme.org/Newsroom/Newsroom-Details/ArticleID/10421/Recommendations-for-Away-Rotations-and-Interviews-for-Fellowship-Applicants-in-2020-2021. Accessed June 16, 2020.

      2. Association of Program Directors in Radiology. APDR/APDIR Position Statement Regarding Residency Recruitment in the 2021-2022 Cycle. APDR 2021. Available at: https://www.apdr.org/-/media/Files/APDR/New-and-Events/APDR_APDIR-Position-Statement-2021-22-Residency-Recruitment.ashx?la=en&hash=E066261A83096672666517EE5B485FDDDD6A3688. Accessed October 1, 2021.

        • Yee JM
        • Moran S
        • Chapman T.
        From beginning to end: a single radiology residency program's experience with web-based resident recruitment during COVID-19 and a review of the literature.
        Acad Radiol. 2021; 28: 1159-1168https://doi.org/10.1016/j.acra.2021.04.009
        • Hill MV
        • Ross EA
        • Crawford D
        • et al.
        Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic.
        Am J Surg. 2021; 222: 99-103https://doi.org/10.1016/j.amjsurg.2020.11.007
        • Donahue LM
        • Morgan HK
        • Peterson WJ
        • et al.
        The carbon footprint of residency interview travel.
        J Grad Med Educ. 2021; 13: 89-94https://doi.org/10.4300/JGME-D-20-00418.1
      3. Gallo K, Becker R, Borin J, et al. Virtual residency interviews reduce cost and carbon emissions. J Urol WoltersKluwer; 0 (0):10.1097/JU.0000000000002197. doi: 10.1097/JU.0000000000002197.

        • Czawlytko C
        • Smith E
        • Awan O
        • et al.
        The effect of virtual interviews and social media on applicant decision-making during the 2020-2021 resident match cycle.
        Acad Radiol. 2021; https://doi.org/10.1016/j.acra.2021.05.028
        • Novin S
        • Yi PH
        • Vanderplas T
        • et al.
        Integrated interventional radiology residency program websites: a development in progress.
        Am J Roentgenol. 2018; 211: 211-216https://doi.org/10.2214/AJR.17.19008
        • Hansberry DR
        • Bornstein J
        • Agarwal N
        • et al.
        An assessment of radiology residency program websites.
        J Am Coll Radiol. 2018; 15: 663-666https://doi.org/10.1016/j.jacr.2017.11.010
        • Wong TY
        • Huang JJ
        • Hoffmann JC
        • et al.
        Resident wellness in radiology as portrayed by departmental websites.
        Acad Radiol. 2021; https://doi.org/10.1016/j.acra.2021.07.016
        • Makary MS
        • Niedermeier M
        • Grewal BK
        • et al.
        Nationwide analysis of integrated and independent interventional radiology residency websites during the COVID-19 pandemic.
        Acad Radiol. 2021; 28: 1304-1312https://doi.org/10.1016/j.acra.2021.03.030
        • Chong A
        • Kagetsu NJ
        • Yen A
        • et al.
        Radiology residency preparedness and response to the COVID-19 pandemic.
        Acad Radiol. 2020; 27: 856-861https://doi.org/10.1016/j.acra.2020.04.001
        • England E
        • Kanfi A
        • Flink C
        • et al.
        Radiology residency program management in the COVID Era – strategy and reality.
        Acad Radiol. 2020; 27: 1140-1146https://doi.org/10.1016/j.acra.2020.05.001
        • Nguyen JK
        • Shah N
        • Heitkamp DE
        • et al.
        COVID-19 and the radiology match: a residency program's survival guide to the virtual interview season.
        Acad Radiol. 2020; 27: 1294-1297https://doi.org/10.1016/j.acra.2020.06.023
        • Wright AS.
        Virtual interviews for fellowship and residency applications are effective replacements for in-person interviews and should continue post-COVID.
        J Am Coll Surg. 2020; 231 (Elsevier): 678-680https://doi.org/10.1016/j.jamcollsurg.2020.09.005
        • Heitkamp NM
        • Snyder AN
        • Ramu A
        • et al.
        Lessons learned: applicant equity and the 2020-2021 virtual interview season.
        Acad Radiol. 2021; https://doi.org/10.1016/j.acra.2021.08.005
        • Fogel HA
        • Liskutin TE
        • Wu K
        • et al.
        The economic burden of residency interviews on applicants.
        Iowa Orthop J. 2018; 38: 9-15
        • Joshi A
        • Bloom DA
        • Spencer A
        • et al.
        Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond.
        Acad Radiol. 2020; 27: 1316-1322https://doi.org/10.1016/j.acra.2020.05.020
      4. National Resident Matching Program. Charting Outcomes in the Match: Senior Students of US DO Medical Schools. 2020. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2020/07/Charting-Outcomes-in-the-Match-2020_DO-Senior_final.pdf. Accessed September 30, 2021.

      5. National Resident Matching Program. Charting Outcomes in the Match: Senior Students of US MD Medical Schools. 2020. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2020/07/Charting-Outcomes-in-the-Match-2020_MD-Senior_final.pdf. Accessed September 30, 2021.

      6. United States Environmental Protection Agency. Greenhouse Gas Equivalencies Calculator. 2021. Available at: https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator. Accessed October 6, 2021.

        • Tanaka ME
        • Brideau HR
        • An TJ
        • et al.
        Utilization of a virtual information session to increase engagement with prospective applicants in the setting of COVID-19.
        Curr Probl Diagn Radiol. 2021; 50: 351-355https://doi.org/10.1067/j.cpradiol.2020.11.005
        • ACR Institute for Radiologic Pathology
        Four Week Radiologic Pathology Correlation Course.
        2021 (Available at:) (Accessed September 28, 2021)

      Linked Article