Evaluation of Residents, Faculty, and Program
Article Outline
Evaluation of Residents
In the evaluation of residents, attention should be directed toward the purpose of the evaluation, the content to be assessed, the evaluators, the residents being evaluated, the settings in which the evaluations take place, the timing of the evaluations, the methods used to collect and summarize the information about each resident, and the management of information by the program. Problems in resident evaluation systems used by local internal medicine residency programs were identified by the American Board of Internal Medicine Hospital Visit Program through visits to training programs. The most common problems cited by program directors, attending faculty, and residents are outlined in Figure 1 (1, pp 63–64). Solutions to these problems include counseling of evaluators; improving communication among all involved regarding performance, criteria, and expectations; and developing an early warning system for residents with problems. Poor feedback to residents was cited as a problem by all three groups and will be addressed later in this article.
Figure 1.
Most common problems cited by program directors, faculty members, and residents regarding evaluations of internal medicine residents (1, pp 63–64).
Rotational and Biannual Evaluations
The following are the Accreditation Council for Graduate Medical Education (ACGME) requirements regarding resident evaluation (2):
The program director is responsible for regular evaluation of residents' knowledge, skill, and overall performance, including the development of professional attitudes consistent with being a physician. Evaluations of each resident's progress and competence should be conducted preferably at the end of each rotation, but not less than four times yearly. The evaluation must concern itself with intellectual abilities, attitudes and interpersonal skills, and clinical and technical competence. There must be provision for appropriate and timely feedback of the content of all evaluations to the resident. The program director or the program director's designee must meet with all the residents at least semiannually to discuss these evaluations and provide feedback on performance. More frequent reviews of performance for residents experiencing difficulties or receiving unfavorable evaluations are required. When a resident fails to progress satisfactorily, a written plan identifying the problems and addressing how they can be corrected must be placed in his or her individual file. Residents should be advanced to positions of higher responsibility only on the basis of their satisfactory progressive scholarship and professional growth. The program must maintain a permanent record of the evaluation and counseling process for each resident. Such records must be accessible to the resident and other authorized personnel. The program must demonstrate that it has developed an effective plan for assessing resident performance throughout the program and for utilizing evaluation results to improve resident performance. This plan should include use of dependable measures to assess residents' competence in patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. There must be a written final evaluation for each resident who completes the program. The evaluation must include a review of the resident's performance during the final period of training and should verify that the resident has demonstrated sufficient professional ability to practice competently and independently. The final evaluation should be part of the resident's permanent record maintained by the institution.
The ACGME does not have specific requirements regarding the conduct of rotational evaluations. Evaluation forms are commonly used to rate the various aspects of resident performance after the resident has completed a rotation in a primary area (eg, chest, nuclear medicine). Depending on the practice of a particular program, the section chief for that area may have primary responsibility for completing the form, may delegate the responsibility to another member of the section, or may solicit written or verbal input from members of the section. Input from as many faculty persons who worked with the resident as possible allows for the most informative evaluation. The number of evaluation items on the form is not as important as the thought processes that go into the evaluation and specific comments that supplement numeric ratings. In fact, research has generally supported a single, summative rating as the most reliable and valid measure of performance (3). The ACGME, however, requires that the evaluation address intellectual abilities, attitudes and character skills, and clinical and technical competence—aspects of performance that are not easily quantified. Written, objective comments—although not currently required by the ACGME—should be mandatory for any ratings of “outstanding,” “marginal,” or “unsatisfactory.” These comments should justify the rating, provide feedback to residents who are excelling or who need specific guidance to improve weaknesses in their performance, and provide information to be included in summative performance evaluations.
It has been shown at one academic institution that in the 2nd, 3rd, and 4th years of radiology residency, there is a positive correlation between rotation evaluation scores and overall scores from the corresponding American College of Radiology (ACR) in-training examination and written portion of the American Board of Radiology (ABR) examination taken during the same year (4). In contrast, in the 1st year of residency, resident rotation evaluation scores do not correlate with ACR in-training examination scores. Thus, residents who are perceived as doing well on their rotations after the 1st year of residency are more likely to do well on standardized written examinations.
For an evaluation system to be effective, it must be used in the educational program. The program director should read and sign every resident evaluation form before it is placed in the resident's permanent record. Any unsatisfactory ratings should be discussed with the resident. Documentation of the content of such discussions will be useful should continued poor performance necessitate restrictions on resident responsibilities, probation, or dismissal from the program. In an ideal world, evaluations would be discussed with all residents, not just those having difficulties. Since this is often not possible, the evaluation forms should at least be available for resident review, and residents should be encouraged to review their evaluations regularly. Copies of the signed reviews should be distributed to residents within 1–3 months after they finish a rotation.
Two phenomena can interfere with the assessment of resident performance: the halo effect and the compensation fallacy (1, p 65). The halo effect accounts for the tendency of the evaluator to rate a resident uniformly high or low on all components based on a general favorable or unfavorable impression. Compensation fallacy is the tendency to exaggerate particular perceived strengths of a resident in the face of clear deficiencies in other areas, so as to create a more favorable overall impression. Once residents have been labeled as outstanding, the halo effect often leads to reluctance to reclassify them. Residents with serious deficiencies may also have major strengths (eg, the personable resident who cannot be trusted, the compassionate resident who cannot apply his knowledge, the bright but unmotivated resident). These residents must not be judged as satisfactory overall, but they often are, due to compensation fallacy.
One must also carefully distinguish residents who have serious problems only during a certain period of time from those whose problems, such as substance abuse or psychiatric illness, continue regardless of rotation schedules or counseling or whose problems migrate from one component of competence to another over time so that overall performance is never quite satisfactory. In the first instance, the resident may be coping with normal life crises (eg, death, divorce, or illness) and temporarily functioning below par. Inappropriate negative evaluations may occur because of personality conflicts, internal program politics, and unfortunate single critical incidents that stigmatize a resident. Such situations should be assessed carefully before residents are judged.
Monthly global evaluations of residents by teaching faculty can be supplemented with objective tests of resident performance. A written monthly test can be administered via a series of computer programs that generate unique monthly examinations from a large question file (5). Combination written and image interpretation examinations can be administered to residents as an educational tool and a means of informally credentialing residents in emergency radiology, before they begin night and weekend emergency room duty (6).
The items included on a rotation evaluation form can include knowledge appropriate for the resident's level (including “book knowledge,” the ability to analyze images, the quality of resident read-out sessions, skill at performing procedures, and care of patients); attitudes regarding work assignments, patients, and other members of the health care team; respect for hospital policies; and attendance (7). Other potential items include willingness to make decisions and accept responsibility; willingness to evaluate one's own performance periodically and plan an improvement program; skill in the use of equipment and knowledge of technique; skill in supervising others; skill in teaching residents, students, and technical staff; and ability to conduct research (8).
Biannual meetings between the program director and each resident are required by the ACGME. The ACGME, however, does not specifically state how the meetings are to be conducted; it describes the process only in general terms. This allows flexibility on the part of programs as to how long the resident meetings should be, over what period of time they should occur, and what issues should be discussed. A formal document describing the resident's performance during the past 6 months and the content of the discussion between the program director and resident should be part of the resident's permanent record (9). Ideally, both the program director and the resident should sign the document. This process should occur more frequently for residents who are performing at a marginal or unsatisfactory level.
These biannual meetings provide an opportunity to ensure that critical information is disseminated to all residents. Ideally, program directors interact regularly with all residents, both as a group and individually. In reality, this interaction may be sporadic given resident absences and faculty scheduling. In addition to discussing the resident's rotational evaluations, the program director can counsel the resident on other performance- and non–performance-related issues. For example, the biannual meetings can be a time to discuss postgraduate fellowship or job placement plans and answer questions regarding that decision. From both reading and personal experience, the program director has knowledge regarding the status of the radiology job market, the choice between academic radiology and private practice, and hiring considerations of radiology groups, and this knowledge can be shared with residents (10). Other issues that can be addressed include scheduling of the physics and diagnostic portions of the written ABR examination, resident performance on the radiological physics examination (RAPHEX) and the ACR in-service examination, resident progress in fulfilling rotation requirements, and review of the resident's procedure log (an ACGME requirement). During the performance meetings, residents can address the strengths and weaknesses of the training program.
To help prevent legal problems, careful documentation is important. For residents with deficiencies, negative reviews should be addressed early, as delay can imply approval. Inconsistent positive and negative evaluations should be resolved and a judgment reached. Residents should be informed early of judgments and expectations of their performance, and the written record should document that they have been informed. Expectations should be enforced, and the results should be recorded in writing. The program director should know and use institutional procedures for academic due process. Giving a resident due process before an important adverse action virtually precludes a successful suit by the resident. Program directors should obtain legal counsel before initiating drug testing, psychiatric evaluation, or a major investigation.
RAPHEX
RAPHEX is published in cooperation with the Radiological and Medical Physics Society of New York (11). A written examination for residents in diagnostic radiology and radiation oncology, it is used as a practice examination or for self-study and can also be used to evaluate the effectiveness of residency program physics courses. It has three sections—general, diagnostic, and therapeutic—with approximately 100 questions in each section. A separate answer booklet is provided, which gives short explanations of why each answer is correct along with worked calculations when appropriate.
RAPHEX can be used for self-assessment of radiologic physics knowledge before taking the board examination, or it can be given by a proctor as a practice examination. When it is given as a practice examination, instructions and computerized answer sheets are sent to the proctor along with the examination and answer booklets. Diagnostic radiology residents are given 3 hours to complete the general and diagnostic sections. After the examination, the proctor sends the completed answer sheets to the Radiological and Medical Physics Society for grading. The answer sheets are generally due in mid-July. Each score and percentile rank is sent back to the proctor after the answer sheets are graded. Physics instructors are urged to review the examination with residents. Copies of present and past examinations can be purchased from Medical Physics Publishing (www.medicalphysics.org).
ACR In-Training Examination
The purpose of the ACR in-training examination is to provide residents with information that is useful to them in evaluating their own progress and to provide program directors with data that is helpful in analyzing and evaluating the program (12). The examination is intended to be a measure of general achievement for residents and program directors in diagnostic radiology. It should not be used as the only measure of an examinee's performance for qualification to any postgraduate program or for certification. All scores are strictly confidential and are reported only to program directors, who must then report them promptly to individual residents. Individual residents' scores should be discussed only between the resident and the program director and should not be used in any departmental discussions or meetings.
The 4-hour examination is generally administered on the second Thursday in February, and all participating residents must take the examination on the same day, unless prior arrangements have been made through the ACR. For example, the examination can be taken at the Armed Forces Institute of Pathology, but only with advance scheduling. At the local institution, the examination cannot be taken earlier or later than the scheduled date. Virtually all accredited programs use the ACR examination. In 1993, 3,447 residents in 238 programs participated (13). Residents are allowed to keep their copy of the examination after completing it, as more than 95% of the questions are newly generated each year.
There is variation among residency programs in terms of which classes take the examination. Some programs test all residents, while others test only certain classes. All residents take the same examination, regardless of their year in training. Program directors are provided with each resident's score and percentile score relative to residents with similar training.
A study comparing subjective rankings of radiology residents at two academic institutions with these residents' scores on the ACR in-training examination showed that faculty and residents were only moderately accurate in their subjective ranking of resident performances on the ACR in-training examination (14). A resident's average ACR in-training examination score, however, has been shown to be a strong predictor of the ABR written board examination score (15). Residents with low in-training examination scores (< 20th percentile) are at risk for poor performance on the written board examination (< 25th percentile) and may benefit from remedial training. This relationship has been shown both locally and across several institutions (16).
ABR Examinations
The ABR administers separate written and oral examinations (17). The written examination includes physics and diagnostic portions, which are separately administered at several sites, generally in early fall. The complete written examination (both physics and diagnostic components) consists of approximately 240 multiple-choice questions. About half of the questions (those on physics) are distributed in the afternoon of the 1st day, and the other half (those on diagnostic radiology) are distributed the following morning. New written examinations are formulated each year in all categories of diagnostic radiology, and the content of the examinations is carefully evaluated to keep current with new information and developments. Residents may apply to take the physics portion of the written examination after 12 months of diagnostic radiology training and the clinical portion after 24 months. Program directors must verify the months of training completed by applicants. Before the oral examination, program directors must also attest to the resident's professional qualifications.
A candidate has three consecutive opportunities to appear for and pass the written examination, beginning with the written examination to which he or she is first declared admissible. If a candidate fails to appear or to pass, he or she must submit a new application and the fee in effect at that time. Candidates must pass both portions of the written examination for admission to the oral examination.
Candidates who have passed the written examination and who will have completed 60 months of approved training by September 30 in the year in which the examination is given are considered for the oral examination. Beginning in 2002, residents completing 5 years of training are issued a 10-year time-limited certificate in diagnostic radiology on successful completion of the oral examination. The oral examination consists of image interpretation and problems of clinical application, including patient care. Electronic display of images is used in all 10 subject categories (musculoskeletal, cardiopulmonary, gastrointestinal, genitourinary, neurologic, vascular and interventional, nuclear, pediatric, and breast radiology, and ultrasound). From the more than 1,000 cases in each category that are stored in an electronic database, 10–15 cases are selected for each of the eight sessions during the 4 days of the examination (18). The databases are continually updated. Candidates who fail (ie, condition) up to three sections of the oral examination can take those sections over in an attempt to pass. Candidates who fail four or more sections must retake the whole examination (all 10 sections). The fee schedule and the schedule of written and oral examinations are available at the ABR Web site (www.theabr.org).
Criteria for Admission to the ABR Examinations
The ABR stipulates that within the required period(s) of residency training, total leaves of absence and vacation may not exceed 6 calendar weeks (30 working days) for residents in a program for 1 year, 12 calendar weeks (60 working days) for residents in a program for 2 years, 18 calendar weeks (90 working days) for residents in a program for 3 years, or 24 calendar weeks (120 working days) for residents in a program for 4 years. If a longer leave of absence is granted, the required period of residency must be extended accordingly. Candidates beginning their graduate medical education after January 1, 1997, are required to have 5 years of approved training with a minimum of 4 years in diagnostic radiology. A minimum of 6 months, but no more than 12 months, must be spent in nuclear radiology in the 4-year diagnostic training program. Candidates may spend no more than 6 months in rotations outside the parent institution. In a 4-year approved diagnostic radiology residency program, not more than 12 months of the required 48 months of training may be spent in a single discipline (research is considered a discipline).
ACGME General Competencies
The radiology residency program “must demonstrate that it has developed an effective plan for assessing resident performance throughout the program and for utilizing evaluation results to improve resident performance. This plan should include use of dependable measures to assess residents' competence in patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice” (2). The Association of Program Directors in Radiology (APDR) Education Committee published a consensus document (19) that defines each of the six competencies, identifies related resident skills and education, and describes methods of assessment. A resident evaluation form addressing these six competencies was developed by the APDR Education Committee (Fig 2) (20). This document can be downloaded from the APDR Web site (www.apdr.org).
Figure 2.
Resident evaluation form.
Feedback
In the setting of graduate medical education, feedback is information that describes residents' performance in a given activity and is intended to guide their future performance in that or a related activity (21). Feedback and evaluation are often used interchangeably, yet they describe distinct and separate processes. Feedback is immediate. Its purpose is descriptive and formative (ie, to improve performance and reinforce appropriate behavior), and it delivers information. It allows for rapid correction of mistakes. Evaluation is usually delayed (occurring at the end of a rotation or course), its purpose is summative, and it is judgmental (22). By providing feedback, faculty members collaborate with the learner to improve areas of weakness and to identify and augment strengths. Both formal and informal feedback mechanisms should exist in training programs. Twelve basic principles can be used to guide the use of feedback for clinical teaching:
Evaluation of Faculty and the Program
The ACGME requirements (2) are as follows:
The program must provide the opportunity for residents to provide written confidential evaluation of the faculty and the program at least annually. Each faculty member must review his or her evaluations. The educational effectiveness of a program must be evaluated in a systematic manner. In particular, the quality of the curriculum and the extent to which the educational goals have been met by residents must be assessed. Anonymous written evaluations by residents should be utilized in this process. The teaching faculty must be organized and have regular documented meetings to review program goals and objectives as well as program effectiveness in achieving them. At least one resident representative should participate in these reviews. The faculty should evaluate at least annually the utilization of the resources available to the program, the contribution of each institution participating in the program, the financial and administrative support of the program, the volume and variety of patients available to the program for educational purposes, the performance of members of the teaching faculty, and the quality of supervision of residents.
Evaluation of Faculty
As stated above, the ACGME requires a confidential evaluation of faculty members by residents at least once a year. The ACGME, however, does not prevent programs from conducting more frequent faculty evaluations. If the goal of a program is to monitor and improve the quality of residency training continually, faculty members should be evaluated with this goal in mind. Residents should be counseled regarding the purpose of faculty evaluations and how the information will be used. The purpose is faculty development, first and foremost. In most academic institutions, evaluations by residents are a required part of a promotion packet for teaching faculty. Thus, residents should not use the evaluation process as an opportunity to vent frustrations with individual faculty members or the program. Obtaining faculty evaluations more frequently than once a year provides more data points, which not only makes the evaluation process more robust but also allows for timelier and thus more effective feedback to faculty about their teaching strengths and weaknesses. Studies have shown that an evaluation system that solicits specific comments from residents on faculty strengths and weaknesses and includes a nonconfrontational forum for presenting results to faculty members can effect improvements in perceived faculty teaching performance, especially among the faculty who are rated below average (23, 24, 25).
Implementation of a 30-item behaviorally targeted radiology faculty appraisal instrument at two academic centers showed that particular instrument to be valid and reliable (26). This form allows ratings of behaviors that residents believe differentiate effective instructors from ineffective ones, enabling an objective and relevant assessment. The items are related to faculty productivity (“assists residents with workload”), teaching (“allows residents to present a case, takes time to teach during read-out and explain diagnoses”), attitude (“is understanding of others”), accessibility (“is available in the reading room”), interpersonal skills (“personalizes relationships with residents”), feedback (“evaluates residents' performance”), expertise or clinical skills (“demonstrates knowledge to others, makes decisions that require clinical judgment”), research (“encourages resident research”), professionalism (“relates respectfully to colleagues”), and general comments.
Student ratings of clinical teaching have been found highly consistent and as reliable as those reported for classroom settings (27). Under appropriate conditions, students' evaluations of teaching are (a) multidimensional, (b) reliable and stable, (c) primarily a function of the instructor who teaches a course rather than the course that is taught, (d) relatively valid against a variety of indicators of effective teaching, (e) relatively unaffected by a variety of variables hypothesized as potential biases (eg, grading leniency, class size, workload, prior subject interest), and (f) useful in improving teaching effectiveness when coupled with appropriate consultations (28). Teaching effectiveness, however, is best judged by a variety of evaluators. In residency programs, evaluators could include residents, medical students, other health care students, the teachers and their colleagues, administrators, and trained observers. Both current and former students and residents can evaluate faculty. A multidimensional system of evaluation should ease faculty concerns that residents' evaluations alone could negatively influence decisions regarding faculty career advancement.
Evaluation of the Program
One measure by which the ACGME evaluates the quality of a residency training program is the performance of graduates on the ABR examination. At least 50% of the graduates from a program in the 5 years preceding program evaluation should have passed the written and oral examinations without condition at their first attempt (2). Another method that can be used to measure a program's quality is quantitative profiling based on computerized storage and retrieval of radiologic reports. This technique can be used to track the range and progress of resident experience, help determine the deployment of residents, and provide empirical data to guide modifications of residency programs (29).
All aspects of the training program should be evaluated cyclically, including the call system, rotational requirements, didactic conferences (including physics instruction), faculty and resident evaluation, resident research participation, the curriculum, the quality of rotations at all of the participating institutions, library facilities, preparation for ABR examinations, chief resident selection and performance, resident recruitment, responsiveness of the program director, resident supervision, call room facilities, the volume and variety of patients and radiologic studies, and procedural training. Residents should be given the opportunity to comment on all of these factors in a written survey. The data from these surveys should be reviewed by a committee of the teaching faculty, which should include at least one resident representative. The program staff should be able to demonstrate that the data are being used to guide changes in the program.
References
- References
- Systems approach to evaluation of residents. In: Henderson M editors. Association of Program Directors in Internal Medicine program manual. 7th. Washington, DC: Association of Program Directors in Internal Medicine; 2000;
- Appendix 1: ACGME program requirements for residency education in diagnostic radiology. Acad Radiol. 2003;10(suppl 1):S102–S108
- . Evaluating teaching effectiveness: a practical guide. In: Beverly Hills, Calif: Sage; 1984;p. 46
- . Correlation between radiology resident rotation performance and examination scores. Acad Radiol. 2000;7:920–926
- . Computer-generated examinations for residents in radiology. AJR Am J Roentgenol. 1986;146:413–414
- . In-house testing in emergency radiology: a method for teaching and credentialing radiology residents. Invest Radiol. 1991;26:836–839
- . Residency evaluations. Acad Radiol. 1996;3:170–173
- . Resident evaluation: is our method effective?. Can Assoc Radiol J. 1976;27:122–125
- . Radiology resident performance reviews. Acad Radiol. 2000;7:350–352
- . Status of the radiology job market, choosing the right career in radiology, and getting the right job: what does the resident need to know?. Acad Radiol. 1998;5:572–577
- RAPHEX 2002. Available at:www.medicalphysics.org. Accessed May 22, 2002.
- American College of Radiology. 2002 diagnostic in-training exam answer module. Available at:www.acr.org/departments/educ/intrain/intro.html. Accessed May 22, 2002.
- . A survey of the American College of Radiology diagnostic radiology in-training examination. Invest Radiol. 1994;29:726–728
- . Assessment of resident knowledge: subjective assessment versus performance on the ACR in-training examination. Acad Radiol. 1999;6:66–71
- . Relationship between American College of Radiology in-training examination scores and American Board of Radiology written examination scores. Acad Radiol. 1996;3:873–878
- . Relationship between American College of Radiology in-training examination scores and American Board of Radiology written examination scores. II. Multi-institutional study. Acad Radiol. 1998;5:374–379
- American Board of Radiology. Booklet of information for diagnostic radiology, 2001–2002. Tucson, Ariz: American Board of Radiology.
- . Current structure and purpose of the American Board of Radiology examination process. Acad Radiol. 2001;8:1260–1261
- General competencies in radiology residency training: definitions, skills, education and assessment. Acad Radiol. 2002;9:721–726
- Radiology resident evaluation: a form that addresses the six competencies of the Accreditation Council for Graduate Medical Education. Acad Radiol. 2002;9:815–816
- . Feedback in clinical medical education. JAMA. 1983;250:777–781
- . Feedback in residency training. In: Henderson M editors. Association of Program Directors in Internal Medicine program manual. 7th. Washington, DC: Association of Program Directors in Internal Medicine; 2000;p. 79–82
- . Improvement of faculty teaching performance: efficacy of resident evaluations. Acad Radiol. 1996;3:63–67
- . Relationship between systematic feedback to faculty and ratings of clinical teaching. Acad Med. 1996;71:1100–1102
- . Teaching effectiveness in the radiology residency: a method of evaluation by residents. Invest Radiol. 1990;25:287–291
- . Radiology faculty evaluation: bi-institutional implementation of an experimental appraisal instrument. Acad Radiol. 1998;5:804–809
- . Evaluating clinical teaching in medicine. J Med Educ. 1981;56:181–186
- . Making students' evaluations of teaching effectiveness effective: the critical issues of validity, bias, and utility. Am Psychol. 1997;52:1187–1197
- . Evaluation of clinical experience in a radiology residency program with quantitative profiling: rationale, methods, and application. Acad Radiol. 1999;6:102–111
PII: S1076-6332(03)80148-4
doi:10.1016/S1076-6332(03)80148-4
© 2003 Acad Radiol. Published by Elsevier Inc. All rights reserved.
