Academic Radiology
Volume 10, Issue 1, Supplement , Pages S44-S47, January 2003

Internal Program Review

  • Jannette Collins, MD, MEd, FCCP

      Affiliations

    • Corresponding Author InformationDepartment of Radiology, University of Wisconsin Hospital and Clinics, E3/311 Clinical Science Center, 600 Highland Ave, Madison, WI 53792-3242

Department of Radiology, University of Wisconsin Hospital and Clinics, E3/311 Clinical Science Center, 600 Highland Ave, Madison, WI 53792-3242, USA

Article Outline

 

An internal review of a residency program is a review conducted by the program's institutional sponsor to assess the program's educational effectiveness, determine its compliance with institutional and Accreditation Council for Graduate Medical Education (ACGME) requirements, and satisfy the ACGME institutional requirement for program review. The internal review may be considered a dress rehearsal for an ACGME site visit or external review. This article describes the preparation for and process of an internal review.

Back to Article Outline

ACGME Institutional Requirements 

The ACGME has published requirements for diagnostic radiology residency programs as well as requirements for institutions that sponsor accredited residency programs. It defines the sponsoring institution as “an organization having the primary purpose of providing educational and/or health care services (eg, a university, a medical school, a hospital, a school of public health, a health department, a public health agency, an organized health care delivery system, a medical examiner's office, a consortium, an educational foundation” (1, p 70).

Sponsoring institutions must have a graduate medical education committee (GMEC) responsible for monitoring and advising on all aspects of residency education. Voting membership on the committee must include residents nominated by their peers, appropriate program directors, other members of the faculty, and the accountable institutional official or his or her designee. The many responsibilities of the GMEC include regular internal reviews of all ACGME-accredited programs, including subspecialty programs (eg, fellowships), to assess their compliance with institutional and program requirements.

Internal reviews must be conducted by the GMEC or a body designated by it, which must include faculty, residents, and administrators from within the institution but from programs other than the one being reviewed. External reviewers also may be included on the review body as determined by the GMEC. The review must follow a written protocol approved by the GMEC. Reviews must be conducted at approximately the midpoint between ACGME external reviews. For example, if a program is anticipating an ACGME review in 5 years, the internal review should occur at approximately the -year point. While assessing the residency program's compliance with each of the program standards, the review should also appraise the following: (a) the educational objectives of each program; (b) the adequacy of available educational and financial resources to meet these objectives; (c) the effectiveness of each program in meeting its objectives; (d) its effectiveness in addressing citations from previous ACGME letters of accreditation and previous internal reviews; (e) the use of dependable measures to assess residents' competence in patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice; and (f) the effectiveness of each program in implementing a process that links educational outcomes with program improvement.

Materials and data to be used in the review process must include the ACGME institutional and program-specific requirements; letters of accreditation from previous ACGME reviews; reports from previous internal reviews of the program; and interviews with the program director, faculty, and residents in the program, and with those outside the program who are deemed appropriate by the committee. A written report of each internal review must be presented to and reviewed by the GMEC to identify areas of noncompliance and appropriate responsive action. In addition, this report or a succinct summary of it must be included in the ACGME institutional review document. Although departmental annual reports are often important sources of information about a residency program, they do not necessarily meet the requirement for a periodic review.

Back to Article Outline

Preparing the Program for the Internal Review 

Program staff should view the internal review as a “mock” external review and prepare for it just as they would for an external review. The best-prepared programs are those in which the staff do not procrastinate, and instead continually monitor themselves for compliance with the published ACGME requirements. The program director should read the published program requirements word for word several times and should frequently refer to them. Those who have not been doing so should start by reading the institutional and program requirements, reviewing prior internal and external review documents, and developing a list of program deficiencies and a plan to correct them.

All documents that may be requested by the external reviewer should be filed in a central location where they can be easily retrieved. These include the written curriculum, resident recruitment policy and guidelines statement, signed institutional affiliation agreements, a document outlining the lines of supervisory responsibility, resident call schedules, resident rotation schedules, conference schedules, conference sign-in sheets documenting resident and faculty attendance, resident evaluations, resident procedure logs, resident evaluations of faculty, resident evaluations of the program, departmental education committee meeting minutes (documenting the regular review of the program's objectives and the program's effectiveness in achieving them), a listing of resident publications, and the program's moonlighting policy. Depending on the institution, these documents may be requested by the internal reviewer.

The internal review of a well-prepared program should reveal no surprising deficiencies. The program director should prepare for the internal review as if he or she were an outsider performing a review. This requires extensive research into the program requirements and prior review documents and a meticulous investigation to determine whether the program is in compliance with all requirements. Program directors who are also members of the institution's GMEC and therefore perform internal reviews of other programs are better prepared for their program's internal and external reviews because they understand the importance of continuous monitoring of the program's compliance with ACGME requirements. They also recognize the advantages of being constantly prepared for a program review and avoiding a scramble to find or create documents at the last minute.

Many institutional GMECs use a template to prepare the internal review report. This template can be a helpful guide for the program staff preparing for a review. The program director can ask for a copy of this template if it is not provided. The template used at the University of Wisconsin, Madison, is shown in the Appendix to this article.

Back to Article Outline

Preparing Residents for the Internal Review 

The purpose of preparing the residents is to explain the process of accreditation and the importance of the internal review and make sure that residents can provide factual information about the program. Both internal and external reviewers will ask to meet with a group of residents to verify information they have been provided regarding the program. A program can do poorly in an internal review if residents provide negative information that is not factual. This is not necessarily the fault of the residents, as they may be unaware of certain aspects of the residency program. Junior residents in particular have not been exposed to some of the curriculum and evaluation activities and may assume they do not exist. They may not have experienced all levels of call responsibilities. Following are questions that residents might be asked by a program evaluator and should be prepared to answer:

1.Did you receive a copy of the program's educational goals and objectives?

2.What were the reasons why you chose this program?

3.Do you evaluate the teaching faculty? How often? Is the evaluation written and confidential?

4.Do you evaluate the program? How often? Is the evaluation written and confidential?

5.How often are you evaluated? Are you given copies of your written evaluations?

6.Do you meet with the program director at least twice a year to review your performance in the program?

7.How are you supervised? Is it too much, too little, or just right?

8.If you were experiencing difficulty in the program, professionally or personally, to whom would you go for help?

9.Explain the call system. Do you have at least 1 day in 7 free from clinical responsibilities? Are you assigned in-house call no more than once every third night, on average?

10.If you thought you were unfairly treated or disciplined, would you be able to describe the due process policy? Would you know where to obtain a written copy of that policy?

11.How were you selected to meet with me? Were you selected by your peers?

12.Do you have the opportunity to attend all required conferences?

13.Describe the educational experiences provided to train residents in patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice.

14.Do you keep a log of all procedures you perform?

15.What research training have you received during residency? Have you participated in research activities during your training?

Residents who are unhappy with aspects of the residency program and do not understand the purpose of the review process may make statements that damage the program or, in the case of an external review, result in its being placed on probation. This may not be the residents' intent, and discussing the review process frankly with residents before the review can prevent miscommunications. If residents are informed of the review process regularly, they will have an opportunity to express any concerns they have to the program director before the review.

Back to Article Outline

Appendix: Internal Review Report Template used by the University of Wisconsin Hospital and Clinics 

Note to Reviewer: The information requested for each section is meant to serve as a guideline to help in writing the report. These guidelines are not meant to be all-inclusive. Please read the ACGME requirements carefully and address all program deficiencies in the report.

Program Reviewed:

Date Report Presented to Graduate Medical Education Oversight Committee:

Internal Reviewer:

Materials Used:

Current list of program trainees

ACGME accreditation letters and follow-up correspondence (last two)

Internal review report and follow-up information and correspondence (most recent)

Internal review survey of training program completed by program director

ACGME program requirements

ACGME institutional requirements

Program's letters of agreement with affiliated hospitals

Completed ACGME program information form from last site visit\

Current ACGME program information form (blank)

House staff survey results and follow-up correspondence (most recent)

Other documents used (list)

Process: Describe the process used to review the program. Document materials used and interviews conducted. This report was based on a review of the materials listed above and interviews with Dr (name of program director), program director (date of interview), and residents (number per year) (date of interview).

Previous ACGME Citations and Current Status: The program was last reviewed by ACGME in (month/year) and received (accreditation status: accreditation, proposed probation, probation) on (date of accreditation letter). The approximate date of the next site visit is (date). List each citation (verbatim) from most recent ACGME accreditation letter. Describe how the program has addressed each citation.

Overview of the Program:

1.Resident recruitment and selection. Describe the length of the training program, number of residents approved by ACGME per year, number of residents recruited per year and the number currently enrolled. State whether a written departmental recruitment policy document exists. Discuss the recruitment and selection process, candidates interviewed and ranked in recent years, and results of recent matches (if applicable).

2.Faculty. State the number of faculty certified in the specialty compared with the requirement.

3.Program director. Discuss number of years the program director has been faculty in the specialty area and compare with requirements. Discuss time allotted to program director administration and compare with requirements.

Requirements: Comment on how the program is meeting both institutional and program-specific requirements in each of the following areas.

1.Didactic program. List the types and frequency of each type of conference required by ACGME. Discuss resident and faculty attendance at conferences and how documented. Discuss compliance with core curriculum requirements. Discuss compliance with curriculum requirements (goals and objective for each rotation or major assignment, distributed to faculty and residents and discussed with residents at the beginning of each rotation or major assignment). Describe the educational experiences provided to train residents in patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice.

2.Clinical experience. List all hospitals where clinical experiences occur and note whether there are current affiliation agreements. Comment on required clinical experiences and list any that are not adequately addressed in training program. Description should include procedures and how they are documented for each resident, inpatient and outpatient training requirements and patient population requirements.

3.Research. State the ACGME requirement for resident research and comment on program compliance. Comment on protected time for research, number of resident publications in recent years, and faculty participation in research.

4.Evaluations
A.Program evaluation of resident. State specific ACGME requirements for frequency of written and verbal evaluations and program's compliance. Comment on program policy for requiring residents to take an in-training examination and resident performance on this examination. List the evaluation tools used to assess resident competence in patient care, medical knowledge, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice. Comment on the program's use of dependable measures to assess competence in these six areas. Comment on program improvement through the use of educational outcomes.

B.Evaluation of program and faculty. State specific ACGME requirements for frequency of evaluation of faculty and program, confidentiality, and program compliance. If requirement includes a survey of program graduates, state requirement and how the program meets the requirement. Comment on method and frequency of faculty evaluation of achievement of program goals and objectives and how resident representation is assured. Discuss resident performance on specialty certifying examinations (ie, how many eligible, how many took examination, and how many passed in the past 5 years).


5.Supervision. State whether there is a written document outlining supervisory lines of responsibility.

6.Duty hours and work environment. Comment on ACGME duty hour requirements and program compliance. Describe resident call duties (ie, in house or from home, frequency, and duration).

7.Space, facilities, and resources. Comment on space available for clinical operations, resident offices and research laboratories and resident reimbursement for expenses (eg, books, travel to meetings, course registration).

Previous Internal Citations and Current Status: The last internal review of this program occurred (date). List citations and suggestions verbatim from the most recent internal review. Describe how the program has addressed each citation.

Conclusions and Recommendations: List each area of noncompliance and recommendations numerically. Include the recommendation for a follow-up report from the program director in 3 months for any citations.

Graduate Medical Education Oversight Committee Action: This section will be completed after the committee has reviewed the report.

Report Distribution:

1.Full report. Provide to program director, department chair, chief executive officer and president of institution, senior vice president for medical affairs, chief of staff of Veterans Affairs hospital (if appropriate).

2.Summary report. Provide to dean of medical school and medical board of institution.

Back to Article Outline

References 

    Reference
  1. Program director's reference guide to ACGME and RRCs. Available at: www.acgme.org/programDir/programDir.asp. Accessed July 17, 2001.

PII: S1076-6332(03)80149-6

doi:10.1016/S1076-6332(03)80149-6

Academic Radiology
Volume 10, Issue 1, Supplement , Pages S44-S47, January 2003

Article Tools