[The following notes were generated by Andrea Belovich, PhD.]

In response to the enthusiastic reception of the IAMSE Winter 2021 WAS series, “The USMLE Step 1 is Going to Pass/Fail, Now what do we do?,” an additional webinar was presented by Dr. Elise Lovell, Clinical Professor of the University of Illinois at Chicago and Co-Chair of the Coalition for Physician Accountability UME-GME Review Committee (UGRC) on February 25^th, 2021. During this sixth and final webinar, made available free-of-charge to registrants, Dr. Lovell provided a cutting-edge update of the UGRC’s work to review the transition from Undergraduate Medical Education (UME) to Graduate Medical Education (GME).

Dr. Lovell began by introducing the composition and mission of the Coalition for Physician Accountability. The Coalition is a collaboration between the national organizations responsible for the education, assessment, accreditation, certification, and licensure processes ranging from UME to practicing physicians. (A full list of member organizations is available on the Coalitions’ website [1].) As a collective, the Coalition’s mission is “to advance health care and promote professional accountability by improving the quality, efficiency, and continuity of the education, training and assessment of physicians” [2].

To provide the audience with context for the development of the Coalition’s UGRC, Dr. Lovell shared the four recommendations resulting from the Invitational Conference on USMLE Scoring (InCUS) in March 2019:

• Consider score reporting changes, e.g., pass/fail; composite score; categorical/tiered scoring for USMLE
• Research on how USMLE is (or is not) related to performance in residency and/or practice
• Continue work to address group (racial/demographic) differences in USMLE
• Undertake comprehensive overview of UME-GME transition system

Dr. Lovell emphasized that while the first recommendation regarding the USMLE Step 1 P/F scoring has received significant attention from the medical education community, the InCUS recognized that the USMLE Step 1 numerical score was merely one factor of many that contributes to the broader problems surrounding the UME-to-GME transition. After briefly discussing the progress being made towards meeting the first three recommendations, Dr. Lovell focused on the fourth InCUS recommendation: the review and overhaul of the current UME-GME transition system. This review process was anticipated to involve eight major areas of focus: 1) Improving the application process, 2) Reducing the number of applications submitted by applicants, 3) Improving transparency of the UME-GME transition process, 4) Identifying potential adjustments to the Match, 5) Improving Program Directors’ ability to more holistically review applicants, 6) Improving the trust/transparency of medical school-based assessments, 7) Reviewing the role of standardized testing in UME-GME transition, and 8) Developing assessments for other important measures beyond knowledge.

As the InCUS developed the recommendation to review the UME-to-GME transition, the Coalition for Physician Accountability was identified as the organization best suited to addressing the task. In September of 2019, the Coalition formed a Planning Committee (PC) to identify the construct and membership of the UME-GME Review Committee (UGRC), develop a process for selecting UGRC membership/leadership, and determine key questions, deliverables, milestones, timeline, and budget for the UGRC.

As a result of the PC’s efforts, membership of the UGRC is comprised of representatives from each of the Coalition member organizations, as well as medical students, resident physicians, members of the public, and support staff. Along with Dr. Lovell, Dr. George Mejicano was selected to co-chair the UGRC. Areas of focus for the UGRC were identified to include fit of applicants for residency programs based on abilities and interests, increase of trust between medical schools and residency programs, and mitigation of reliance on licensing exams during the application process. In addition, the UGRC’s focus encompasses ensuring fairness for all types of applicants (allopathic, osteopathic, and international medical graduates) while considering cost and the potential for systemic bias and diversity issues throughout the transition from UME to GME. The broader scope of the UGRC was also determined to include consideration of specialty-specific competencies, including when and how specialty-specific preparatory training for residency should occur, as well as where responsibility should rest in resolving any gaps in residency readiness.

Once established, the UGRC created four work groups to address the charges from the Coalition’s Planning Committee, and added their own overall considerations of diversity, equity, inclusion, fairness, student wellbeing, and the public good. The work groups were also asked to proactively anticipate outcome assessment of their recommendations and to plan for research for continual modification to support the formation of an iterative review process of the UME-GME transition going forward.

Work Group A’s focus is on ensuring residency readiness by identifying competencies students should be learning during UME and improving the advising process as students choose residencies for application. Work Group B’s charge is to improve the mechanics of the application/election process from the UME perspective, which includes information sharing by medical schools with residency programs, residency application content, and application mechanics (e.g., away rotations, etc.). Work Group C’s task is to improve the mechanics of the application/election process from the GME perspective, including information sharing by residency programs with medical schools, interview processes, and the match process with an emphasis on considering financial and opportunity costs to students. Work Group D is concerned with post-match optimization and improving the “soft transition” into residency during the fourth year of medical school. In particular, this group is addressing how to ensure that junior residents are prepared to meet specialty-specific competencies and how responsibility for any readiness gaps should be shared in the interest of patient safety and students’ professional development. This process is anticipated to involve information sharing/feedback from the residency program back to UME in order to more successfully educate and prepare future students.

Dr. Lovell then provided an overview of the UGRC’s process and work-to-date. Following up from a well-planned and intentional virtual brainstorming session held January 14^th and 15^th, 2021, the UGRC’s work groups have achieved/established the following: 1) Communication framework including bi-directional Stakeholder engagement; 2) Central Repository for reference literature; 3) Work Group level-setting (defined present status of UME-GME transition and data resources); 4) Root cause analysis of current problems; 5) Consideration of “what’s good/worth keeping” from current state; 6) Envisioned success, including measurement of outcomes; 7) Solicited Stakeholder visions of the transition’s future Ideal State.

Dr. Lovell then invited audience members to follow UGRC meeting updates on physicianaccountability.org, and provided an expected timeline for the delivery of the URGC’s recommendations, which are currently being built. These draft recommendations will be reviewed by the UGRC in March 2021, and made available for public review in April 2021. Final UGRC recommendations are expected to be delivered to the Coalition for Physician Accountability in May 2021.

The URGC is currently seeking UME-to-GME innovation input from stakeholders, which will be accepted through March 12^th, 2021 [3].

References:

1. Coalition for Physician Accountability. https://physicianaccountability.org/members/
2. Coalition for Physician Accountability. https://physicianaccountability.org/about/
3. “UME-to-GME Review Committee (UGRC) Seeks Input on “Innovations” being Used in UME-to-GME Transition.” Coalition for Physician Accountability. https://physicianaccountability.org/wp-content/uploads/2021/02/UGRC-Innovations-Invitation-1.pdf

[The following notes were generated by Andrea Belovich, PhD.]

The fifth session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on February 4th, 2021 by Drs. John (Jack) Boulet and Robert Cain. Prior to retiring in 2020, Dr. Boulet served as Vice President of Research and Data Resources for the Foundation for Advancement of International Medical Education and Research (FAIMER) and the Educational Commission for Foreign Medical Graduates (ECFMG). Dr. Cain is the President & CEO of the American Association of Colleges of Osteopathic Medicine (AACOM). In this webinar, “USMLE Step 1: Osteopathic and International Perspectives,” the speakers discussed how the USMLE Step 1 shift to pass/fail (P/F) is likely to impact International and Osteopathic medical training programs.

Dr. Boulet began by discussing recent changes to the USMLE Step exam series and how international medical graduates (IMGs) may be impacted. Nearly one-fourth of active physicians and residents in the United States are IMGs, and future IMGs applying to U.S. residency programs may be impacted by the changes to the USMLE Step Exam Series beyond the residency application and ranking process. In order for IMGs to be admitted to residency or fellowship programs accredited by the Accreditation Council for Graduate Medical Education (ACGME), they must first receive ECFMG Certification [1].

Dr. Boulet described the current ECFMG Certification requirements, which include examination in both medical science and clinical skills. Whereas medical science knowledge is assessed by the USMLE Step 1/Step 2 Clinical Knowledge (CK) exams, clinical skills have been assessed using the USMLE Step 2 Clinical Skills (CS) exam before its suspension in early 2020 and recent elimination in January 2021. According to Dr. Boulet, the ECFMG now allows applicants to demonstrate clinical skills through alternative avenues, including having an license to practice medicine in another country or having passed a standardized clinical skills exam for medical licensure. Other avenues include graduating from a medical school that 1) is accredited by an agency recognized by the World Federation for Medical Education, 2) participates in the U.S. Federal Student Loan Program, or 3) issues degree a jointly with U.S. medical school accredited by the Liaison Committee on Medical Education. Finally, without the communication requirements assessed by the USMLE Step 2 CS, IMGs must also pass the Occupational English Test in medicine to receive ECFMG Certification.

Dr. Boulet next discussed how the reporting of USMLE Step 1 scores as P/F is likely to affect IMGs during the residency application process. On average, IMGs have a ~60% match rate with U.S. residencies compared to a >90% for medical graduates trained in the United States. IMGs therefore tend to submit a great number of applications (~150), increasing competition for interviews. Dr. Boulet shared that since IMGs can no longer depend upon a strong numerical USMLE Step 1 scores to stand out from their peers, they must now reconsider how to distinguish themselves during the transition to Graduate Medical Education (GME).

Dr. Boulet suggested that the likely use of the numerically scored Step 2 CK exam to screen residency applications may disadvantage certain IMGs, depending on whether their school’s curriculum that provides early clinical experience or can be altered to do so. Schools may differ in this ability, depending on whether the school uses a 4-year curriculum (such as the Caribbean schools) or a 6-7 year curriculum. Schools with a small minority of graduates who apply to U.S. residencies may also not wish to restructure their curriculum, which could disadvantage their graduates seeking U.S. residency training. Given these concerns, IMGs’ motivation to secure U.S. residency programs may decrease.

In terms of the educational impact of the USMLE Step 1 P/F transition, Dr. Boulet anticipated that students may be less likely to neglecting preclinical coursework in favor of Step 1 preparation. However, the decreased pressure may reduce the perceived value of the basic sciences, which may result in reduced motivation for students to perform as well in the basic sciences. Overall, the lack of pressure to score highly on Step 1 may allow time for a more balanced education, which is expected to positively affect wellness and reduce burnout.

Nevertheless, the question remains in regard to how IMGs will distinguish themselves during the Undergraduate Medical Education (UME) to GME transition. In addition to the increased reliance on Step 2 CK numeric scores, U.S. residency program directors may rely on the international reputation of medical schools, including whether they are ECFMG accredited. Dr. Boulet concluded by emphasizing the need to reevaluate the residency selection process, and highlighted efforts by institutions such as the Coalition for Physician Accountability, the ECFMG, and the National Board of Osteopathic Medical Examiners (NBOME) to make the UME to GME transition more efficient. These efforts include exploring assessment innovations to allow ranking across multiple dimensions needed for success in residency, creating structured holistic reviews of credential portfolios, and developing more holistic metrics such as a “Resident Readiness Index.”

Dr. Cain continued the webinar by sharing the overall organizational impact of the USMLE Step 1 transition to P/F on the osteopathic medicine community, despite the community having its own independent licensing body, the NBOME. Dr. Cain emphasized that osteopathic students and graduates are becoming an increasingly large part of the physician workforce, with 25% of medical students graduating with a Doctor of Osteopathy (DO) degree. Approximately 20% of medical schools offer the DO degree, an 20% of those schools are located in rural and underserved regions. Altogether, this underscores the need for DO graduates to have access to high quality residency training, which is becoming more competitive due to increased volume of applications, including those from IMG and MD graduates. Dr. Cain also expressed concerns that residency programs who have previously accepted DO students may be less likely to do so as a result of the USMLE Step 1 P/F transition, which may result in a loss of DO students in leadership and other prominent positions.

The impact of many residency programs requiring DO students to take the USMLE Step 1 to be considered is associated with significant costs. For students, this cost is felt financially, personally, and relationally, while schools feel the impact of the USMLE Step 1 requirement on their preclinical curricula. These costs raise an ethical question regarding the fairness of requiring students to take two exams, particularly when considering the additional opportunity cost: during the time students must prepare to this extra exam, they must forgo other extracurricular or leadership opportunities.

Dr. Cain then discussed whether it is therefore a benefit or a detriment to require DO students to take the USMLE Step 1 exam, particularly after the transition to P/F occurs. The requirement serves to benefit DO students when a numerical score is provided, as it allows students (particularly high-performing individuals) to distinguish themselves. However, does that benefit change when the Step 1 exam is P/F, if both the COMLEX Level 1 and the USMLE Step 1 test the same material and the COMLEX is also acceptable for medical licensure?

To explore this question, Dr. Cain discussed the advantages and disadvantages of associated with the USMLE Step 1 scoring system transition to P/F. Advantages include decreased stress and cost for students, and schools will experience less pressure to “teach to the test.” Furthermore, the reduced pressure to score highly on an assessment designed for allopathic medical education aligns with and supports the development of the osteopathic professional identity. However, disadvantages include concerns regarding reduced transparency surrounding the residency application screening process due to the loss of the numeric Step 1 score and whether program directors will rely upon school reputation or their own familiarity with the individual DO programs. Finally, a greater proportion of DO students pursue clinical clerkship training in non-hospital based settings than MD students, such as ambulatory and physical rehabilitation training, which disadvantages them from scoring as highly on the USMLE Step 2 CK exam.

Dr. Cain next explained the process for DO licensure, underscoring that DO students may achieve licensure either through the NBOME’s COMLEX exams or through the National Board of Medical Examiners’ (NBME) USMLE exams. While DO students are required to pass the COMLEX exams to graduate and most commonly use this pathway to achieve licensure, they are not required to take both COMLEX and USMLE exams. For DO students, this raises the concern that the USMLE exams have been misused in regards to their primary purpose to inform licensure when used as a tool for residency placement, particularly when there are differences in performance compared to allopathic students.

Dr. Cain also provided a more in-depth look at the purpose of the NBOME’s COMLEX exam series, and explained importance to osteopathic medical education. In addition to the biomedical, clinical, and health systems sciences integral to allopathic medical education, a significant component of the osteopathic medical curriculum includes over 200 hours of biomechanical science, complexity science, and osteopathic philosophy. All of these aspects need to be assessed in order to determine knowledge, skill, and behavior of students as they progress through their training. The COMLEX exams are designed to test the osteopathic-specific components at each stage of development, in addition to the biomedical, clinical, and health systems sciences. In all, the COMLEX is not designed as an add-on exam, but is rather embedded throughout the four-year experience.

To conclude the webinar, Dr. Cain emphasized that work is currently being done by the Coalition for Physician Accountability to review the transition from UME to GME, including the development of a more holistic approach to residency recruitment and interviewing. To summarize the general sentiment of the osteopathic community, Dr. Cain stated that the change of the USMLE Step 1 scoring to P/F could have a negative impact on DO students, when there really should be no impact. This highlights a great need for continuing education around the use of the COMLEX exam and understanding the tools for determining residency readiness.

References:

1. About ECFMG Certification. Educational Commission for Foreign Medical Graduates. https://www.ecfmg.org/certification/