[The following notes were generated by Andrea Belovich, PhD.]
The third session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 21st, 2021 by Dr. Jonathan Amiel, Interim Co-Vice Dean for Education, Senior Associate Dean for Curricular Affairs, and Associate Professor of Psychiatry at the Columbia University Vagelos College of Physicians and Surgeons. In this webinar, “USMLE Step 1 P/F: A UME Curriculum Dean’s Perspective,” Dr. Amiel discussed how the transition of the USMLE Step 1 to pass/fail (P/F) is anticipated to impact undergraduate medical education (UME) curriculum.
Dr. Amiel began with an overview of the USMLE Step exam licensure program, outlining the recommendations of the 2019 Invitational Conference on USMLE Scoring (InCUS) that culminated in the decision to change the USMLE Step 1 from a numerical scoring system to P/F after January 2022. He then discussed the role of a Curriculum Dean in UME, and how, from this perspective, the Step 1 P/F shift would likely impact UME curricula. Quoting Sir Isaac Newton, “And to every action there is always an equal and opposite or contrary reaction,” this portion of the webinar was framed as an exploration of both the changes that the Step 1 P/F shift is likely to introduce to UME, and the potential implications of UME’s responses to those changes.
He explained that a curriculum dean’s role is to coordinate between several departments to create a coherent and cohesive educational program. He listed major factors that could be impacted as UME adjusts to the Step 1 P/F transition: 1) Educational program mission, objectives, values, and ethics, 2) Accreditation, 3) Logistics, 4) Curriculum architecture and engineering, 5) Faculty development and 6) Resource management.
In terms of the mission, objective, values, and ethics of educational programs and accreditation, Dr. Amiel expressed optimism that pre-clerkship faculty may be able to take a more holistic approach to the knowledge and learning skills students acquire in pre-clerkship curricula. Rather than “teaching to the test,” institutions may utilize Step 1’s shift to P/F as an opportunity to work with the USMLE program and the NBME to approach Step 1 from a competency-based perspective. Dr. Amiel emphasized shifting the exam’s focus towards assessing what students need to know to be competent in clinical settings. Dr. Amiel stated that he did not anticipate a major impact in the accreditation arena, with the caveat that the NBME may decide to increase the passing threshold for Step 1. If implemented, this increased threshold could introduce a higher fail rate that may affect accreditation.
Whether or not the Step 1 passing threshold is raised, curriculum deans must consider how the P/F shift will affect the logistics of preparing students for standardized exams and how student support services may be impacted. Learners who have difficulty with standardized tests will still require support for licensure exams, and institutions need to ensure that they are still providing this support. This includes greater support for the USMLE Step 2 CK exam, which may now grow in importance for residency application ranking, as it will still be scored numerically. Additionally, a thoughtful approach to revising assessment methods is needed. For example, NBME subject exams are often currently used to prepare students for USMLE, but Dr. Amiel urged participants to consider other assessment methods to help prepare students for passing the Step 1 exam, including utilizing more open-ended questions and applied-knowledge assessments.
Next, Dr. Amiel discussed how the architecture of UME curricula may be impacted by the Step 1 P/F shift. He predicted that the most significant architectural impact would be on dedicated study time for the Step exams. Schools may need to dedicate more time to Step 2 CK preparation, which will affect the senior curriculum. While it may be tempting to reduce the dedicated study time for Step 1 to create space for Step 2 CK preparation, students who experience difficulty with standardized testing may be put at greater risk of failing Step 1. Elective rotations may also be reduced, but these are important for residency interviews and gaining clinical experience. Ultimately, these issues may result in individualized study time/tracks for students.
With the potential changes to assessment methods, Dr. Amiel also anticipated that faculty development programs will need to provide new training to faculty. He emphasized this as an opportunity to reframe “assessment of learning” as “assessment for learning,” and to help faculty broaden their thinking about how to test medical knowledge early in the curriculum. He then discussed the potential impact of the Step 1 P/F shift on resources and resource management. As institutions have learned during the COVID-19 pandemic era, resources need to be used wisely, and the process of making changes to curricula (especially in terms of architecture, dedicated study time, and timing of exams) is resource-intensive. Parallel curricula for different graduating classes may be required, which can result in confusing messaging for students. Changes in testing may also require more resources dedicated to testing support.
To summarize, Dr. Amiel outlined UMSE’s likely reactions to some of the changes that anticipated to result from the USMLE Step 1 shift to P/F. Firstly, students may be required to take Step 2 CK earlier in their curriculum than they do now in order to obtain scores needed for residency applications. This would then require UME curricula to prepare students for the Step 2 CK earlier as well. Dedicated study time for Step 1 may be decreased, but new dedicated time would likely be introduced for Step 2 CK preparation. Schools that administer the Step 1 exam later in students’ training (like Columbia University) will need to consider shifting both the Step 1 and Step 2 CK exams earlier. Specialties may also begin to develop and administer their own exams to sort and rank applicants. If this occurs, curriculum design would be further impacted in order to help students be prepared and competitive for their desired specialties.
Dr. Amiel suggested that shifting the USMLE Step 1 to P/F will be positive for medical education, but stakeholders must take care to mitigate any negative impacts that could result from the reactions to this shift. He stressed the importance of basing initiatives in education theory and best-practices to assess medical knowledge, promoting competency-based medical education. He also mentioned working with the NBME and USMLE program to establish optimal curricular times dedicated to the Step 1 exam as part of the gateway into the clinical phase of UME curricula. He suggested that students should be able to demonstrate their competency in a basic body of knowledge and some clinical reasoning skills before moving to clinic, and that, in its current form, the USMLE Step 1 itself may currently not be sufficient for this purpose. This raises the question of whether the USMLE Step 1 is even the optimal test for the transition from pre-clerkship training to the clinic. However, Dr. Amiel did stress that all parts of an assessment program should be designed to help further learner’s progress and learning, and should include experience with high-stakes assessments.
Finally, Dr. Amiel cautioned that any new methods of programmatic assessment should be mindful of holistic review and how generating data about learner performance may impact residency applications. In particular, institutions must be careful to employ anti-racist pedagogy and assessment practices to avoid increasing inequities and disadvantaging students who may not have had as many enriched educational experiences as others. Dr. Amiel concluded with the recognition that the USMLE Step 1 P/F shift will result in an abundance of scholarly activity opportunities.