This month the IAMSE Publications Committee review is taken from the article titled Difficulty and Discriminative Ability of Various Categories of Multiple Choice Questions in Medical School Preclerkship Examinations published in the Medical Science Educator, Volume 24, (pp 387-393), by Stanley J. Nazian and Frazier T. Stevenson.
Experienced writers of medical science examination items can easily agree that not all multiple choice vignette questions are created equal. The more pragmatic question is: does vignette complexity matter when measuring outcomes? Stanley Nazian and Past IAMSE President Frazier Stevenson examined part of this question in their 2014 work published in Medical Science Educator, where they describe an investigation on vignette complexity as it relates to assessment item performance on internal course examinations.
Nazian and Stevenson categorized all examination questions in their preclinical medical course examinations as one of four question types: recall, pseudo-vignette, vignette 1, or vignette 2. Recall questions feature strict recall of information with no vignette. Pseudo-vignette questions contain a vignette to place the recall item into a real-world context, but the vignette is not required to answer the question correctly. Vignette 1 and Vignette 2 questions are those where a student must derive one or two pieces of information from the vignette, respectively, in order to answer the question correctly.
The authors describe the distribution of these question types across both preclinical competencies (e.g. immunology-infectious disease, pharmacology, etc.) and specific didactic courses (e.g. Neurological, Cardiovascular and Pulmonary, etc.) in their curriculum. They then determine the differences in difficulty, discrimination index, and point biserial correlation coefficient across the different question types in year 1 versus year 2.
This is an important work, and one worth digging back a few years to revisit because there has not been a tremendous amount of follow up on this topic at other schools despite an increasing reliance on vignette-style questions in standardized licensing exams for medical science. Many questions still remain: Do other schools follow similar distribution trends of question types across competencies? Does performance on vignette-style questions vary depending on competency topic? Do standardized testing scores improve with increased proportions of complex vignette questions? Nazian and Stevenson postulate that performance improves on complex vignette questions from year 1 to 2 due to experience with these question types or true learning, leading one to wonder if vignette-based questions of varying complexity actually prepare our learners better for their clinical experiences?
As discussions about modifying the United States Medical Licensing Examinations intensify in the US, now is an opportune time for all of us to examine our internal assessments, and this work by Nazian and Frazer provides a great framework with which to do so.
Kelly M. Quesnelle, PhD
Associate Professor
Western Michigan University Homer Stryker M.D. School of Medicine
Chair, IAMSE Publications Committee