For centuries, medical educators have used traditional means for assessing medical knowledge and skills: paper-and-pencil tests, microscope-based exams, and clinical skills exams with simulated patients.  Some of these trusted methods, however, have serious drawbacks.  For instance, in a typical microscope-based exam students are given little time to examine a specimen and are not allowed to review their answers.  Not uncommonly, specimens change or become damaged during the examination process. 

Recent advances in web-based and robotic technology have remedied some of the disadvantages of traditional assessment methods.  These new formats of assessment, however, are expensive and introduce a new set of challenges.  For instance, security concerns are raised because students need to be tested in shifts due to limited seating capacity in computer laboratories.  Also, students could conceivably use the Internet inappropriately during the exam using “instant messaging” or surfing the World Wide Web to find answers.  It has been UCLA’s experience, however, that reminding students of the Honor Code is sufficient to avert widespread cheating.

In this seminar, Dr. Uijtdehaage also discussed emerging trends in the field of computer-based assessment.  “Virtual patients” are computer-based simulations with which students can interact to sharpen their diagnostic reasoning and procedural skills without risks to patients.  Virtual patients range from relatively simple web-based applications to very complex, high-fidelity computer-driven mannequins. These simulations can be used to assess clinical skills to the extent to which they can track and document students’ clinical decisions and treatment choices.

Computer adaptive testing (CAT) is being adopted increasingly in standardized testing but has not yet found widespread use in medical education.  It was introduced in this presentation as a potential novel method to measure medical knowledge with great precision.  Based on Item Response Theory, CAT selects a unique sequence of test items to estimate a student’s proficiency. Difficulty level of the questions is based on the student’s performance on previous questions. CAT, however, requires a large set of questions with established psychometric properties such as difficulty level.  Therefore, this method may not be feasible for individual institutions unless medical colleges collaborate.

Fortunately, recent technological advances have facilitated collaborations among institutions.  For instance, several XML metadata schemas have been developed to describe the content and characteristics of test items such as the IMS Question and Test Interoperability Specification (www.imsglobal.org/question/).  As increasing numbers of medical schools use electronic course management systems that are compatible with such metadata schemas, we may see more sharing, banking and re-deployment of test items in the near future.

In conclusion, computer-based testing resolves some problems associated with conventional assessment methods but at the same time introduces new challenges.  Because computer-based assessment opens new ways to improve the validity and reliability of testing, it is worthwhile exploring how sharing of test items among medical schools can address the increased cost.  Finally, but importantly, writing effective test items is and remains an art regardless of the sophistication of the assessment method.

View Slides from this Presentation