Abstracts on Assessment

The following abstracts have been accepted for presentation at this meeting in Poster format in the category of Assessment.  Those selected for Oral Poster Presentations are so designated in the far right column.
 
 

A1	Correlation OF STUDENT PERFORMANCE in assessments of KNOWLEDGE, CLINICAL PERFORMANCE AND CLINICAL SKILLS IN A second year medical curriculum: A Two Year Perspective Peter T. Borgia1*, Ph.D.,, and George A. Dunaway*, Ph.D., 1Department of Medical Microbiology and Immunology, and 2Department of Pharmacology, Southern Illinois University, School of Medicine, Springfield, IL 62794-1220 U.S.A. The second year curriculum at Southern Illinois University School of Medicine is organized into four units: 1) Infection and Host Defenses; 2) Cardiovascular, Respiratory and Renal; 3) Neuromuscular and Behavior; and 4) Endocrine, Reproductive and Gastrointestinal. Each unit is directed by a basic scientist and a physician and employs a multidisciplinary faculty. Objectives are largely drawn from pharmacology, pathology, microbiology, immunology, radiology, and clinical medicine. Topics in epidemiology, preventive medicine, evidence-based medicine, genetics and neoplasia are integrated within each unit. A variety of learning activities are used. Groups of six students meet with a tutor twice a week in Problem-Based Learning (PBL) sessions to discuss cases that are presented in a written form or as standardized patients. A variety of large group multidisciplinary case-based sessions cases are employed by some of the units. A limited number of single discipline and large group lecture sessions occur in each unit. Multiple-choice, self-assessment questions created/chosen by the faculty are available to the students. The questions relate to important unit topics and aid students in monitoring their learning. Students participate in unit relevant clinical activities and spend time on a weekly basis with a primary care physician mentor. Student assessment in the curriculum is multifaceted, and the components include: 1) tutor and mentor evaluations of knowledge, self-directed learning, clinical reasoning, and interpersonal skills; 2) a multiple choice knowledge and problem solving examination (MCQ) written by faculty in the USMLE format; 3) a clinical performance examination (CCX) based on three cases that are presented as standardized patients; and 4) a series of OSCE stations designed to assess focal clinical skills. We have statistically analyzed the performance of individual students in each assessment component across the 2002-2003 academic year and have correlated individual student performance across the components. The results of the analysis following completion of the first two units indicate a strong correlation between student performances in the MCQ examination across units. Significant but less strong correlations were found between: 1) the CCX examination performance across the units; and 2) the MCQ examination performance and the CCX examination performance within the units. Results of analyses from two years classes will be presented at the meeting.	ORAL
A2	INTEGRATED, CASE-BASED, R-TYPE MULTIPLE CHOICE ASSESSMENTS: A NOVEL APPROACH TO STUDENT EVALUATION IN THE BASIC SCIENCES Peter Cauwenbergs, B.Sc., M.Sc., Ph.D., D.C. Department of Anatomy, Canadian Memorial Chiropractic College, Toronto, ON CANADA M4G 3E6 In an effort to better integrate basic science content with the clinical sciences, a novel approach to student assessments has been implemented in 1st Year anatomy courses at the Canadian Memorial Chiropractic College. In preparation for this assessment change, it was first necessary to employ an integrated, case-based approach in lecture presentations, using the same clinical scenarios in each content area, which included gross human anatomy, histology and embryology. Questions from each content area, related to particular clinical scenarios taken from those used in lectures, were combined into a common examination which students wrote in one sitting. The assessment, which was entirely case-based in a Type "R" format, was subdivided into four 50 minute sections with each section containing questions examining gross anatomical, histological and embryological content. The class was divided into four groups and rotated through each section of the test. This format allowed us to utilize either practical stations, having gross anatomical specimens and/or microscope slides, or paper stations with several multiple choice questions relating to the clinical case. Students recorded their answers on computer cards. Although this approach to student assessment required considerably more time to prepare the questions, as well as more faculty to proctor the examination, it had several positive outcomes for both students and instructors. Students benefited from a more integrated and applied understanding of the anatomical sciences and gained a better appreciation for the relevance of each content area to their future clinical practice, while instructors acquired an appreciation for presentations in related content areas, thus helping to reduce redundancies between presentations in each area. The number of examinations each student was required to write were considerably reduced, while still allowing instructors to monitor student performance in each content area. The computerized marking also provided fast feedback for both students and instructors. This assessment format appeared to have little effect on overall student performance, which was comparable to that of previous years in each content area. Preliminary results indicate increased levels of satisfaction from both students and faculty.
A3	ASSESSING THE QUALITY OF PROBLEM BASED LEARNING: INTERACTION OF GROUP, TUTOR, AND CASE Barbara J Davis, Ph.D.*, and Elaine Dannefer, Ph.D., University of Rochester School of Medicine and Dentistry, Rochester, NY 14642 U.S.A. The double helix curriculum (DHC) at the University of Rochester School of Medicine and Dentistry is a hybrid curriculum that includes extensive use of problem based learning (PBL). The goals of the present study were to assess the quality of PBL groups, tutors, and cases, and to explore relationships between group function., tutor skills, and case quality. An assessment instrument was designed with the aid of student and faculty focus groups. The assessment instrument was administered to all PBL groups (n=12 groups/year with 8-9 students per group). A mean score based on five attributes was calculated for each category (group, tutor, and case). Data were collected for two consecutive years in Human Structure and Function, a 14-week integrated course in anatomy, physiology, and histology that is offered in the fall semester of Year One at DHC. Feedback from the first year of assessment was shared with course directors, tutors, and case writers. Data was analyzed using SPSS. The results of this study showed that there were significant correlations between group function, tutor skills, and case quality. Lower functioning groups rated cases lower than other groups, and groups with low-rated tutors also tended to rate case lower than other groups. Cases were rated higher in Year Two of the assessment compared with the same cases in Year One. Most tutor scores also improved during the second year of assessment. These results suggest that there are important interactions between group, tutor, and case, and indicate that feedback can be used to improve tutor skills and case quality in problem based learning.
A4	EVALUATION OF THE PERFOMANCE OF THE STUDENTS AND ITS IMPACT ON THE TEACHERS Dresch, S., Dastugue, M., De Medicis, P*., Bernabo, J., Ruvinsky R., Torino, M. y Lanosa, G. School of Medicine, Buenos Aires University, Buenos Aires ARGENTINA In the context of a curricular change the School of Medicine of the University of Buenos Aires decides to carry on an evaluation of the performance of the students at the end of the Clinical Cycle. For that purpose an instrument ( pencil and paper ) applicable to a great number of students is developed, having a relevance to the aims of Cycle and being capable of producing an impact on teaching. A model of an objective and structured test , with multi-item and common basis for the solving of clinical cases is used (EOEM). Its utility is assessed in this report (reliability, validity, acceptability, impact and cost). The 50 Teaching Units that form the School are divided in 5 regions that are called Regional Teaching (RT). The leading teachers of each RT are trained to develop clinical cases (simulators) according to a table of specifications. The contents are validated, the technical requirements are supervised and previous tests are carried on. 251 students took the test. The reliability index was adequate in relation to the standard value, validity was guaranteed during the development of the test and it was considered relevant and without any ambiguity both by teachers and students . The low cost was taken up by the available resources. The indicators of impact on teachers were satisfactory (motivated the research of the precision of the objectives of the program, increasing the exchange of opinions between the different RT teachers, etc). It will be convenient to analyze the obtained achievements respect the validity and accuracy to include as an evaluating instrument.
A5	IMPACT OF DIFFERENT STANDARD-SETTING METHODS ON EXAM AND COURSE GRADES Pamela L. Derstine, Ph.D.*, Loyola University Chicago Stritch School of Medicine, Maywood, IL 60153 U.S.A. Robust methods for setting P/F cut scores for high stakes exams have been validated and widely used. Course directors may not be aware of such methods or the implications for student learning when the common methods (pre-established score ranges or normative [standard deviations derived from actual performance]) are used for standard setting. Both commonly used methods may appear to be objective approaches, but pre-set score ranges imply a judgment about exam difficulty and/or student ability without explicit consideration of these characteristics; use of standard deviations perpetuates competitive learning, which is inconsistent with outcomes-based competency assessment. The purpose of this study was to investigate the two common methods as well as three validated methods for: ease of use by an individual course director; failure rates for exams; application to determining exam grade cut scores; and utility for combining exam grades into a global course grade. The validated methods were Hofstee, Ebel, and Direct Borderline (a simplification of the modified Angoff method). Use of these methods will be explained and exam and course grade distributions will be compared by method in the poster. Both the normative and Hofstee methods utilized student performance data and resulted in higher failure rates. The Ebel and Direct Borderline methods utilized item-centered judgments and resulted in lower failure rates. Of the 3 validated methods, Hofstee is the easiest for a single course director to use since judgments for each item are not needed. (Cut-off scores are determined by consideration of borderline student characteristics.) It is also the most subjective of the 3 methods while the Ebel method appears least subjective. Item writers can submit item difficulty and relevancy ratings while the course director can determine % of students for each question category, thus separating student performance from test items. All 3 methods can be used to determine final course grades, but students will not easily be able to track progress toward their final grade. This could result in student dissatisfaction. In addition, the three validated methods are time-consuming to use and not practical to determine grade cut-scores for each exam. However, the Hofstee method is a good approach for determining P/F cut scores for both exams and final course outcome.	ORAL
A6	TEST ITEM DIFFICULTY RELATED TO ITEM DISCRIMINATION AND CONTENT: AN APPROACH TO IMPROVED TEST CONSTRUCTION Robert A. Lavine, Ph.D.*, Department of Pharmacology & Physiology, The George Washington University School of Medicine and Health Sciences, Washington, DC 20037 U.S.A. Two measures of test items are item difficulty, based upon the proportion of students examined who do not choose the correct response) and item discrimination, based upon the correlation between a correct response and a more general measure of ability, that may be obtained from the total test score or the category within which the test score falls. Items may eliminated if their difficulty level is too high or too low, or if their discrimination is too low or below zero, so that the items selected provide a desirable test score distribution and a satisfactory level of discrimination. A hypothesis considered here is that many items that are overly difficult by the above definition may (1) lose the ability to discriminate adequately and (2) may suffer from problems in item construction, content, or presentation of the associated material. Initial data were obtained from test results of first-year medical students in physiology and neurobiology courses. In a physiology examination, when item discrimination was plotted against item difficulty, data correlated well (r=.84) up to the difficulty level at which approximately 80% of the class obtained the correct answer. Above that level there was considerable scatter and poor correlation (r=-.08). In a neurobiology examination, results were similar (r=.84 and r=-.01 for the two ranges of difficulty). Increased difficulty above a certain threshold did not produce a commensurate rise in discrimination, and was associated with many high-scoring students choosing incorrect answers for such items. Initial examination of item content provided some support for the hypothesis that problems in item construction, such as logic or wording, in narrowness of content, or in presentation of the subject matter, may contribute to the statistical characteristics observed. The availability of software, to be described here, that allows for sorting and retrieval of test items with desired levels of difficulty and/or discrimination, should assist in improved tests construction in medical education when item content is taken into account along with these statistical measures.
A7	STUDENT SELF-CONFIDENCE AS AN ASSESSMENT TOOL FOR THE DEVELOPMENT OF CLINICAL REASONING SKILLS IN A PROBLEM-BASED LEARNING COURSE Jon S. Patterson, D.V.M., Ph.D.*, Department of Pathobiology and Diagnostic Investigation, Michigan State University College of Veterinary Medicine, East Lansing, MI 48824 U.S.A. Over a 4-year period (1999-2002), second-year veterinary students in a course taught in problem-based learning (PBL) format were surveyed regarding their self-confidence in performing 3 clinical reasoning skills: 1) making patient Problem Lists, and grouping related problems; 2) making Rule-out Lists, and ranking the rule-outs as more likely or less likely; and 3) choosing appropriate diagnostic tests, and interpreting test results to arrive at a definitive diagnosis. The purpose of the study was to determine whether self-confidence in these skills increased over the course of a 15-week semester, during which the students practiced the skills in small-group case discussions and in an independent written assignment. Results of the survey were then interpreted in light of subjective evaluations of students’ competency in these skills, to assess whether increased confidence is associated with increased competence. On both the first and the last day of class, students were asked to rate their confidence in each of the 3 skills on a scale of 0-10, with 0 indicating "no confidence" and 10 indicating "extremely confident." During the semester, PBL groups discussed 12 clinical cases. All cases included Task Questions on making Problem Lists, making Rule-out Lists, and choosing appropriate diagnostic tests, and groups’ Problem Lists and Rule-out Lists were collected for approximately two-thirds of the cases and critiqued by the course moderator. In addition, students independently completed a written assignment which emphasized the 3 clinical reasoning skills evaluated in the confidence survey; this assignment was then critiqued by clinicians. For the self-confidence study, mean confidence scores were calculated for the class for both time points, and for each of the 3 clinical reasoning skills. The means of the beginning-semester and end-semester scores were compared by a paired-samples t-test. Results indicated that there was a significant increase in students’ confidence in all 3 clinical reasoning skills in each of the 4 years (P<.001) over the 15-week semester. Assessment of PBL groups’ Problem Lists and Rule-out Lists by the course moderator generally indicated improved competency in making clinically useful lists over the course of the semester. The written assignments were generally scored as very good to excellent by clinicians. In conclusion, students’ self-confidence in performing certain clinical reasoning skills increased as a result of practice in a PBL course, and confidence may be a valid assessment
A8	BASIC SCIENCE IN A PROBLEM-BASED AND IN A TRADITIONAL CURRICULUM – ASSESSMENT OF STUDENTS ACHIEVEMENT USING MULTIPLE CHOICE AND MODIFIED ESSAY QUESTIONS Joerg Pelz, Ph.D.*, Martin Digweed, Reformstudiengang Medizin and Institut fuer Humangenetik, Charité Universitätsmedizin Berlin, Berlin GERMANY The teaching of basic sciences in traditional medical curricula has emphasized the acquisition of knowledge rather than its use. The most common selected-response item format (multiple-choice questions – MCQ) is applied traditionally to measure cognitive knowledge and educational achievement efficiently and in a straightforward manner. MCQs are criticized for their presentation of non-authentic content. The Charite (Berlin) offers the possibility to compare two groups of students from two different curricula (traditional and problem based). We tested both groups with MCQs and modified essay question, a clinical problem-solving exercise, in which a patient clinical vignette was presented, with short essay-type questions interspersed. The greatest advantage of this type of examination is that it requires examinees to produce written responses. The MEQ-format has been shown to be fairly reliable. The focus of both tests was on biological problems. Students of the traditional curriculum yielded significantly higher marks in the MCQ, while this difference wanes in the MEQ. Both formats covered similar problems, so comparisons on an individual level could be conducted. The traditional curriculum favors the acquisition of lower cognitive levels, while the more complex, higher order cognitive tasks than recognizing correct answers from a list were similarly favored in both curricula.	ORAL
A9	A COMPREHENSIVE COURSE EVALUATION SYSTEM UTILIZING COMBINED ON-LINE STUDENT AND FACULTY EVALUATIONS, STUDENT PERFORMANCE MEASURES AND COURSE DIRECTOR MEETINGS Thomas V. Peterson, Ph.D*., Rebecca A. Baker, B.A., R. Kelly Hester, Ph.D. and J. Ben Green, III, M.D. Office of Medical Education, Texas A&M University System Health Science Center College of Medicine, College Station, TX 77843 U.S.A. Required course and clerkship evaluations often consist of student course satisfaction surveys and/or course performance measures. Our evaluation system utilizes a more comprehensive approach taking into account multiple types of information about each course, i.e. triangulation. 1. Student perspective about the course is obtained by means of an on-line questionnaire which addresses the following broad areas with specific items related to each: a) course/block organization; b) quality of course teaching materials; c) student workload and examinations; d) quality of teaching; e) faculty availability; f) value of labs, small group exercises and clinical experiences; and g) integration of material in the course/block. Student input is also obtained from student as members of the Curriculum Study Group (CSG), the standing Curriculum Committee subcommittee responsible for the evaluation process. 2. Faculty perspective is obtained by use of a separate on-line questionnaire completed by faculty teaching in each course or block. This instrument addresses: a) appropriateness of course/block educational goals and objectives; b) course/block content; c) total course/block time allotment; d) sequencing of material within the course/block and within the curriculum; e) appropriateness of the exam schedule; f) cooperation of faculty in the course/block; g) coordination with other courses/ blocks; and h) enjoyment of teaching in the course/block. 3) Student achievement and effectiveness of the curriculum is obtained by reviewing course grades and NBME subject examination scores. After review of the data by the CSG, the Course/Block Coordinators are invited to meet with the CSG to discuss the course evaluation and formulate recommendations for improvement if needed. A final report is then submitted to the College of Medicine Curriculum Committee. We feel this system of triangulation has provided a more comprehensive evaluation of our educational program than previous methodologies.
A10	IMPROVEMENT OF MEDICAL SCHOOL ESSAY QUESTIONS USING STATISTICS Edward G. Simanton, Ph.D.*, Kurt E. Borg, Ph.D., Roger W. Schauer, MD., and Richard C. Vari, Ph.D. Office of Medical Education, School of Medicine and Health Sciences, University of North Dakota, Grand Forks, ND 58202 U.S.A. The University of North Dakota School of Medicine and Health Science medical curriculum is a PBL-hybrid model with traditional lectures and labs focused on systems biology and pathobiology structured around patient cases. The first two years are divided into eight blocks. An important component of the assessment, administered at the end of each block, is a 20 to 40 essay question case examination. While the case examinations (including model answers and grading criteria) are modified every year to reflect curricular changes, a large part of the case examinations are used from year to year with little or no change. Case examinations are scored by teams of faculty as follows: Faculty members are assigned test questions by specialty. Faculty members grade all student answers for the questions they are assigned using previously developed model answers and grading criteria. As faculty members complete their assigned items, student scores are entered into a spreadsheet program. After all scores have been entered into the spread sheet, item statistics are run on all items and evaluated for difficulty and discriminant value. Questions not meeting minimal test item performance are eliminated before final student scores are calculated. In the spring of 2004, data from the previous year’s Block III and Block VII case examinations were analyzed for item difficulty and discriminant value (correlation with total score). Items were classified as: Excellent difficulty (average score of 70 – 90%) Excellent discriminant value (R >.300), Too easy (average score of 100%), Too difficult (average score <50%), Too low discriminant value (R <.000). Item performance ratings and suggestions for interpretation were provided to the case examination preparation team. Item performance in 2003 and 2004 were as follows: Excellent difficulty (2003 - 33%, 2004 - 35%) Excellent discriminant value (2003 – 24%, 2004 – 39%), Too easy (2003 - 3%, 2004 – 2%), Too difficult (2003 – 5%, 2004 – 3%), Too low discriminant value (2003 – 13%, 2004 – 6%). These results indicate that statistical analysis of previous tests provides valuable feedback for improvement of essay questions. This process of analysis, feedback and improvement gives medical educators greater confidence in the use of case-based essay questions to assess student learning.
A11	Formative Evaluation of Structure and Organization of Curriculum in a Japanese Medical School Hitoshi Sohma1*, Atsushi Miyamoto2, Hideo Kanoh3, Kohzoh Imai4, Kowichi Jimbow5 1Director of Medical Education, 2Hospital Pharmacy (Ex-Director of Medical Education), 3Department of Biochemistry (Ex-vice-Dean), 4President (Ex-vice-Dean), and 5Department of Dermatology (Ex-Dean) of Sapporo Medical University School of Medicine, Sapporo 060-8556 JAPAN Formative evaluation of serial lectures by students has been performed yearly since 2001 to improve teaching skills of individual teaching staff at Sapporo Medical University School of Medicine. The evaluation, done soon after regular examinations, consisted of five grades and was on the basis of the following 10 questions: 1) Are overall structure and arrangement of curriculum adequate?; 2) Are individual lectures and syllabus consistent?; 3) Are teaching staff motivated in their lectures?; 4) Are learning contents valuable?; 5) Are lecture contents comprehensive?; 6) Do lectures stimulate intellectual curiosity?; 7) Is medical knowledge acquired from lectures?; 8) Are teaching materials well prepared by teaching staff?; 9) Are examination volumes appropriate?; 10) Are contents of examinations adequate? The evaluation scores for the academic year 2001 and 2002 were compared and analyzed statistically. The average score of all subjects had increased by 0.15 points in the year 2002. The magnitude of the increase was 0.18 and 0.27 for the subjects of liberal arts and clinical medicine, respectively, while it was 0.03 for the basic medicine subjects, suggesting an improvement in the effort of each instructor. For example, some instructors show learning objectives at the beginning of each lecture, revise their syllabus frequently, introduce animation into lectures, and discuss about instructional materials and educational methods with other teaching staff, etc. The standard deviation of the scores for 8 questions (2 questions on examinations were omitted) was smaller in the clinical medicine subjects compared with liberal arts and basic medicine subjects. We speculate that this result was due to the fact that the content of the clinical medicine subjects directly connected to the National Examination for Medical Practitioner. It seems that it is easier to create a teaching plan which will be achieve the required learning objectives in clinical medicine subjects compared with in other subjects. Continuous formative evaluation of serial lectures and analysis of results is needed for the improvement of teaching skills.
A12	COMPUTERIZED PATIENT SIMULATIONS AS AN ASSESSMENT OF STUDENT KNOWLEDGE AND REASONING Carol F. Whitfield, Ph.D.*, Department of Cellular and Molecular Physiology & Office of Medical Education, Penn State College of Medicine, Hershey, PA 17033 U.S.A. In the first two years of our medical school’s curriculum, problem-based learning (PBL) accounts for a significant amount of student contact hours, increasing from about 7% at the beginning of year one, to about 50% in some of the year two courses. We use PBL to teach students how to approach a clinical case, as well as to develop a basic science knowledge base. At the end of each semester for the past five years, we have assessed second-year medical students’ ability to approach a clinical case and use the knowledge they gained, and recently made it a requirement that they pass two of these exams in the second year. DxR Clinician® is used as the assessment tool. Faculty members write the clinical cases for the assessment using subjects from the courses students have taken during that semester. Students, not knowing the subject of the case, are instructed to produce a broad differential, do an interview, physical exam, order laboratory or diagnostic tests, make the diagnosis, justify it, rule out competing hypotheses and develop a treatment plan. Each student’s exam is reviewed for these components and scored as pass or fail. The score generated by the program is not the sole factor determining pass or fail. Results show a consistent bimodal distribution over the years and the percent of students failing each exam varies with case content. Of those failing, 50% fail due to knowledge deficits (not remembering key tests for a diagnosis, for example). Another 30% of students get onto the wrong track, cannot reason past the block or panic. A final 20% are due to illness, fatigue or lack of effort. About 50% of the students who fail also have other academic deficiencies. In addition, we have determined that some students do not know the definition of "diagnosis", and instead, list signs and symptoms as their diagnosis. The large amount of information available within the students’ records saved within the DxR program gives us the ability to identify specific areas where students have difficulties and determine if it is their approach to a clinical case or a deficiency in their fund of knowledge that accounts for their failure.
A13	EVALUATION OF THE EFFECTIVENESS OF INTEGRATING A MEDICAL SCHOOL CURRICULUM Stephanie Wragg, Ph.D.*, Andria M. Thomas, Ph.D. and Peggy J. Wagner, Ph.D., Medical College of Georgia, Augusta, GA 30912 U.S.A. The Medical College of Georgia (MCG) seeks to train health practitioners to meet the healthcare needs of urban and rural citizens of Georgia. Every year, approximately 360 students enroll in the Phase 1/2 program of the School of Medicine (SOM). An ongoing evolutionary process for integrating the Phase 1/2 Curriculum has lead to the design of a modular curriculum for implementation in academic year 2004-2005. Academic year 2003-2004 is being used to plan a more coherent and coordinated curriculum, plus an evaluation strategy to assess student perceptions towards this new curriculum. The management methods for evaluation of program effectiveness and monitoring of syllabus content that will be used by the Evaluation Team of the SOM are described. Data from years prior to implementation of the new curriculum will be used to establish the baseline for subsequent comparison. Following implementation, the Evaluation Team proposes the following layered approaches to direct its work: Quantitative examination of satisfaction and performance-related data obtained as part of the standard educational process (evaluation of faculty and courses, academic GPA, MCAT, NBME and USMLE Step 1 scores); Qualitative assessment through focus groups of whether student perceptions match curricular objectives; Annual student survey using a validated tool for quantitatively measuring cognitive behavior, attitudes towards social issues in medicine and learning environment; Annual faculty survey to determine if their perceptions parallel those of the students. The pre-clinical years of medical school are a critical foundation in the development of future physicians. Assessing the effectiveness of curriculum changes at MCG on a continuous annual basis will identify successful areas, and guide modification in parts where students may show deficiencies or dissatisfaction.	ORAL