MEDICAL SCHOOL LECTURE ATTENDANCE: WHO, WHY, AND WHAT IS THE EFFECT ON ACADEMIC PERFORMANCE?
Award Nominee
Robert Bloodgood, John Jackson & James Martindale
University of Virginia School of Medicine
Charlottesville VA 22908-0732
USA
PURPOSE
Medical student lecture attendance is consistently high for 3 semesters and declines in the 4th semester. In recent years, this decline has moved into the 3rd semester. This study asks: who chooses to skip lectures, what is the motivation and is there an effect on academic performance?
METHODS
This study used a web-based questionnaire plus three databases (admissions data; attendance data; course grades).
RESULTS
Top reasons cited for decreased lecture attendance in the 4th semester of the curriculum were: need to prepare for USMLE Step 1, pressure to study for cumulative exams and quality of lectures. Students cite USMLE Step 1 as the major stressor in Year 2. There was no correlation between level of attendance and course grades. There was no correlation between lecture attendance and ability of the student (based on admissions data) . Both undergraduate grade point average (uGPA) and combined MCAT scores were predictors of academic performance in the 1st and 2nd year courses with MCAT scores showing a stronger positive correlation than uGPA.
CONCLUSION
There is no correlation between the level of lecture attendance and the academic performance of medical students in the first two years of the curriculum. Using admissions metrics as reflective of ability, there was no preference for more or less able students choosing to skip lectures. The proximity of the USMLE Step 1 exam appeared to be a major factor in the decline in lecture attendance at the end of the 2nd year.
USING FORMATIVE AND SUMMATIVE CBA TO STIMULATE STUDENTSâ STUDY BEHAVIOR
Roel Sijstermans & Nynke Bos
Academic Medical Center – University of Amsterdam
Amsterdam 1105 AZ
The Netherlands
PURPOSE
In 2006 the Academic Medical Center â University of Amsterdam has started a new medical curriculum. One of the goals was to stimulate students to study regularly during the course and not only the last week before the final exam.
METHODS
During an eight week course there are two moments where students can earn extra credits by making a summative computer-based assessment. The test consists of 15 randomized MC questions and includes subjects discussed in the previous weeks. After question analysis students receive, besides their score, individual tailored feedback on their exam. The 5 most difficult questions are also discussed plenary by the teacher.
Before entering the laboratory or the dissection room students have to fill out an electronic formative test. At least an 80% score must be achieved before they can start their practical work. They can take the online test several times, 24/7, before reaching the 80% score.
RESULTS
30% of the participants pass the extra credit exam. An average of 73% of the students (n=31 2) expresses that the bonus assessments stimulate them to study regularly during a course. Averages of 4% of the students pass their final exam because of the bonus credit earned.
CONCLUSIONS
It seems that the use of midterm bonus assessments stimulates students to study regularly. However, it can also be a social desirable answer. After all, which student doesnât want to earn extra credits? Only a few students pass their final exam by using the bonus credits. A positive factor is that students can test their knowledge during a course. Individual feedback is thereby necessary to give students an insight in their knowledge gaps, on which a teacher can also anticipate.
The admission tests for practical education seem to be more successful. Teachers experience that students have more theoretical knowledge and are better prepared when they enter the laboratory or dissection room.
ASSESSMENT OF LEARNING EFFICACY OF SMALL GROUP WORK USING A KEY WORD TEST
Dirk Ruiter, Riekje de Jong, Peter de Wilde, Ron Leunissen, Marleen Olde Bekkink & Goos van Muijen
Radboud University Medical Centre
Nijmegen 6500 HB
The Netherlands
PURPOSE
In modern (bio)medical curricula small group work is a central element. As deep learning and elaboration of learning are essential in small group work we hypothesized that key words representative for main concepts and principles could be used as a target for testing the efficacy of learning.
METHODS
We have designed a prototype of a key word test. The test was introduced to all 400 bachelor biomedical students of the RUMC during a course on Oncological Pathology, as an experimental instrument for monitoring learning efficacy. It consists of four questions on the nature and background of oncogenesis and two questions on the student’s perception of their own efforts and degree of understanding. It was taken at the beginning (t=0), and at the end (t=1) of the small group session. The short test was well accepted by both students and teachers.
RESULTS
The score of t=1 exceeded that of t=0, especially regarding the two questions concerning key words highly representative for the particular small group learning goals(p<0CONCLUSIONS/FUTURE DIRECTIONS
Our results suggest that it is feasible to monitor the learning efficacy of small group work using a key word test. The mechanisms of the learning efficacy deserve further investigation.
USING MULTIPLE MCAT SCORES IN THE ASSESSMENT OF STUDENT PERFORMANCE
Award Nominee
Stephen Manuel
University of Cincinnati College of Medicine
Cincinnati 45230
US Minor Outlying Islands
PURPOSE
The purpose of this study was to determine the use of multiple MCAT scores (or combinations thereof) in the assessment of students performance as measured by second year class rank. This study determined the use of multiple Medical College Admissions Test (MCAT) scores in the assessment of student performance as measured by second year class rank. The most recent research in this area used pre-1985 scores at a single institution.1
METHODS
770 students with multiple MCAT scores enrolled in one of three medical schools from 1991-2007 comprised the sample. Using the first MCAT attempt and the most MCAT attempt, the average score was computed. Also computed from the first and most recent MCAT was the highest score. A Pearson correlation analysis (p < .05) was performed between these four scores and class rank at the end of the second year.
RESULTS
For Verbal Reasoning there were no significant correlations. For Physical Science the highest correlation was the first MCAT attempt (r = -.117) and for Biological Science the highest correlation was the first MCAT attempt (r = -.155).
CONCLUSION /FUTURE DIRECTIONS
When reviewing MCAT scores for basic science performance schools should consider using the first attempt score on the Biological Science and Physical Science scores and realize the weak correlations.
1 Hojat M.,Veloski JJ., Zeleznik, C. Predictive validity of the MCAT for students with two sets of scores. Journal of Medical Education 1985;60:91 1-8.
CORRELATIONS BETWEEN CRITICAL THINKING SKILLS AND MCAT PERFORMANCE REVISITED
Bruce Newton, Kevin Phelan & Carol Thrush
University of Arkansas School of Medicine
Little Rock AR 72205
USA
PURPOSE
In 1991, the MCAT underwent a dramatic change in content and score reporting. Two studies reported a positive correlation between the pre-1991 MCAT total and subtest performance and critical thinking (CT) skills using the Watson-Glaser Critical Thinking Assessment (WGCTA; Scott and Markert, 1994; Miller et al., 1993). However, the existing MCAT has not been evaluated for correlation to CT skills.
METHODS
This paper reports on the initial findings of a multi-year study of the CT skills of medical students at our institution and relationships between CT skills and MCAT performance in the combined 2007 and 2008 matriculating classes. The WGCTA was administered during orientation week to 291 volunteer participants (94% recruitment rate).
RESULTS
The total MCAT and WGCTA scores were 28.5 +/- 4.01 (mean +/- S.D.) (range 16-39) and 65.9 +/- 6.34 (range 46-77), respectively. There was a significant positive correlation between the total MCAT and WGCTA scores (0.45, p < 0.01). MCAT verbal, physical science and biology subtests each exhibited positive correlations with total WGCTA scores (0.46, 0.31 and 0.29, respectively; p < 0.01), but the writing subtest lacked such a relationship (0.06, p > 0.05). The WGCTA inference, deduction and interpretation subtests exhibited positive correlations with total MCAT scores (0.35, 0.36 and 0.43, respectively; p < 0.01). The recognition of assumptions and evaluation of arguments subtests exhibited lower, though still significant, positive correlations (0.17, p < 0.01 and 0.12, p < 0.05, respectively).
CONCLUSIONS
The study results indicate that CT skills of first-year medical students remain positively correlated to total and specific MCAT subtest performance. Comparisons with previously published studies will be discussed.
Supported by the Medical Education Foundation Fund of Arkansas.
A FACULTY DEVELOPMENT WORKSHOP: GUIDING EDUCATIONAL RESEARCH TO CREATE SCHOLARLY PUBLICATIONS
Jack Scott
Louisiana State University Health Sciences Center
New Orleans, LA 70112
USA
PURPOSE
Publishing scholarly work is important for medical scientists. Writing skills are attained in a faculty development workshop where hands-on and interactive applications are conducted in a supportive, peer environment. Our model has significant findings for participant knowledge, skills and satisfaction. Individual participants received consultation for their manuscripts at the IAMSE Annual Meeting (2008) workshop predicated on Glassickâs (1997) scholarly criteria.
METHODS
What is the value of a conference workshop on writing for publication? A pre-post session questionnaire in the six-hour workshop targeted knowledge, intentions and self-efficacy measures and post-session reflection on learning satisfaction. Questionnaires included a Likert-type 5 point scale for session items (e.g., âTo what extent are you able to use time effectively to write a manuscript?â) and open-ended questions for other items (e.g., âWhat was most beneficial about todayâs session?â).
RESULTS
Questionnaires were completed (n= 12) with data analyzed using Wilcoxsonâs signed rank test. Significance (p < 0.05) was achieved on five of the eight items. The pre and post-session means were 3.09 (SD = 0.73) and 3.98 (SD= 0.60), respectively. Average item score gain = > 0.89. Item rankings included: Content relevance = 4.77; Interaction with peers = 4.69; Interaction with facilitators = 4.62; Activity relevance = 4.46; Effectiveness of presentation = 4.46 and Overall = 4.46. Added quantitative and qualitative data will be provided.
CONCLUSIONS
A workshop model meets needs of medical education scientists in writing for publication, especially for JIAMSE. We shall refine the model with similar target populations.
DEVELOPMENT TOWARDS A NATIONAL EXAMINATION FOR RESIDENTS IN MEDICAL MICROBIOLOGY IN THE NETHERLANDS
Mascha Verheggen, Ed Kuiper, Johan Mouton, Arno Muijtjens, Frank Tiel van & Maarten Visser
Maastricht University
Department of Educational Development and Research FHML
Maastricht 6200 MD
The Netherlands
PURPOSE
In 2003 the idea matured in the Netherlands for a yearly national exam for residents in medical microbiology. The main reason was a demand for an instrument that could provide feedback to residents and the educational supervisors regarding the residentsâ growth of knowledge during the resident training.
METHODS
An exam was developed to assess knowledge and the appliance of knowledge on skills for different sub domains within the medical microbiology field. Test items were developed according to a blueprint based on the end level of the medical microbiology resident training. All test items were reviewed by a national review committee which analyzed content, format and relevance of the items.
RESULTS
Until now 5 national exams have been organized for residents in medical microbiology in the Netherlands. Participation was high (95%, an average of 60 residents per exam) and residents were divided into year classes (1-5). For each of the 4 national examinations residents who were further advanced in their training scored higher, on average, than their colleagues of lower year classes. For the first 3 year classes this increase was statistically significant.
Furthermore, investigation of the results of 20 residents who participated in all 4 national exams from 2005 to 2008 showed that their ranking score (within the total group of 60 residents attending an exam) significantly increased over the years.
FUTURE DIRECTIONS
These results indicate that a valid and useful feedback instrument has been developed to measure the growth of knowledge of residents in medical microbiology.
â
PASSIVE/ACTIVE LEARNING PERCEPTION DISPARITY BETWEEN MEDICAL STUDENTS, NEW AND EXPERIENCED TEACHERS
Award Nominee
Rachel Aland, Philip Adds, Peter Bazira & Edward Gosden
St George’s, University of London
London SW17 ORE
United Kingdom
PURPOSE
The advantages of an active rather than passive approach to learning, in preclinical years and future careers, of medical students, are well understood. Teachers may influence students’ approach to learning through pedagogy. Changes were implemented in an anatomy unit taught in the medical course at our university to foster an active approach to learning. As part of these changes, this study compared students perceptions of their learning styles with the perceptions of their teachers.
METHODS
201 UK school leaving students enrolled in the first year of a five year course in medicine were asked to self assess their learning styles by completing a questionnaire designed to evaluate their passivity or activity in learning. Demonstrators (more senior students and newly qualified doctors) with limited teaching experience were also asked to assess the classes that they taught using a modified version of the student questionnaire. The same modified questionnaire was also completed by experienced (>10 years) lecturers familiar with active-passive approaches to learning.
RESULTS
The majority of students (86.7%) rated themselves as active learners. There was wide variation in demonstrator scoring, with the overall mean perceiving students as neutral, in between active and passive styles. In contrast, the lecturers universally scored the class as passive in learning style.
CONCLUSIONS/ FUTURE DIRECTIONS
This data questions whether first year medical students have insight into their own learning styles and demonstrates that they view their approach to learning in quite a different way to experienced lecturers. The data from the demonstrators implies that demonstrators with limited experience in teaching may not be able to accurately distinguish between active and passive approaches to learning, and may require additional training to foster an active approach in students that they teach.
USE OF A CAPSTONE WRITING ASSIGNMENT AND SURVEYS TO EVALUATE THE IMPACT OF CURRICULAR CHANGE
Amanda Fales-Williams*, Charles Johnson, Brandon Plattner & Kevin Saunders
Iowa State University
Ames IA 50011-1250
USA
PURPOSE
At the Iowa State University College of Veterinary Medicine (ISUCVM), senior students spend 2 weeks in an afternoon Necropsy rotation. Recently, this clinical rotation was paired with a restructured, morning Clinical Pathology course, as both courses rely on pathology case data from the teaching hospital. We sought to validate this curricular change.
METHODS
We measured the impact of the change by comparing scores (before and after implementation) on a written assignment, the Case Correlation Assignment (CCA), and by surveying students and informally interviewing faculty after the change. The CCA is a written case report based on the medical record of animals treated and necropsied at the ISUCVM. We hypothesized that students taking the co-scheduled courses would demonstrate improved scores or subscores on the CCA relative to students from the previous two years. Ninety papers (30/year) were evaluated by two outside raters, using a 21-item, 5-level rubric.
RESULTS
An independent samples t-test was performed on scores from three rubric items. No significant difference was noted between the years from any rater, or when papers were compared by top, middle and bottom scores. However, the sample students taking the co-scheduled pathology courses performed somewhat better on one specific item within the rubric (Ante-mortem Data Interpretation). In response to surveys and interviews, students and faculty both endorsed the change.
CONCLUSIONS
Anecdotally, co-scheduling the pathology courses in the clinical year had a positive impact on instructors and students. Within the CCA, the Ante-mortem Data Interpretation item had the strongest trend toward improvement.
A PROCESS FOR PEER EVALUATION OF TEACHING
Diane Hills*, Glenna Ewing, Luke Mortensen & Maria Patestas
Des Moines University
Des Moines IA 50312
USA
PURPOSE
Student evaluations of faculty teaching have limitations. We have developed a process of faculty peer review that is both formative and summative. We will provide details of a model of peer review that is designed to evaluate the quality of a faculty memberâs teaching and encourage improvement as well as reflection on teaching effectiveness in a supportive climate.
METHODS
The process includes: an interview with the faculty member to discuss teaching and assessment strategies; a review of the faculty memberâs teaching philosophy; one or more peer teaching observations; and a review of examination questions, handouts, slide sets, etc.
Department chairpersons initiate the review process by composing an evaluation team of two faculty members who have received training in preparation for peer evaluation. The evaluation instrument focuses broadly on the teaching domains of: clarity and organization, style, group interaction, content, professionalism, and assessment. The ensuing summary report is fashioned to allow the faculty member to reflect on: teaching-learning efficacy; communication skills; academic rigor; suitability and cohesiveness of course objectives and teaching content; student engagement in learning; quality of student understanding through performance; and scholarly teaching effort and ability.
RESULTS
Since its implementation, reviewed faculty members have found the process to be agreeably pertinent, influential in their teaching, and fair in its approach.
FUTURE DIRECTIONS
The sustained utility of this model will be reflected in future reevaluations of these same faculty members toward determining improved teaching efficacy and student learning.
COMPETENCY-BASED ASSESSMENT OF COMMUNICATION SKILLS EARLY IN THE PRE-CLERKSHIP CURRICULUM
Marieke Kruidering
University of California
San Francisco, CA 94143
USA
INTRODUCTION
To prepare medical students for clinical immersion and ACGME competencies, we adapted the communications domain into a menu applicable to the pre-clerkship curriculum, and instituted competency-based assessments for first-year students.
OBJECTIVES
Our aim was to pilot and evaluate an oral presentation exercise to assess communication skills in the pre-clerkship curriculum.
Methods
The first pre-clerkship course assessed a 5-minute oral presentation during anatomy lab. Videos created by instructors served as examples for students. Instructors rated students ânot yet atâ, âatâ, or âaboveâ expected competency for 3 items from the menu and provided written and oral feedback. Faculty and students rated the usefulness of the activity.
RESULTS
Faculty noted that the presentations increased student confidence and stimulated anatomy learning. For the second year of the program, refinement of anchors and additional faculty development reduced faculty rater variability. Students and faculty rated usefulness of the assessment highly (students:4.17 sd=0.89, faculty:4.75 sd =0.45 5=maximum).
DISCUSSION
We successfully introduced formal assessment of communication skills through a presentation exercise. Key ingredients were faculty development and clear expectations for all stakeholders. Through videos depicting sample presentations, faculty observers developed a consistent understanding of the assessment rating scale, which a rating training session further refined. Students were apprised of expectations through online videos and the syllabus.
CONCLUSION
Formal assessment of communication skills early in the curriculum is an effective way of introducing students to competency-based measures. The next steps include assessing additional competency domains.
LONGITUDINAL ASSESSMENT OF PHARMOCOLOGY AND PHARMACOTHERAPY IN MEDICAL SCHOOL
Kalyani Premkumar
ES&D
College of Medicine
University of Saskatchewan
Saskatoon SK S7N 5E5
Canada
PURPOSE
Concept maps have been identified as a useful tool for organizing existing knowledge or to create new knowledge. It has also been used to communicate complex ideas, and to assess understanding or diagnose misunderstanding. Concept maps mirror the processes of thinking and learning and provides a window into a students mind. Experiments show that subjects using concept maps outperform those who donât in long term retention tests. A concept map has three parts: concept, proposition and learning and several concepts arranged on a page linked by propositions constitute a concept map.
METHODS
In our medical school, concepts maps are used as an assessment tool in two modules of a first year medical course. Following an orientation to concept map creation, students were given a problem in cardiovascular physiology and in immunology in two separate modules. Students created concept maps individually or in self-selected groups and submitted the assignment online or as hard copies. Assignments were corrected using specific criteria. An online survey focusing on studentâs perceptions on the use of concept maps and its effect on their learning was administered at the end of the modules.
RESULTS
Most students used commercial software to create concept maps and worked in groups. Preliminary survey results indicate that students find these assignments engaging and help with their learning. In this presentation, the results of the survey will be discussed and examples of student concept maps will be shown. Conclusion/Future directions
Given the benefits of concept maps we hope to introduce more such experiences in this course and other courses in the medical curriculum.
RETENTION OF BASIC SCIENCE INFORMATION BY SENIOR MEDICAL STUDENTS
Award Nominee
Dave Swanson
NBME
Philadelphia PA 19104
USA
PURPOSE
Over the past 30 years, NBME studies of the retention of basic science information have consistently shown performance declines as trainees progress through medical education. This research extends that work by analyzing patterns of performance of Step 2 and Step 3 examinees on Step 1 test items.
METHODS
244 content and statistically representative Step 1 items were rotated through unscored positions on 2008-09 Step 2 Clinical Knowledge (CK) and 2009-10 Step 3 test forms, and the performance of first-time examinees from US and Canadian schools was analyzed to identify item characteristics affecting examinee performance.
RESULTS
Across all 244 items, the mean item difficulty on Step 1 was 75.2%; on Step 2 CK, this value declined 4.4% to 70.8%; results for Step 3 were not available when this abstract was prepared, but will be available for presentation at IAMSE. Performance improvement (10%) on Step 2 CK was only observed for items written by the Behavioral Sciences Test Material Development Committee (TMDC). Performance for items written by the Pathology TMDC was similar on Step 1 and Step 2 CK, but performance was worse on Step 2 CK for all other Step 1 TMDCs, with the largest declines seen for Biochemistry (18%), Microbiology (10%), and Pharmacology (8%). Performance on items presented in the context of patient vignettes declined relatively little (3%), with larger declines observed for experimental vignettes (13%) and non-vignettes (14%). Step 2 CK performance on items concerning normal structure and function in the organ systems declined 7%, while Step 2 CK performance was better than Step 1 performance for items related to abnormal processes in the organ systems.
CONCLUSION
Shifts in examinee performance were similar to those observed in five previous NBME studies. These results were somewhat disappointing: one might anticipate that increased use of integrated basic science curricula would improve retention of basic science information. Additional research on teaching, learning and testing of traineesâ understanding of basic science is desirable.
â
USE OF A COMPETENCY TRACKING SYSTEM TO MONITOR STUDENT PERFORMANCE FROM DAY ONE: A PILOT STUDY
e-Demo
Machelle Davison & Matt Vassar
Oklahoma State University Center for Health Sciences
Tulsa OK 74107
USA
PURPOSE
The assessment of studentsâ competencies is a timely and important issue in medical education. To address this matter, we developed a competency-tracking system to evaluate performance deficiencies, information redundancy, and ineffective training methods for purposes of monitoring knowledge/skill performance and mastery. The purpose of this eDemo is to introduce and demonstrate the capabilities of our system.
METHODS
We developed a competency tracking system to align the American Osteopathic Associationâs core competencies to learning objectives, activities, and evaluations in all medical courses and required clerkships. A second year medical course piloted the use of the system in identifying students with problems early on. Key metrics (i.e. question/objective= define hypoxia, subtopic=hypoxia, topic=cell injury, competency=medical knowledge) were used to determine student deficiency in the medical knowledge competency.
RESULTS
Our competency tracking system has practical utility: (1) defining exactly what a student should look like and at what level through the curriculum, (2) internal review based on up-to-date, non-static information, (3) communication among faculty and clerkship directors who can now access the objectives, activities, and evaluation of competencies for all courses and clerkships (4) early student remediation and (5) instant reporting to administrators and accrediting bodies.
CONCLUSION
A competency tracking system can be a dynamic portal for all faculty, clerkship directors, and administrators to support consistency in the curriculum, evaluate student performance, and identify problem areas for students earlier. Our demonstration will alert other medical educators to options they may pursue with their institutions regarding the tracking of medical competencies.
DEVELOPMENT OF A COURSE EVALUATION TOOL WITH HIGHER ACCURACY AND USEFULNESS IN MEDICAL EDUCATION
Machelle Davison & Matt Vassar
Oklahoma State University Center for Health Sciences
Tulsa OK 74107
USA
PURPOSE
Student course evaluations are typically regarded as the most valid source of data in the assessment of teaching effectiveness (McKeachie, 1997). While it is estimated that almost 80% of North American medical schools utilize student course evaluation assessments, most research to date has focused on undergraduate populations as opposed to the medical school environment (Abrahams & Friedman, 1996). Medical schools have unique environments where students have little to no choice regarding the courses, the professors under whom they must study, and courses taught by multiple instructors which makes evaluation of courses different.
METHODS
A variety of research methods were used to develop a course evaluation instrument that addresses the two principal issues related to course evaluations seen in literature including the accuracy and usefulness of course evaluations (Tang, 1997). These methods included: a literature review, Q-Factor Analysis (both faculty and students), and focus groups (both faculty and students). New course questions were piloted in Fall 2008 and additional feedback was solicited from faculty and students.
RESULTS
Fourteen course questions were developed and piloted. Faculty rated each question with a high degree of clarity and accuracy. Both faculty and students indicated greater satisfaction with providing and receiving feedback for curricular improvements. Qualitative data was collected with overall positive comments.
CONCLUSIONS
Our research has resulted in a course evaluation instrument that yields better and more accurate feedback for faculty and a higher satisfaction with feedback given and received among both students and faculty.
WHATâS IN ONE MARK? TELESCOPING INTEGRATED AND ONGOING ASSESSMENTS
Tomlin Paul, Donna Beman, Joseph Branday & Elaine Williams
University of the West Indies
Kingston 7
Jamaica
PURPOSE
To determine the cumulative representative value of marks derived in an assessment of students at the end of the first three years (Stage 1) of medical school.
METHODS
A log of all of the component assessments, their value and content contribution over three years was mapped for the Stage 1 assessment in the medical school at the University of the West Indies, Jamaica. The retrospective cumulative value of a mark in the final result was determined by proportionately distributing the cumulative value of all examinations over the final 100 marks reported as the Stage 1 result. The distribution was stratified for written and practical examinations.
RESULTS
The student performance in a cumulative total of 2,252 items contributed to the final Stage 1 mark (22.5 item exposure per final mark earned). This included testing in 16 course examinations, 3 clinical clerkships, 4 OSPEs (the in-course assessments); and, two comprehensive examinations, a practical and data interpretation examination covering a wide breadth of content over 3 years (the final exam). The weightings of the in-course assessments were not equal. When the proportional representation of the assessment model was applied to the final result it was seen that one mark was represented by 9.1 items of equal value.
CONCLUSIONS
One mark in the final cumulated Stage 1 result carries substantial value in quota and breadth of knowledge. Final assessment scores in this integrated programme run over several years must consider the representational value of these scores in decision making.
VARIATIONS IN PASS/FAIL CUT-POINTS IN MULTIDISCIPLINARY UNDERGRADUATE MEDICAL EXAMINATIONS
Donna Beman, Joseph Branday, Annette Crawford-Sykes & Tomlin Paul
Faculty of Medical Sciences
Kingston
Jamaica
PURPOSE
In 2006 the decision was taken to replace the fixed pass mark system which was in place in the Stage 1 (pre-clinical) part of the undergraduate medical programme at the University of the West Indies, Jamaica with formal standard setting. This paper describes the initial experience with introducing the modified Angoff method of standard setting to the programme.
METHODS
Course assessments are comprised of multiple-choice questions primarily from the basic sciences but include laboratory sciences and clinical disciplines. The maximum possible score for each examination is 100 percent. Under the modified Angoff method of standard setting, a panel of âexpertâ judges makes a determination of the performance of borderline students in each question. The average value for all questions is the pass mark. The difference between the cut point derived by standard setting and the former fixed pass mark was computed for each of the nine examinations reviewed. Feedback on the process was obtained from judges.
RESULTS
Judgesâ assessments generally produced lower pass/fail cut-points compared with the previous fixed score (mean difference of -3.2; range -9.2 to 2.4). Six of the nine examinations had standard set cut-points within 5 marks of the fixed pass mark.
Participating judges found the process was too time-consuming and were uneasy about making judgement on questions which were not in the discipline in which they teach.
CONCLUSION
Introduction of the modified Angoff standard setting process has produced slightly lower pass marks for the majority of examinations in this series. Faculty recruited as judges have reservations about the process.
ASSESSING THE PERFORMANCE OF DISTRACTERS IN MCQ EXAMINATIONS USING A âTHEORY OF EQUAL DISTRACTIONâ
Elaine Williams, Donna Beman, Joseph Branday, Joan Leitch & Tomlin Paul
Department of Pathology
Kingston
Jamaica
PURPOSE
To assess the performance of distracters in multiple choice questions (MCQs) in the Stage 1 Undergraduate Bachelor of Medicine, Bachelor of Surgery (MB BS) examinations at the University of the West Indies Mona.
METHODS
Studentsâ performance data on 532 multiple choice questions from 8 examinations were obtained from computer grading. The observed frequency with which students chose the incorrect options for each question was determined after exclusion of very easy and very difficult questions judged by the calculated difficulty index. The probability of choosing a fully functioning distracter (expected frequency) was determined by equally distributing the number of students who got the question incorrect assuming an equal distraction model. The absolute differences between observed and expected frequencies were computed for the 1,175 distracters in the data set. The distribution, mean and median of the differences were determined. Differences approaching zero between observed and expected frequencies are deemed to be indicative of higher performing MCQs.
RESULTS
The distributions of differences between observed and expected frequencies for all examinations were positively skewed. The median difference ranged between 8.7 and 12.7 percent. Two examinations had modal differences close to zero (<2).
CONCLUSIONS
On the basis of distracter analysis, the eight examinations assessed in this study appear to have similar performance of distracters. There is room for improvement in item writing to enhance distracter functioning. The median absolute difference of the observed and expected frequencies of studentsâ choice of distracters is proposed as an indicator for monitoring the quality of MCQs.
DEMONSTRATING Q METHODOLOGY FOR MEDICAL EDUCATION
e-Demo
Matt Vassar & Machelle Davison
Oklahoma State University
Tulsa OK 74107
USA
PURPOSE
Medical education research is often concerned with assessing the attitudes or perceptions of individuals on various issues. For example, researchers might be interested in examining physiciansâ perceptions regarding evidence-based medicine or in understanding faculty attitudes of course evaluations. In such cases, subjective data are collected through a variety of means, and the researcher must determine the interpretation of the results. This presentation will introduce and demonstrate a method to evaluate such data. Q methodology is grounded in the study of human subjectivity and is designed to examine these types of attitudes and perceptions. Participants are asked to sort a series of statements that are structured to represent a broad range of ideas related to the topic. Statements are then subjected to factor analysis procedures to derive clusters of individuals with shared viewpoints. In this presentation, we present a step-by-step approach of applying the method and analyzing data.
METHODS
FlashQ is an online tool capable of collecting participant data. This free program enables researchers to collect Q-sort data online in an interactive fashion similar to in-person sorting. FlashQ will be demonstrated.
RESULTS
Results can easily be tabulated using a freeware package called PQMethod 2.11. We will demonstrate the integration of FlashQ with PQMethod and show participants the data analysis process.
CONCLUSIONS
Q Methodology is a useful approach in the assessment of attitudes or opinions. It can also be used for program evaluation purposes, and thus can address a wide array of questions.
CURRICULUM, INSTRUCTIONAL METHODS, AND EVALUATION IN MEDICAL EDUCATION: A TEN YEAR REVIEW
Matt Vassar & Machelle Davison
Oklahoma State University Center for Health Sciences
Tulsa OK 74107
USA
PURPOSE
The purpose of this study is to describe trends related to curriculum, instructional methods, and evaluation over a 10 year period across 5 prominent medical journals.
METHODS
All published articles from five prominent medical education journals over a ten year period were retrieved for analysis. These medical journals include: Academic Medicine, Journal of the International Association of Medical Science Educators, Medical Education, Medical Teacher, and Teaching and Learning in Medicine. A coding key was constructed based on specific elements related to medical curriculum, instructional methods, and evaluation. Coding is currently in progress, with each principal investigator (2 raters) independently coding one-half of all articles comprising the total sample. Prior to coding, 20 articles were randomly selected and coded by both investigators. Cohenâs kappa suggested high inter-rater agreement.
RESULTS
As our ten year review and content analysis is currently underway, finalized results cannot yet be reported. Upon completion, however, our results will describe trends in medical education over the previous 10 years related to the advancement and organization of curriculum practices, instructional approaches regarding the delivery of information, and the evaluation methods being utilized, both in and out of the classroom, for assessing mastery and fluency of such information. Attention will also be given to the incorporation and assessment of core competencies delineated by ACGME and AACOM.
CONCLUSIONS/ FUTURE DIRECTIONS
Results from this study will not only provide information regarding evolving trends in medical education but will also provide avenues for additional research and future advancements concerning these issues.