The following abstracts have been accepted for presentation at this meeting in Poster format in the category of Assessment.  Those selected for Oral Poster Presentations are so designated in the far right column.
 
 

A1	IS PROBLEM-BASED LEARNING ACHIEVING ITS GOALS? LEARNING ISSUE WRITE-UPS AS A WINDOW ON STUDENT PERFORMANCE H.M. Amerongen*, Ph.D., J. Arnold, B.A., S.C. Ellis, M.A., Ed.S., and J.L.Keller, Ph.D.  Department of Cell Biology and Anatomy & Division of Academic Resources, College of Medicine, University of Arizona, Tucson, Arizona 85724 U.S.A.  Students at the University of Arizona College of Medicine work with clinical cases in a problem-based learning (PBL) format as a small part of the first-year, first-semester Gross Anatomy, and Histology and Cell Biology courses. The learning goals of PBL in this setting are fourfold: (1) to increase information gathering and reporting skills; (2) to increase critical thinking skills; (3) to apply basic science content covered in lectures to clinical problems; and (4) to develop teamwork skills. We examined students’ achievement of the first three of these goals by analyzing students’ written learning issue reports. We developed three grading rubrics and used them to assess information gathering, critical thinking and use of basic science information in each of three PBL cases throughout the semester. Our data suggest students improved on information gathering and reporting over the course of the semester. In contrast, students in general demonstrated poor to moderate critical thinking skills initially, and did not improve. With regard to the use of basic science information in clinical contexts, we found the majority of students focused exclusively on clinical aspects (e.g., symptoms and diagnosis) without demonstrating understanding of the basic science underpinnings of the case. Hence, two of the three goals examined are not being met. These data provide useful information on PBL performance that has not been obtainable from facilitators’ student evaluations. There is in fact no correlation (-.01) between scores on facilitator evaluations and rubric scores from learning issue write-ups. This information is being used to adjust our PBL format. We expect this will make PBL a more effective tool for teaching critical thinking and the application of basic science to clinical problems.	ORAL
A2	ASSESSING AN INSTRUMENT FOR THE PEER EVALUATION OF TEACHERS ON AN INTERNAL MEDICINE HOSPITAL SERVICE T.J. Beckman MD1*, M.C. Lee MD1, C.H Rohren MD1, V.S. Pankratz2 PhD. 1 Department of Internal Medicine, Mayo Clinic, Rochester, MN, 55906,  2 Department of Health Sciences Research, Mayo Clinic, Rochester, MN, 55906  U.S.A. Objective:  To determine the validity and reliability of an instrument for the peer evaluation of teachers on the Mayo General Internal Medicine hospital services. Methods:  The Stanford Teaching Evaluation Form (SFPD-26) has been validated for the student and resident evaluation of faculty.  The Mayo Teaching Evaluation Form (MTEF) is an instrument we developed for the peer assessment of clinical teaching.  Our instrument is based on the Stanford educational framework, which consists of the following seven components: 1) learning climate (LC), 2) control of teaching session (CS), 3) communication of goals (CG), 4) understanding and retention (UR), 5) evaluation (EV), 6) feedback (FB), and 7) self-directed learning (SDL).  The MTEF is a 25-item, Likert-scaled instrument utilizing items from the SFPD-26, the Mayo resident electronic evaluation form, and three additional items.  In this study, three physician-evaluators used the MTEF to evaluate clinical teaching performances of ten Consultants on the General Internal Medicine hospital services.  Calculating Chronbach’s alphas and Kappa scores determined the validity and reliability of the MTEF. Results: The MTEF is valid across all three evaluators (Chronbach’s alpha = 0.894).  In our educational setting, the most valid Stanford categories are SDL (alpha = 0.907), LC (alpha = 0.851), CG (alpha = 0.828), and EV (alpha = 0.710).  The least valid Stanford categories are FB (alpha = .648), UR (alpha = 0.570), and CS (alpha = -0.147).  The MTEF also contains at least one reliable item from each of the seven Stanford categories except CG and FB.  The categories of EV and SDL contained the largest numbers of reliable items. Conclusions:  In this limited sample, the MTEF appears valid for the peer assessment of inpatient teaching on Mayo General Internal Medicine hospital services.  The MTEF also contains reliable items for most of the Stanford educational categories.  Hence, the MTEF may be a useful tool for the peer assessment of inpatient teaching at our institution.	ORAL
A3	REDESIGNING A FIRST-YEAR BASIC SCIENCE COURSE FOR LEARNING AND ASSESSING IN A COMPETENCY-BASED EDUCATION CURRICULUM Pamela L. Derstine, Ph.D.*, Loyola University Chicago Stritch School of Medicine, Maywood, IL 60153  U.S.A. An essential principle of Competency-Based Education (CBE) is the use of observable outcomes with assessments as judgments of competence based on defined criteria.  Basic Science faculty are accustomed to using Learning Objectives as the defining criteria for knowledge, assessing students using written exams.  They are less familiar with how the principles of CBE are applied to other competencies.  We recently adopted school-wide UGME goals and objectives, modeled after the ACGME Competencies.  Our approach to redesigning this course was to:  (1) identify current course components containing elements other than knowledge outcomes; (2) develop course goals based on observable outcomes; (3) design appropriate assessment tools; (4) incorporate reflection as the basis for learning and self growth. The current daily course format for Cell and Molecular Biology consists of lecture, followed by student-centered small group problem solving sessions (SGPSS), ending with whole class, faculty-led “recap” problem discussion.1  Previous course goals were that students will: (1) gain an understanding of the principles and concepts of cell and molecular biology; (2) develop an appreciation for how these principles and concepts are important to medicine; (3) demonstrate an ability to think critically using these principles and concepts. Goal 1 was assessed by written exams. We assumed Goals 2 and 3 were met through SGPSS, and outcomes were not assessed.  New course goals were written for competency in Medical Knowledge and Lifelong Learning and in Communication and Professionalism.  This poster will present the details of the change process outlined above for implementing the redesigned course (developing revised goals, designing new assessment tools, creating faculty development opportunities) and on the reflective Learning Portfolios.  Details for the implementation of learning and assessing outcomes in Lifelong Learning2 and Communication and Professionalism3 are presented in two other posters. 1.  Derstine, P.L., (1996) ”Maximizing student-centered learning”, Academic Medicine 71 (5): 538. 2.  Derstine, P.L. (2002) “Learning and assessing lifelong-learning outcomes in a first-year basic science course: a competency-based genetics education project” IAMSE 6th Annual Meeting. 3.  Derstine, P.L. (2002) “Incorporating communication and professionalism competency learning and assessment in a first-year basic science course” IAMSE 6th Annual Meeting.
A4	LEARNING AND ASSESSING LIFELONG-LEARNING OUTCOMES IN A FIRST-YEAR BASIC SCIENCE COURSE: A COMPETENCY-BASED GENETICS EDUCATION PROJECT Pamela L. Derstine, Ph.D.*, Loyola University Chicago Stritch School of Medicine, Maywood, IL 60153  U.S.A. Our recent LCME self-study revealed deficiencies both in genetics content and opportunities for students to develop lifelong learning skills.  In June 2001 we adopted a set of UGME goals and objectives related to the ACGME Outcomes Project, but we have little first-hand experience designing courses and measuring outcomes based on competencies other than Medical Knowledge.  Our curriculum features learning in case-based small groups, accompanied by lectures.  The objectives of this project were to incorporate genetics while minimally increasing lecture time and give faculty and students experience in a competency-based education approach to learning and assessment.  Cell and Molecular Biology is a learner-centered 9-week block course in semester one1. Course goals were rewritten to address multiple competencies and an independent learning project based on the archived online lecture series "Frontiers in Clinical Genetics" (www.frontiersingenetics.com/calendar99/calendar99.htm) was designed.  This project required individuals to identify a learning issue from the group-viewed lecture, use online searches to research the topic, write a critical analysis, and work with small group members to prepare a cohesive 30-minute conference presentation combining all individual topics, using PowerPoint slides.  Conferences were videotaped to permit self-assessment by presenters.  Faculty did not supervise the project, but evaluated papers and, along with other attendees, evaluated presentations.  Three 4-point Likert scale assessment tools containing specific criteria for the paper, presentation and group planning process were developed.  Written formative feedback was solicited in each form.  The numeric assessment data were combined with written exam scores to determine the Medical Knowledge and Lifelong Learning Competency course grade. The project design allowed effective assessment of specific outcomes in knowledge, lifelong learning, communication, and professionalism competencies and could be easily adapted to other schools and curriculum formats.  Planned improvements include: refining assessment criteria, integrating FCG content with other course content, providing a suggested project time-line, post-conference debriefing of each group with faculty. 1. Derstine, P. L. (1996), ”Maximizing student-centered learning”, Academic Medicine 71 (5): 538.	ORAL
A5	INCORPORATING COMMUNICATION AND PROFESSIONALISM COMPETENCY LEARNING AND ASSESSMENT IN A FIRST-YEAR BASIC SCIENCE COURSE Pamela L. Derstine, Ph.D.*, Loyola University Chicago Stritch School of Medicine, Maywood, IL 60153 U.S.A.  Few medical basic science courses expect or assess development of abilities other than knowledge.  Communication skills and professionalism are commonly associated with the clinical domain.  Pre-clinical "doctoring" courses usually focus on doctor-patient interactions when dealing with these competencies; assessment of interactions among peers and team members is largely ignored.  Acquiring knowledge and skills in a highly competitive environment and in the absence of nurturing non-cognitive competencies encompassing attitudes and behaviors throughout the curriculum may be linked to dysfunctional behaviors in later practice.1 The objective for this project was to give faculty and students first-hand experience in competency-based learning and assessment in these affective domains within a basic science course.  Cell and Molecular Biology is a learner-centered 9-week block course in semester one. Grading is standards-based, rather than normative, to promote cooperative learning. Each day begins with lecture, followed by a learner-centered small group problem solving session (SGPSS), concluding with whole-class discussion of the problems.  Non-content-expert faculty facilitate the small group process but do not lead or overtly participate in discussions.  A 19-item, 5-point Likert scale tool was used by students to self- and peer-assess course goals for communication and professionalism multiple times within SGPSSs.  Facilitators used the tool to observe students daily, give formative feedback midcourse, summative feedback at course conclusion, and competency grades.  Frequent use of the tool enabled students and faculty to develop a shared understanding of knowledge, skills, attitudes and behaviors for communication/professionalism within the course, and to document improvement in abilities.  Student and faculty post-course feedback indicated widespread acceptance of combining cognitive and non-cognitive competency outcomes in a basic science course.  This approach would be most successful when groups have the same facilitator for the entire semester or course.  Planned improvements include refining assessment criteria, simplifying the SGPSS tool and providing additional faculty development opportunities for more consistent use of assessment tools. 1.  Wear, D. and Bickel, J., Eds. (2000). Educating for Professionalism, University of Iowa Press.
A6	INTEGRATING BASIC SCIENCES INTO PERFORMANCE-BASED CLINICAL ASSESSMENT ACROSS A FOUR-YEAR MEDICAL SCHOOL CURRICULUM Linda Morrison, Mary Aiello, Amber Barnhart, Peter Borgia, Terri Cameron, Linda Distlehorst, Sharon Hull, Tracy Lower, John Tomkowiak and Hurley Myers*, Southern Illinois University School of Medicine. Carbondale, IL 62901, U.S.A. Southern Illinois University School of Medicine (SIU) implemented a new undergraduate medical curriculum in August 2000. This new curriculum, which is a small group, problem-based program, was developed around eight guiding principles, one of which stated that “all curricular events should be evaluated and that such evaluations should be diverse in style and performance-based.” This directive created a new challenge for faculty, particularly basic science faculty who now had to assess basic science knowledge in a performance-based manner. To meet this need, faculty created a variety of different assessment instruments. One of these instruments, the Clinical Competency Exam (CCX), was adopted for use in all curricular units for each of the four years because it allowed faculty to assess most of the required evaluation categories (e.g., basic content knowledge, clinical skills, clinical reasoning, interpersonal skills, and professionalism) with one assessment tool. The CCX exam consists of three major parts. First, students perform an H&P on a standardized patient. Second, they access CCX software, where they document their findings, enter hypotheses, order and interpret labs, make a diagnosis, and recommend a plan for the patient’s management. Third, they answer a series of related basic science and clinical content knowledge questions (i.e., MCQ, T/F, or short answer), which are either inserted at various points throughout the computer exam or given to students as a “pencil and paper” exam after they have completed the computer part of CCX. In year one, for example, students are given one formative (practice) CCX and one summative CCX in each of three first year curricular units (i.e., a total of six CCX exams). Basic science content knowledge is assessed with a pencil and paper exam administered immediately following CCX and graded separately. In years two, three and four, content knowledge is tested using questions that are given at various points throughout the computer portion of each CCX exam.  Data from all CCX exams, whether collected during a curricular unit, throughout one year, or over multiple years, are imported into a Multi-Case Performance Utility, where faculty can view a student’s patterns of strengths and/or weaknesses in each of the key evaluation categories, including basic content knowledge. Promotion committees and the senior exam CCX committee at SIU use these findings to tailor feedback to students and/or customize remediation programs based on a student’s individual patterns of weakness.  Findings have also helped faculty identify areas of strengths and weaknesses in this new curriculum.	ORAL
A7	REINFORCING FACILITATOR SKILLS THROUGH CRITICAL REFLECTION Jack R. Scott, Ed.D., M.P.H.*; School of Medicine; University of Texas Medical Branch, Galveston, TX 77555-0664 U.S.A.  The University of Texas Medical Branch has conducted PBL facilitator training since initiation of the Integrated Medical Curriculum (IMC) in 1995. To date, 425 faculty members (Basic Science and Clinical Educator) have been trained in small group process skills. The Introductory course provides basic skills in establishing a positive learning environment, guiding the group reasoning process, gaining commitment, promoting self-directed learning, encouraging interactive problem-solving, and monitoring student performance. However, individual peer coaching has not been afforded on a voluntary basis to allow faculty reflection on their facilitation within the basic science courses. Peer coaching session, although brief, are focused upon group management, difficulties encountered, strategies for improvement, generation of personal theories/principles, and reinforcement of positive actions. Several faculty requested videotaped segments for peer assessment of their group performance. Videotapes are reviewed with the peer coach for self-assessment and quality improvements. Thus far, both basic and clinical scientists (n= 37) have rated these coaching sessions helpful in advancing their group facilitation abilities. Brookfield's (1995) model of Critical Reflection is used as a pedagogical approach to uncover or challenge assumptions that shape a deliberate, individual theory. Therefore, it is our intent to continue to identify basic science faculty needs with data that enhances their best practices in collaborative PBL facilitation.
A8	CRITERIA FOR EVALUATION IN A HORIZONTAL INTEGRATED PROBLEM-BASED LEARNING J.V. Bareng, M.D.*, University of the East Ramon Magsaysay Memorial Medical Center, Aurora Blvd., Quezon City 1113, Philippines Level I is an integration of the first year medical subjects: Anatomy, Biochemistry, Physiology, Epidemiology, Microbiology - Parasitology (normal flora of the different region of the body), and healthy child (embryology and pediatric parameters).  At the end of each module, the student’s performance is assessed based on the following criteria:   Tutorial grade 25% Long Examination 25% Final Examination 30% Practical Examination (anatomy) 10% Laboratory Examination (physiology) 0.5% Reports  0.5%   100%  The tutorial grade covers the assessment of the cog native, psychomotor and attitudinal domain.  Evaluating the knowledge of the students plays the greatest role in both tutorial grade and examination.