A critical aspect of review is explicit effort toward achieving consensus on the basis or criteria to be used.  As an example, participants were asked to provide input regarding observable attributes they associate with “effective teaching.”  Within a few minutes, participants witnessed first-hand the wide variability of individuals’ definitions for what constitutes being effective as a teacher, and concluded that one cannot assume common understandings of criteria associated with broad categories (e.g., teaching, research, scholarship, service).  There must be colleague discussion and agreement prior to the review process.  Methods of peer review included approaches such as direct observation of teaching/learning, reviews of instructional and/or student assessment materials, course critiques, portfolio reviews, and combined approaches (e.g., direct observation, videotape, and post-observation conference or debriefing).  For each of these methods, data collection tools are selected or designed, and training of reviewers in the use of the methods and tools is necessary. 

Participants were invited to explore ways in individuals might identify appropriate peers to conduct reviews.  For example, when would it be advantageous to select a peer reviewer from within one’s own discipline and/or department and when would it be advantageous to select someone from a different discipline and/or department?  Are there certain review activities that are more appropriate for individuals in particular roles?  For example, a colleague within one’s own department/discipline might be a valuable addition to a peer review team for the purpose of direct observation of the teaching and learning process and to provide explicit input about content structure, accuracy, up-to-datedness, and clarity.  On the other hand, an external peer reviewer from another department or discipline, but at the same school, might best contribute by observing the teaching and learning process and providing input regarding the effectiveness of various teaching/learning process elements.  Still, an external reviewer from another school who is in the same content domain might be asked to review and critique instructional or student assessment materials.  This discussion led naturally to the issue of determining the number of reviewers that might be appropriate, ranging anywhere from one person to several, depending on the scope and purpose for using peer review.  The preparation of reviewers for their roles is critical for long term success, especially in terms of clarity of purpose, review criteria, consistent use of methods, measurement procedures and tools (e.g., ensuring validity and reliability).  

Among the issues associated with using peer review for faculty evaluation was the institutional support for such an enterprise, both in terms of organizational policies and expectations, and in terms of human, technical and financial resources.  Similarly, a related issue is the extent to which faculty members at an institution would value such an option.  Certainly, the participants agreed that faculty members must perceive benefits outweighing any risks.  

Peer review programs might be initiated through informal faculty focus groups, and discussions with departmental leadership, chairs of promotion and honors committees, academic deans, and faculty development directors.  Potential resources for promoting and facilitating peer review of faculty can include books and journals that report on implementation models.  In addition, faculty development and medical education research specialists and offices are rich resources and award, recognition and incentive programs, internal and external grant opportunities, and participation in professional organizations and meetings can provide incentives.

Experience and professional reports suggest that starting small, ensuring low risk for faculty members using informal or pilot activities first (e.g., formative feedback only consultations), and focus on only one aspect of a faculty members’ career responsibilities (e.g., course review or teaching methods review) might facilitate successful adoption, then implementation, and ultimately expansion, enhancement, and large-scale institutionalization. 

Participants closed the session by providing very positive feedback about the effectiveness and utility of the discussion for learning about the effective use of peer review in faculty evaluation.

(Editor's Note:  Dr. Chauvin is now at Louisiana State University where she is the Director of the Office of Medical Education Research and Development (OMERAD) and Professor in the Department of Public Health and Preventive Medicine.  She can be reached at schauv@lsuhsc.edu)