Information technology is changing the way we evaluate courses and faculty. Moving evaluations online introduces new and sometimes complex problems that must be addressed by any effective electronic evaluation system:

1) Authentication - The system must have a way to identify users. Online systems generally use passwords, but there are other alternatives. All password based systems are vulnerable.  Longer, more complex passwords are not necessarily more secure.  Password management is a non-trivial task in most systems.

2) Authorization - Once a user authenticates, the system must decide if they are authorized to perform the requested operation(s).  Access control may occur at the level of individuals, groups, or roles. Access management is also a non-trivial task in most settings. Limit the number of (trusted!) individuals who can create, view, or change passwords. Require some form of ID. Allow users to request new passwords for any reason at any time. Make accounts “inactive” or remove them when users leave.

3) Anonymity - It is often desirable (and sometimes required) that evaluation data be collected anonymously. Anonymity must coexist with authentication and authorization. This may be difficult. This must be balanced against the need to authenticate and authorize individuals to submit data. Threats to anonymity are system administrators, submission records, free text entry which must be read by an administrator, and the small “N” problem. Credibility must be maintained. Avoid partial truths, make things explicit and use a one-way process that separates authentication data from submission data.

4) Submission Management - The submission process requires constant care and attention. Forms must be created and become available at the correct times, and disappear when they are no longer required. Participants must be aware when submissions are due. Administrators must be able to monitor the process in “real time” when possible. Make it easy to delegate responsibility and automate repetitive tasks. Data from other sources (including paper forms) must be integrated, and data must be archived.

5) Report Generation - Once data have accumulated in the system, they must be analyzed and reported. A well designed system will automate this process while providing a degree of flexibility. This flexibility should include direct access to raw data (with proper authorization) for specialized analysis and external use. Types of data to consider: scored (aka a test or quiz); scaled (no “correct” answer); survey (structured information); and free text.  Mean and standard deviation are frequently reported, but these values are misleading. Use the median, not the mean. Show the quartiles above and below the median value and show the extreme high and low values.  With the help of a biostatistician we have developed an alternative way to present these data. We measure the “distance” from the expected performance and report only statistically significant values. Be careful not to create the appearance of meaning where there is none.


In this IAMSE webcast audio seminar, Dr. Rathe described the criteria for electronic evaluations and related his experiences in designing and implementing the online evaluation of courses, lecturers and students.  He also included difficulty index and point biserial as an important measure. A discussion of student compliance was also part of the presentation.  The system he designed is available to other schools at http://evalsuite.medinfo.ufl.edu.

View Slides from this Presentation