Purpose: The aim of this study was to investigate to what extent ratings of tutor performance remain stable in the long term. At many schools, teaching performance is assessed and these evaluations are consulted as part of the decision-making process for promotion, tenure, and salary. Since this information may have summative value, it is crucial that the reliability of the data be assessed. A previous study had shown that a single evaluation of a tutor is reliable when the responses of six students are used (interrater reliability). The present study focused on the stability of tutor evaluations over repeated occasions of evaluation.
Method: A generalizability study was conducted to estimate the number of occasions required to demonstrate stability. The study took place during three academic years (1992-93, 1993-94, and 1994-95) at the problem-based medical school of the University of Limburg (now Maastricht University). A total of 291 ratings were analyzed (97 tutors rated during three sequential tutoring occasions). Two types of scores were used: an aggregate score calculated from ratings of 13 items and an overall judgment.
Results: The results indicate that when the scores are used to interpret the precision of individual scores, two evaluation occasions should be available for the overall judgment and four occasions for the aggregate score. If the tutor scores are consulted only to determine whether performances are above or below a cutoff score, a reliable decision can be made after only a single occasion of evaluation.
Conclusion: The results demonstrate that data collected over an extended period of time can be reliably used as part of the decision-making process for promotion, salary, and tenure.