Objectives: The Objective Structured Clinical Examination (OSCE) is an assessment format commonly used to assess undergraduate nursing students. However, in spite of its prominence, relatively little research has been conducted into how OSCE assessors form judgements about student performances, and whether divergent processes of judgement formation have the potential to negatively impact the inter-rater reliability (IRR) of awarded scores. This qualitative study aimed to uncover the cognitive processes which assessors employ when assessing OSCE performances.
Design, setting, participants: In order to investigate this, a convenience, purposive sample of 12 assessors watched four videos of students completing single-station OSCEs: two videos of blood pressure measurement, and two of naso-gastric tube insertion.
Methods: Assessors were asked to "think aloud" while watching the videos, and also participated in a semi-structured interview about their assessment practices.
Results: Thematic analysis of the qualitative data revealed three themes: observation, processing, and integration. Within each theme, a number of sub-themes were identified, which explain the cognitive mechanisms used by assessors when watching, judging and grading student performances.
Conclusions: Notably, the presence of these mechanisms was not uniform across the sample, indicating that assessors utilise different approaches when viewing and interpreting the same performances. This has the potential to threaten the IRR of awarded scores, and thus the validity of decisions made on the basis of those scores.
Keywords: Assessments; Inter-rater reliability; OSCE; Reliability; Simulation; Validity.
Copyright © 2023. Published by Elsevier Ltd.