Workplace-based assessments of junior doctors: do scores predict training difficulties?

Abstract

Objectives: Workplace-based assessment (WPBA) is an increasingly important part of postgraduate medical training and its results may be used as evidence of professional competence. This study evaluates the ability of WPBA to distinguish UK Foundation Programme (FP) doctors with training difficulties and its effectiveness as a surrogate marker for deficiencies in professional competence.

Methods: We conducted a retrospective observational study using anonymised records for 1646 trainees in a single UK postgraduate deanery. Data for WPBAs conducted from August 2005 to April 2009 were extracted from the e-portfolio database. These data included all scores submitted by trainees in FP years 1 and 2 on mini-clinical evaluation exercise (mini-CEX), case-based discussion (CbD), direct observation of procedural skills (DOPS) and mini-peer assessment tool (mini-PAT) assessments. Records of trainees in difficulty, as identified by their educational supervisors, were tagged as index cases. Main outcome measures were odds ratios (ORs) for associations between mean WPBA scores and training difficulties. Further analyses by the reported aetiology of the training difficulty (health-, conduct- or performance-related) were performed.

Results: Of the 1646 trainees, 92 had been identified as being in difficulty. Mean CbD and mini-CEX scores were lower for trainees in difficulty and an association was found between identified training difficulties and average scores on the mini-CEX (OR = 0.54; p = 0.034) and CbD (OR = 0.39; p = 0.002). A receiver operator characteristic curve analysis of mean WPBA scores for diagnosing 'in difficulty' status yielded an area under the curve of 0.64, indicating weak predictive value. There was no statistical evidence that mean scores on DOPS and mini-PAT assessments differed between the two groups.

Conclusions: Analysis of a large dataset of WPBA scores revealed significant associations between training difficulties and lower mean scores on both the mini-CEX and CbD. Models show that using WPBA scores is, however, not a valid way of screening for trainees in difficulty. Workplace-based assessments have value as formative assessments that prompt supervision, feedback and reflection. They should not be relied upon to certify competence and their use for such ends may reduce their effectiveness in training. Their results should be interpreted in the context of multiple other methods of assessment, with the aim of achieving a genuinely holistic and representative assessment of professional competence.