Background: Personal statements and referees' reports are widely used on medical school application forms, particularly in the UK, to assess the suitability of candidates for a career in medicine. However there are few studies which assess the validity of such information for predicting unhappiness or dissatisfaction with a career in medicine. Here we combine data from a long-term prospective study of medical student selection and training, with an experimental approach in which a large number of assessors used a paired comparison technique to predict outcome.
Methods: Data from a large-scale prospective study of students applying to UK medical schools in 1990 were used to identify 40 pairs of doctors, matched by sex, for whom personal statements and referees' reports were available, and who in a 2002/3 follow-up study, one pair member was very satisfied and the other very dissatisfied with medicine as a career. In 2005, 96 assessors, who were experienced medical school selectors, doctors, medical students or psychology students, used information from the doctors' original applications to judge which member of each pair of doctors was the happier, more satisfied doctor.
Results: None of the groups of assessors were significantly different from chance expectations in using applicants' personal statements and the referees' reports to predict actual future satisfaction or dissatisfaction, the distribution being similar to binomial expectations. However judgements of pairs of application forms from pairs of doctors showed a non-binomial distribution, indicating consensus among assessors as to which doctor would be the happy doctor (although the consensus was wrong in half the cases). Assessors taking longer to do the task concurred more. Consensus judgements seem mainly to be based on referees' predictions of academic achievement (even though academic achievement is not actually a valid predictor of happiness or satisfaction).
Conclusion: Although widely used in medical student selection to assess motivation, interest and commitment to a medical career, the personal statement and the referee's report cannot validly be used by assessors, including experienced medical school selectors, to identify doctors who will subsequently be dissatisfied with a medical career.