Inter-rater agreement in pPOSSUM scores of geriatric trauma patients: a prospective evaluation

Abstract

Purpose: Risk prediction models are widely used in the perioperative setting to identify high-risk patients who may benefit from additional care and to aid clinical decision-making. pPOSSUM is such a prediction model, however, little is known about the inter-rater agreement when scoring subjective parameters. This study assessed the inter-rater agreement between clinicians of different specialties and work-level when scoring 30 clinical case reports of geriatric hip fracture patients with pPOSSUM.

Methods: Eighteen clinicians of the department of Surgery (three specialists, four residents), Anaesthesiology (four specialists, two residents) and Emergency Medicine (three specialists, two residents) who were familiar with the pPOSSUM scoring system were asked to calculate the scores. The kappa statistic and the statistical method of Fleiss were used to analyse inter-rater agreement.

Results: The response rate was 100%. Among surgeons, Anaesthesiologists and Emergency department doctors (ED), the overall mean kappa values were 0.42, 0.08 and 0.20, respectively. Among surgery, anaesthesiology and ED residents the overall mean kappa values were 0.21, 0.33 and 0.37, respectively. Within the department of Surgery, Anaesthesiology and Emergency Medicine the overall mean kappa values were 0.23, 0.12 and 0.22, respectively. An overall mean kappa value of 0.19 was seen among all specialists. All residents had an overall mean kappa value of 0.21 and all clinicians had an overall mean kappa value of 0.21.

Conclusion: The overall inter-rater agreement of clinicians and interdisciplinary agreement when scoring geriatric hip fracture patients with pPOSSUM was low and prone to subjectivity in our study. A higher work-experience level did not lead to better agreement. When pPOSSUM is calculated without clinical assessment by the same clinician, caution is advised to prevent over-reliance on the pPOSSUM risk prediction model.

Level of evidence: III.

Keywords: Geriatric trauma care; Inter-rater agreement; Observer variation; Predicting tool; pPossum.