Accuracy of commercial reporting systems to monitor quality of care in burns

Abstract

Introduction: Capse Healthcare Knowledge Systems (CHKS) is a global commercial organisation that operates health benchmarking programmes in the UK and internationally. In absence of a specialty-specific quality monitoring programme for burn services, CHKS has been producing comparative quality data for burn services for a number of years. The major quality indicator reported by CHKS is mortality as a Risk Adjusted Mortality Index (RAMI). The accuracy of RAMI is unknown in comparison to published burn-specific mortality prediction models.

Methods: A retrospective study design was used to collect data for patients admitted to the Adult Burn Service at University Hospital South Manchester (UHSM) between January 2006 and December 2010. Data was collected from two sources, CHKS and Manchester Burn Injury Database (MBID). The demographic and injury characteristics of survivors and non-survivors were compared and Receiver Operator Curve (ROC), equivalence and non-inferiority analyses were used to assess accuracy of RAMI in comparison to Abbreviated Burn Severity Index (ABSI), Belgian Outcome of Burn Injury (BOBI) score, Baux score (Baux) and McGwin score (McGwin).

Results: The accuracy of RAMI to discriminate between survivors and non-survivors (area under curve=0.79, 95% CI 0.50-0.81) was significantly inferior to that of ABSI, BOBI, Baux and McGwin scores. Equivalence and non-inferiority testing of ROC curves also showed RAMI score to be inferior to ABSI, BOBI, Baux and McGwin scores at 5% significance level.

Conclusion: CHKS RAMI provides an inaccurate and inferior monitoring of mortality as a quality indicator in burn patients compared to burn specific mortality prediction models. This study raises concerns about the ability of commercially reported systems to accurately monitor quality indicators of relevance to burn care.

Keywords: Burn; Mortality; Outcome; Quality indicator(s); Risk-adjustment.