Inter-rater Agreement for Abstraction of the Early Management Bundle, Severe Sepsis/Septic Shock (SEP-1) Quality Measure in a Multi-Hospital Health System

Seth R Bauer; Judith A Gonet; Rebecca F Rosario; Lori A Griffiths; Tracy Kingery; Anita J Reddy

doi:10.1016/j.jcjq.2018.10.002

Inter-rater Agreement for Abstraction of the Early Management Bundle, Severe Sepsis/Septic Shock (SEP-1) Quality Measure in a Multi-Hospital Health System

Jt Comm J Qual Patient Saf. 2019 Feb;45(2):108-111. doi: 10.1016/j.jcjq.2018.10.002. Epub 2018 Nov 30.

Authors

Seth R Bauer¹, Judith A Gonet², Rebecca F Rosario³, Lori A Griffiths⁴, Tracy Kingery⁵, Anita J Reddy⁶

Affiliations

¹ is Clinical Coordinator, Critical Care Pharmacy Services, Department of Pharmacy, Cleveland Clinic, Cleveland, Ohio, USA. Electronic address: bauers@ccf.org.
² was Abstractor (now retired), Quality & Patient Safety, Cleveland Clinic.
³ is Data Registry Manager for Core Measures, Quality & Patient Safety, Cleveland Clinic.
⁴ is Director, Quality Data Registries, Quality & Patient Safety, Cleveland Clinic.
⁵ is Abstractor, Quality & Patient Safety, Cleveland Clinic.
⁶ is Staff Physician and Quality Officer, Medical ICU, Respiratory Institute, Cleveland Clinic.

PMID: 30503812
DOI: 10.1016/j.jcjq.2018.10.002

Abstract

Background: The Early Management Bundle, Severe Sepsis/Septic Shock (SEP-1) quality measure is complex to abstract, which may lead to discrepancies between abstractors. This study was designed to evaluate inter-rater agreement between abstractors at individual hospitals in a health system and a lead abstractor on abstraction elements and measure compliance for SEP-1.

Methods: Patient cases qualifying for abstraction for SEP-1 over a four-month period in 2016 were initially abstracted at a local hospital and then centrally by a lead abstractor. Abstraction results were retrospectively compared to determine inter-rater agreement.

Results: A total of 580 SEP-1 cases were abstracted locally and centrally. Each site contributed a median (interquartile range) of 63 (49, 86) cases. There was complete concordance of measure-related elements in 391 cases (67%) (inter-rater agreement: κ = 0.40, p < 0.01). The most common discrepancy (60 cases) was severe sepsis presentation time. There was a weak correlation between SEP-1 compliance adjudicated locally and centrally (r² = 0.41, p < 0.01). The average change in monthly SEP-1 measure compliance at each site after central adjudication was a 1% increase but ranged from a 49% decrease to a 40% increase.

Conclusions: Concordance on SEP-1 abstraction elements between local and expert adjudicators was fair, and SEP-1 performance varied considerably from initial site-reported performance. The detailed nature of SEP-1 can lead to unreliable abstraction, which may lead to inaccurate reporting of compliance with the measure and affect comparability of performance between hospitals. Abstraction by a dedicated team for SEP-1 can reduce variability and improve efficiency.

Keywords: healthcare; quality improvement; quality indicators; sepsis.

MeSH terms

Centers for Medicare and Medicaid Services, U.S.
Clinical Coding / standards*
Humans
Observer Variation*
Patient Care Bundles / standards*
Quality Indicators, Health Care / standards*
Retrospective Studies
Sepsis / therapy*
Shock, Septic / therapy
United States