Development and Validation of an Electronic Adverse Event Model for Patient Safety Surveillance in Interventional Radiology

Abstract

Background: Comprehensive adverse event (AE) surveillance programs in interventional radiology (IR) are rare. Our aim was to develop and validate a retrospective electronic surveillance model to identify outpatient IR procedures that are likely to have an AE, to support patient safety and quality improvement.

Methods: We identified outpatient IR procedures performed in the period from October 2017 to September 2019 from the Veterans Health Administration (n = 135,283) and applied electronic triggers based on posyprocedure care to flag cases with a potential AE. From the trigger-flagged cases, we randomly sampled n = 1,500 for chart review to identify AEs. We also randomly sampled n = 600 from the unflagged cases. Chart-reviewed cases were merged with patient, procedure, and facility factors to estimate a mixed-effects logistic regression model designed to predict whether an AE occurred. Using model fit and criterion validity, we determined the best predicted probability threshold to identify cases with a likely AE. We reviewed a random sample of 200 cases above the threshold and 100 cases from below the threshold from October 2019 to March 2020 (n = 20,849) for model validation.

Results: In our development sample of mostly trigger-flagged cases, 444 of 2,096 cases (21.8%) had an AE. The optimal predicted probability threshold for a likely AE from our surveillance model was >50%, with positive predictive value of 68.9%, sensitivity of 38.3%, and specificity of 95.3%. In validation, chart-reviewed cases with AE probability >50% had a positive predictive value of 63% (n = 203). For the period from October 2017 to March 2020, the model identified approximately 70 IR cases per month that were likely to have an AE.

Conclusions: This electronic trigger-based approach to AE surveillance could be used for patient-safety reporting and quality review.

Keywords: Adverse event detection; US Department of Veterans Affairs; data methods; data mining; interventional; patient safety; radiology.