Predictability and stability testing to assess clinical decision instrument performance for children after blunt torso trauma

Aaron E Kornblith; Chandan Singh; Gabriel Devlin; Newton Addo; Christian J Streck; James F Holmes; Nathan Kuppermann; Jacqueline Grupp-Phelan; Jeffrey Fineman; Atul J Butte; Bin Yu

doi:10.1371/journal.pdig.0000076

Predictability and stability testing to assess clinical decision instrument performance for children after blunt torso trauma

PLOS Digit Health. 2022 Aug 8;1(8):e0000076. doi: 10.1371/journal.pdig.0000076. eCollection 2022 Aug.

Authors

Aaron E Kornblith^{1

2}, Chandan Singh³, Gabriel Devlin², Newton Addo¹, Christian J Streck⁴, James F Holmes⁵, Nathan Kuppermann^{5

6}, Jacqueline Grupp-Phelan^{1

2}, Jeffrey Fineman², Atul J Butte⁷, Bin Yu^{3

8}

Affiliations

¹ Department of Emergency Medicine, University of California, San Francisco, San Francisco, United States of America.
² Department of Pediatrics, University of California, San Francisco, San Francisco, United States of America.
³ Department of Electrical Engineering & Computer Science, University of California, Berkeley, Berkeley, United States of America.
⁴ Department of Surgery, Medical University of South Carolina, Children's Hospital, Charleston, United States of America.
⁵ Department of Emergency Medicine, University of California, Davis, Davis, United States of America.
⁶ Department of Pediatrics, University of California, Davis, Davis, United States of America.
⁷ Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, United States of America.
⁸ Departments of Statistics, University of California, Berkeley, Berkeley, United States of America.

Abstract

Objective: The Pediatric Emergency Care Applied Research Network (PECARN) has developed a clinical-decision instrument (CDI) to identify children at very low risk of intra-abdominal injury. However, the CDI has not been externally validated. We sought to vet the PECARN CDI with the Predictability Computability Stability (PCS) data science framework, potentially increasing its chance of a successful external validation.

Materials & methods: We performed a secondary analysis of two prospectively collected datasets: PECARN (12,044 children from 20 emergency departments) and an independent external validation dataset from the Pediatric Surgical Research Collaborative (PedSRC; 2,188 children from 14 emergency departments). We used PCS to reanalyze the original PECARN CDI along with new interpretable PCS CDIs developed using the PECARN dataset. External validation was then measured on the PedSRC dataset.

Results: Three predictor variables (abdominal wall trauma, Glasgow Coma Scale Score <14, and abdominal tenderness) were found to be stable. A CDI using only these three variables would achieve lower sensitivity than the original PECARN CDI with seven variables on internal PECARN validation but achieve the same performance on external PedSRC validation (sensitivity 96.8% and specificity 44%). Using only these variables, we developed a PCS CDI which had a lower sensitivity than the original PECARN CDI on internal PECARN validation but performed the same on external PedSRC validation (sensitivity 96.8% and specificity 44%).

Conclusion: The PCS data science framework vetted the PECARN CDI and its constituent predictor variables prior to external validation. We found that the 3 stable predictor variables represented all of the PECARN CDI's predictive performance on independent external validation. The PCS framework offers a less resource-intensive method than prospective validation to vet CDIs before external validation. We also found that the PECARN CDI will generalize well to new populations and should be prospectively externally validated. The PCS framework offers a potential strategy to increase the chance of a successful (costly) prospective validation.

Copyright: © 2022 Kornblith et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Grants and funding

This work was supported in part by NSF TRIPODS Grant 1740855, DMS-1613002, 1953191, 2015341, IIS 1741340, ONR grant N00014-17-1-2176. Moreover, this work is supported in part by the Center for Science of Information (CSoI), an NSF Science and Technology Center, under grant agreement CCF-0939370. This project was supported in part by the National Center for Advancing Translational Sciences, National Institutes of Health, through UCSF-CTSI Grant Number UL1 TR001872. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.