Reporting Quality of Discrete Event Simulations in Healthcare-Results From a Generic Reporting Checklist

Abstract

Objectives: The aims of this study were to formulate a generic reporting checklist for healthcare-related discrete event simulation (DES) studies and to critically appraise the existing studies.

Methods: Based on the principles of accessibility and generality, assessment items were derived from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-Society for Medical Decision Making (SMDM) Task Force reports. The resulting checklist was applied to all 211 DES studies identified in a previous review. The proportion of fulfilled checklist items served as an indicator of reporting quality. A logistic regression was conducted to investigate whether study characteristics (eg, publication before or after the publication of the ISPOR-SMDM reports) increased the likelihood of fulfilling more than the mean number of items fulfilled by the appraised DES studies.

Results: An 18-item checklist was formulated covering model conceptualization, parameterization and uncertainty assessment, validation, generalizability, and stakeholder involvement. The reporting quality of the DES models fluctuated around the mean of 63.7% (SD 11.0%) over the period studied. A modest nonsignificant improvement in reporting quality was found after the publication of the ISPOR-SMDM reports (64.5% vs 62.9%). Items with the lowest performance were related to predictive validation (2.8% of studies), cross validation (8.5%), face validity assessment (26.5%), and stakeholder involvement (27.5%). Models applied to health economic evaluation (HEE), country under study, and industry sponsorship were significantly associated with the odds of achieving above-average reporting quality.

Conclusions: The checklist is applicable across various model-based analyses beyond HEEs. Adherence to the ISPOR-SMDM guidelines should be improved, particularly regarding model validation.

Keywords: discrete event simulation; healthcare decision modeling; reporting quality checklist.