Validating health economic models with the Probabilistic Analysis Check dashBOARD (PACBOARD)

Xavier G L V Pouwels; Karel Kroeze; Naomi van der Linden; Michelle M A Kip; Hendrik Koffijberg

doi:10.1016/j.jval.2024.04.008

Validating health economic models with the Probabilistic Analysis Check dashBOARD (PACBOARD)

Value Health. 2024 Apr 17:S1098-3015(24)02340-4. doi: 10.1016/j.jval.2024.04.008. Online ahead of print.

Authors

Xavier G L V Pouwels¹, Karel Kroeze², Naomi van der Linden³, Michelle M A Kip⁴, Hendrik Koffijberg⁴

Affiliations

¹ Section of Health Technology & Services Research, Technical Medical Centre, Faculty of Behavioural, Management, and Social Sciences, University of Twente, the Netherlands, Overijssel, Enschede. Electronic address: x.g.l.v.pouwels@utwente.nl.
² Behavioural Data Science incubator, Faculty of Behavioural, Management, and Social Sciences, University of Twente, the Netherlands, Overijssel, Enschede.
³ Section of Health Technology & Services Research, Technical Medical Centre, Faculty of Behavioural, Management, and Social Sciences, University of Twente, the Netherlands, Overijssel, Enschede; Institute for Health Systems Science, Faculty of Technology, Policy and Management, Delft University of Technology, the Netherlands, South Holland, Delft.
⁴ Section of Health Technology & Services Research, Technical Medical Centre, Faculty of Behavioural, Management, and Social Sciences, University of Twente, the Netherlands, Overijssel, Enschede.

PMID: 38641056
DOI: 10.1016/j.jval.2024.04.008

Abstract

Objectives: Health economic (HE) models are often considered as "black boxes" because they are not publicly available and lack transparency, which prevents independent scrutiny of HE models. Additionally, validation efforts and validation status of HE models are not systematically reported. Methods to validate HE models in absence of their full underlying code are therefore urgently needed to improve health policy making.This study aimed to develop and test a generic dashboard to systematically explore the workings of HE models and validate their model parameters and outcomes.

Methods: The Probabilistic Analysis Check dashBOARD (PACBOARD) was developed using insights from literature, health economists, and a data scientist.Functionalities of PACBOARD are 1) exploring and validating model parameters and outcomes using standardised validation tests and interactive plots, 2) visualising and investigating the relationship between model parameters and outcomes using metamodelling, and 3) predicting health economic outcomes using the fitted metamodel.To test PACBOARD, two mock HE models were developed and errors were introduced in these models, e.g. negative costs inputs, utility values exceeding 1. PACBOARD metamodelling predictions of incremental net monetary benefit were validated against the original model's outcomes.

Results: PACBOARD automatically identified all errors introduced in the erroneous HE models. Metamodel predictions were accurate compared to the original model outcomes.

Conclusions: PACBOARD is a unique dashboard aiming at improving the feasibility and transparency of validation efforts of HE models. PACBOARD allows users to explore the working of HE models using metamodelling based on HE models' parameters and outcomes.

Keywords: health economic model; metamodels/emulators; probabilistic sensitivity analysis; sensitivity analysis; validation.