Purpose: The purpose of this study is to provide guidance for determining when incorporating a decision-analytic model alongside a systemic review would be of added value for decisionmaking purposes. The purpose of systematic reviews is to synthesize the current scientific literature on a particular topic in the form of evidence reports and technology assessments to assist public and private organizations in developing strategies that improve the quality of health care and decisionmaking. However, there is often not enough evidence to fully address the questions that are relevant for decisionmakers. Decision models may provide added value alongside systematic reviews by adding a formal structure, which can be informed by the evidence.
Methods: Our framework is informed by two sets of interviews and a focus group discussion; literature reviews to summarize best modeling practices and to profile the modeling literature; and an exploration of the feasibility of developing a database of published models. We interviewed Evidence-based Practice Center (EPC) members, some of whom have successfully incorporated models in EPC reports, to document lessons learned from those experiences. We also interviewed members of the U.S. Preventive Services Task Force (USPSTF) and cancer modelers who were involved in the recent efforts to use modeling with a systematic review to update USPSTF cancer screening guidelines, to evaluate the process of conducting a simultaneous systematic review and modeling exercise, and to evaluate stakeholder-perceived needs and whether needs were met. We reviewed and summarized the literature on best practices for modeling. This was supplemented by a focus group discussion with modeling experts to elicit, characterize, and precisely qualify best practices in decision and simulation modeling. These included: model formulation and characterization, model development and construction, handling and presentation of modeling assumptions, definition and presentation of parameters, outcomes to incorporate into the model, model analysis, model testing, validation, and implementation (including results presentation and communication). We developed a profile of the current modeling literature by conducting a systematic review of the medical literature and the grey literature to document publications that used a decision model and for what purpose (e.g., disease of interest, interventions evaluated). We also developed a prototype database to serve as a preliminary step in developing a resource that could be used to determine if, and how many, models exist on a particular disease of interest.
Results: The resulting report consists of six chapters. Decision and Simulation Modeling Alongside Systematic Reviews provides an overview of models and describes the differences and synergies between systematic reviews and decision analysis. In Overview of Decision Models Used in Research, we provide a “scan” of the medical literature over the past 5 years in terms of the use of models in studies that compare intervention strategies using multiple sources of data. Use of Modeling in Systematic Reviews: The EPC Perspective documents the extent to which EPCs have incorporated models into data and presents results from key informant interviews with EPC members. We present a framework for deciding when a decision model can inform decisionmaking alongside a systematic review in Suggested Framework for Deciding When a Modeling Effort Should Be Added to a Systematic Review. Potential Modeling Resources explores several possible approaches to use when undertaking a modeling effort and discusses some of the challenges. Lastly, Best Practices for Decision and Simulation Modeling reviews the literature on best practices for modeling, supplemented by a focus group discussion with modeling experts, and lessons learned about the process of conducting a modeling exercise alongside a systematic review using recent experience with the USPSTF.
Conclusions: We suggest a process for deciding when conducting a decision analysis in conjunction with a systematic review would be of value to decisionmakers.