Objectives: Reviewers often employ a “best evidence” approach to address the key questions, but what is meant by “best” is often unclear. The goal of this project was to create a decision framework for “best evidence” approaches in systematic reviews. This document is not intended to be prescriptive, but rather to provide a conceptual construct to enhance the transparency of inclusion decisions made during the course of a systematic review.
Review Methods: We set out to accomplish the following tasks: (1) create a list of possible inclusion criteria, and for each criterion, create a list of factors that might affect a reviewer's decision to use it, (2) create a list of evidence prioritization strategies, and (3) list the ways in which evidence prioritization strategies might be formally evaluated. In a series of conference calls, collaborators from five Evidence-based Practice Centers discussed methods for accomplishing the tasks noted above. After the initial conference call, the project leaders prepared a series of discussion documents specific to the first three tasks. Subsequent conference calls were scheduled to discuss comments and suggestions from the collaborators, whose feedback was incorporated in revisions in the task documents. The document was then externally reviewed by experts from other institutions, and revisions were made based on reviewer comments.
Results: For Task 1, we identified 21 potential inclusion criteria and 15 modifying factors a reviewer should consider when deciding which criteria to employ. The inclusion criteria were divided into three categories: criteria pertaining to study design, criteria pertaining to study conduct and reporting, and criteria pertaining to relevance. A flow chart of the decision process provides a guide to reviewers, and tables illustrate the factors influencing decisions about each inclusion criterion. For Task 2, we identified four strategies for prioritizing evidence. For Task 3, we identified a number of potential approaches that might be used to formally evaluate these strategies in the future.
Conclusion: Systematic reviewers routinely prioritize evidence in numerous ways. This paper provides a framework for understanding the possibilities, considering influential factors, and choosing among the myriad options. This will help enhance the transparency of review processes, which in turn may help users determine how different reviews of the same topic can reach different conclusions.