Objectives: No guidelines exist in the approach that Evidence Review Groups (ERGs) should take to appraise search methodologies in the manufacturer's submission (MS) in Single Technology Appraisals (STA). As a result, ERGs are left to appraise searches using their own approach. This study investigates the limitations of manufacturers' search methodologies as critiqued by ergs in published sta reports and to provide a recommended checklist.
Methods: Limitations from search critiques in 83 ERG reports published in the NIHR Web site between 2006 and May 2011 were extracted. The limitations were grouped into themes. Comparisons were made between limitations reported in the clinical effectiveness versus cost-effectiveness searches.
Results: Twelve themes were identified, six relating to the search strategy, source, limits, filters, translation, reporting, and missing studies. The search strategy theme contained the most limitations. Missing studies were frequently found by the ERG group in the clinical effectiveness searches. The omission of searches by manufacturers for unpublished and ongoing trials was frequently reported by the ERG. By contrast, failure of the manufacturer to report strategies was the most common limitation in the cost-effectiveness searches. Themes with the most frequent limitations in both types of searches are search strategy, reporting and source.
Conclusions: It is recommended that a checklist that has reporting, source and search strategy elements be used in the appraisal of manufacturer's searches during the STA process.