Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study

Melissa L Rethlefsen; Tara J Brigham; Carrie Price; David Moher; Lex M Bouter; Jamie J Kirkham; Sara Schroter; Maurice P Zeegers

doi:10.1016/j.jclinepi.2023.111229

Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study

J Clin Epidemiol. 2024 Feb:166:111229. doi: 10.1016/j.jclinepi.2023.111229. Epub 2023 Dec 3.

Authors

Melissa L Rethlefsen¹, Tara J Brigham², Carrie Price³, David Moher⁴, Lex M Bouter⁵, Jamie J Kirkham⁶, Sara Schroter⁷, Maurice P Zeegers⁸

Affiliations

¹ Health Sciences Library & Informatics Center, University of New Mexico, MSC 09 5100, 1 University of New Mexico, Albuquerque, NM 87131-0001, USA; Department of Epidemiology, Maastricht University, Maastricht, The Netherlands. Electronic address: mlrethlefsen@gmail.com.
² Library Services-Florida, Mayo Clinic Libraries, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224, USA.
³ Albert S. Cook Library, Towson University, 8000 York Road, Towson, MD 21252, USA.
⁴ Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, The Ottawa Hospital, General Campus, Centre for Practice Changing Research Building, 501 Smyth Road, PO BOX 201B, Ottawa, Ontario K1H 8L6, Canada.
⁵ Department of Epidemiology and Data Science, Amsterdam UMC, Vrije Universiteit Amsterdam, De Boelelaan 1089a, 1081 HV Amsterdam, The Netherlands; Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands.
⁶ Centre for Biostatistics, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
⁷ BMJ, BMA House, Tavistock Square, London WC1H 9JR, UK; Faculty of Public Health & Policy, London School of Hygiene & Tropical Medicine, Keppel Street, London WC1E 7HT, UK.
⁸ Department of Epidemiology, Maastricht University, Maastricht, The Netherlands; MBP Holding, Heerlen, The Netherlands.

PMID: 38052277
DOI: 10.1016/j.jclinepi.2023.111229

Abstract

Objectives: To determine the reproducibility of biomedical systematic review search strategies.

Study design and setting: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records.

Results: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible.

Conclusion: Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.

Keywords: Database searches; Reporting guidelines; Reproducibility; Search strategies; Systematic reviews; Transparency.

MeSH terms

Cross-Sectional Studies
Databases, Factual
MEDLINE
Reproducibility of Results
Research Design*
Systematic Reviews as Topic*