GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

Holger J Schünemann; Carlos Cuello; Elie A Akl; Reem A Mustafa; Jörg J Meerpohl; Kris Thayer; Rebecca L Morgan; Gerald Gartlehner; Regina Kunz; S Vittal Katikireddi; Jonathan Sterne; Julian Pt Higgins; Gordon Guyatt; GRADE Working Group

doi:10.1016/j.jclinepi.2018.01.012

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

J Clin Epidemiol. 2019 Jul:111:105-114. doi: 10.1016/j.jclinepi.2018.01.012. Epub 2018 Feb 9.

Authors

Affiliations

¹ Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada. Electronic address: schuneh@mcmaster.ca.
² Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada.
³ Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada; AUB GRADE Center, Clinical Research Institute, American University of Beirut, PO Box 11-0236, Riad El Solh, Beirut, 1107 2020, Lebanon.
⁴ Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada; Department of Medicine, University of Kansas Medical Center, 3901 Rainbow Blvd, MS3002, Kansas City, KS, 66160, USA.
⁵ Cochrane Germany, Medical Center University of Freiburg, Breisacher Strasse 153, Freiburg, 79110, Germany.
⁶ Integrated Risk Information System (IRIS) Division, National Center for Environmental Assessment, Environmental Protection Agency, 1200 Pennsylvania Avenue, N.W.Washington, DC 20460, USA.
⁷ Department for Evidence-Based Medicine and Clinical Epidemiology, Danube University Krems, Dr Karl Dorrek Straße 30, Krems, 3500, Austria.
⁸ Basel Institute of Clinical Epidemiology, University Hospital Basel, Hebelstrasse 10, Basel, 4031, Switzerland.
⁹ MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, Top Floor, 200 Renfield Street, Glasgow, G2 3QB, Scotland.
¹⁰ Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK.
¹¹ Department of Health Research Methods, Evidence, and Impact and McGRADE Center, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S4K1, Canada.

Abstract

Objective: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

Study design and setting: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

Results: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

Conclusions: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

Keywords: Certainty of the evidence; GRADE; Nonrandomized studies; Quality of evidence; ROBINS; Risk of bias.

MeSH terms

Bias*
Clinical Trials as Topic / standards
Evidence-Based Medicine / methods
Evidence-Based Medicine / standards*
Humans
Observational Studies as Topic / standards
Practice Guidelines as Topic* / standards
Risk Assessment
Risk Factors
Systematic Reviews as Topic
Uncertainty

Abstract

MeSH terms

Grants and funding