Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies

Dawid Pieper; Livia Puljak; Marien González-Lorenzo; Silvia Minozzi

doi:10.1016/j.jclinepi.2018.12.004

Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies

J Clin Epidemiol. 2019 Apr:108:26-33. doi: 10.1016/j.jclinepi.2018.12.004. Epub 2018 Dec 10.

Authors

Dawid Pieper¹, Livia Puljak², Marien González-Lorenzo³, Silvia Minozzi⁴

Affiliations

¹ Institute for Research in Operative Medicine, Evidence-based Health Services Research, Faculty of Health, School of Medicine, Witten/Herdecke University, Ostmerheimer Str, Cologne 200 51109, Germany. Electronic address: Dawid.pieper@uni-wh.de.
² Catholic University of Croatia, Ilica 242, Zagreb 10000, Croatia.
³ Department of Biomedical Sciences, Humanitas University, Milan, Italy; IBD Center, Humanitas Clinical and Research Center, Milan, Italy.
⁴ Department of Biomedical Sciences for Health, University of Milan, Milan, Italy.

PMID: 30543911
DOI: 10.1016/j.jclinepi.2018.12.004

Abstract

Objective: To compare A Measurement Tool to Assess Systematic Reviews (AMSTAR 2) with a tool to assess risk of bias in systematic reviews (ROBIS) in terms of validity, reliability, and applicability.

Study design and setting: We analyzed 30 systematic reviews (SRs) that included randomized and nonrandomized studies, with Cochrane and non-Cochrane SRs sampled in 1:1 ratio. Four reviewers assessed independently all 30 SRs with AMSTAR 2, followed by ROBIS. We calculated Fleiss' Kappa as a measure of inter-rater reliability (IRR) across 4 raters.

Results: The IRR for scoring the overall confidence in the SRs with AMSTAR 2 and the overall domain in ROBIS was fair (AMSTAR 2: κ = 0.30, 95% [confidence interval] CI: 0.17 to 0.43; ROBIS: κ = 0.28, 95% CI: 0.13 to 0.42). AMSTAR 2 confidence in review ratings strongly correlated with the overall domain rating in ROBIS (Spearman r_s = 0.84). Mean time for scoring AMSTAR 2 was slightly higher than for ROBIS (18 vs. 16 min), with huge differences between the reviewers.

Conclusion: Both AMSTAR 2 and ROBIS can be applied to SRs including both randomized controlled trials (RCTs) and non-RCTs. Measurement properties of ROBIS seemed not to be much different when comparing with other studies that include only SRs of RCTs.

Keywords: AMSTAR; AMSTAR 2; Methodological quality; ROBIS; Risk of bias; Systematic reviews.

MeSH terms

Bias
Clinical Trials as Topic*
Data Interpretation, Statistical
Evidence-Based Medicine*
Humans
Observer Variation
Randomized Controlled Trials as Topic*
Reproducibility of Results
Systematic Reviews as Topic*