Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden

Maya M Jeyaraman; Rasheda Rabbani; Leslie Copstein; Reid C Robson; Nameer Al-Yousif; Michelle Pollock; Jun Xia; Chakrapani Balijepalli; Kimberly Hofer; Samer Mansour; Mir S Fazeli; Mohammed T Ansari; Andrea C Tricco; Ahmed M Abou-Setta

doi:10.1016/j.jclinepi.2020.09.033

Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden

J Clin Epidemiol. 2020 Dec:128:140-147. doi: 10.1016/j.jclinepi.2020.09.033. Epub 2020 Sep 25.

Authors

Affiliations

¹ George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba R3E 0T6, Canada; Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba R3E 0T6, Canada. Electronic address: maya.jeyaraman@umanitoba.ca.
² George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba R3E 0T6, Canada; Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba R3E 0T6, Canada.
³ George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba R3E 0T6, Canada.
⁴ Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, Ontario M5B 1T8, Canada.
⁵ Institute of Health Economics, 1200-10405 Jasper Avenue, Edmonton, Alberta T5J 3N4, Canada.
⁶ Nottingham Ningbo GRADE Centre, The University of Nottingham, Ningbo, 199 East Taikang Road, Ningbo, China; Division of Epidemiology and Public Health, School of Medicine, University of Nottingham Medical School, Nottingham, NG7 2UH, UK.
⁷ Pharmalytics Group, 422 Richards St, Suite 170, Vancouver, Canada.
⁸ Centre Hospitalier de l'Université de Montreal, 1000, St Denis St, Montreal, Quebec H2X 0C1, Canada; Faculty of Medicine, Department of Medicine, Université de Montréal, Roger-Gaudry Building, 2900 Edouard Montpetit Blvd, Montreal, Quebec H3T 1J4, Canada; Centre de recherche du Centre, Hospitalier de l'Université de Montréal, 900 St Denis St, Montreal, Quebec H2X 0A9, Canada.
⁹ Evidinno Outcomes Research Inc., 1750 Davie Street, Suites 601 & 602, Vancouver, British Columbia V6G 1W3, Canada.
¹⁰ School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Room 101, 600 Peter Morand Crescent, Ottawa, Ontario, K1G 5Z3, Canada.
¹¹ Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, Ontario M5B 1T8, Canada; Epidemiology Division, Dalla Lana School of Public Health & Institute of Health, Management, and Policy Evaluation, University of Toronto, 27 King's College Circle, Toronto, Ontario M5S 1A1, Canada; Queen's Collaboration for Health Care Quality, Joanna Briggs Institute Centre of Excellence, Queen's University, 92 Barrie Street, Room 214, Kingston, Ontario, K7L 3N6, Canada.

PMID: 32987166
DOI: 10.1016/j.jclinepi.2020.09.033

Abstract

Objective: To assess the real-world interrater reliability (IRR), interconsensus reliability (ICR), and evaluator burden of the Risk of Bias (RoB) in Nonrandomized Studies (NRS) of Interventions (ROBINS-I), and the ROB Instrument for NRS of Exposures (ROB-NRSE) tools.

Study design and setting: A six-center cross-sectional study with seven reviewers (2 reviewer pairs) assessing the RoB using ROBINS-I (n = 44 NRS) or ROB-NRSE (n = 44 NRS). We used Gwet's AC₁ statistic to calculate the IRR and ICR. To measure the evaluator burden, we assessed the total time taken to apply the tool and reach a consensus.

Results: For ROBINS-I, both IRR and ICR for individual domains ranged from poor to substantial agreement. IRR and ICR on overall RoB were poor. The evaluator burden was 48.45 min (95% CI 45.61 to 51.29). For ROB-NRSE, the IRR and ICR for the majority of domains were poor, while the rest ranged from fair to perfect agreement. IRR and ICR on overall RoB were slight and poor, respectively. The evaluator burden was 36.98 min (95% CI 34.80 to 39.16).

Conclusions: We found both tools to have low reliability, although ROBINS-I was slightly higher. Measures to increase agreement between raters (e.g., detailed training, supportive guidance material) may improve reliability and decrease evaluator burden.

Keywords: Evaluator burden; Interconsensus reliability; Interrater reliability; Nonrandomized studies; ROBINS-I; RoB instrument for NRS of exposures.

MeSH terms

Bias
Consensus*
Cross-Sectional Studies
Epidemiologic Research Design*
Humans
Observer Variation
Reproducibility of Results
Research Personnel / statistics & numerical data*
Risk Assessment