Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

Maya M Jeyaraman; Reid C Robson; Leslie Copstein; Nameer Al-Yousif; Michelle Pollock; Jun Xia; Chakrapani Balijepalli; Kimberly Hofer; Samer Mansour; Mir S Fazeli; Mohammed T Ansari; Andrea C Tricco; Rasheda Rabbani; Ahmed M Abou-Setta

doi:10.1016/j.jclinepi.2021.04.017

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

J Clin Epidemiol. 2021 Aug:136:157-167. doi: 10.1016/j.jclinepi.2021.04.017. Epub 2021 May 9.

Authors

Affiliations

¹ George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada; Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, R3E 0T6, Canada. Electronic address: maya.jeyaraman@umanitoba.ca.
² Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, Ontario, M5B 1T8, Canada.
³ George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada.
⁴ Institute of Health Economics, 1200-10405 Jasper Avenue, Edmonton, Alberta, T5J 3N4, Canada.
⁵ Division of Epidemiology and Public Health, School of Medicine, University of Nottingham Medical School, Nottingham, NG7 2UH, UK; Nottingham Ningbo GRADE Centre, The University of Nottingham Ningbo, 199 East Taikang Road, Ningbo, China.
⁶ Pharmalytics Group, 422 Richards St, Suite 170, Vancouver, Canada.
⁷ Evidinno Outcomes Research Inc., 1750 Davie Street, Suites 601 & 602, Vancouver, British Columbia, V6B 2Z4, Canada.
⁸ Centre Hospitalier de l'Université de Montreal, 2900, boul. Édouard-Montpetit, Montréal (Québec) H3T 1J4, Canada; Faculty of Medicine, Department of Medicine, Université de Montréal, Roger-Gaudry Building, 2900 Edouard Montpetit Blvd, Montreal, Quebec H3T 1J4, Canada; Centre de recherche du Centre Hospitalier de l'Université de Montréal, 900 St Denis St, Montreal, Quebec H2 × 0A9, Canada.
⁹ School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Room 101, 600 Peter Morand Crescent, Ottawa, Ontario, K1G 5Z3, Canada.
¹⁰ Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, Ontario, M5B 1T8, Canada; Epidemiology Division & Institute of Health, Management, and Policy Evaluation, Dalla Lana School of Public Health, University of Toronto, 27 King's College Circle, Toronto, Ontario, M5S 1A1, Canada; Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, 92 Barrie Street, Room 214, Kingston, Ontario, K7L 3N6, Canada.
¹¹ George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada; Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, R3E 0T6, Canada.

PMID: 33979663
DOI: 10.1016/j.jclinepi.2021.04.017

Abstract

Objectives: To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE).

Study design and setting: In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC₁ statistic to calculate IRR and ICR.

Results: After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase.

Conclusion: Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

Keywords: Evaluator burden; Inter-consensus reliability; Inter-rater reliability; Non-randomized studies; ROBINS-I; RoB instrument for NRS of exposures.

Publication types

Comparative Study

MeSH terms

Adult
Biomedical Research / standards*
Biomedical Research / statistics & numerical data
Canada
Cross-Sectional Studies
Epidemiologic Research Design*
Female
Humans
Male
Middle Aged
Observer Variation*
Peer Review / standards*
Psychometrics / methods
Reproducibility of Results
Research Design / standards*
Research Design / statistics & numerical data
Research Personnel / education*
United Kingdom