Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews

Abstract

Objectives: The aim of the study was to analyze adequacy of risk of bias (RoB) judgments for selective reporting in Cochrane systematic reviews.

Study design and setting: We extracted RoB assessments, including judgment (low, high, or unclear risk) and supporting comment from Cochrane reviews of randomized controlled trials using computer parser. We analyzed sources of information mentioned in supporting comments. We compared judgments of Cochrane authors with guidance from the Cochrane Handbook for Systematic Reviews of Interventions (Cochrane Handbook) and categorized them into adequate or inadequate.

Results: At least 60% of judgments for risk of selective reporting bias of trials in analyzed Cochrane reviews were not in line with the Cochrane Handbook. Few Cochrane authors mentioned the trial protocol as a source of data for assessing selective reporting. Most of the inadequate judgments were made among trials that were judged with low risk of selective reporting bias; more than 90%. In 9% of analyzed RoB tables, Cochrane authors did not use this RoB domain at all.

Conclusion: Cochrane authors frequently make RoB judgments about selective reporting that are not in line with Cochrane Handbook and not mentioning trial protocol. Interventions aimed at helping Cochrane authors to make adequate RoB assessments in Cochrane reviews would be beneficial.

Keywords: Bias; Cochrane; Epidemiologic methods; Reporting bias; Selective reporting; Systematic reviews.