Agreement of Risk-of -Bias varied in systematic reviews on acupuncture and was associated with methodological quality

Abstract

Objectives: The objective of the study was to evaluate the consistency of risk of bias assessments for overlapping randomized controlled trials (RCTs) included in systematic reviews (SRs) on acupuncture.

Study design and setting: Databases were searched for acupuncture SRs. A weighted kappa (κ) statistic was calculated, and logistic regression was used to explore the factors of disagreements.

Results: We included 241 RCTs from 109 SRs on acupuncture. The percentage disagreements ranged from 25% to 44%, with moderate agreement for random sequence generation (κ = 0.57), allocation concealment (κ = 0.50), and incomplete outcome data (κ = 0.50), besides fair agreement for blinding of participants and personnel (κ = 0.44), blinding of outcome assessment (κ = 0.31), and selective reporting (κ = 0.39). Only 19% RCTs were evaluated completely consistent. Methodological quality (random sequence generation, odds ratio (OR) = 3.46), international cooperation (allocation concealment, OR = 0.14; incomplete outcome data, OR = 0.14; selective reporting, OR = 0.05), and risk of bias reporting completeness score (selective reporting, OR = 0.53) significantly affected the relative odds of disagreements.

Conclusion: The level of agreement varied from fair to moderate agreement depending on the risk of bias domain. Methodological quality appears to be an overarching factor to account for disagreements.

Keywords: Acupuncture; Agreement; Bias; Epidemiologic methods; Reproducibility; Systematic review.