Evaluating the Strength of Evidence in Favor of Rehabilitation Effects: A Bayesian Analysis

Abstract

Background: Relying solely on null hypothesis significance testing to investigate rehabilitation interventions may result in researchers erroneously concluding the presence of a treatment effect.

Objective: We sought to quantify the strength of evidence in favour of rehabilitation treatment effects by calculating Bayes factors (BF₁₀s) for significant findings. Additionally, we sought to examine associations between BF₁₀s, P-values, and Cohen's d effect sizes.

Methods: We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups from individual findings within 175 meta-analyses. Investigators independently classified the interventions according to the Rehabilitation Treatment Specification System. We calculated t-statistics, P-values, effect sizes, and BF₁₀s for each finding. We isolated statistically significant findings (P≤0.05); applied evidential categories to BF₁₀s, P-values, and effect sizes; and examined relationships descriptively.

Results: We analysed 1935 rehabilitation findings. Across intervention types, 25% of significant findings offered only anecdotal evidence in favour of a treatment effect; only 48% indicated strong evidence. This pattern persisted within intervention types and when conducting robustness analyses. Smaller P-values and larger effect sizes were associated with stronger evidence in favour of a treatment effect. However, a notable portion of findings with P-value 0.01 to 0.05 (63%) or a large effect size (18%) offered anecdotal evidence in favour of an effect.

Conclusions: For a substantial portion of statistically significant rehabilitation findings, the data neither support nor refute the presence of a treatment effect. This was the case among a notable portion of large treatment effects and for most findings with P-value>0.01. Rehabilitation evidence would be improved by researchers adopting more conservative levels of significance, complementing the use of null hypothesis significance testing with Bayesian techniques and reporting effect sizes.

Keywords: Bayes factor; Bayesian analysis; Meta-analysis; Meta-research; Null hypothesis significance testing; Statistical power.