A Bayesian analysis of non-significant rehabilitation findings: Evaluating the evidence in favour of truly absent treatment effects

Abstract

Background: Relying solely on null hypothesis significance testing (NHST) to investigate rehabilitation interventions may result in researchers erroneously concluding the absence of a treatment effect.

Objective: We aimed to distinguish between truly null treatment effects and data that are insensitive to detecting treatment effects by calculating Bayes factors (BF₀₁s) for non-significant findings in the rehabilitation literature. Additionally, to examine associations between BF₀₁, sample size, and observed P-values.

Method: We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that clearly evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups for individual findings within 175 meta-analyses. Two independent investigators classified the interventions into 4 categories using the Rehabilitation Treatment Specification System. We calculated t-statistics and associated P-values for each finding in order to extract non-significant results (P>0.05). We calculated BF₀₁s for 5790 non-significant results and classified BF₀₁s based on the strength of evidence in favour of the null hypothesis (i.e., anecdotal, moderate, and strong) across and within intervention types. We examined correlations between BF₀₁, sample size, and P-values across and within intervention types.

Results: Across all intervention types, most (71.9%) findings were deemed anecdotal, and this pattern remained within distinct intervention types (58.4-76.0%). Larger sample sizes tended to be associated with greater strength in favour of the null hypothesis, both across and within intervention types. Larger P-values were not associated with greater strength in favour of the null hypothesis; this finding was present both across and within intervention types.

Conclusion: Our findings indicate that most non-significant rehabilitation findings are unable to distinguish between the true absence of a treatment effect and data that are merely insensitive to detecting a treatment effect. Findings also suggest that rehabilitation researchers may improve the strength of their statistical conclusions by increasing sample size and that Bayes factors may offer unique benefits relative to P-values.

Keywords: Bayes factor; Bayesian analysis; Meta-analysis; Meta-research; Null hypothesis significance testing; Statistical power.