A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common

Abstract

Objectives: To determine the prevalence of poor interpretation practices, such as conflating evidence of absence with absence of evidence and over-emphasis of statistical non-significance in abstract conclusions, in a sample of randomized controlled trials (RCTs) with non-statistically significant primary outcomes published after the 2016 American Statistical Association statement on the interpretation of P-values.

Design and setting: Review of 50 two-arm individually randomized superiority trials with non-statistically significant results in four high impact journals published between 2017 and 2020, to determine the proportion that conclude evidence of no impact (thus, likely conflating evidence of absence with absence of evidence) or place emphasis on statistical non-significance (technically correct but arguably uninformative) in the abstract conclusion.

Results: Of the 50 RCTs with non-statistically significant results for primary outcomes, 28 (56%) of abstract were classified as concluding there was no difference between the two treatments; 19 (38%) placed an over-emphasis on statistical significance; only one acknowledged any uncertainty and the remaining 2 (4%) concluded that one treatment was more effective. Only four studies provided any justification for a finding of no difference, for example that the confidence interval gave no support to values of importance.

Conclusions: RCTs with non-statistically significant primary outcomes almost always present their conclusion in the abstract as evidence of no impact or ambiguously as "not statistically significant" without giving due attention to values supported by the confidence interval.

Keywords: Clinical importance; Confidence intervals; Conflating absence of evidence; Misinterpretation; Non-statistically significant; P-values; Randomised controlled trials; Reporting; Treatment effects.