The bias for statistical significance in sport and exercise medicine

Abstract

Objectives: We aimed to examine the bias for statistical significance using published confidence intervals in sport and exercise medicine research.

Design: Observational study.

Methods: The abstracts of 48,390 articles, published in 18 sports and exercise medicine journals between 2002 and 2022, were searched using a validated text-mining algorithm that identified and extracted ratio confidence intervals (odds, hazard, and risk ratios). The algorithm identified 1744 abstracts that included ratio confidence intervals, from which 4484 intervals were extracted. After excluding ineligible intervals, the analysis used 3819 intervals, reported as 95 % confidence intervals, from 1599 articles. The cumulative distributions of lower and upper confidence limits were plotted to identify any abnormal patterns, particularly around a ratio of 1 (the null hypothesis). The distributions were compared to those from unbiased reference data, which was not subjected to p-hacking or publication bias. A bias for statistical significance was further investigated using a histogram plot of z-values calculated from the extracted 95 % confidence intervals.

Results: There was a marked change in the cumulative distribution of lower and upper bound intervals just over and just under a ratio of 1. The bias for statistical significance was also clear in a stark under-representation of z-values between -1.96 and +1.96, corresponding to p-values above 0.05.

Conclusions: There was an excess of published research with statistically significant results just below the standard significance threshold of 0.05, which is indicative of publication bias. Transparent research practices, including the use of registered reports, are needed to reduce the bias in published research.

Keywords: Confidence interval; Estimation; Misconduct; Publication bias; Selective reporting; p-hacking.