Meta-analysis prediction intervals are under reported in sport and exercise medicine

Abstract

Aim: Prediction intervals are a useful measure of uncertainty for meta-analyses that capture the likely effect size of a new (similar) study based on the included studies. In comparison, confidence intervals reflect the uncertainty around the point estimate but provide an incomplete summary of the underlying heterogeneity in the meta-analysis. This study aimed to estimate (i) the proportion of meta-analysis studies that report a prediction interval in sports medicine; and (ii) the proportion of studies with a discrepancy between the reported confidence interval and a calculated prediction interval.

Methods: We screened, at random, 1500 meta-analysis studies published between 2012 and 2022 in highly ranked sports medicine and medical journals. Articles that used a random effect meta-analysis model were included in the study. We randomly selected one meta-analysis from each article to extract data from, which included the number of estimates, the pooled effect, and the confidence and prediction interval.

Results: Of the 1500 articles screened, 866 (514 from sports medicine) used a random effect model. The probability of a prediction interval being reported in sports medicine was 1.7% (95% CI = 0.9%, 3.3%). In medicine the probability was 3.9% (95% CI = 2.4%, 6.6%). A prediction interval was able to be calculated for 220 sports medicine studies. For 60% of these studies, there was a discrepancy in study findings between the reported confidence interval and the calculated prediction interval. Prediction intervals were 3.4 times wider than confidence intervals.

Conclusion: Very few meta-analyses report prediction intervals and hence are prone to missing the impact of between-study heterogeneity on the overall conclusions. The widespread misinterpretation of random effect meta-analyses could mean that potentially harmful treatments, or those lacking a sufficient evidence base, are being used in practice. Authors, reviewers, and editors should be aware of the importance of prediction intervals.

Keywords: confidence interval; forest plot; heterogeneity; random effects model; review.