The widespread misuse of effect sizes

Abstract

Objectives: Studies comparing multiple groups (i.e., experimental and control) often examine the efficacy of an intervention by calculating within group effect sizes using Cohen's d. This method is inappropriate and largely impacted by the pre-test variability as opposed to the variability in the intervention itself. Furthermore, the percentage change is often analyzed, but this is highly impacted by the baseline values and can be potentially misleading. Thus, the objective of this study was to illustrate the common misuse of the effect size and percent change measures.

Design: Here we provide a realistic sample data set comparing two resistance training groups with the same pre-test to post-test change.

Methods: Statistical tests that are commonly performed within the literature were computed.

Results: Analyzing the within group effect size favors the control group, while the percent change favors the experimental group. The most appropriate way to present the data would be to plot the individual responses or, for larger samples, provide the mean change and 95% confidence intervals of the mean change. This details the magnitude and variability within the response to the intervention itself in units that are easily interpretable.

Conclusions: This manuscript demonstrates the common misuse of the effect size and details the importance for investigators to always report raw values, even when alternative statistics are performed.

Keywords: Cohen’s d; Muscle hypertrophy; Muscle size; Percentage change; Statistics.