Randomized controlled trials published on patients with hernias have a high percentage of unreproducible statistics

Abstract

Introduction: Surgeons rely on randomized controlled trials (RCT) to compare the effectiveness of treatments. RCTs require careful planning and substantial effort to complete. Because of the careful study design, statistics performed are often easy to reproduce such as Chi-squared or t-test. Issues such as statistical discordance, or reporting statistical results that cannot be reproduced, should be uncommon.

Methods: RCTs pertaining to hernias were identified in PubMed using the search terms "hernia" and "randomized controlled trial." Studies were selected using a random number generator. Studies were included if the primary outcome could be reproduced using the data and statistical test reported in the manuscript. Discordance between the obtained p-value from our analysis and the published p-value was assessed. Primary outcome was the number of studies that reported p-values that crossed the level of statistical significance (p-value = 0.05) but on reproduction analysis did not.

Results: Of the 100 included RCTs, five reported p-values that crossed the "p = 0.05" threshold that our team was unable to reproduce using the statistical test reported in the manuscript. An additional three studies reported p-values that crossed the "p = 0.05" threshold that our team was unable to reproduce using the appropriate statistical test (i.e., Fisher's exact test when all expected cell counts < 5). All eight studies published p-values < 0.05, whereas, our re-analysis demonstrated p ≥ 0.05.

Conclusion: Eight percent of the RCTs analyzed in this study reported p-values < 0.05 that on reproduction analysis was ≥ 0.05. The next steps should be to determine reasons for discordance and how to prevent this from happening.

Keywords: Discordance; Hernia; Randomized controlled trials; Statistics; p-value.