Are large randomised controlled trials in severe sepsis and septic shock statistically disadvantaged by repeated inadvertent underestimates of required sample size?

Abstract

Objectives: We sought to understand why randomised controlled trials in septic shock have failed to demonstrate effectiveness in the face of improving overall outcomes for patients and seemingly promising results of early phase trials of interventions.

Design: We performed a retrospective analysis of large critical care trials of severe sepsis and septic shock. Data were collected from the primary trial manuscripts, prepublished statistical plans or by direct communication with corresponding authors.

Setting: Critical care randomised control trials in severe sepsis and septic shock.

Participants: 14 619 patients randomised in 13 trials published between 2005 and 2015, enrolling greater than 500 patients and powered to a primary outcome of mortality.

Intervention: Multiple interventions including the evaluation of treatment strategies and novel therapeutics.

Primary and secondary outcome measures: Our primary outcome measure was the difference between the anticipated and actual control arm mortality. Secondary analysis examined the actual effect size and the anticipated effect size employed in sample size calculation.

Results: In this post hoc analysis of 13 trials with 14 619 patients randomised, we highlight a global tendency to overestimate control arm mortality in estimating sample size (absolute difference 9.8%, 95% CI -14.7% to -5.0%, p<0.001). When we compared anticipated and actual effect size of a treatment, there was also a substantial overestimation in proposed values (absolute difference 7.4%, 95% CI -9.0% to -5.8%, p<0.0001).

Conclusions: An interpretation of our results is that trials are consistently underpowered in the planning phase by employing erroneous variables to calculate a satisfactory sample size. Our analysis cannot establish if, given a larger sample size, a trial would have had a positive result. It is disappointing so many promising phase II results have not translated into durable phase III outcomes. It is possible that our current framework has biased us towards discounting potentially life-saving treatments.

Keywords: clinical trials; randomized control trials; sample size calculation; sepsis; septic shock.