The distillation method: A novel approach for analyzing randomized trials when exposure to the intervention is diluted

Abstract

Objective: To introduce a novel analytical approach for randomized controlled trials that are underpowered because of low participant enrollment or engagement.

Data sources: Reanalysis of data for 805 patients randomized as part of a pilot complex care intervention in 2015-2016 in a large delivery system. In the pilot randomized trial, only 64.6% of patients assigned to the intervention group participated.

Study design: A case study and simulation. The "Distillation Method" capitalizes on the frequently observed correlation between the probability of subjects' participation or engagement in the intervention and the magnitude of benefit they experience. The novel method involves three stages: first, it uses baseline covariates to generate predicted probabilities of participation. Next, these are used to produce nested subsamples of the randomized intervention and control groups that are more concentrated with subjects who were likely to participate/engage. Finally, for the outcomes of interest, standard statistical methods are used to re-evaluate intervention effectiveness in these concentrated subsets.

Data extraction methods: We assembled secondary data on patients who were randomized to the pilot intervention for one year prior to randomization and two follow-up years. Data included program enrollment status, membership data, demographics, utilization, costs, and clinical data.

Principal findings: Using baseline covariates only, Generalized Boosted Regression Models predicting program enrollment performed well (AUC 0.884). We then distilled the full randomized sample to increasing levels of concentration and reanalyzed program outcomes. We found statistically significant differences in outpatient utilization and emergency department utilization (both follow-up years), and in total costs (follow-up year two only) at select levels of population concentration.

Conclusions: By offering an internally valid analytic framework, the Distillation Method can increase the power to detect effects by redefining the estimand to subpopulations with higher enrollment probabilities and stronger average treatment effects while maintaining the original randomization.

Keywords: biostatistical methods; internal validity; learning health system; participant responsiveness; randomized controlled trial.