Systematic review and narrative review lead experts to different cancer trial predictions: a randomized trial

Abstract

Objectives: The objective of the study was to assess the impact of narrative review (NR) vs. systematic review (SR) on expert assessments of a clinical trial.

Study design and setting: Experts in colon and rectal surgery were randomized to read an NR or SR for an ongoing clinical trial involving surgery for colorectal cancer. Experts from the United States and Canada completed online or paper surveys between December 2018 and June 2019. After reading the NR or SR, experts predicted the trial's outcome and evaluated the trial under a hypothetical ethical review.

Results: Experts who read the NR (n = 55) compared with those who read the SR (n = 56) were more likely to predict a higher absolute risk reduction, 58% vs. 33%, P = 0.018, mean predictions 10.6% vs. 6.6%, mean difference 4.0% [95% confidence interval: 0.3%, 7.8%]. Experts who read the NR were more likely to evaluate the trial more favorably under a hypothetical ethical review, 48% vs. 26%, P = 0.039, 20.0% vs. 8.9% "strongly in favor" of trial being pursued.

Conclusion: An NR and an SR for the same trial led to different judgments of likely outcomes and ethical appropriateness. These differences point to a potential source of unaddressed bias in ethical review.

Keywords: Ethical review; Evidence-based medicine; Judgment; Narrative review; Prediction; Systematic review.