Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

Yoon Duk Hong; Jeroen P Jansen; John Guerino; Marc L Berger; William Crown; Wim G Goettsch; C Daniel Mullins; Richard J Willke; Lucinda S Orsini

doi:10.1186/s12916-021-02176-1

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

BMC Med. 2021 Dec 6;19(1):307. doi: 10.1186/s12916-021-02176-1.

Authors

Yoon Duk Hong¹, Jeroen P Jansen^{2

3}, John Guerino⁴, Marc L Berger⁵, William Crown⁶, Wim G Goettsch^{7

8}, C Daniel Mullins⁹, Richard J Willke¹⁰, Lucinda S Orsini¹¹

Affiliations

¹ University of Maryland School of Pharmacy, Baltimore, MD, USA. yhon039@umaryland.edu.
² Department of Clinical Pharmacy, School of Pharmacy, University of California-San Francisco, San Francisco, CA, USA.
³ PrecisionHEOR, Oakland, CA, USA.
⁴ Ipsos Healthcare, Parsippany, NJ, USA.
⁵ Independent Consultant, New York, NY, USA.
⁶ The Heller School for Social Policy and Management, Brandeis University, Waltham, MA, USA.
⁷ Utrecht Centre of Pharmaceutical Policy, Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht University, Utrecht, The Netherlands.
⁸ National Health Care Institute, Diemen, The Netherlands.
⁹ University of Maryland School of Pharmacy, Baltimore, MD, USA.
¹⁰ ISPOR-The Professional Society for Health Economics and Outcomes Research, Lawrenceville, NJ, USA.
¹¹ COMPASS Pathways plc, London, UK.

Abstract

Background: There have been ongoing efforts to understand when and how data from observational studies can be applied to clinical and regulatory decision making. The objective of this review was to assess the comparability of relative treatment effects of pharmaceuticals from observational studies and randomized controlled trials (RCTs).

Methods: We searched PubMed and Embase for systematic literature reviews published between January 1, 1990, and January 31, 2020, that reported relative treatment effects of pharmaceuticals from both observational studies and RCTs. We extracted pooled relative effect estimates from observational studies and RCTs for each outcome, intervention-comparator, or indication assessed in the reviews. We calculated the ratio of the relative effect estimate from observational studies over that from RCTs, along with the corresponding 95% confidence interval (CI) for each pair of pooled RCT and observational study estimates, and we evaluated the consistency in relative treatment effects.

Results: Thirty systematic reviews across 7 therapeutic areas were identified from the literature. We analyzed 74 pairs of pooled relative effect estimates from RCTs and observational studies from 29 reviews. There was no statistically significant difference (based on the 95% CI) in relative effect estimates between RCTs and observational studies in 79.7% of pairs. There was an extreme difference (ratio < 0.7 or > 1.43) in 43.2% of pairs, and, in 17.6% of pairs, there was a significant difference and the estimates pointed in opposite directions.

Conclusions: Overall, our review shows that while there is no significant difference in the relative risk ratios between the majority of RCTs and observational studies compared, there is significant variation in about 20% of comparisons. The source of this variation should be the subject of further inquiry to elucidate how much of the variation is due to differences in patient populations versus biased estimates arising from issues with study design or analytical/statistical methods.

Keywords: Observational data; Pharmaceuticals; Real-world evidence.

Publication types

Review

MeSH terms

Humans
Observational Studies as Topic
Pharmaceutical Preparations*
Randomized Controlled Trials as Topic
Research Design*

Substances

Pharmaceutical Preparations