An umbrella review of effect size, bias, and power across meta-analyses in emergency medicine

Austin J Parish; Denley M K Yuan; Jason R Raggi; Oluyemi O Omotoso; Jason R West; John P A Ioannidis

doi:10.1111/acem.14312

An umbrella review of effect size, bias, and power across meta-analyses in emergency medicine

Acad Emerg Med. 2021 Dec;28(12):1379-1388. doi: 10.1111/acem.14312. Epub 2021 Jun 28.

Authors

Austin J Parish^{1

2}, Denley M K Yuan², Jason R Raggi², Oluyemi O Omotoso², Jason R West², John P A Ioannidis^{1

3

4

5

6}

Affiliations

¹ Meta-Research Innovation Center at Stanford (METRICS, Stanford University, Stanford, California, USA.
² Department of Emergency Medicine, Lincoln Medical Center, Bronx, New York, USA.
³ Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, California, USA.
⁴ Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, USA.
⁵ Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, USA.
⁶ Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, USA.

PMID: 34133813
DOI: 10.1111/acem.14312

Abstract

Objectives: The objective of this study was to conduct an umbrella review of therapeutic studies relevant to emergency medicine, analyzing patterns in effect size, power, and signals of potential bias across an entire field of clinical research.

Methods: We combined topic- and journal-driven searches of PubMed and Google Scholar for published articles of systematic reviews and meta-analyses (SRMA) relevant to emergency medicine (last search in November 2020). Data were screened and extracted by six investigators. Redundant meta-analyses were removed. Whenever possible for each comparison we extracted one meta-analysis on mortality with the most events and one meta-analysis on a nonmortality outcome with the most studies. From each meta-analysis we extracted all individual study effects; outcomes were converted to odds ratios (ORs) and placed on a common scale where an OR < 1.0 represents a reduction in a harmful outcome with an experimental treatment versus control. Outcomes were analyzed at the level of individual studies and at the level of summary effects across meta-analyses.

Results: A total of 332 articles contained 431 eligible meta-analyses with a total of 3,129 individual study outcomes; of these, 2,593 (83%) were from randomized controlled trials. The median OR across all studies was 0.70. Within each meta-analysis, the earliest study effect on average demonstrated larger benefit compared to the overall summary effect. Only 57 of 431 meta-analyses (13%) both favored the experimental intervention and did not show any signal of small study effects or excess significance, and of those only 12 had at least one study with 80% or higher power to detect an OR of 0.70. Of these, no interventions significantly decreased mortality in well-powered trials. Although the power of studies increased somewhat over time, the majority of studies were underpowered.

Conclusions: Few interventions studied within SRMAs relevant to emergency medicine seem to have strong and unbiased evidence for improving outcomes. The field would benefit from more optimally powered trials.

Keywords: bias; effect size; meta-analysis; meta-research; power; umbrella review.

Publication types

Meta-Analysis
Review

MeSH terms

Emergency Medicine*
Humans
Odds Ratio
Randomized Controlled Trials as Topic
Systematic Reviews as Topic