Interpretation of epidemiologic studies very often lacked adequate consideration of confounding

Lars G Hemkens; Hannah Ewald; Florian Naudet; Aviv Ladanie; Jonathan G Shaw; Gautam Sajeev; John P A Ioannidis

doi:10.1016/j.jclinepi.2017.09.013

Interpretation of epidemiologic studies very often lacked adequate consideration of confounding

J Clin Epidemiol. 2018 Jan:93:94-102. doi: 10.1016/j.jclinepi.2017.09.013. Epub 2017 Sep 21.

Authors

Lars G Hemkens¹, Hannah Ewald², Florian Naudet³, Aviv Ladanie², Jonathan G Shaw⁴, Gautam Sajeev⁵, John P A Ioannidis⁶

Affiliations

¹ Department of Clincial Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, Basel, Switzerland; Department of Medicine, Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA.
² Department of Clincial Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, Basel, Switzerland.
³ Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, USA.
⁴ Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA, USA.
⁵ Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA, USA.
⁶ Department of Medicine, Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA. Electronic address: jioannid@stanford.edu.

PMID: 28943377
DOI: 10.1016/j.jclinepi.2017.09.013

Abstract

Background and objective: Confounding bias is a most pervasive threat to validity of observational epidemiologic research. We assessed whether authors of observational epidemiologic studies consider confounding bias when interpreting the findings.

Study design and setting: We randomly selected 120 cohort or case-control studies published in 2011 and 2012 by the general medical, epidemiologic, and specialty journals with the highest impact factors. We used Web of Science to assess citation metrics through January 2017.

Results: Sixty-eight studies (56.7%, 95% confidence interval: 47.8-65.5%) mentioned "confounding" in the Abstract or Discussion sections, another 20 (16.7%; 10.0-23.3%) alluded to it, and there was no mention or allusion at all in 32 studies (26.7%; 18.8-34.6%). Authors often acknowledged that for specific confounders, there was no adjustment (34 studies; 28.3%) or deem it possible or likely that confounding affected their main findings (29 studies; 24.2%). However, only two studies (1.7%; 0-4.0%) specifically used the words "caution" or "cautious" for the interpretation because of confounding-related reasons and eventually only four studies (3.3%; 0.1-6.5%) had limitations related to confounding or any other bias in their Conclusions. Studies mentioning that the findings were possibly or likely affected by confounding were more frequently cited than studies with a statement that findings were unlikely affected (median 6.3 vs. 4.0 citations per year, P = 0.04).

Conclusions: Many observational studies lack satisfactory discussion of confounding bias. Even when confounding bias is mentioned, authors are typically confident that it is rather irrelevant to their findings and they rarely call for cautious interpretation. More careful acknowledgment of possible impact of confounding is not associated with lower citation impact.

Keywords: Bias; Bibliometrics; Confounding; Observational studies; Research reporting.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Bias
Case-Control Studies
Cohort Studies
Confounding Factors, Epidemiologic*
Humans
Journal Impact Factor
Observational Studies as Topic / standards*