Reproducibility of individual effect sizes in meta-analyses in psychology

Esther Maassen; Marcel A L M van Assen; Michèle B Nuijten; Anton Olsson-Collentine; Jelte M Wicherts

doi:10.1371/journal.pone.0233107

Reproducibility of individual effect sizes in meta-analyses in psychology

PLoS One. 2020 May 27;15(5):e0233107. doi: 10.1371/journal.pone.0233107. eCollection 2020.

Authors

Esther Maassen¹, Marcel A L M van Assen^{1

2}, Michèle B Nuijten¹, Anton Olsson-Collentine¹, Jelte M Wicherts¹

Affiliations

¹ Department of Methodology and Statistics, Tilburg University, Tilburg, the Netherlands.
² Department of Sociology, Utrecht University, Utrecht, the Netherlands.

Abstract

To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis. Results showed that almost half (k = 224) of all sampled primary effect sizes could not be reproduced based on the reported information in the meta-analysis, mostly because of incomplete or missing information on how effect sizes from primary studies were selected and computed. Overall, this led to small discrepancies in the computation of mean effect sizes, confidence intervals and heterogeneity estimates in 13 out of 33 meta-analyses. We provide recommendations to improve transparency in the reporting of the entire meta-analytic process, including the use of preregistration, data and workflow sharing, and explicit coding practices.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Confidence Intervals
Meta-Analysis as Topic
Psychology / methods*
Reproducibility of Results

Grants and funding

This research was supported by a Consolidator Grant 726361 (IMPROVE) from the European Research Council (ERC, https://erc.europa.eu), awarded to J.M. Wicherts. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.