Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity

Michael O'Donnell; Amelia S Dev; Stephen Antonoplis; Stephen M Baum; Arianna H Benedetti; N Derek Brown; Belinda Carrillo; Andrew L Choi; Paul Connor; Kristin Donnelly; Monica E Ellwood-Lowe; Ruthe Foushee; Rachel Jansen; Shoshana N Jarvis; Ryan Lundell-Creagh; Joseph M Ocampo; Gold N Okafor; Zahra Rahmani Azad; Michael Rosenblum; Derek Schatz; Daniel H Stein; Yilu Wang; Don A Moore; Leif D Nelson

doi:10.1073/pnas.2103313118

Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity

Proc Natl Acad Sci U S A. 2021 Nov 2;118(44):e2103313118. doi: 10.1073/pnas.2103313118.

Authors

Michael O'Donnell¹, Amelia S Dev², Stephen Antonoplis³, Stephen M Baum⁴, Arianna H Benedetti⁵, N Derek Brown⁴, Belinda Carrillo³, Andrew L Choi⁴, Paul Connor⁶, Kristin Donnelly⁴, Monica E Ellwood-Lowe³, Ruthe Foushee⁷, Rachel Jansen³, Shoshana N Jarvis⁴, Ryan Lundell-Creagh³, Joseph M Ocampo³, Gold N Okafor³, Zahra Rahmani Azad⁸, Michael Rosenblum⁹, Derek Schatz¹⁰, Daniel H Stein⁴, Yilu Wang¹¹, Don A Moore⁴, Leif D Nelson⁴

Affiliations

¹ McDonough School of Business, Georgetown University, Washington, DC 20057; michael.odonnell@georgetown.edu.
² Department of Psychology, University of Miami, Miami, FL 33146.
³ Berkeley Department of Psychology, University of California, Berkeley, CA 94720.
⁴ Berkeley Haas School of Business, University of California, Berkeley, CA 94720.
⁵ Airbnb, San Francisco, CA 94103.
⁶ College of Arts and Sciences, New York University, New York, NY 10003.
⁷ Department of Psychology, University of Chicago, Chicago, IL 60637.
⁸ University of Tübingen 72074 Tübingen, Germany.
⁹ Stern School of Business, New York University, New York, NY 10012.
¹⁰ McKinsey & Company, Chicago, IL 60601.
¹¹ School of Psychological and Cognitive Sciences, Peking University, Beijing, China 100871.

Abstract

Empirical audit and review is an approach to assessing the evidentiary value of a research area. It involves identifying a topic and selecting a cross-section of studies for replication. We apply the method to research on the psychological consequences of scarcity. Starting with the papers citing a seminal publication in the field, we conducted replications of 20 studies that evaluate the role of scarcity priming in pain sensitivity, resource allocation, materialism, and many other domains. There was considerable variability in the replicability, with some strong successes and other undeniable failures. Empirical audit and review does not attempt to assign an overall replication rate for a heterogeneous field, but rather facilitates researchers seeking to incorporate strength of evidence as they refine theories and plan new investigations in the research area. This method allows for an integration of qualitative and quantitative approaches to review and enables the growth of a cumulative science.

Keywords: evidentiary value; meta-analysis; open science; reproducibility; scarcity.

Publication types

Meta-Analysis
Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, Non-P.H.S.
Review

MeSH terms

Empirical Research*
Food Insecurity
Humans
Pain Measurement
Reproducibility of Results*
Research Design
Resource Allocation