A number of papers by Young and collaborators have criticized epidemiological studies and meta-analyses of air pollution hazards using a graphical method that the authors call a P value plot, claiming to find zero effects, heterogeneity, and P hacking. However, the P value plot method has not been validated in a peer-reviewed publication. The aim of this study was to investigate the statistical and evidentiary properties of this method.
Methods: A simulation was developed to create studies and meta-analyses with known real effects , integrating two quantifiable conceptions of evidence from the philosophy of science literature. The simulation and analysis is publicly available and automatically reproduced.
Results: In this simulation, the plot did not provide evidence for heterogeneity or P hacking with respect to any condition. Under the right conditions, the plot can provide evidence of zero effects; but these conditions are not satisfied in any actual use by Young and collaborators.
Conclusion: The P value plot does not provide evidence to support the skeptical claims about air pollution hazards made by Young and collaborators.
Keywords: Air pollution; P hacking; P value plot; Simulation methods.
Copyright © 2022 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of The Environmental Epidemiology. All rights reserved.