Indirect evidence of reporting biases was found in a survey of medical research studies

Abstract

Objectives: To explore indirect evidence of reporting biases by examining the distribution of P-values reported in published medical articles and to compare P-values distributions across different contexts.

Study design and setting: We selected a random sample (N = 1,500) of articles published in PubMed in March 2014. We extracted information on study type, design, medical discipline, and P-values for the first reported outcome and primary outcome (if specified) from each article. We plotted the P-values transformed to the z-score scale and used caliper tests to investigate threshold effects.

Results: Out of the 1,500 randomly selected records, 758 (50.5%) were included. We retrieved or calculated 758 P-values for first reported outcomes and 389 for primary outcomes (specified in only 51% of included studies). The first reported and the primary outcome differed in 28% (110/389) of the included studies. The distributions of P-values for first reported outcomes and primary outcomes showed a notable discontinuity at the common thresholds of statistical significance (P-value = 0.05 and P-value = 0.01). We also found marked discontinuities in the distributions of z-scores across various medical disciplines, study designs, and types.

Conclusion: Reporting biases are still common in medical research. We discuss their implications, strategies to detect them, and recommended practices to avoid them.

Keywords: Bias; Methodology; Publication bias; Reporting bias; p-curve; p-hacking.