An exploratory test for an excess of significant findings

TLDR

The published clinical research literature may be distorted by the pursuit of statistically significant results, leading to potential excesses of significant findings. The study aims to develop an exploratory test that evaluates whether an excess of formally significant findings exists in the published literature due to biases such as publication bias, selective reporting, or fabricated data. The test estimates the expected number of statistically significant studies and compares it to the observed count, using alpha thresholds (default 0.05) and varying effect‑size priors, and is best applied across domains of many meta‑analyses sharing common characteristics. Applying the test to eight large meta‑analyses of clinical trials and ten meta‑analyses of neuroleptic agents in schizophrenia revealed an excess of significant studies in six of the eight large meta‑analyses and in the neuroleptic domain, with results differing from commonly used publication‑bias tests.

Abstract

The published clinical research literature may be distorted by the pursuit of statistically significant results.We aimed to develop a test to explore biases stemming from the pursuit of nominal statistical significance.The exploratory test evaluates whether there is a relative excess of formally significant findings in the published literature due to any reason (e.g., publication bias, selective analyses and outcome reporting, or fabricated data). The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies. The main application uses alpha = 0.05, but a range of alpha thresholds is also examined. Different values or prior distributions of the effect size are assumed. Given the typically low power (few studies per research question), the test may be best applied across domains of many meta-analyses that share common characteristics (interventions, outcomes, study populations, research environment).We evaluated illustratively eight meta-analyses of clinical trials with >50 studies each and 10 meta-analyses of clinical efficacy for neuroleptic agents in schizophrenia; the 10 meta-analyses were also examined as a composite domain. Different results were obtained against commonly used tests of publication bias. We demonstrated a clear or possible excess of significant studies in 6 of 8 large meta-analyses and in the wide domain of neuroleptic treatments.The proposed test is exploratory, may depend on prior assumptions, and should be applied cautiously.An excess of significant findings may be documented in some clinical research fields.