Replicator degrees of freedom allow publication of misleading failures to replicate

TLDR

Psychology has increasingly conducted large‑scale replication tests, and the debate examined here typifies many recent replication studies, indicating broad relevance. The study demonstrates that replicator degrees of freedom enable researchers to easily produce and publish false‑negative replication results while claiming methodological rigor. By analyzing data from the debate, the authors illustrate how flexibility in experimental design and data analysis can make a true finding appear unreproducible. The analysis shows that replicator degrees of freedom and flexible design/analysis practices readily produce false‑negative replication results, that current remedies such as preregistration are insufficient, and it outlines implications for stronger methodological standards.

Abstract

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that “replicator degrees of freedom” make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate that we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field’s standard remedies, including preregistration. Implications for how the field could develop more effective methodological standards for replication are discussed.