An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017)

TLDR

Concerns over research quality have spurred reforms aimed at improving transparency and reproducibility to enhance credibility, yet meta‑research has not fully captured the broader impact of these initiatives. This study establishes a baseline by estimating the prevalence of transparency and reproducibility indicators in social science literature from 2014–2017. The authors manually examined a random sample of 250 articles to assess the presence of these indicators. Among 250 sampled social science articles, only 11% provided materials, 0% protocols, 7% raw data, 1% scripts, none were pre‑registered, and few reported funding or conflicts; replication studies were rare, and less than half were publicly available, indicating minimal adoption of transparency practices that could undermine research credibility and efficiency.

Abstract

Serious concerns about research quality have catalyzed a number of reform initiatives intended to improve transparency and reproducibility and thus facilitate self-correction, increase efficiency, and enhance research credibility. Meta-research has evaluated the merits of some individual initiatives; however, this may not capture broader trends reflecting the cumulative contribution of these efforts. In this study, we manually examined a random sample of 250 articles in order to estimate the prevalence of a range of transparency and reproducibility-related indicators in the social sciences literature published between 2014-2017. Few articles indicated availability of materials (16/151, 11% [95% confidence interval, 7% to 16%]), protocols (0/156, 0% [0% to 1%]), raw data (11/156, 7% [2% to 13%]), or analysis scripts (2/156, 1% [0% to 3%]), and no studies were pre-registered (0/156, 0% [0% to 1%]). Some articles explicitly disclosed funding sources (or lack of; 74/236, 31% [25% to 37%]) and some declared no conflicts of interest (36/236, 15% [11% to 20%]). Replication studies were rare (2/156, 1% [0% to 3%]). Few studies were included in evidence synthesis via systematic review (17/151, 11% [7% to 16%]) or meta-analysis (2/151, 1% [0% to 3%]). Less than half the articles were publicly available (101/250, 40% [34% to 47%]). Minimal adoption of transparency and reproducibility-related research practices could be undermining the credibility and efficiency of social science research. The present study establishes a baseline that can be revisited in the future to assess progress.