An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)

TLDR

Concerns over research quality have spurred reforms aimed at improving transparency and reproducibility to enable self‑correction, increase efficiency, and enhance credibility, yet meta‑research has not fully captured broader trends in these efforts. The study aimed to estimate the prevalence of transparency and reproducibility indicators in social‑science literature from 2014 to 2017. The authors manually examined a random sample of 250 articles to achieve this. They found that only 11% of articles provided materials, 0% protocols, 7% raw data, 1% scripts, none were pre‑registered, and overall transparency practices were minimal, with less than half of the articles publicly available, indicating that such low adoption may undermine the credibility and efficiency of social‑science research.

Abstract

Serious concerns about research quality have catalysed a number of reform initiatives intended to improve transparency and reproducibility and thus facilitate self-correction, increase efficiency and enhance research credibility. Meta-research has evaluated the merits of some individual initiatives; however, this may not capture broader trends reflecting the cumulative contribution of these efforts. In this study, we manually examined a random sample of 250 articles in order to estimate the prevalence of a range of transparency and reproducibility-related indicators in the social sciences literature published between 2014 and 2017. Few articles indicated availability of materials (16/151, 11% [95% confidence interval, 7% to 16%]), protocols (0/156, 0% [0% to 1%]), raw data (11/156, 7% [2% to 13%]) or analysis scripts (2/156, 1% [0% to 3%]), and no studies were pre-registered (0/156, 0% [0% to 1%]). Some articles explicitly disclosed funding sources (or lack of; 74/236, 31% [25% to 37%]) and some declared no conflicts of interest (36/236, 15% [11% to 20%]). Replication studies were rare (2/156, 1% [0% to 3%]). Few studies were included in evidence synthesis via systematic review (17/151, 11% [7% to 16%]) or meta-analysis (2/151, 1% [0% to 3%]). Less than half the articles were publicly available (101/250, 40% [34% to 47%]). Minimal adoption of transparency and reproducibility-related research practices could be undermining the credibility and efficiency of social science research. The present study establishes a baseline that can be revisited in the future to assess progress.