Sustainable data analysis with Snakemake

TLDR

Data analysis typically involves many heterogeneous steps, and while reproducibility is widely recognized as essential, it alone does not guarantee lasting impact or sustainability. The study proposes that adaptability and transparency are equally important, and examines the properties required for an analysis to be reproducible, adaptable, and transparent. Using Snakemake, the authors demonstrate a workflow that ensures reproducibility, adaptability, and transparency, providing an ergonomic, unified representation of all analysis steps from raw data to interactive plots.

Abstract

<ns4:p>Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid.</ns4:p><ns4:p>Here, we analyze the properties needed for a data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.</ns4:p>