Tidy Data

TLDR

Data cleaning consumes much effort, yet research on simplifying it is scarce, while tidy datasets—structured with variables as columns, observations as rows, and each observational unit as a table—facilitate manipulation, modeling, and visualization. The paper focuses on data tidying as a key component of data cleaning. The authors propose a framework that, with a small set of tools, can tidy diverse unstructured datasets and enables the development of analysis tools that both consume and produce tidy data. A case study shows that a consistent data structure and matching tools eliminate routine data manipulation tasks.

Abstract

A huge amount of effort is spent cleaning data to get it ready for analysis, but there has been little research on how to make data cleaning as easy and effective as possible. This paper tackles a small, but important, component of data cleaning: data tidying. Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table. This framework makes it easy to tidy messy datasets because only a small set of tools are needed to deal with a wide range of un-tidy datasets. This structure also makes it easier to develop tidy tools for data analysis, tools that both input and output tidy datasets. The advantages of a consistent data structure and matching tools are demonstrated with a case study free from mundane data manipulation chores.