Automating large-scale data quality verification

TLDR

Modern companies rely on data for every decision, yet missing or incorrect information compromises downstream processes, making data quality verification essential but tedious. The authors introduce a scalable system that automates data quality verification to meet production use‑case requirements. Their platform offers a declarative API that combines built‑in constraints with user‑defined checks, executes the validation as Spark aggregation queries, supports incremental validation, and employs machine‑learning techniques for constraint suggestions, predictability estimation, and anomaly detection. Experimental evaluation on diverse datasets demonstrates the system’s feasibility and performance in real‑world scenarios.

Abstract

Modern companies and institutions rely on data to guide every single business process and decision. Missing or incorrect information seriously compromises any decision process downstream. Therefore, a crucial, but tedious task for everyone involved in data processing is to verify the quality of their data. We present a system for automating the verification of data quality at scale, which meets the requirements of production use cases. Our system provides a declarative API, which combines common quality constraints with user-defined validation code, and thereby enables 'unit tests' for data. We efficiently execute the resulting constraint validation workload by translating it to aggregation queries on Apache Spark. Our platform supports the incremental validation of data quality on growing datasets, and leverages machine learning, e.g., for enhancing constraint suggestions, for estimating the 'predictability' of a column, and for detecting anomalies in historic data quality time series. We discuss our design decisions, describe the resulting system architecture, and present an experimental evaluation on various datasets.