Holistic data cleaning: Putting violations into context

TLDR

Data cleaning is critical, and declarative quality rules are a promising approach, yet prior work has treated formalisms such as FDs, CFDs, and MDs in isolation and often applied them in pipelines. The study introduces a unified framework for data cleaning. The framework lets users define denial constraints with predicates, subsumes existing formalisms, and models their interactions as a conflict hypergraph to compute repair contexts. Experiments on real datasets demonstrate that the holistic approach yields higher‑quality repairs more efficiently than prior methods.

Abstract

Data cleaning is an important problem and data quality rules are the most promising way to face it with a declarative approach. Previous work has focused on specific formalisms, such as functional dependencies (FDs), conditional functional dependencies (CFDs), and matching dependencies (MDs), and those have always been studied in isolation. Moreover, such techniques are usually applied in a pipeline or interleaved. In this work we tackle the problem in a novel, unified framework. First, we let users specify quality rules using denial constraints with ad-hoc predicates. This language subsumes existing formalisms and can express rules involving numerical values, with predicates such as "greater than" and "less than". More importantly, we exploit the interaction of the heterogeneous constraints by encoding them in a conflict hypergraph. Such holistic view of the conflicts is the starting point for a novel definition of repair context which allows us to compute automatically repairs of better quality w.r.t. previous approaches in the literature. Experimental results on real datasets show that the holistic approach outperforms previous algorithms in terms of quality and efficiency of the repair.