Journal of Data and Information Quality · 2020 · 21 citations · 24 references
ComputingComputer ScienceData ManagementData MiningQuality AssuranceData CleaningData PreparationData PreprocessingData QualityDatabase NormalizationDuplicate DetectionDuplicate Record RemovalNormalizationPair-matchingPrecisionPrecision-recall TradeoffRedundancy EliminationSimilarity DetectionWhitespace
Data errors represent a major issue in most application workflows. Before any important task can take place, a certain data quality has to be guaranteed by eliminating a number of different errors that may appear in data. Typically, most of these errors are fixed with data preparation methods, such as whitespace removal. However, the particular error of duplicate records, where multiple records refer to the same entity, is usually eliminated independently with specialized techniques. Our work is the first to bring these two areas together by applying data preparation operations under a systematic approach prior to performing duplicate detection. Our process workflow can be summarized as follows: It begins with the user providing as input a sample of the gold standard, the actual dataset, and optionally some constraints to domain-specific data preparations, such as address normalization. The preparation selection operates in two consecutive phases. First, to vastly reduce the search space of ineffective data preparations, decisions are made based on the improvement or worsening of pair similarities. Second, using the remaining data preparations an iterative leave-one-out classification process removes preparations one by one and determines the redundant preparations based on the achieved area under the precision-recall curve (AUC-PR). Using this workflow, we manage to improve the results of duplicate detection up to 19% in AUC-PR.
24
Data mining: concepts and techniques
Jiawei Han, Micheline Kamber · Choice Reviews Online · 2012 · 28.9K citations
Deep Learning for Entity Matching
Sidharth Mudgal, Han Li, Theodoros Rekatsinas et al. · 2018 · 531 citations
Data preparation for data mining
Shichao Zhang, Chengqi Zhang, Qiang Yang · Applied Artificial Intelligence · 2003 · 509 citations
Swoosh: a generic approach to entity resolution
Omar Benjelloun, Héctor García-Molina, David Menestrina et al. · The VLDB Journal · 2008 · 486 citations