Entity resolution with iterative blocking

TLDR

Entity resolution identifies duplicate records, but exhaustive similarity computation is costly, so blocking partitions records into subsets, yet most techniques process blocks independently without leveraging results from other blocks. This work proposes an iterative blocking framework that propagates matching results from processed blocks to subsequent blocks. The framework repeatedly processes blocks until no new matches appear, implemented as a scalable system that updates block contents based on prior matches. Iterative blocking improves accuracy and efficiency over simple blocking, as shown by experiments on large datasets where it yields more matches and reduces processing time.

Abstract

Entity Resolution (ER) is the problem of identifying which records in a database refer to the same real-world entity. An exhaustive ER process involves computing the similarities between pairs of records, which can be very expensive for large datasets. Various blocking techniques can be used to enhance the performance of ER by dividing the records into blocks in multiple ways and only comparing records within the same block. However, most blocking techniques process blocks separately and do not exploit the results of other blocks. In this paper, we propose an iterative blocking framework where the ER results of blocks are reflected to subsequently processed blocks. Blocks are now iteratively processed until no block contains any more matching records. Compared to simple blocking, iterative blocking may achieve higher accuracy because reflecting the ER results of blocks to other blocks may generate additional record matches. Iterative blocking may also be more efficient because processing a block now saves the processing time for other blocks. We implement a scalable iterative blocking system and demonstrate that iterative blocking can be more accurate and efficient than blocking for large datasets.