What's hot and what's not: tracking most frequent items dynamically

TLDR

Database systems track statistics such as frequently occurring hot items used in histograms for selectivity estimation, yet existing algorithms struggle to handle deletions without rescanning. The paper proposes new methods to dynamically identify hot items in a relation undergoing insertions and deletions. The approach uses compact data structures that monitor transactions and, via group‑testing techniques, can output hot items on demand without rescanning, offering provable space and time guarantees. Experiments confirm that the algorithms accurately report all hot items with user‑specified probability, regardless of insertion/deletion rates.

Abstract

Most database management systems maintain statistics on the underlying relation. One of the important statistics is that of the “hot items” in the relation: those that appear many times (most frequently, or more than some threshold). For example, end-biased histograms keep the hot items as part of the histogram and are used in selectivity estimation. Hot items are used as simple outliers in data mining, and in anomaly detection in many applications.We present new methods for dynamically determining the hot items at any time in a relation which is undergoing deletion operations as well as inserts. Our methods maintain small space data structures that monitor the transactions on the relation, and, when required, quickly output all hot items without rescanning the relation in the database. With user-specified probability, all hot items are correctly reported. Our methods rely on ideas from “group testing.”” They are simple to implement, and have provable quality, space, and time guarantees. Previously known algorithms for this problem that make similar quality and performance guarantees cannot handle deletions, and those that handle deletions cannot make similar guarantees without rescanning the database. Our experiments with real and synthetic data show that our algorithms are accurate in dynamically tracking the hot items independent of the rate of insertions and deletions.