Maintaining variance and k-medians over data stream windows

Brain Babcock, Mayur Datar, Rajeev Motwani, Liadan O'Callaghan

2003 · 277 citations · 23 references

Concepts

TL;DR

The sliding window model discounts stale data by considering only the most recent N elements in a continuous stream. The authors propose a novel technique to maintain both variance and k‑median clustering within such sliding windows. Their method estimates variance with relative error ε using O(1/ε² log N) memory and delivers a constant‑factor approximation for k‑median clustering with O(k/τ⁴ N²τ log² N) memory. The variance estimator achieves a relative error of at most ε while requiring only O(1/ε² log N) memory.

Abstract

The sliding window model is useful for discounting stale data in data stream applications. In this model, data elements arrive continually and only the most recent N elements are used when answering queries. We present a novel technique for solving two important and related problems in the sliding window model---maintaining variance and maintaining a k--median clustering. Our solution to the problem of maintaining variance provides a continually updated estimate of the variance of the last N values in a data stream with relative error of at most e using O(1/e2 log N) memory. We present a constant-factor approximation algorithm which maintains an approximate k--median solution for the last N data points using O(k/τ4 N2τ log2N) memory, where τ

References

23