Optimization and simplification of hierarchical clusterings

Abstract

Clustering is often used to discover structure in data. Clustering systems differ in the objective function used to evaluate clustering quality and the control strategy used to search the space of clusterings. In general, a search strategy cannot both (1) consistently construct clusterings of high quality and (2) be computationally inexpensive. However, we can partition the search so that a system inexpensively constructs 'tentative' clusterings for initial examination, followed by iterative optimization, which continues to search in background for improved clusterings. This paper evaluates hierarchical redistribution, which appears to be a novel optimization strategy in the clustering literature. A final component of search prunes tree-structured clusterings, thus simplifying them for analysis. In particular, resampling is used to significantly simplify hierarchical clusterings.