Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

Yu. A. Malkov, D. A. Yashunin

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2018 · 1.6K citations · 48 references

Concepts

TL;DR

The authors introduce Hierarchical NSW (HNSW), a novel approximate K‑nearest neighbor search method based on navigable small‑world graphs with a controllable hierarchy. HNSW is a fully graph‑based index that incrementally builds a multi‑layer proximity graph, selects the maximum layer for each element using an exponentially decaying probability, and starts searches from the top layer while employing a heuristic neighbor selection to boost performance, yielding logarithmic complexity and enabling straightforward distributed implementation. Evaluation shows that HNSW strongly outperforms existing open‑source vector‑only approaches in metric‑space search.

Abstract

We present a new approach for the approximate K-nearest neighbor search based on navigable small world graphs with controllable hierarchy (Hierarchical NSW, HNSW). The proposed solution is fully graph-based, without any need for additional search structures (typically used at the coarse search stage of the most proximity graph techniques). Hierarchical NSW incrementally builds a multi-layer structure consisting of a hierarchical set of proximity graphs (layers) for nested subsets of the stored elements. The maximum layer in which an element is present is selected randomly with an exponentially decaying probability distribution. This allows producing graphs similar to the previously studied Navigable Small World (NSW) structures while additionally having the links separated by their characteristic distance scales. Starting the search from the upper layer together with utilizing the scale separation boosts the performance compared to NSW and allows a logarithmic complexity scaling. Additional employment of a heuristic for selecting proximity graph neighbors significantly increases performance at high recall and in case of highly clustered data. Performance evaluation has demonstrated that the proposed general metric space search index is able to strongly outperform previous opensource state-of-the-art vector-only approaches. Similarity of the algorithm to the skip list structure allows straightforward balanced distributed implementation.

References

48