Outlier detection for high dimensional data

TLDR

Outlier detection in high‑dimensional data is challenging because proximity‑based methods lose meaning in sparse spaces, making every point appear almost equally outlying and complicating the identification of meaningful anomalies. The paper aims to introduce novel projection‑based techniques for detecting outliers in high‑dimensional datasets. These techniques analyze the behavior of data projections to identify outliers, leveraging projection patterns rather than proximity.

Abstract

The outlier detection problem has important applications in the field of fraud detection, network robustness analysis, and intrusion detection. Most such applications are high dimensional domains in which the data can contain hundreds of dimensions. Many recent algorithms use concepts of proximity in order to find outliers based on their relationship to the rest of the data. However, in high dimensional space, the data is sparse and the notion of proximity fails to retain its meaningfulness. In fact, the sparsity of high dimensional data implies that every point is an almost equally good outlier from the perspective of proximity-based definitions. Consequently, for high dimensional data, the notion of finding meaningful outliers becomes substantially more complex and non-obvious. In this paper, we discuss new techniques for outlier detection which find the outliers by studying the behavior of projections from the data set.