Nonlinear Discriminant Analysis Using Kernel Functions

TLDR

Linear discriminant analysis reduces dimensionality and classifies data using linear decision boundaries, but these boundaries often fail to separate classes in many applications. The authors propose a nonlinear generalization of discriminant analysis that employs kernel functions to overcome this limitation. Their algorithm formulates an EM approach in kernel space, enabling unsupervised, supervised, and semi‑supervised discriminant analysis with partially labeled data.

Abstract

Fishers linear discriminant analysis (LDA) is a classical multivariate technique both for dimension reduction and classification. The data vectors are transformed into a low dimensional subspace such that the class centroids are spread out as much as possible. In this subspace LDA works as a simple prototype classifier with linear decision boundaries. However, in many applications the linear boundaries do not adequately separate the classes. We present a nonlinear generalization of discriminant analysis that uses the kernel trick of representing dot products by kernel functions. The presented algorithm allows a simple formulation of the EM-algorithm in terms of kernel functions which leads to a unique concept for unsupervised mixture analysis, supervised discriminant analysis and semi-supervised discriminant analysis with partially unlabelled observations in feature spaces.