Linear discriminant analysis: A detailed tutorial

TLDR

Linear Discriminant Analysis is widely used for dimensionality reduction in machine learning but is often treated as a black‑box technique that many practitioners do not fully understand. This tutorial aims to provide a clear, intuitive understanding of LDA and guide readers on how to apply it across various applications. The authors present the fundamental definitions and step‑by‑step procedures of LDA, illustrate class‑dependent and class‑independent computations with visual aids and numerical examples, discuss common challenges such as small‑sample and non‑linearity problems, and evaluate the impact of eigenvectors and SSS on classification performance through experiments on multiple datasets.

Abstract

Linear Discriminant Analysis (LDA) is a very common technique for dimensionality reduction problems as a pre-processing step for machine learning and pattern classification applications. At the same time, it is usually used as a black box, but (sometimes) not well understood. The aim of this paper is to build a solid intuition for what is LDA, and how LDA works, thus enabling readers of all levels be able to get a better understanding of the LDA and to know how to apply this technique in different applications. The paper first gave the basic definitions and steps of how LDA technique works supported with visual explanations of these steps. Moreover, the two methods of computing the LDA space, i.e. class-dependent and class-independent methods, were explained in details. Then, in a step-by-step approach, two numerical examples are demonstrated to show how the LDA space can be calculated in case of the class-dependent and class-independent methods. Furthermore, two of the most common LDA problems (i.e. Small Sample Size (SSS) and non-linearity problems) were highlighted and illustrated, and state-of-the-art solutions to these problems were investigated and explained. Finally, a number of experiments was conducted with different datasets to (1) investigate the effect of the eigenvectors that used in the LDA space on the robustness of the extracted feature for the classification accuracy, and (2) to show when the SSS problem occurs and how it can be addressed.