Self-Organizing Maps in data analysis - notes on overfitting and overinterpretation.

Abstract

The Self-Organizing Map, SOM, is a widely used tool in exploratory data analysis. Visual inspection of the SOM can be used to list potential dependencies between variables, that are then validated with more principled statistical methods. In this paper we discuss the use of the SOM in searching for dependencies in the data. We point out that simple use of the SOM may lead to excessive number of false hypotheses. We formulate the exact probability density model for which the SOM training gives the Maximum Likelihood estimate and show how the model parameters (neighborhood and kernel width) can be chosen to avoid overfitting. The conditional distributions from the true density model offer a consistent way to quantify and test the dependencies between variables.