Neural Network Classifiers Estimate Bayesian <i>a posteriori</i> Probabilities

TLDR

Neural network classifiers often output estimates of Bayesian posterior probabilities, which, when accurate, sum to one. Bayesian probabilities are obtained when the network is trained to produce one‑hot outputs using a squared‑error or cross‑entropy loss function. Monte Carlo simulations with MLP, RBF, and polynomial networks show that such outputs accurately estimate Bayesian probabilities, with accuracy depending on network complexity, training data size, and representativeness; interpreting outputs as probabilities facilitates combining classifiers, setting rejection thresholds, correcting class‑probability mismatches, optimizing risk functions, and redefining performance metrics.

Abstract

Many neural network classifiers provide outputs which estimate Bayesian a posteriori probabilities. When the estimation is accurate, network outputs can be treated as probabilities and sum to one. Simple proofs show that Bayesian probabilities are estimated when desired network outputs are 1 of M (one output unity, all others zero) and a squared-error or cross-entropy cost function is used. Results of Monte Carlo simulations performed using multilayer perceptron (MLP) networks trained with backpropagation, radial basis function (RBF) networks, and high-order polynomial networks graphically demonstrate that network outputs provide good estimates of Bayesian probabilities. Estimation accuracy depends on network complexity, the amount of training data, and the degree to which training data reflect true likelihood distributions and a priori class probabilities. Interpretation of network outputs as Bayesian probabilities allows outputs from multiple networks to be combined for higher level decision making, simplifies creation of rejection thresholds, makes it possible to compensate for differences between pattern class probabilities in training and test data, allows outputs to be used to minimize alternative risk functions, and suggests alternative measures of network performance.