International Conference on Acoustics, Speech, and Signal Processing · 2002 · 174 citations · 5 references
ComputingComputer EngineeringComputer ScienceArtificial IntelligenceInformation TheoryMachine LearningMaximum Likelihood EstimationMutual InformationNeural NetworksPattern RecognitionProbabilistic Machine LearningSupervised LearningBayesian NetworksKullback-leibler DivergenceMaximum Mutual InformationNeural Network ClassificationNeural Network ModelingNeural Network PredictionProbabilistic ClassificationStatistical Learning Theory
It is shown that training a neural network using a mean-square-error criterion gives network outputs that approximate posterior class probabilities. Based on this probabilistic interpretation of the network operation, information-theoretic training criteria such as maximum mutual information and the Kullback-Liebler measure are investigated. It is shown that both of these criteria are equivalent to the maximum-likelihood estimation (MLE) of the network parameters. MLE of a network allows for the comparison of network models using the Akaike information criterion and the minimum-description length criterion.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
5
Modeling by shortest data description
J. Rissanen · Automatica · 1978 · 6K citations
Information Theory and Statistics.
Meyer Dwass, S. Kullback · American Mathematical Monthly · 1960 · 1.5K citations