arXiv (Cornell University) · 2016 · 370 citations · 28 references
ComputingComputer ScienceMachine LearningOptimizationStatistical LearningDeep LearningNeural NetworksOptimization For Machine LearningRepresentation LearningVariational FormulationVariational InferenceAdversarial RobustnessInformation BottleneckInformation Bottleneck AnalysisNeural Network ModelingOptimization AlgorithmsReparameterizationVariational BayesVariational Regularization
We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.
28
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Auto-Encoding Variational Bayes
Diederik P. Kingma, Max Welling · UvA-DARE (University of Amsterdam) · 2013 · 15.6K citations · Full text
Information theory, inference, and learning algorithms
Choice Reviews Online · 2004 · 6.6K citations