2015 · 857 citations · 27 references
ComputingComputer ScienceArtificial IntelligenceComputer VisionMachine LearningDeep LearningImage Feature EngineeringNeural NetworksPattern RecognitionRepresentation LearningSupervised LearningDeep Feature ExtractionDeep Image FeaturesDeep Learning-based ClassificationLearned Feature RepresentationsPretrainingStochastic Gradient DescentTraining Effectiveness AnalysisVanishing Gradient Problem
Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by study-ing a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robust-ness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce “com-panion objective ” to the individual hidden layers, in addition to the overall objec-tive at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advan-tage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN). 1
27
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia et al. · 2015 · 46.8K citations
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006 · 16.4K citations
Backpropagation Applied to Handwritten Zip Code Recognition
Yann LeCun, Bernhard E. Boser, J. S. Denker et al. · Neural Computation · 1989 · 11.9K citations