arXiv (Cornell University) · 2017 · 26 citations · 41 references
ComputingComputer ScienceArtificial Intelligence In EducationIntelligent Tutoring SystemsMachine LearningDeep LearningLearner ModelingLoss FunctionNeural NetworksTransfer LearningAdversarial LossClassification AccuracyClassification Accuracy AnalysisClassification Accuracy AssessmentKnowledge DistillationShallow Neural NetworksStudent-teacher RelationshipsTeacher-student Learning
There is an increasing interest on accelerating neural networks for real-time applications. We study the student-teacher strategy, in which a small and fast student network is trained with the auxiliary information learned from a large and accurate teacher network. We propose to use conditional adversarial networks to learn the loss function to transfer knowledge from teacher to student. The proposed method is particularly effective for relatively small student networks. Moreover, experimental results show the effect of network size when the modern networks are used as student. We empirically study the trade-off between inference time and classification accuracy, and provide suggestions on choosing a proper student network.
41
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky et al. · 2014 · 34.3K citations