Applied Intelligence · 2022 · 68 citations · 25 references
ComputingComputer ScienceEducational TechnologyArtificial Intelligence In EducationComputer VisionMachine LearningClassificationEnsemble LearningEnsemble MethodsNeural NetworksSelf-supervised LearningSupervised LearningDeep Image FeaturesDeep Learning-based ClassificationKnowledge DistillationLimited Data SetsModel Performance AnalysisModel Performance EvaluationSupervised ClassificationTeacher-student Learning
A single model usually cannot learn all the appropriate features with limited data, thus leading to poor performance when test data are used. To improve model performance, we propose a teacher-student collaborative knowledge distillation (TSKD) method based on knowledge distillation and self-distillation. The method consists of two parts: learning in the teacher network and self-teaching in the student network. Learning in the teacher network allows the student network to use knowledge from the teacher network. Self-teaching in the student network is to build a multi-exit network based on self-distillation and provide deep features as supervised information for training. In the inference stage, we use ensembles to vote on the classification results of multiple sub-models in the student network. The experimental results demonstrate the superior performance of our method compared with a traditional knowledge distillation method and a self-distillation-based multi-exit network.
25
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin et al. · 2018 · 9K citations
Sergey Zagoruyko, Nikos Komodakis · 2016 · 6K citations
Knowledge Distillation: A Survey
Jianping Gou, Baosheng Yu, Stephen J. Maybank et al. · International Journal of Computer Vision · 2021 · 3.5K citations · Full text
Computing, Chemical Engineering, Knowledge Representation +2