Queen Mary Research Online (Queen Mary University of London) · 2018 · 144 citations · 0 references
Open access
ComputingComputer ScienceComputer VisionKnowledge TransferMachine LearningDeep LearningEnsemble LearningEnsemble MethodsNeural NetworksPattern RecognitionRepresentation LearningComputational EfficiencyComputational Efficiency AnalysisDeep EnsemblesDeep Learning-based ClassificationKnowledge DistillationTransferable Representation Learning
Knowledge distillation trains compact, generalisable models for low‑memory, fast‑running applications, but offline methods depend on a strong pre‑trained teacher and a complex two‑phase training, while online approaches avoid this but lack a high‑capacity teacher. The authors propose an On‑the‑fly Native Ensemble (ONE) strategy for one‑stage online distillation. ONE trains a single multi‑branch network and simultaneously builds a strong teacher on‑the‑fly to improve the target network’s learning. Extensive evaluations on CIFAR10, CIFAR100, SVHN, and ImageNet demonstrate that ONE significantly improves generalisation of various deep neural networks while maintaining computational efficiency.
Knowledge distillation is effective to train the small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a high-capacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) learning strategy for one-stage online distillation. Specifically, ONE only trains a single multi-branch network while simultaneously establishing a strong teacher on-the-fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance of a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.