Cross-Layer Distillation with Semantic Calibration

TLDR

Knowledge distillation often uses intermediate teacher layers as targets, but manually pairing layers can cause semantic mismatch and degrade performance. This work introduces Semantic Calibration for Cross‑layer Knowledge Distillation (SemCKD) to automatically assign appropriate teacher layers to each student layer. SemCKD employs an attention mechanism that learns a distribution over teacher layers, allowing each student layer to distill from multiple teacher layers rather than a single fixed one. Experiments across diverse teacher–student architectures show that SemCKD consistently outperforms state‑of‑the‑art methods, confirming its effectiveness and flexibility.

Abstract

Recently proposed knowledge distillation approaches based on feature-map transfer validate that intermediate layers of a teacher model can serve as effective targets for training a student model to obtain better generalization ability. Existing studies mainly focus on particular representation forms for knowledge transfer between manually specified pairs of teacher-student intermediate layers. However, semantics of intermediate layers may vary in different networks and manual association of layers might lead to negative regularization caused by semantic mismatch between certain teacher-student layer pairs. To address this problem, we propose Semantic Calibration for Cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple layers rather than a single fixed intermediate layer from the teacher model for appropriate cross-layer supervision in training. Consistent improvements over state-of-the-art approaches are observed in extensive experiments with various network architectures for teacher and student models, demonstrating the effectiveness and flexibility of the proposed attention based soft layer association mechanism for cross-layer distillation.