Knowledge Distillation

Knowledge Distillation is a model compression technique in which a smaller, more efficient neural network (the student) is trained to emulate the behavior of a larger, more complex network (the teacher) by transferring learned representations or output distributions, thereby preserving performance while reducing computational demands for deployment in resource-constrained edge environments.

1.6K

Publications

118.3K

Citations

6.1K

Authors

981

Institutions

Publications per year

2017–2026

1.6K

Parents

Edge Artificial Intelligence

Publications

1.6K

Distilling the Knowledge in a Neural Network

Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text

Computing, Computer Science, Acoustics +15

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Victor Sanh, Lysandre Debut, Julien Chaumond et al. · arXiv (Cornell University) · 2019 · 4.6K citations · Full text

Computing, Computer Science, Knowledge Representation +15

Knowledge Distillation: A Survey

Jianping Gou, Baosheng Yu, Stephen J. Maybank et al. · International Journal of Computer Vision · 2021 · 3.5K citations · Full text

Computing, Chemical Engineering, Knowledge Representation +2

FitNets: Hints for Thin Deep Nets

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou et al. · arXiv (Cornell University) · 2014 · 2K citations · Full text

Education, Educational Technology, Artificial Intelligence In Education +8

A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning

Junho Yim, Donggyu Joo, Ji‐Hoon Bae et al. · 2017 · 1.6K citations

Computing, Computer Science, Knowledge Representation +14

1–5 of 1.6K

Authors

6.1K

DT
Dacheng Tao
The University of Sydney, Australia
19 publications · 4.7K citations
17
h-index
MS
Mingli Song
Zhejiang University, China
14 publications · 912 citations
12
h-index
XW
Xinchao Wang
National University of Singapore, Singapore
12 publications · 716 citations
11
h-index
CX
Chang Xu
The University of Sydney, Australia
9 publications · 569 citations
9
h-index
YW
Yunhe Wang
The University of Sydney, Australia
9 publications · 554 citations
9
h-index

1–5 of 6.1K

Institutions

981

1–8 of 981

Venues

arXiv (Cornell University)
Repository · 302 publications · 35.6K citations
60
h-index
Proceedings of the AAAI Conference on Artificial Intelligence
Conference · 79 publications · 5.5K citations
39
h-index
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Conference · 38 publications · 5.3K citations
32
h-index
IEEE Transactions on Pattern Analysis and Machine Intelligence
TPAMI · 25 publications · 3.1K citations
23
h-index
IEEE Transactions on Geoscience and Remote Sensing
26 publications · 1.2K citations
22
h-index

1–5