arXiv (Cornell University) · 2019 · 33 citations · 33 references
ComputingComputer ScienceComputer VisionKnowledge TransferMachine LearningDeep LearningTransfer LearningAdversarial SearchBenchmark DatasetsKnowledge DistillationPrivacyPrivacy-preserving Model TrainingTransferable Representation Learning
Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel method which trains a student to match the predictions of its teacher without using any data or metadata. We achieve this by training an adversarial generator to search for images on which the student poorly matches the teacher, and then using them to train the student. Our resulting student closely approximates its teacher for simple datasets like SVHN, and on CIFAR10 we improve on the state-of-the-art for few-shot distillation (with 100 images per class), despite using no data. Finally, we also propose a metric to quantify the degree of belief matching between teacher and student in the vicinity of decision boundaries, and observe a significantly higher match between our zero-shot student and the teacher, than between a student distilled with real data and the teacher. Code available at: https://github.com/polo5/ZeroShotKnowledgeTransfer
33
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato et al. · 2014 · 6.6K citations
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · arXiv (Cornell University) · 2015 · 4.7K citations · Full text
Reading digits in natural images with unsupervised feature learning
Yuval Netzer · 2024 · 4.6K citations · Full text