Apprentice: Using Knowledge Distillation Techniques To Improve\n Low-Precision Network Accuracy

Abstract

Deep learning networks have achieved state-of-the-art accuracies on computer\nvision workloads like image classification and object detection. The performant\nsystems, however, typically involve big models with numerous parameters. Once\ntrained, a challenging aspect for such top performing models is deployment on\nresource constrained inference systems - the models (often deep networks or\nwide networks or both) are compute and memory intensive. Low-precision numerics\nand model compression using knowledge distillation are popular techniques to\nlower both the compute requirements and memory footprint of these deployed\nmodels. In this paper, we study the combination of these two techniques and\nshow that the performance of low-precision networks can be significantly\nimproved by using knowledge distillation techniques. Our approach, Apprentice,\nachieves state-of-the-art accuracies using ternary precision and 4-bit\nprecision for variants of ResNet architecture on ImageNet dataset. We present\nthree schemes using which one can apply knowledge distillation techniques to\nvarious stages of the train-and-deploy pipeline.\n