Understanding Knowledge Distillation in Non-autoregressive Machine\n Translation

Abstract

Non-autoregressive machine translation (NAT) systems predict a sequence of\noutput tokens in parallel, achieving substantial improvements in generation\nspeed compared to autoregressive models. Existing NAT models usually rely on\nthe technique of knowledge distillation, which creates the training data from a\npretrained autoregressive model for better performance. Knowledge distillation\nis empirically useful, leading to large gains in accuracy for NAT models, but\nthe reason for this success has, as of yet, been unclear. In this paper, we\nfirst design systematic experiments to investigate why knowledge distillation\nis crucial to NAT training. We find that knowledge distillation can reduce the\ncomplexity of data sets and help NAT to model the variations in the output\ndata. Furthermore, a strong correlation is observed between the capacity of an\nNAT model and the optimal complexity of the distilled data for the best\ntranslation quality. Based on these findings, we further propose several\napproaches that can alter the complexity of data sets to improve the\nperformance of NAT models. We achieve the state-of-the-art performance for the\nNAT-based models, and close the gap with the autoregressive baseline on WMT14\nEn-De benchmark.\n