MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

TLDR

NLP has achieved great success with huge pre‑trained models, but their large size and high latency hinder deployment on resource‑limited mobile devices. This paper proposes MobileBERT to compress and accelerate the popular BERT model for mobile deployment. MobileBERT is a thin version of BERT_LARGE equipped with bottleneck structures and a balanced mix of self‑attentions and feed‑forward networks, trained via knowledge transfer from a specially designed teacher model. MobileBERT is 4.3× smaller and 5.5× faster than BERT_BASE while achieving competitive GLUE and SQuAD scores, with a 77.7 GLUE score and 62 ms latency on a Pixel 4 phone, and dev F1 of 90.0/79.2 on SQuAD v1.1/v2.0.

Abstract

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).