Exploring the Limits of Language Modeling

TLDR

The study investigates how to extend RNNs for large‑scale language modeling by addressing corpus size, vocabulary size, and long‑term structure, and releases the resulting models for community use. The authors conduct an exhaustive evaluation of character‑CNN and LSTM techniques on the One Billion Word Benchmark to address corpus size, vocabulary size, and long‑term language structure. The best single model reduces perplexity from 51.3 to 30.0 with 20× fewer parameters, and an ensemble further lowers perplexity to 23.7, setting a new record.

Abstract

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.