2017 · 21 citations · 18 references
ComputingComputer ScienceComputational LinguisticsCorpus LinguisticsNatural Language ProcessingLanguage ModelingLanguage ModelsLexical TypologyMultilingual Natural Language ProcessingCharacter-level Language ModelingCharacter-level RepresentationMultilingual Corpus ComparisonNeural Language Models
Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the "bursty" distribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus; MWC) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.
18
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget et al. · 2010 · 5.4K citations
Neural Architecture Search with Reinforcement Learning
Barret Zoph, Quoc V. Le · arXiv (Cornell University) · 2016 · 3.9K citations · Full text