Minerva Access (University of Melbourne) · 2017 · 29 citations · 25 references
ComputingComputer ScienceComputational LinguisticsData StructuresNatural Language ProcessingCorpus QueryingLanguage ModelingLanguage ModelsN-gram AnalysisQuery ProcessingLarge Language ModelsLarge-scale Natural Language ProcessingN-gram StatisticsNeural Language Models
Efficient methods for storing and querying are critical for scaling high-order m-gram language models to large corpora. We propose a language model based on compressed suffix trees, a representation that is highly compact and can be easily held in memory, while supporting queries needed in computing language model probabilities on-the-fly. We present several optimisations which improve query runtimes up to 2500×, despite only incurring a modest increase in construction time and memory usage. For large corpora and high Markov orders, our method is highly competitive with the state-of-the-art KenLM package. It imposes much lower memory requirements, often by orders of magnitude, and has runtimes that are either similar (for training) or comparable (for querying).
25
SRILM - an extensible language modeling toolkit
Andreas Stolcke · 2002 · 4.4K citations
A Block-sorting Lossless Data Compression Algorithm
Michael T. Burrows, D. J. Wheeler · 1994 · 2.4K citations
Suffix Arrays: A New Method for On-Line String Searches
Udi Manber, Gene Myers · SIAM Journal on Computing · 1993 · 2K citations