ComputingComputer ScienceDistributed SystemsNatural Language ProcessingParallel ComputingDistributed ComputingDistributed Data ProcessingLanguage ModelingLanguage ModelsMachine TranslationN-gram AnalysisCompressionDistributed Database ManagementLanguage Processing ModelsLarge Language ModelsLarge-scale Natural Language ProcessingMapreduceMapreduce AlgorithmsStorage
This thesis presents the work of building a large scale distributed ngram language model using a MapReduce platform named Hadoop and a distributed database called Hbase. We propose a method focusing on the time cost and storage size of the model, exploring different Hbase table structures and compression approaches. The method is applied to build up training and testing processes using Hadoop MapReduce framework and Hbase. The experiments evaluate and compare different table structures on training 100 million words for unigram, bigram and trigram models, and the results suggest a table based on half ngram structure is a good choice for distributed language model. The results of this work can be applied and further developed in machine translation and other large scale distributed language processing areas.
8
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
SRILM - an extensible language modeling toolkit
Andreas Stolcke · 2002 · 4.4K citations
Fay W. Chang, Sanjay Ghemawat, Wilson C. Hsieh et al. · ACM Transactions on Computer Systems · 2008 · 3.4K citations
Improved backing-off for M-gram language modeling
Reinhard Kneser, Hermann Ney · 2002 · 1.5K citations
Large Language Models in Machine Translation
Thorsten Brants, Ashok C. Popat, Peng Xu et al. · 2007 · 550 citations