Procedia Technology · 2014 · 43 citations · 8 references
ComputingComputer ScienceComputational LinguisticsInformation RetrievalNatural Language ProcessingComparable CorporaComparable Corpus AnalysisMachine TranslationMultilingual Natural Language ProcessingParallel CorporaComparable Corpus MiningCross-lingual Information RetrievalCross-lingual RetrievalMonolingual Comparable CorporaMultilingual Parallel CorporaParallel Corpus MiningParallel Text CorporaStatistical Machine Translation
Parallel sentences are a relatively scarce but extremely useful resource for many applications including cross-lingual retrieval and statistical machine translation. This research explores our methodology for mining such data from previously obtained comparable corpora. The task is highly practical since non-parallel multilingual data exist in far greater quantities than parallel corpora, but parallel sentences are a much more useful resource. Here we propose a web crawling method for building subject-aligned comparable corpora from Wikipedia articles. We also introduce a method for extracting truly parallel sentences that are filtered out from noisy or just comparable sentence pairs. We describe our implementation of a specialized tool for this task as well as training and adaption of a machine translation system that supplies our filter with additional information about the similarity of comparable sentence pairs.
8
Philipp Koehn, Richard Zens, Chris Dyer et al. · 2007 · 4.9K citations · Full text
SRILM - an extensible language modeling toolkit
Andreas Stolcke · 2002 · 4.4K citations
A Study of Translation Edit Rate with Targeted Human Annotation
Matthew Snover, Bonnie J. Dorr, Richard Schwartz et al. · 2006 · 2.4K citations
Parallel Data, Tools and Interfaces in OPUS
Jörg Tiedemann · Language Resources and Evaluation · 2012 · 1.3K citations
KenLM: Faster and Smaller Language Model Queries
Kenneth Heafield · 2011 · 1.1K citations