2010 · 98 citations · 3 references
Language StudiesComputer ScienceComputational LinguisticsCorpus LinguisticsNatural Language ProcessingCorpus AnalysisCorpus AnnotationCorpus CompilationCorpus DesignLanguage ModelingText SegmentationAnchor Text AnalysisCorpus StatisticsData DistributionN-gram RepresentationN-gram StatisticsPhrase SegmentationSegmentation Corpus ConstructionWeb-based Corpora
This document describes the properties and some applications of the Microsoft Web N-gram corpus. The corpus is designed to have the following characteristics. First, in contrast to static data distribution of previous corpus releases, this N-gram corpus is made publicly available as an XML Web Service so that it can be updated as deemed necessary by the user community to include new words and phrases constantly being added to the Web. Secondly, the corpus makes available various sections of a Web document, specifically, the body, title, and anchor text, as separates models as text contents in these sections are found to possess significantly different statistical properties and therefore are treated as distinct languages from the language modeling point of view. The usages of the corpus are demonstrated here in two NLP tasks: phrase segmentation and word breaking. 1
3
Exploring web scale language models for search query processing
Jian Huang, Jianfeng Gao, Jiangbo Miao et al. · 2010 · 68 citations