2009 · 33 citations · 10 references
ComputingComputer ScienceInformation RetrievalNatural Language ProcessingText AnalyticsCorpus AnalysisCorpus QueryingCorpus RetrievalDuplicate DetectionCorpus CreationDuplicate Content DetectionLarge-scale Natural Language ProcessingWeb Corpus QueryingWeb-based Corpora
Most phenomena in natural languages are distributed in accordance with Zipf's law, so many words, phrases and other items occur rarely and we need very large corpora to provide evidence about them. Previous work shows that it is possible to create very large (multi-billion word) corpora from the web. The usability of such corpora is often limited by duplicate contents and a lack of efficient query tools. This paper describes BiWeC, a Big Web Corpus of English texts currently comprising 5.5b words fully processed, and with a target size of 20b. We present a method for detecting near-duplicate text documents in multi-billion-word text collections and describe how one corpus query tool, the Sketch Engine, has been re-engineered to efficiently encode, process and query such corpora on low-cost hardware.
10
Adam Kilgarriff, Pavel Rychlý, Pavel Smrž et al. · 2004 · 528 citations
Adam Kilgarriff, Pavel Rychlý, Pavel Smrž et al. · Proceedings of the 11th EURALEX International Congress · 2004 · 342 citations
Introducing and evaluating ukWaC, a very large Web-derived corpus of English
Adriano Ferraresi, Eros Zanchetta, Marco Baroni et al. · 2008 · 325 citations
Adam Kilgarriff · Computational Linguistics · 2007 · 286 citations · Full text
Manatee/Bonito - A Modular Corpus Manager
Pavel Rychlý · RASLAN · 2007 · 132 citations