WIT transactions on information and communication technologies · 2006 · 27 citations · 6 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingText AnalyticsLexical AnalysisPart-of-speech TaggingSyntactic AnalysisText MiningText PreprocessingWord Sense DisambiguationLinguistic PatternsPunctuation HandlingPunctuation MarksRule-based Part-of-speech TaggingText Mining FeaturesTokenization
Tokenization is commonly understood as the first step of any kind of natural language text preparation. The major goal of this early (pre-linguistic) task is to convert a stream of characters into a stream of processing units called tokens.
6
Tokenization as the initial phase in NLP
Jonathan J. Webster, Chunyu Kit · 1992 · 364 citations · Full text
Lexical analysis and stoplists
Christopher Fox · Information Retrieval · 1992 · 166 citations
Periods, Capitalized Words, etc.
Andrei Mikheev · Computational Linguistics · 2002 · 93 citations
Critical tokenization and its properties
Jin Guo · 1997 · 56 citations