UNSUPERVISED WORD INDUCTION USING MDL CRITERION

Abstract

Unsupervised learning of units (phonemes, words, phrases, etc.) is important to the design of statistical speech and NLP systems. This paper presents a general source-coding framework for induc-ing words from natural language text without word boundaries. An efficient search algorithm is developed to optimize the mini-mum description length (MDL) induction criterion. Despite some seemingly over-simplified modeling assumption, we achieved good results on several word induction problems.