Vowel and Diacritic Restoration for Social Media Texts

Abstract

In this paper, we focus on two important problems of social media text normaliza-tion, namely: vowel and diacritic restora-tion. For these two problems, we pro-pose a hybrid model consisting both a dis-criminative sequence classifier and a lan-guage validator in order to select one of the morphologically valid outputs of the first stage. The proposed model is lan-guage independent and has no need for manual annotation of the training data. We measured the performance both on syn-thetic data specifically produced for these two problems and on real social media data. Our model (with 97.06 % on syn-thetic data) improves the state of the art results for diacritization of Turkish by 3.65 percentage points on ambiguous cases and for the vowel restoration by 45.77 percent-age points over a rule based baseline with 62.66 % accuracy. The results on real data are 95.43 % and 69.56 % accordingly. 1