A Character-Level Machine Translation Approach for Normalization of SMS Abbreviations

Abstract

This paper describes a two-phase method for expanding abbreviations found in in-formal text (e.g., email, text messages, chat room conversations) using a machine translation system trained at the charac-ter level during the first phase. In this way, the system learns mappings between character-level “phrases ” and is much more robust to new abbreviations than a word-level system. We generate transla-tion models that are independent of the way in which the abbreviations are formed and show that the results show little degra-dation compared to when type-dependent models are trained. Our experiments on a large data set show our proposed system performs well when tested both on isolated abbreviations and, with the incorporation of a second phase utilizing an in-domain language model, in the context of neigh-boring words. 1