Developing Language-tagged Corpora for Code-switching Tweets

Suraj Maharjan, Elizabeth Blair, Steven Bethard, Thamar Solorio

2015 · 33 citations · 19 references

DOIFull text

Open access

Concepts

Abstract

Code-switching, where a speaker switches between languages mid-utterance, is frequently used by multilingual populations worldwide. Despite its prevalence, limited effort has been devoted to develop computational approaches or even basic linguistic resources to support research into the processing of such mixedlanguage data. We present a user-centric approach to collecting code-switched utterances from social media posts, and develop language universal guidelines for the annotation of codeswitched data. We also present results for several baseline language identification models on our corpora and demonstrate that language identification in code-switched text is a difficult task that calls for deeper investigation.

References

19