A simple rule-based part of speech tagger

TLDR

Automatic part‑of‑speech tagging is an NLP area where statistical techniques have outperformed rule‑based methods. This paper presents a simple rule‑based part‑of‑speech tagger that automatically acquires its rules. The tagger achieves accuracy comparable to stochastic taggers. The tagger offers many advantages, including a vast reduction in stored information, a small set of meaningful rules, ease of improvement, better portability across tag sets, corpus genres, and languages, and demonstrates that stochastic methods are not the only viable approach, encouraging further exploration of rule‑based tagging.

Abstract

Automatic part of speech tagging is an area of natural language processing where statistical techniques have been more successful than rule-based methods. In this paper, we present a simple rule-based part of speech tagger which automatically acquires its rules and tags with accuracy comparable to stochastic taggers. The rule-based tagger has many advantages over these taggers, including: a vast reduction in stored information required, the perspicuity of a small set of meaningful rules, ease of finding and implementing improvements to the tagger, and better portability from one tag set, corpus genre or language to another. Perhaps the biggest contribution of this work is in demonstrating that the stochastic method is not the only viable method for part of speech tagging. The fact that a simple rule-based tagger that automatically learns its rules can perform so well should offer encouragement for researchers to further explore rule-based tagging, searching for a better and more expressive set of rule templates and other variations on the simple but effective theme described below.