Stop Word Lists in Free Open-source Software Packages

Abstract

Open-source software (OSS) packages for natural language processing often include stop word lists. Users may apply them without awareness of their surprising omissions (e.g. hasn't but not hadn't) and inclusions (e.g. computer), or their incompatibility with particular tokenizers. Motivated by issues raised about the Scikitlearn stop list, we investigate variation among and consistency within 52 popular English-language stop lists, and propose strategies for mitigating these issues.