Experiences with Parallelisation of an Existing NLP Pipeline: Tagging Hansard

Abstract

This poster describes experiences processing the two-billion-word Hansard corpus using a fairly standard NLP pipeline on a high performance
\ncluster. Herein we report how we were able to parallelise and apply a “traditional” single-threaded batch-oriented application
\nto a platform that differs greatly from that for which it was originally designed. We start by discussing the tagging toolchain, its specific
\nrequirements and properties, and its performance characteristics. This is contrasted with a description of the cluster on which it was to
\nrun, and specific limitations are discussed such as the overhead of using SAN-based storage. We then go on to discuss the nature of the
\nHansard corpus, and describe which properties of this corpus in particular prove challenging for use on the system architecture used.
\nThe solution for tagging the corpus is then described, along with performance comparisons against a na¨ıve run on commodity hardware.
\nWe discuss the gains and benefits of using high-performance machinery rather than relatively cheap commodity hardware. Our poster
\nprovides a valuable scenario for large scale NLP pipelines and lessons learnt from the experience.