More than Words: Using Token Context to Improve Canonicalization of Historical German

Bryan Jurish

LDV-Forum/Journal for language technology and computational linguistics · 2010 · 33 citations · 44 references

DOIFull text

Open access

Concepts

Abstract

Historical text presents numerous challenges for
\ncontemporary natural language processing techniques. In particular, the absence of consistent orthographic conventions in historical text presents difficulties for any system requiring reference to a static lexicon indexed by orthographic form. Canonicalization approaches seek to address these issues by associating one or more extant ``canonical cognates'' with each word of the input text and deferring application analysis to these canonical forms. Type-wise conflation techniques treating each input word in isolation often suffer from a pronounced precision -- recall trade-off pattern: high-precision techniques such as conservative transliteration have comparatively poor recall, whereas high-recall techniques such as phonetic conflation tend to be disappointingly imprecise. In this paper, I present a technique for disambiguation of type conflation sets at the token level using a Hidden Markov Model whose lexical probability matrix is dynamically computed from the candidate conflations, and evaluate its performance on a manually annotated corpus of historical German.
\n

References

44