<scp>Canine</scp>: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting

Transactions of the Association for Computational Linguistics · 2022 · 120 citations · 69 references

DOIFull text

Open access

Concepts

TL;DR

End‑to‑end neural models have largely replaced pipelined NLP systems, yet most still require tokenization, and fixed‑vocabulary subword approaches, while less brittle, are not equally suited to all languages and can limit adaptability. The authors introduce Canine, a tokenization‑free neural encoder that processes raw character sequences and offers a pre‑training strategy that can use characters alone or incorporate subwords as a soft inductive bias. Canine achieves efficient, fine‑grained input handling by downsampling to reduce sequence length and feeding the result into a deep transformer stack for contextual encoding. On the multilingual TyDi QA benchmark, Canine surpasses a comparable mBert model by 5.7 F1 despite having fewer parameters.

Abstract

Abstract Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present Canine, a neural encoder that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. Canine outperforms a comparable mBert model by 5.7 F1 on TyDi QA, a challenging multilingual benchmark, despite having fewer model parameters.

References

69