Transactions of the Association for Computational Linguistics · 2022 · 120 citations · 69 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsMultilingual Natural Language ProcessingRepresentation LearningWord EmbeddingsEncoder DesignInductive BiasLanguage Model PretrainingNeural Language ModelsSubword ModelingSubword RepresentationSubword TokenizationTokenization
End‑to‑end neural models have largely replaced pipelined NLP systems, yet most still require tokenization, and fixed‑vocabulary subword approaches, while less brittle, are not equally suited to all languages and can limit adaptability. The authors introduce Canine, a tokenization‑free neural encoder that processes raw character sequences and offers a pre‑training strategy that can use characters alone or incorporate subwords as a soft inductive bias. Canine achieves efficient, fine‑grained input handling by downsampling to reduce sequence length and feeding the result into a deep transformer stack for contextual encoding. On the multilingual TyDi QA benchmark, Canine surpasses a comparable mBert model by 5.7 F1 despite having fewer parameters.
Abstract Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present Canine, a neural encoder that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. Canine outperforms a comparable mBert model by 5.7 F1 on TyDi QA, a challenging multilingual benchmark, despite having fewer model parameters.
69