Concepedia

Subword Tokenization

Subword Tokenization is a text preprocessing method that segments words into subword units by balancing the trade-off between character-level and word-level representations, thereby enabling efficient handling of out-of-vocabulary terms and reducing vocabulary sparsity in natural language processing tasks.

56

Publications

2.9K

Citations

213

Authors

66

Institutions

Publications per year

2017–2026

51

Parents

Publications

56

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Taku Kudo, John T. E. Richardson · 2018 · 286 citations · Full text

, , +13

CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

2021 · 269 citations · Full text

, , +12

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

Linting Xue, Aditya Barua, Noah Constant et al. · Transactions of the Association for Computational Linguistics · 2022 · 238 citations · Full text

, , +15

A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition

Aren Jansen, Emmanuel Dupoux, Sharon Goldwater et al. · 2013 · 137 citations · Full text

, , +15

BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

Benjamin Heinzerling, Michael Strube · arXiv (Cornell University) · 2017 · 130 citations · Full text

, , +17

1–5 of 56

Authors

213

  1. CR
    Colin Raffel

    Google (United States), United States

    3 publications · 413 citations

    3

    h-index

  2. TL
    Thomas Lavergne

    Norwegian Meteorological Institute, Norway

    2 publications · 151 citations

    2

    h-index

  3. HN
    Hiroshi Noji

    Nara Institute of Science and Technology, Japan

    2 publications · 151 citations

    2

    h-index

  4. LX
    Linting Xue

    Google (United States), United States

    2 publications · 307 citations

    2

    h-index

  5. CP
    Chanjun Park

    Korea University, South Korea

    2 publications · 58 citations

    2

    h-index

1–5 of 213

Institutions

66

  1. Google (United States)

    Mountain View, United States

    6 h-index

    27 publications

    3.4K citations

  2. University of Cambridge

    Cambridge, United Kingdom

    3 h-index

    5 publications

    210 citations

  3. Johns Hopkins University

    Baltimore, United States

    3 h-index

    11 publications

    1.2K citations

  4. Commissariat à l'Énergie Atomique et aux Énergies Alternatives

    Paris, France

    2 h-index

    3 publications

    179 citations

  5. Stanford University

    Stanford, United States

    2 h-index

    6 publications

    340 citations

  6. Sulzer (Germany)

    Bruchsal, Germany

    2 h-index

    2 publications

    91 citations

  7. Korea University

    Seoul, South Korea

    2 h-index

    6 publications

    174 citations

  8. National Institute of Advanced Industrial Science and Technology

    Tsukuba, Japan

    2 h-index

    6 publications

    300 citations

1–8 of 66

Venues

  1. arXiv (Cornell University)

    Repository · 15 publications · 624 citations

    12

    h-index

  2. 2

    h-index

  3. IEEE Access

    2 publications · 47 citations

    2

    h-index

  4. Proceedings of the AAAI Conference on Artificial Intelligence

    Conference · 2 publications · 139 citations

    2

    h-index

  5. 2

    h-index

1–5