Concepedia

CoLT5: Faster Long-Range Transformers with Conditional Computation

Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay,

2023 · 25 citations · 25 references

DOIFull text

Open access

Concepts

Abstract

Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

References

25

Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer

Colin Raffel, Noam Shazeer, Adam Roberts et al. · arXiv (Cornell University) · 2019 · 8.3K citations · Full text

, , +14

Longformer: The Long-Document Transformer

arXiv (Cornell University) · 2020 · 2.2K citations · Full text

, , +14

Natural Questions: A Benchmark for Question Answering Research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield et al. · Transactions of the Association for Computational Linguistics · 2019 · 2K citations · Full text

, , +12

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan et al. · arXiv (Cornell University) · 2020 · 1.5K citations · Full text

, , +16

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Jingqing Zhang, Yao Zhao, Mohammad Saleh et al. · arXiv (Cornell University) · 2019 · 981 citations · Full text

, , +14