Masked Language Modeling for Proteins via Linearly Scalable Long-Context\n Transformers

Abstract

Transformer models have achieved state-of-the-art results across a diverse\nrange of domains. However, concern over the cost of training the attention\nmechanism to learn complex dependencies between distant inputs continues to\ngrow. In response, solutions that exploit the structure and sparsity of the\nlearned attention matrix have blossomed. However, real-world applications that\ninvolve long sequences, such as biological sequence analysis, may fall short of\nmeeting these assumptions, precluding exploration of these models. To address\nthis challenge, we present a new Transformer architecture, Performer, based on\nFast Attention Via Orthogonal Random features (FAVOR). Our mechanism scales\nlinearly rather than quadratically in the number of tokens in the sequence, is\ncharacterized by sub-quadratic space complexity and does not incorporate any\nsparsity pattern priors. Furthermore, it provides strong theoretical\nguarantees: unbiased estimation of the attention matrix and uniform\nconvergence. It is also backwards-compatible with pre-trained regular\nTransformers. We demonstrate its effectiveness on the challenging task of\nprotein sequence modeling and provide detailed theoretical analysis.\n