Generating Long Sequences with Sparse Attention

ABSTRACT

Transformers are powerful sequence models, but their self-attention mechanism scales quadratically with sequence length, making them computationally prohibitive for long inputs like high-resolution images, audio, or entire books. We introduce Sparse Transformers, which modify the full attention matrix into factorized, sparse attention patterns. By employing specific strided and local attention kernels, we reduce the memory and compute complexity from O(N^2) to O(N * sqrt(N)). This efficiency allows us to train generative models on sequences of tens of thousands of tokens. We achieve state-of-the-art autoregressive modeling results on CIFAR-10, Enwik8, and ImageNet-64, demonstrating that dense attention is not strictly necessary for long-range dependency modeling.