Topic: Transformers

← Back to home

1. Attention Is All You Need

   Paper • Jun 12, 2017 • arXiv • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

   The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder ...

2. Generating Long Sequences with Sparse Attention

   Paper • Apr 23, 2019 • arXiv • Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever

   Transformers are powerful sequence models, but their self-attention mechanism scales quadratically with sequence length, making them computationally prohibitive for long inputs like high-resolution...

3. Improving Language Understanding by Generative Pre-Training

   Paper • Jun 11, 2018 • OpenAI • Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever

   Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large un...

4. RT-1: Robotics Transformer for Real-World Control at Scale

   Paper • Dec 13, 2022 • arXiv • Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Google DeepMind

   By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets. We i...

5. Perceiver: General Perception with Iterative Attention

   Paper • Mar 4, 2021 • arXiv • Andrew Jaegle, Felix Gimeno, Andrew Brock, Oriol Vinyals, Andrew Zisserman, Joao Carreira

   Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, and touch. We introduce the Perceiver, an architecture tha...

6. Generative Pretraining from Pixels

   Paper • Jun 17, 2020 • OpenAI • Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever

   Inspired by the success of unsupervised representation learning in natural language processing with models like GPT-2, we examine whether similar models can learn useful representations for images....