Attention Is All You Need

2 min read

The Transformer drops recurrence and convolution from sequence-to-sequence models and keeps only attention: an encoder and a decoder built from stacked self-attention and feed-forward layers, with scaled dot-product attention split across several heads and sinusoidal position encodings. Every position can attend to every other in one step, so a layer needs a constant number of sequential operations, against one per position for a recurrent layer, at a cost of O(n²·d) per layer.

Findings

  • WMT 2014 English-German: 28.4 BLEU, more than 2 BLEU above the earlier best results including ensembles, after 3.5 days on 8 P100 GPUs.
  • WMT 2014 English-French: 41.8 BLEU for a single model, at a small fraction of the training cost of earlier best models.
  • Ablations: one attention head is 0.9 BLEU worse than the best setting and too many heads also hurts; shrinking the key size hurts; learned position embeddings match the sinusoids almost exactly.
  • English constituency parsing on WSJ (about 40K training sentences) beats the BerkeleyParser even without extra data; only the Recurrent Neural Network Grammar does better.

Limits

  • Cost grows with the square of sequence length. The authors suggest restricting attention to a neighbourhood for long inputs and leave it to future work.
  • Text only. Images, audio and video were listed as future work.
  • Decoding stays autoregressive.

Relevance

Every VLM, VLA and LLM judge in this cluster runs on this architecture, so their shared failure modes (context limits, sensitivity to position and order) start here. See foundation-models-for-robotics.