Research paper
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
- Published
- NeurIPS, 2017
- Finished
- Finished April 2026
aillmmachine-learning
Key takeawaySelf-attention alone, without recurrence, is enough to model sequences — and it parallelises.
Notes
- Scaled dot-product attention:
softmax(QKᵀ / √d_k) V. - Multi-head attention lets the model attend to different representation subspaces at once.
- Positional encodings replace the ordering that recurrence used to provide.