The document discusses Ashish Vaswani's talk on the Transformer architecture, highlighting its use of attention mechanisms for sequence transduction tasks. The proposed model outperforms traditional recurrent and convolutional neural networks in machine translation and other applications by enhancing parallelization and reducing training time. It introduces the concept of self-attention and positional encoding, showcasing significant improvements in translation quality.

2. Introduction
Introduction

3. Introduction
From Ashish Vaswani’s Talk
… the purpose is … not going to be just to talk about a
particular model, but, as … starting point is always asking a question
what are the … what’s the structure in my dataset or what are the
symmetries in my dataset and is there a model that exists that has the
Inductive biases to model these properties that exist in my dataset.
… deep learning, … is all about representation learning … building the
right tools for learning representations is important factor in
achieving empirical success.

4. Abstract
Abstract

5. Abstract
The dominant sequence transduction models are based on complex recurrent
or convolutional neural networks that include an encoder and a decoder. The
best performing models also connect the encoder and decoder through an
attention mechanism. We propose a new simple network architecture, the
Transformer, based solely on attention mechanisms, dispensing with
recurrence and convolutions entirely. Experiments on two machine translation
tasks show these models to be superior in quality while being more
parallelizable and requiring significantly less time to train.
Our model achieves 28.4 BLEU on the WMT 2014 English to-German translation
task, improving over the existing best results, including ensembles, by over 2
BLEU. We show that the Transformer generalizes well to other tasks by
applying it successfully to English constituency parsing both with large and
limited training data.

6. Background
Background

7. Background
Sequence-to-sequence with pure RNN

8. Background
Sequence-to-sequence with Attention

9. Background
What Attention do in VQA

10. Transformer
Paradigm Shift in Seq2Seq !
Transformer

11. Transformer
Why not RNN?
This inherently sequential nature precludes parallelization within training
examples, which becomes critical at longer sequence lengths, as memory
constraints limit batching across examples. Recent work has achieved
significant improvements in computational efficiency through factorization
tricks [21] and conditional computation [32], while also improving model
performance in case of the latter. The fundamental constraint of
sequential computation, however, remains.

12. Transformer
Convolutional Neural Networks
for Sentence Classification
In these models, the number of operations
required to relate signals from two arbitrary
input or output positions grows in the
distance between positions, linearly for
ConvS2S and logarithmically for ByteNet. This
makes it more difficult to learn dependencies
between distant positions

13. Transformer
Transformer Architecture

14. Transformer
Transformer Architecture
Positional encoding

15. Transformer
Positional Encoding
𝑃𝐸(𝑝𝑜𝑠, 2𝑖) = sin(𝑝𝑜𝑠/100002𝑖/𝑑 𝑚𝑜𝑑𝑒𝑙)
𝑃𝐸(𝑝𝑜𝑠,2𝑖 + 1) = cos(𝑝𝑜𝑠/100002𝑖/𝑑 𝑚𝑜𝑑𝑒𝑙)

16. Transformer
Self Attention

17. Transformer
Self Attention

18. Transformer
Self Attention
𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑄, 𝐾, 𝑉 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥
𝑄𝐾 𝑇
𝑑 𝑘
𝑉

19. Transformer
Multi-Head Attention

20. Transformer
Multi-Head Attention

21. Transformer
Decoder : Masked Multi-Head Attention
Due to teacher forcing

22. Transformer
Transformer Architecture
Final encoding outputs are used for Key
and Value vector of multi-head attention
in Decoder

23. Transformer
Regularization
1. 𝐿𝑎𝑏𝑒𝑙 𝑆𝑚𝑜𝑜𝑡ℎ𝑖𝑛𝑔
2. 𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙 𝐷𝑟𝑜𝑝𝑜𝑢𝑡

24. Results
Results

25. Results

26. References
References

27. References
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia
Polosukhin. “Attention Is All You Need.” (NIPS) 2017.
Paper link: https://arxiv.org/pdf/1706.03762.pdf
Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 14 – Transformers and Self-Attention
Youtube: https://youtu.be/5vcj8kSwBCY
Transformer (Attention Is All You Need) by Minsuk Heo
Youtube: https://youtu.be/mxGCEWOxfe8