This document describes a new model called Attention is All You Need that uses attention mechanisms without recurrent or convolutional layers. It introduces a model architecture that uses multi-head attention and feed-forward networks along with techniques like dropout and label smoothing. The model is evaluated on machine translation tasks using WMT 2014 English-German and English-French datasets and outperforms existing sequence models.

1. Attention is All
You Need
June 21, 2021
Hee Dae Kwon

2. contents
• Introduction
• Model architecture
• Method
• Experiments
• Results

3. Introduction
• Previous
Recurrent 모델 사용
=> 병렬처리 제한
Attemtion 추가
=> 여전한 시퀀스 길이의 제약

4. Introduction
• Solution
= RNN 모델 제거
Attention mechanism만 활용

5. Model Architecture
•
•

6. Method
• Q = Query
• K = Key
• V = Value

7. Method
•
• Multi-Head Attention

8. Method
• Position-wise Feed-Forward Networks
• Embeddings and Softmax
•

9. experiments
텍스트를 입력하십시오
Dataset Optimizer Reqularization
WMT 2014
English-German dataset
WMT 2014
English-French dataset
Adam optimize Dropout
Label Smoothing

10. Results

11. • 감사합니다