SlideShare a Scribd company logo

Attention is all you need 설명

•
•
J
Junho Lee

구글에서 발표한 Attention 논문 설명입니다.

Education
1 of 14
Download to read offline
Attention is All you Need Ashish Vaswani et al.(2017) 한국 인공지능 연구소 이준호
• 기존에 사용되던 RNN / CNN 방식과 다른 FNN과 Skip Connection 기반의 모듈을 N개 쌓아서 LSTM, GRU같은 아키텍 처 없이 long-term dependency를 효과적으로 해결한 새로운 인코더-디코더 방식 • Source 단어와 Output 사이 distance가 짧아 Attention으로 정 보가 직접적으로 연결 • RNN의 순차적 특성으로 병렬 처리가 힘들다는 단점을 극복해 서 약 30배 가량의 학습속도 개선
Motivation(Google Research Blog) • “I arrived at the bank after crossing the …” • I arrived at the bank after crossing the road (Bank  은행) • I arrived at the bank after crossing the river (Bank  강둑) RNN은 순차적 언어 처리  word distance 커질수록 분석을 위한 타임스탭 숫자가 증가. CNN을 이용한 최신 언어 모델의 경우에도 word distance에 따른 단계(레이 어)가 증가한다.
Self-Attention Mechanism 밑줄된 부분에 따라서 it에 대한 가중치에 영향을 미침
Model Architecture • Left side : Encoder • Right side : Decoder • 각 단계에서 자동 회귀를 실시 • 즉, 이전 단계에서 나온 symbol을 다음 단계에 추 가한다.
Encoder • encode는 N = 6의 identical 레이어를 가진다. • 각 레이어는 2개의 Sub 레이어를 동반한다. • 처음은 Multi-head self attention mechanism • 뒤 이어서 Fully connected된 feed-forward network가 온다. • 모델에서 처리하는 기본 차원은 d_model = 512라고 가정
Decoder • Encoder같이 N = 6의 identical 레이어를 가짐 • Decoder는 3개의 Sub레이어를 가진다. • Encoder와 비슷하지만, input-feeding을 위해 디코더의 최종 출력이 다시 입력으로 들어간다.
• Input Embedding  pre-softmax linear transformation  text  probability vector • Positional Encoding - 기본 조건 : d_model = 512 (to summation) - use sine and cosine funtions pos is position i is the dimension - RNN / CNN 방식을 사용하는 것이 아니기 때문에 time step의 개념을 보 존해야한다. - sinusoidal 방법을 이용하면 inference시 긴 시퀀스에 대해서 ‘extrapolate’(추론) 할 수 있게 될 뿐 아니라, 어떠한 offset k에 대해서도 PE_{pos + k} = a * PE_{pos} + b 형태로 표현 가능하다고 한다.... 어렵네요 ㅠ
Q : 디코더 이전 레이어 hidden state K : 인코더의 output key V : 인코더의 output value 출처 : http://dalpo0814.tistory.com/category/Machine%20Learning
Scaled Dot Product Q와 K의 유사도를 계산 너무 큰 값이 지배적이지 않도록 스케일링 • Scale 과 normalize의 차이 Scale – mean 0, Var 1으로 조정, 주로 overflow & underflow를 방지 Normalize – 전체 구간을 설정, 데이터 군 내에서 특정 데이터의 위치를 확인 - 수식 : (요소값 – 최소값) / (최대값 – 최소값) 유사도  가중치 (총합 == 1) 가중치를 V와 곱연산 정리 : 이번 상태의 key와 value 페어인 {K, V}가 이전 상태인 Q와 어떤 연관이 있을것이다. 그러므로 K와 Q의 유사도를 계산하고 그걸 V에 반영 결국 V에 더 많은 Q의 정보가 전달된다.
Multi-Head Attention • 지금까지 얘기했던 V, K, Q가 h개의 Scaled Dot-Product 를 통과하면서 h번의 선형 projection 뒤 결과들을 concat 해서 사용하면 더 좋다는 사실이 발견되었다. Projection으로 인한 dimension decrease 때문에 계산량이 큰 폭으로 증가하지 않는다.
Position-wise Feed-Forward Networks Function • 인코더와 디코더에 fully connected로 연결되어있으며, 개별적 으로 동일하게 적용된다. • ReLU activation을 포함함 두가지 선형변환으로 구성
출처 : https://research.googleblog.com/2017/08/tr ansformer-novel-neural-network.html
마무리하며 • 페이퍼를 가지고 스터디 자료를 만들면서 최대한 많은 정보를 담으려고 노력했습니다만, 개인적인 부족함으로 100% 이해하 지 못한 부분이 있어서 정보 전달이 떨어지지 않을까 싶습니다 • 관련 페이퍼는 https://arxiv.org/abs/1706.03762에서 직접 확인 하실수 있습니다. • Google Research Blog와 http://dalpo0814.tistory.com/category/Machine%20Learning를 열심히 참고했습니다.

Recommended

Attention is all you need
Attention is all you needAttention is all you need
Attention is all you needHoon Heo
 
[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You Need[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You NeedDaiki Tanaka
 
Word embeddings, RNN, GRU and LSTM
Word embeddings, RNN, GRU and LSTMWord embeddings, RNN, GRU and LSTM
Word embeddings, RNN, GRU and LSTMDivya Gera
 
Anomaly Detection based on Diffusion
Anomaly Detection based on DiffusionAnomaly Detection based on Diffusion
Anomaly Detection based on Diffusionssuserbaebf8
 
Attention Is All You Need
Attention Is All You NeedAttention Is All You Need
Attention Is All You NeedIllia Polosukhin
 
Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Jeong-Gwan Lee
 
RNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataRNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataYao-Chieh Hu
 
LSTM
LSTMLSTM
LSTM佳蓉 倪
 

Recommended

Attention is all you need
Attention is all you needAttention is all you need
Attention is all you needHoon Heo
 
[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You Need[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You NeedDaiki Tanaka
 
Word embeddings, RNN, GRU and LSTM
Word embeddings, RNN, GRU and LSTMWord embeddings, RNN, GRU and LSTM
Word embeddings, RNN, GRU and LSTMDivya Gera
 
Anomaly Detection based on Diffusion
Anomaly Detection based on DiffusionAnomaly Detection based on Diffusion
Anomaly Detection based on Diffusionssuserbaebf8
 
Attention Is All You Need
Attention Is All You NeedAttention Is All You Need
Attention Is All You NeedIllia Polosukhin
 
Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Jeong-Gwan Lee
 
RNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataRNN & LSTM: Neural Network for Sequential Data
RNN & LSTM: Neural Network for Sequential DataYao-Chieh Hu
 
LSTM
LSTMLSTM
LSTM佳蓉 倪
 
ShuffleNet - PR054
ShuffleNet - PR054ShuffleNet - PR054
ShuffleNet - PR054Jinwon Lee
 
LSTM Tutorial
LSTM TutorialLSTM Tutorial
LSTM TutorialRalph Schlosser
 
여러 컨볼루션 레이어 테크닉과 경량화 기법들
여러 컨볼루션 레이어 테크닉과 경량화 기법들여러 컨볼루션 레이어 테크닉과 경량화 기법들
여러 컨볼루션 레이어 테크닉과 경량화 기법들Dongyi Kim
 
Software Development for Space-group Analysis: Magnetic Space Group and Irred...
Software Development for Space-group Analysis: Magnetic Space Group and Irred...Software Development for Space-group Analysis: Magnetic Space Group and Irred...
Software Development for Space-group Analysis: Magnetic Space Group and Irred...Kohei Shinohara
 
Attention mechanism 소개 자료
Attention mechanism 소개 자료Attention mechanism 소개 자료
Attention mechanism 소개 자료Whi Kwon
 
A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)Shuntaro Yada
 
Recurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: TheoryRecurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: TheoryAndrii Gakhov
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkYan Xu
 
Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural NetworksSharath TS
 
Activation function
Activation functionActivation function
Activation functionRakshithGowdakodihal
 
NLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Bootcamp 2018 : Representation Learning of text for NLPNLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Bootcamp 2018 : Representation Learning of text for NLPAnuj Gupta
 
Sarcasm Detection: Achilles Heel of sentiment analysis
Sarcasm Detection: Achilles Heel of sentiment analysisSarcasm Detection: Achilles Heel of sentiment analysis
Sarcasm Detection: Achilles Heel of sentiment analysisAnuj Gupta
 
Mask R-CNN
Mask R-CNNMask R-CNN
Mask R-CNNJaehyun Jun
 
Attention scores and mechanisms
Attention scores and mechanismsAttention scores and mechanisms
Attention scores and mechanismsJaeHo Jang
 
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...Simplilearn
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Larry Guo
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning健程 杨
 
Attention
AttentionAttention
AttentionSEMINARGROOT
 
PR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
PR-108: MobileNetV2: Inverted Residuals and Linear BottlenecksPR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
PR-108: MobileNetV2: Inverted Residuals and Linear BottlenecksJinwon Lee
 
CNN Tutorial
CNN TutorialCNN Tutorial
CNN TutorialSungjoon Choi
 
Bidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionBidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionWoodam Lim
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural NetworksSanghoon Yoon
 

More Related Content

What's hot

ShuffleNet - PR054
ShuffleNet - PR054ShuffleNet - PR054
ShuffleNet - PR054Jinwon Lee
 
여러 컨볼루션 레이어 테크닉과 경량화 기법들
여러 컨볼루션 레이어 테크닉과 경량화 기법들여러 컨볼루션 레이어 테크닉과 경량화 기법들
여러 컨볼루션 레이어 테크닉과 경량화 기법들Dongyi Kim
 
Software Development for Space-group Analysis: Magnetic Space Group and Irred...
Software Development for Space-group Analysis: Magnetic Space Group and Irred...Software Development for Space-group Analysis: Magnetic Space Group and Irred...
Software Development for Space-group Analysis: Magnetic Space Group and Irred...Kohei Shinohara
 
Attention mechanism 소개 자료
Attention mechanism 소개 자료Attention mechanism 소개 자료
Attention mechanism 소개 자료Whi Kwon
 
A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)Shuntaro Yada
 
Recurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: TheoryRecurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: TheoryAndrii Gakhov
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkYan Xu
 
Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural NetworksSharath TS
 
NLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Bootcamp 2018 : Representation Learning of text for NLPNLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Bootcamp 2018 : Representation Learning of text for NLPAnuj Gupta
 
Sarcasm Detection: Achilles Heel of sentiment analysis
Sarcasm Detection: Achilles Heel of sentiment analysisSarcasm Detection: Achilles Heel of sentiment analysis
Sarcasm Detection: Achilles Heel of sentiment analysisAnuj Gupta
 
Attention scores and mechanisms
Attention scores and mechanismsAttention scores and mechanisms
Attention scores and mechanismsJaeHo Jang
 
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...Simplilearn
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Larry Guo
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning健程 杨
 
PR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
PR-108: MobileNetV2: Inverted Residuals and Linear BottlenecksPR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
PR-108: MobileNetV2: Inverted Residuals and Linear BottlenecksJinwon Lee
 

What's hot (20)

ShuffleNet - PR054
ShuffleNet - PR054ShuffleNet - PR054
ShuffleNet - PR054
 
LSTM Tutorial
LSTM TutorialLSTM Tutorial
LSTM Tutorial
 
여러 컨볼루션 레이어 테크닉과 경량화 기법들
여러 컨볼루션 레이어 테크닉과 경량화 기법들여러 컨볼루션 레이어 테크닉과 경량화 기법들
여러 컨볼루션 레이어 테크닉과 경량화 기법들
 
Software Development for Space-group Analysis: Magnetic Space Group and Irred...
Software Development for Space-group Analysis: Magnetic Space Group and Irred...Software Development for Space-group Analysis: Magnetic Space Group and Irred...
Software Development for Space-group Analysis: Magnetic Space Group and Irred...
 
Attention mechanism 소개 자료
Attention mechanism 소개 자료Attention mechanism 소개 자료
Attention mechanism 소개 자료
 
A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)
 
Recurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: TheoryRecurrent Neural Networks. Part 1: Theory
Recurrent Neural Networks. Part 1: Theory
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural Network
 
Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural Networks
 
Activation function
Activation functionActivation function
Activation function
 
NLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Bootcamp 2018 : Representation Learning of text for NLPNLP Bootcamp 2018 : Representation Learning of text for NLP
NLP Bootcamp 2018 : Representation Learning of text for NLP
 
Sarcasm Detection: Achilles Heel of sentiment analysis
Sarcasm Detection: Achilles Heel of sentiment analysisSarcasm Detection: Achilles Heel of sentiment analysis
Sarcasm Detection: Achilles Heel of sentiment analysis
 
Mask R-CNN
Mask R-CNNMask R-CNN
Mask R-CNN
 
Attention scores and mechanisms
Attention scores and mechanismsAttention scores and mechanisms
Attention scores and mechanisms
 
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
Deep Learning Tutorial | Deep Learning TensorFlow | Deep Learning With Neural...
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10) Deep Learning: Recurrent Neural Network (Chapter 10)
Deep Learning: Recurrent Neural Network (Chapter 10)
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning
 
Attention
AttentionAttention
Attention
 
PR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
PR-108: MobileNetV2: Inverted Residuals and Linear BottlenecksPR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
PR-108: MobileNetV2: Inverted Residuals and Linear Bottlenecks
 
CNN Tutorial
CNN TutorialCNN Tutorial
CNN Tutorial
 

Similar to Attention is all you need 설명

Bidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionBidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionWoodam Lim
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural NetworksSanghoon Yoon
 
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)Lee Seungeun
 
Chapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive netsChapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive netsKyeongUkJang
 
Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)Jaemin Cho
 
Chapter 17 monte carlo methods
Chapter 17 monte carlo methodsChapter 17 monte carlo methods
Chapter 17 monte carlo methodsKyeongUkJang
 
개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow양 한빛
 
Papago/N2MT 개발이야기
Papago/N2MT 개발이야기Papago/N2MT 개발이야기
Papago/N2MT 개발이야기NAVER D2
 
Howto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear RegressionHowto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear RegressionHyo jeong Lee
 
순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요Byoung-Hee Kim
 
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...Gyubin Son
 
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksOh Yoojin
 
[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경NAVER D2
 
Deep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniquesDeep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniquesKang Pilsung
 
A Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding AutoencoderA Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding AutoencoderLee Seungeun
 
네이버 NLP Challenge 후기
네이버 NLP Challenge 후기네이버 NLP Challenge 후기
네이버 NLP Challenge 후기Jangwon Park
 
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)Adonis Han
 
CNN for sentence classification
CNN for sentence classificationCNN for sentence classification
CNN for sentence classificationKyeongUkJang
 
2.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-32.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-3Haesun Park
 
Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks ISang Jun Lee
 

Similar to Attention is all you need 설명 (20)

Bidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionBidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehension
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural Networks
 
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
 
Chapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive netsChapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive nets
 
Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)
 
Chapter 17 monte carlo methods
Chapter 17 monte carlo methodsChapter 17 monte carlo methods
Chapter 17 monte carlo methods
 
개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow
 
Papago/N2MT 개발이야기
Papago/N2MT 개발이야기Papago/N2MT 개발이야기
Papago/N2MT 개발이야기
 
Howto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear RegressionHowto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear Regression
 
순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요
 
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
 
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
 
[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경
 
Deep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniquesDeep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniques
 
A Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding AutoencoderA Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding Autoencoder
 
네이버 NLP Challenge 후기
네이버 NLP Challenge 후기네이버 NLP Challenge 후기
네이버 NLP Challenge 후기
 
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
 
CNN for sentence classification
CNN for sentence classificationCNN for sentence classification
CNN for sentence classification
 
2.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-32.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-3
 
Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks I
 

Attention is all you need 설명

  • 1. Attention is All you Need Ashish Vaswani et al.(2017) 한국 인공지능 연구소 이준호
  • 2. • 기존에 사용되던 RNN / CNN 방식과 다른 FNN과 Skip Connection 기반의 모듈을 N개 쌓아서 LSTM, GRU같은 아키텍 처 없이 long-term dependency를 효과적으로 해결한 새로운 인코더-디코더 방식 • Source 단어와 Output 사이 distance가 짧아 Attention으로 정 보가 직접적으로 연결 • RNN의 순차적 특성으로 병렬 처리가 힘들다는 단점을 극복해 서 약 30배 가량의 학습속도 개선
  • 3. Motivation(Google Research Blog) • “I arrived at the bank after crossing the …” • I arrived at the bank after crossing the road (Bank  은행) • I arrived at the bank after crossing the river (Bank  강둑) RNN은 순차적 언어 처리  word distance 커질수록 분석을 위한 타임스탭 숫자가 증가. CNN을 이용한 최신 언어 모델의 경우에도 word distance에 따른 단계(레이 어)가 증가한다.
  • 4. Self-Attention Mechanism 밑줄된 부분에 따라서 it에 대한 가중치에 영향을 미침
  • 5. Model Architecture • Left side : Encoder • Right side : Decoder • 각 단계에서 자동 회귀를 실시 • 즉, 이전 단계에서 나온 symbol을 다음 단계에 추 가한다.
  • 6. Encoder • encode는 N = 6의 identical 레이어를 가진다. • 각 레이어는 2개의 Sub 레이어를 동반한다. • 처음은 Multi-head self attention mechanism • 뒤 이어서 Fully connected된 feed-forward network가 온다. • 모델에서 처리하는 기본 차원은 d_model = 512라고 가정
  • 7. Decoder • Encoder같이 N = 6의 identical 레이어를 가짐 • Decoder는 3개의 Sub레이어를 가진다. • Encoder와 비슷하지만, input-feeding을 위해 디코더의 최종 출력이 다시 입력으로 들어간다.
  • 8. • Input Embedding  pre-softmax linear transformation  text  probability vector • Positional Encoding - 기본 조건 : d_model = 512 (to summation) - use sine and cosine funtions pos is position i is the dimension - RNN / CNN 방식을 사용하는 것이 아니기 때문에 time step의 개념을 보 존해야한다. - sinusoidal 방법을 이용하면 inference시 긴 시퀀스에 대해서 ‘extrapolate’(추론) 할 수 있게 될 뿐 아니라, 어떠한 offset k에 대해서도 PE_{pos + k} = a * PE_{pos} + b 형태로 표현 가능하다고 한다.... 어렵네요 ㅠ
  • 9. Q : 디코더 이전 레이어 hidden state K : 인코더의 output key V : 인코더의 output value 출처 : http://dalpo0814.tistory.com/category/Machine%20Learning
  • 10. Scaled Dot Product Q와 K의 유사도를 계산 너무 큰 값이 지배적이지 않도록 스케일링 • Scale 과 normalize의 차이 Scale – mean 0, Var 1으로 조정, 주로 overflow & underflow를 방지 Normalize – 전체 구간을 설정, 데이터 군 내에서 특정 데이터의 위치를 확인 - 수식 : (요소값 – 최소값) / (최대값 – 최소값) 유사도  가중치 (총합 == 1) 가중치를 V와 곱연산 정리 : 이번 상태의 key와 value 페어인 {K, V}가 이전 상태인 Q와 어떤 연관이 있을것이다. 그러므로 K와 Q의 유사도를 계산하고 그걸 V에 반영 결국 V에 더 많은 Q의 정보가 전달된다.
  • 11. Multi-Head Attention • 지금까지 얘기했던 V, K, Q가 h개의 Scaled Dot-Product 를 통과하면서 h번의 선형 projection 뒤 결과들을 concat 해서 사용하면 더 좋다는 사실이 발견되었다. Projection으로 인한 dimension decrease 때문에 계산량이 큰 폭으로 증가하지 않는다.
  • 12. Position-wise Feed-Forward Networks Function • 인코더와 디코더에 fully connected로 연결되어있으며, 개별적 으로 동일하게 적용된다. • ReLU activation을 포함함 두가지 선형변환으로 구성
  • 14. 마무리하며 • 페이퍼를 가지고 스터디 자료를 만들면서 최대한 많은 정보를 담으려고 노력했습니다만, 개인적인 부족함으로 100% 이해하 지 못한 부분이 있어서 정보 전달이 떨어지지 않을까 싶습니다 • 관련 페이퍼는 https://arxiv.org/abs/1706.03762에서 직접 확인 하실수 있습니다. • Google Research Blog와 http://dalpo0814.tistory.com/category/Machine%20Learning를 열심히 참고했습니다.