SlideShare a Scribd company logo

Attention is all you need 설명

•
•
J
Junho Lee

구글에서 발표한 Attention 논문 설명입니다.

1 of 14
Download to read offline
Attention is All you Need Ashish Vaswani et al.(2017) 한국 인공지능 연구소 이준호
• 기존에 사용되던 RNN / CNN 방식과 다른 FNN과 Skip Connection 기반의 모듈을 N개 쌓아서 LSTM, GRU같은 아키텍 처 없이 long-term dependency를 효과적으로 해결한 새로운 인코더-디코더 방식 • Source 단어와 Output 사이 distance가 짧아 Attention으로 정 보가 직접적으로 연결 • RNN의 순차적 특성으로 병렬 처리가 힘들다는 단점을 극복해 서 약 30배 가량의 학습속도 개선
Motivation(Google Research Blog) • “I arrived at the bank after crossing the …” • I arrived at the bank after crossing the road (Bank  은행) • I arrived at the bank after crossing the river (Bank  강둑) RNN은 순차적 언어 처리  word distance 커질수록 분석을 위한 타임스탭 숫자가 증가. CNN을 이용한 최신 언어 모델의 경우에도 word distance에 따른 단계(레이 어)가 증가한다.
Self-Attention Mechanism 밑줄된 부분에 따라서 it에 대한 가중치에 영향을 미침
Model Architecture • Left side : Encoder • Right side : Decoder • 각 단계에서 자동 회귀를 실시 • 즉, 이전 단계에서 나온 symbol을 다음 단계에 추 가한다.
Encoder • encode는 N = 6의 identical 레이어를 가진다. • 각 레이어는 2개의 Sub 레이어를 동반한다. • 처음은 Multi-head self attention mechanism • 뒤 이어서 Fully connected된 feed-forward network가 온다. • 모델에서 처리하는 기본 차원은 d_model = 512라고 가정
Decoder • Encoder같이 N = 6의 identical 레이어를 가짐 • Decoder는 3개의 Sub레이어를 가진다. • Encoder와 비슷하지만, input-feeding을 위해 디코더의 최종 출력이 다시 입력으로 들어간다.
• Input Embedding  pre-softmax linear transformation  text  probability vector • Positional Encoding - 기본 조건 : d_model = 512 (to summation) - use sine and cosine funtions pos is position i is the dimension - RNN / CNN 방식을 사용하는 것이 아니기 때문에 time step의 개념을 보 존해야한다. - sinusoidal 방법을 이용하면 inference시 긴 시퀀스에 대해서 ‘extrapolate’(추론) 할 수 있게 될 뿐 아니라, 어떠한 offset k에 대해서도 PE_{pos + k} = a * PE_{pos} + b 형태로 표현 가능하다고 한다.... 어렵네요 ㅠ
Q : 디코더 이전 레이어 hidden state K : 인코더의 output key V : 인코더의 output value 출처 : http://dalpo0814.tistory.com/category/Machine%20Learning
Scaled Dot Product Q와 K의 유사도를 계산 너무 큰 값이 지배적이지 않도록 스케일링 • Scale 과 normalize의 차이 Scale – mean 0, Var 1으로 조정, 주로 overflow & underflow를 방지 Normalize – 전체 구간을 설정, 데이터 군 내에서 특정 데이터의 위치를 확인 - 수식 : (요소값 – 최소값) / (최대값 – 최소값) 유사도  가중치 (총합 == 1) 가중치를 V와 곱연산 정리 : 이번 상태의 key와 value 페어인 {K, V}가 이전 상태인 Q와 어떤 연관이 있을것이다. 그러므로 K와 Q의 유사도를 계산하고 그걸 V에 반영 결국 V에 더 많은 Q의 정보가 전달된다.
Multi-Head Attention • 지금까지 얘기했던 V, K, Q가 h개의 Scaled Dot-Product 를 통과하면서 h번의 선형 projection 뒤 결과들을 concat 해서 사용하면 더 좋다는 사실이 발견되었다. Projection으로 인한 dimension decrease 때문에 계산량이 큰 폭으로 증가하지 않는다.
Position-wise Feed-Forward Networks Function • 인코더와 디코더에 fully connected로 연결되어있으며, 개별적 으로 동일하게 적용된다. • ReLU activation을 포함함 두가지 선형변환으로 구성
출처 : https://research.googleblog.com/2017/08/tr ansformer-novel-neural-network.html
마무리하며 • 페이퍼를 가지고 스터디 자료를 만들면서 최대한 많은 정보를 담으려고 노력했습니다만, 개인적인 부족함으로 100% 이해하 지 못한 부분이 있어서 정보 전달이 떨어지지 않을까 싶습니다 • 관련 페이퍼는 https://arxiv.org/abs/1706.03762에서 직접 확인 하실수 있습니다. • Google Research Blog와 http://dalpo0814.tistory.com/category/Machine%20Learning를 열심히 참고했습니다.

Recommended

Attention is all you need
Attention is all you needAttention is all you need
Attention is all you need
Hoon Heo
 
[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You Need[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You Need
Daiki Tanaka
 
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Universitat Politècnica de Catalunya
 
Attention
AttentionAttention
Attention
SEMINARGROOT
 
Attention Is All You Need
Attention Is All You NeedAttention Is All You Need
Attention Is All You Need
Illia Polosukhin
 
Anomaly Detection based on Diffusion
Anomaly Detection based on DiffusionAnomaly Detection based on Diffusion
Anomaly Detection based on Diffusion
ssuserbaebf8
 
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
홍배 김
 
CNN Attention Networks
CNN Attention NetworksCNN Attention Networks
CNN Attention Networks
Taeoh Kim
 

Recommended

Attention is all you need
Attention is all you needAttention is all you need
Attention is all you need
Hoon Heo
 
[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You Need[Paper Reading] Attention is All You Need
[Paper Reading] Attention is All You Need
Daiki Tanaka
 
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Attention is all you need (UPC Reading Group 2018, by Santi Pascual)
Universitat Politècnica de Catalunya
 
Attention
AttentionAttention
Attention
SEMINARGROOT
 
Attention Is All You Need
Attention Is All You NeedAttention Is All You Need
Attention Is All You Need
Illia Polosukhin
 
Anomaly Detection based on Diffusion
Anomaly Detection based on DiffusionAnomaly Detection based on Diffusion
Anomaly Detection based on Diffusion
ssuserbaebf8
 
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
홍배 김
 
CNN Attention Networks
CNN Attention NetworksCNN Attention Networks
CNN Attention Networks
Taeoh Kim
 
Deep learning with tensorflow
Deep learning with tensorflowDeep learning with tensorflow
Deep learning with tensorflow
Charmi Chokshi
 
Deep Learning for Computer Vision: Attention Models (UPC 2016)
Deep Learning for Computer Vision: Attention Models (UPC 2016)Deep Learning for Computer Vision: Attention Models (UPC 2016)
Deep Learning for Computer Vision: Attention Models (UPC 2016)
Universitat Politècnica de Catalunya
 
Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Attention is All You Need (Transformer)
Attention is All You Need (Transformer)
Jeong-Gwan Lee
 
[한국어] Neural Architecture Search with Reinforcement Learning
[한국어] Neural Architecture Search with Reinforcement Learning[한국어] Neural Architecture Search with Reinforcement Learning
[한국어] Neural Architecture Search with Reinforcement Learning
Kiho Suh
 
NLP using transformers
NLP using transformers NLP using transformers
NLP using transformers
Arvind Devaraj
 
ConvNetの歴史とResNet亜種、ベストプラクティス
ConvNetの歴史とResNet亜種、ベストプラクティスConvNetの歴史とResNet亜種、ベストプラクティス
ConvNetの歴史とResNet亜種、ベストプラクティス
Yusuke Uchida
 
Densely Connected Convolutional Networks
Densely Connected Convolutional NetworksDensely Connected Convolutional Networks
Densely Connected Convolutional Networks
Hosein Mohebbi
 
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
Deep Learning JP
 
Introduction For seq2seq(sequence to sequence) and RNN
Introduction For seq2seq(sequence to sequence) and RNNIntroduction For seq2seq(sequence to sequence) and RNN
Introduction For seq2seq(sequence to sequence) and RNN
Hye-min Ahn
 
Conditional Image Generation with PixelCNN Decoders
Conditional Image Generation with PixelCNN DecodersConditional Image Generation with PixelCNN Decoders
Conditional Image Generation with PixelCNN Decoders
suga93
 
rnn BASICS
rnn BASICSrnn BASICS
rnn BASICS
Priyanka Reddy
 
Wasserstein GAN 수학 이해하기 I
Wasserstein GAN 수학 이해하기 IWasserstein GAN 수학 이해하기 I
Wasserstein GAN 수학 이해하기 I
Sungbin Lim
 
Attention mechanism 소개 자료
Attention mechanism 소개 자료Attention mechanism 소개 자료
Attention mechanism 소개 자료
Whi Kwon
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
Seq2Seq (encoder decoder) model
Seq2Seq (encoder decoder) modelSeq2Seq (encoder decoder) model
Seq2Seq (encoder decoder) model
佳蓉 倪
 
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
Edge AI and Vision Alliance
 
Transformer Introduction (Seminar Material)
Transformer Introduction (Seminar Material)Transformer Introduction (Seminar Material)
Transformer Introduction (Seminar Material)
Yuta Niki
 
딥러닝 기반의 자연어처리 최근 연구 동향
딥러닝 기반의 자연어처리 최근 연구 동향딥러닝 기반의 자연어처리 최근 연구 동향
딥러닝 기반의 자연어처리 최근 연구 동향
LGCNSairesearch
 
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
Deep Learning JP
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networks
Si Haem
 
Bidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionBidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehension
Woodam Lim
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural Networks
Sanghoon Yoon
 

More Related Content

What's hot

Deep learning with tensorflow
Deep learning with tensorflowDeep learning with tensorflow
Deep learning with tensorflow
Charmi Chokshi
 
Deep Learning for Computer Vision: Attention Models (UPC 2016)
Deep Learning for Computer Vision: Attention Models (UPC 2016)Deep Learning for Computer Vision: Attention Models (UPC 2016)
Deep Learning for Computer Vision: Attention Models (UPC 2016)
Universitat Politècnica de Catalunya
 
Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Attention is All You Need (Transformer)
Attention is All You Need (Transformer)
Jeong-Gwan Lee
 
[한국어] Neural Architecture Search with Reinforcement Learning
[한국어] Neural Architecture Search with Reinforcement Learning[한국어] Neural Architecture Search with Reinforcement Learning
[한국어] Neural Architecture Search with Reinforcement Learning
Kiho Suh
 
NLP using transformers
NLP using transformers NLP using transformers
NLP using transformers
Arvind Devaraj
 
ConvNetの歴史とResNet亜種、ベストプラクティス
ConvNetの歴史とResNet亜種、ベストプラクティスConvNetの歴史とResNet亜種、ベストプラクティス
ConvNetの歴史とResNet亜種、ベストプラクティス
Yusuke Uchida
 
Densely Connected Convolutional Networks
Densely Connected Convolutional NetworksDensely Connected Convolutional Networks
Densely Connected Convolutional Networks
Hosein Mohebbi
 
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
Deep Learning JP
 
Introduction For seq2seq(sequence to sequence) and RNN
Introduction For seq2seq(sequence to sequence) and RNNIntroduction For seq2seq(sequence to sequence) and RNN
Introduction For seq2seq(sequence to sequence) and RNN
Hye-min Ahn
 
Conditional Image Generation with PixelCNN Decoders
Conditional Image Generation with PixelCNN DecodersConditional Image Generation with PixelCNN Decoders
Conditional Image Generation with PixelCNN Decoders
suga93
 
rnn BASICS
rnn BASICSrnn BASICS
rnn BASICS
Priyanka Reddy
 
Wasserstein GAN 수학 이해하기 I
Wasserstein GAN 수학 이해하기 IWasserstein GAN 수학 이해하기 I
Wasserstein GAN 수학 이해하기 I
Sungbin Lim
 
Attention mechanism 소개 자료
Attention mechanism 소개 자료Attention mechanism 소개 자료
Attention mechanism 소개 자료
Whi Kwon
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
Seq2Seq (encoder decoder) model
Seq2Seq (encoder decoder) modelSeq2Seq (encoder decoder) model
Seq2Seq (encoder decoder) model
佳蓉 倪
 
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
Edge AI and Vision Alliance
 
Transformer Introduction (Seminar Material)
Transformer Introduction (Seminar Material)Transformer Introduction (Seminar Material)
Transformer Introduction (Seminar Material)
Yuta Niki
 
딥러닝 기반의 자연어처리 최근 연구 동향
딥러닝 기반의 자연어처리 최근 연구 동향딥러닝 기반의 자연어처리 최근 연구 동향
딥러닝 기반의 자연어처리 최근 연구 동향
LGCNSairesearch
 
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
Deep Learning JP
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networks
Si Haem
 

What's hot (20)

Deep learning with tensorflow
Deep learning with tensorflowDeep learning with tensorflow
Deep learning with tensorflow
 
Deep Learning for Computer Vision: Attention Models (UPC 2016)
Deep Learning for Computer Vision: Attention Models (UPC 2016)Deep Learning for Computer Vision: Attention Models (UPC 2016)
Deep Learning for Computer Vision: Attention Models (UPC 2016)
 
Attention is All You Need (Transformer)
Attention is All You Need (Transformer)Attention is All You Need (Transformer)
Attention is All You Need (Transformer)
 
[한국어] Neural Architecture Search with Reinforcement Learning
[한국어] Neural Architecture Search with Reinforcement Learning[한국어] Neural Architecture Search with Reinforcement Learning
[한국어] Neural Architecture Search with Reinforcement Learning
 
NLP using transformers
NLP using transformers NLP using transformers
NLP using transformers
 
ConvNetの歴史とResNet亜種、ベストプラクティス
ConvNetの歴史とResNet亜種、ベストプラクティスConvNetの歴史とResNet亜種、ベストプラクティス
ConvNetの歴史とResNet亜種、ベストプラクティス
 
Densely Connected Convolutional Networks
Densely Connected Convolutional NetworksDensely Connected Convolutional Networks
Densely Connected Convolutional Networks
 
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
[DL輪読会]Parallel WaveNet: Fast High-Fidelity Speech Synthesis
 
Introduction For seq2seq(sequence to sequence) and RNN
Introduction For seq2seq(sequence to sequence) and RNNIntroduction For seq2seq(sequence to sequence) and RNN
Introduction For seq2seq(sequence to sequence) and RNN
 
Conditional Image Generation with PixelCNN Decoders
Conditional Image Generation with PixelCNN DecodersConditional Image Generation with PixelCNN Decoders
Conditional Image Generation with PixelCNN Decoders
 
rnn BASICS
rnn BASICSrnn BASICS
rnn BASICS
 
Wasserstein GAN 수학 이해하기 I
Wasserstein GAN 수학 이해하기 IWasserstein GAN 수학 이해하기 I
Wasserstein GAN 수학 이해하기 I
 
Attention mechanism 소개 자료
Attention mechanism 소개 자료Attention mechanism 소개 자료
Attention mechanism 소개 자료
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
 
Seq2Seq (encoder decoder) model
Seq2Seq (encoder decoder) modelSeq2Seq (encoder decoder) model
Seq2Seq (encoder decoder) model
 
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
“Introduction to DNN Model Compression Techniques,” a Presentation from Xailient
 
Transformer Introduction (Seminar Material)
Transformer Introduction (Seminar Material)Transformer Introduction (Seminar Material)
Transformer Introduction (Seminar Material)
 
딥러닝 기반의 자연어처리 최근 연구 동향
딥러닝 기반의 자연어처리 최근 연구 동향딥러닝 기반의 자연어처리 최근 연구 동향
딥러닝 기반의 자연어처리 최근 연구 동향
 
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
[DL輪読会]Reinforcement Learning with Deep Energy-Based Policies
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networks
 

Similar to Attention is all you need 설명

Bidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionBidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehension
Woodam Lim
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural Networks
Sanghoon Yoon
 
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
Lee Seungeun
 
Chapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive netsChapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive nets
KyeongUkJang
 
Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)
Jaemin Cho
 
Chapter 17 monte carlo methods
Chapter 17 monte carlo methodsChapter 17 monte carlo methods
Chapter 17 monte carlo methods
KyeongUkJang
 
개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow
양 한빛
 
Papago/N2MT 개발이야기
Papago/N2MT 개발이야기Papago/N2MT 개발이야기
Papago/N2MT 개발이야기
NAVER D2
 
Howto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear RegressionHowto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear Regression
Hyo jeong Lee
 
순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요
Byoung-Hee Kim
 
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
Gyubin Son
 
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Oh Yoojin
 
[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경
NAVER D2
 
Deep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniquesDeep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniques
Kang Pilsung
 
A Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding AutoencoderA Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding Autoencoder
Lee Seungeun
 
네이버 NLP Challenge 후기
네이버 NLP Challenge 후기네이버 NLP Challenge 후기
네이버 NLP Challenge 후기
Jangwon Park
 
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
Adonis Han
 
CNN for sentence classification
CNN for sentence classificationCNN for sentence classification
CNN for sentence classification
KyeongUkJang
 
2.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-32.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-3
Haesun Park
 
Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks I
Sang Jun Lee
 

Similar to Attention is all you need 설명 (20)

Bidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehensionBidirectional attention flow for machine comprehension
Bidirectional attention flow for machine comprehension
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural Networks
 
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
CNN 초보자가 만드는 초보자 가이드 (VGG 약간 포함)
 
Chapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive netsChapter 10 sequence modeling recurrent and recursive nets
Chapter 10 sequence modeling recurrent and recursive nets
 
Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)Deep Learning for Chatbot (2/4)
Deep Learning for Chatbot (2/4)
 
Chapter 17 monte carlo methods
Chapter 17 monte carlo methodsChapter 17 monte carlo methods
Chapter 17 monte carlo methods
 
개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow개발자를 위한 공감세미나 tensor-flow
개발자를 위한 공감세미나 tensor-flow
 
Papago/N2MT 개발이야기
Papago/N2MT 개발이야기Papago/N2MT 개발이야기
Papago/N2MT 개발이야기
 
Howto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear RegressionHowto_Tensorflow+Linear Regression
Howto_Tensorflow+Linear Regression
 
순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요순환신경망(Recurrent neural networks) 개요
순환신경망(Recurrent neural networks) 개요
 
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
 
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
 
[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경[224] backend 개발자의 neural machine translation 개발기 김상경
[224] backend 개발자의 neural machine translation 개발기 김상경
 
Deep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniquesDeep neural networks cnn rnn_ae_some practical techniques
Deep neural networks cnn rnn_ae_some practical techniques
 
A Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding AutoencoderA Beginner's guide to understanding Autoencoder
A Beginner's guide to understanding Autoencoder
 
네이버 NLP Challenge 후기
네이버 NLP Challenge 후기네이버 NLP Challenge 후기
네이버 NLP Challenge 후기
 
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
(Kor ver.)NLP embedding(word2vec) tutorial & implementation(Tensorflow)
 
CNN for sentence classification
CNN for sentence classificationCNN for sentence classification
CNN for sentence classification
 
2.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-32.supervised learning(epoch#2)-3
2.supervised learning(epoch#2)-3
 
Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks I
 

Attention is all you need 설명

  • 1. Attention is All you Need Ashish Vaswani et al.(2017) 한국 인공지능 연구소 이준호
  • 2. • 기존에 사용되던 RNN / CNN 방식과 다른 FNN과 Skip Connection 기반의 모듈을 N개 쌓아서 LSTM, GRU같은 아키텍 처 없이 long-term dependency를 효과적으로 해결한 새로운 인코더-디코더 방식 • Source 단어와 Output 사이 distance가 짧아 Attention으로 정 보가 직접적으로 연결 • RNN의 순차적 특성으로 병렬 처리가 힘들다는 단점을 극복해 서 약 30배 가량의 학습속도 개선
  • 3. Motivation(Google Research Blog) • “I arrived at the bank after crossing the …” • I arrived at the bank after crossing the road (Bank  은행) • I arrived at the bank after crossing the river (Bank  강둑) RNN은 순차적 언어 처리  word distance 커질수록 분석을 위한 타임스탭 숫자가 증가. CNN을 이용한 최신 언어 모델의 경우에도 word distance에 따른 단계(레이 어)가 증가한다.
  • 4. Self-Attention Mechanism 밑줄된 부분에 따라서 it에 대한 가중치에 영향을 미침
  • 5. Model Architecture • Left side : Encoder • Right side : Decoder • 각 단계에서 자동 회귀를 실시 • 즉, 이전 단계에서 나온 symbol을 다음 단계에 추 가한다.
  • 6. Encoder • encode는 N = 6의 identical 레이어를 가진다. • 각 레이어는 2개의 Sub 레이어를 동반한다. • 처음은 Multi-head self attention mechanism • 뒤 이어서 Fully connected된 feed-forward network가 온다. • 모델에서 처리하는 기본 차원은 d_model = 512라고 가정
  • 7. Decoder • Encoder같이 N = 6의 identical 레이어를 가짐 • Decoder는 3개의 Sub레이어를 가진다. • Encoder와 비슷하지만, input-feeding을 위해 디코더의 최종 출력이 다시 입력으로 들어간다.
  • 8. • Input Embedding  pre-softmax linear transformation  text  probability vector • Positional Encoding - 기본 조건 : d_model = 512 (to summation) - use sine and cosine funtions pos is position i is the dimension - RNN / CNN 방식을 사용하는 것이 아니기 때문에 time step의 개념을 보 존해야한다. - sinusoidal 방법을 이용하면 inference시 긴 시퀀스에 대해서 ‘extrapolate’(추론) 할 수 있게 될 뿐 아니라, 어떠한 offset k에 대해서도 PE_{pos + k} = a * PE_{pos} + b 형태로 표현 가능하다고 한다.... 어렵네요 ㅠ
  • 9. Q : 디코더 이전 레이어 hidden state K : 인코더의 output key V : 인코더의 output value 출처 : http://dalpo0814.tistory.com/category/Machine%20Learning
  • 10. Scaled Dot Product Q와 K의 유사도를 계산 너무 큰 값이 지배적이지 않도록 스케일링 • Scale 과 normalize의 차이 Scale – mean 0, Var 1으로 조정, 주로 overflow & underflow를 방지 Normalize – 전체 구간을 설정, 데이터 군 내에서 특정 데이터의 위치를 확인 - 수식 : (요소값 – 최소값) / (최대값 – 최소값) 유사도  가중치 (총합 == 1) 가중치를 V와 곱연산 정리 : 이번 상태의 key와 value 페어인 {K, V}가 이전 상태인 Q와 어떤 연관이 있을것이다. 그러므로 K와 Q의 유사도를 계산하고 그걸 V에 반영 결국 V에 더 많은 Q의 정보가 전달된다.
  • 11. Multi-Head Attention • 지금까지 얘기했던 V, K, Q가 h개의 Scaled Dot-Product 를 통과하면서 h번의 선형 projection 뒤 결과들을 concat 해서 사용하면 더 좋다는 사실이 발견되었다. Projection으로 인한 dimension decrease 때문에 계산량이 큰 폭으로 증가하지 않는다.
  • 12. Position-wise Feed-Forward Networks Function • 인코더와 디코더에 fully connected로 연결되어있으며, 개별적 으로 동일하게 적용된다. • ReLU activation을 포함함 두가지 선형변환으로 구성
  • 14. 마무리하며 • 페이퍼를 가지고 스터디 자료를 만들면서 최대한 많은 정보를 담으려고 노력했습니다만, 개인적인 부족함으로 100% 이해하 지 못한 부분이 있어서 정보 전달이 떨어지지 않을까 싶습니다 • 관련 페이퍼는 https://arxiv.org/abs/1706.03762에서 직접 확인 하실수 있습니다. • Google Research Blog와 http://dalpo0814.tistory.com/category/Machine%20Learning를 열심히 참고했습니다.