SlideShare a Scribd company logo

Graph attention network - deep learning paper review

•
•
T
taeseon ryu

Graph attention network - deep learning paper review 딥러닝 논문읽기 모임에서 주정헌님이 사용하신 발표 자료 입니다.

Science
1 of 25
Download to read offline
Graph Attention Network 딥러닝 논문읽기 “자연어처리팀” 팀원 :주정헌(발표자), 김은희, 황소현, 신동진, 문경언 2021.02.21
Contents • Attention-mechanism • Graph Neural Network • Graph Attention Network • Inductive-learning & Transductive-learning • Experiments • Q & A
Attention- mechanism
Attention-mechanism Key Similar function Value Similarity Amazon 0.31 AWS, Amazon prime 0.35 Google 0.28 Youtube, Android 0.20 Facebook 0.46 Instagram, Whatsapp 0.53 Tesla 0.19 Model S, Model X 0.11 Nvidia 0.04 Geforce, ARM 0.15 Apple Query Query * Key Similar function * Value Output = 0.35 + 0.20 + 0.53 + 0.11 + 0.15 Attention score == Similar function
Attention-mechanism • Document classification based RNN Attention : Query와 Key의 유사도를 구한 후 value와 weighted sum KEY, VALUE : RNN의 hidden-state QUERY : 학습될 문맥벡터(context vector) input key value Attention score document classes x1 h1 h1 a1 = h1 * c Feature = a1 + a2 + a3 + a4 + a5 class1 x2 h2 h2 a2 = h2 * c class2 x3 h3 h3 a3 = h3 * c class3 x4 h4 h4 a4 = h4 * c class4 x5 h5 h5 a5 = h5 * c class5 [0.12, 0.35, 0.041, 0.05, 0.711] Query : c RNN Feature 는 문서의 특성을 갖고 있다고 여긴다 !! RNN Similarity function???
Attention-mechanism Similarity function (Alignment model) Alignment는 벡터간의 유사도를 구하는 방법을 나타냄 Additive / concat = Bahdanua attention(2014), GAT Scaled-dot product = Transformer(2017) Others = Luong(2015) et al 출처 : https://www.youtube.com/watch?v=NSjpECvEf0Y (강현규, 김성범)
Graph Neural Network • Node(vertex) : 데이터 그 자체의 특성값 • Edge : 데이터들 사이의 관계를 나타내는 값
Graph Neural Network (부록) • Language modeling NLP 분야에서는 문장 혹은 단어들 간의 시퀀스(관계) 를 수치화하여 다차원으로 표현될 수 있는 값을 Euclidean space에 표현할 수 있게 한다. 이를 언어모델링이라 표현하고 이렇게 나온 값들을 Representation 혹은 Embedding이라 부른다.
Graph Neural Network CORA PPI(protein-protein interaction)
Graph Neural Network Tesla Apple Nvidia Google Amazon Facebook IOS IPHONE IPAD MAC React pytorch Facebook instagram MODEL Neural link spacex bitcoin geforce arm cuda tegra aws sagemaker kinddle Amazon go Android pixel youtube Chrome
Graph Neural Network GNN Layer Graph data Adjacency Matrix 0 1 0 1 0 1 1 0 1 0 1 0 0 1 0 1 1 1 1 0 1 0 1 0 0 1 1 1 0 1 1 0 1 0 1 0 1:apple, 2:google, 3:facebook 4: tesla, 5:nvidia, 6:amazon
Graph Neural Network • Aggregation a1 = aggregate(. , ) 인접 노드의 임베딩 벡터를 요약한다. • Combine h1 = combine(a1, ) h1 = • Readout aws sagemaker kinddle Amazon go Android pixel youtube Chrome IOS IPHONE IPAD MAC h1 h2 h3 h4 h1 h2 h3 h4
Graph Neural Network 출처 : https://www.youtube.com/watch?v=NSjpECvEf0Y (강현규, 김성범)
Graph Attention Network 1) Nodes 들 간의 linear transform 2) Nodes들 간의 Attention 3) Activation -> Softmax -> Attention score 4) Hidden-states와 Aij를 weighted sum 5) Self-attention을 이용한 multi-head attention 6) 새로운 hidden-state로 target node의 feature 학습
Graph Attention Network IOS IPHONE IPAD MAC React pytorch Facebook instagram MODEL Neural link spacex bitcoin geforce arm cuda tegra aws sagemaker kinddle Amazon go Android pixel youtube Chrome 1-layered Feed-Forward Leaky-relu m a12 m a14 m a16 a21 0 a23 0 a25 0 0 a32 0 a34 a35 a36 a41 m a43 m a45 m m a52 a53 a54 m a56 a61 m a63 m a65 m
Inductive-learning & Transductive-learning • 귀납법(Induction) 개별적 사실을 근거로 확률적으로 일반화된 추론 방식 1) 참새는 날 수 있다. 2) 독수리는 날 수 있다. - 모든 새는 날 수 있다.(일반화된 추론) • 연역법(Deduction) 기존 전제(혹은 가설)이 참이면 결론도 반드시(necessarily) 참인 추론 방식 1) 소크라테스는 사람이다. 2) 모든 사람은 죽는다. - 소크라테스는 죽는다.(일반화된 사실)
Inductive-learning & Transductive-learning
Inductive-learning & Transductive-learning W Input graph Prediction labele d unlabel ed Original dataset mask Test set ⇒ Accuracy = 67% Min-cut
Inductive-learning & Transductive-learning
Experiments l Inductive learning (GraphSAGE) -Hamilton et al., 2017 ü GraphSAGE-GCN ü Graph Convolution Network ü GraphSAGE-mean ü 자신 node와 이웃 node의 평균값 사용 ü GraphSAGE-LSTM ü 이웃 node를 램덤으로 섞은 후 LSTM 방식으로 aggregating ü GraphSAGE-pool ü Max/mean을 통해서 이웃을 pooling 하는 방법
Experiments
Experiments Explainable model Color = 노드 분류 Edge = 멀티헤드 어텐션 평균값 1개 노드를 updat하는 과정을 알려면 해당 edges의 노드들의 클래스를 관찰할 수 있다. t-SNE 고차원 -> 저차원으로 임베딩 시키는 알고리즘 1. High dimentsion에서 i,j가 이웃 node일 확률 2. low dimentsion에서 i,j가 이웃 node일 확률 3. t-SNE 목적함수 (KLD) -> minimize with sgd
Reference • GAT : https://arxiv.org/abs/1710.10903 (논문 원문) • Youtube : https://www.youtube.com/watch?v=NSjpECvEf0Y (고려대학교 산업공학과 김성범 교수님) • Books : http://cv.jbnu.ac.kr/index.php?mid=ml (전북대학교 오일석 교수님)
Graph attention network - deep learning paper review

Recommended

Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks
Yunjey Choi
 
深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
深層生成モデルの理論と導出(Variational Auto-encoderからADGM)深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
Tetsuo Tashiro
 
Graph Neural Networks
Graph Neural NetworksGraph Neural Networks
Graph Neural Networks
tm1966
 
動作認識におけるディープラーニングの最新動向１　3D-CNN
動作認識におけるディープラーニングの最新動向１　3D-CNN動作認識におけるディープラーニングの最新動向１　3D-CNN
動作認識におけるディープラーニングの最新動向１　3D-CNN
WEBFARMER. ltd.
 
Pythonによる機械学習実験の管理
Pythonによる機械学習実験の管理Pythonによる機械学習実験の管理
Pythonによる機械学習実験の管理
Toshihiro Kamishima
 
Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向
Ohnishi Katsunori
 
異常検知と変化検知で復習するPRML
異常検知と変化検知で復習するPRML異常検知と変化検知で復習するPRML
異常検知と変化検知で復習するPRML
Katsuya Ito
 
Graph kernels
Graph kernelsGraph kernels
Graph kernels
Luc Brun
 

Recommended

Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks
Yunjey Choi
 
深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
深層生成モデルの理論と導出(Variational Auto-encoderからADGM)深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
Tetsuo Tashiro
 
Graph Neural Networks
Graph Neural NetworksGraph Neural Networks
Graph Neural Networks
tm1966
 
動作認識におけるディープラーニングの最新動向１　3D-CNN
動作認識におけるディープラーニングの最新動向１　3D-CNN動作認識におけるディープラーニングの最新動向１　3D-CNN
動作認識におけるディープラーニングの最新動向１　3D-CNN
WEBFARMER. ltd.
 
Pythonによる機械学習実験の管理
Pythonによる機械学習実験の管理Pythonによる機械学習実験の管理
Pythonによる機械学習実験の管理
Toshihiro Kamishima
 
Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向
Ohnishi Katsunori
 
異常検知と変化検知で復習するPRML
異常検知と変化検知で復習するPRML異常検知と変化検知で復習するPRML
異常検知と変化検知で復習するPRML
Katsuya Ito
 
Graph kernels
Graph kernelsGraph kernels
Graph kernels
Luc Brun
 
[DL輪読会]Adversarial Feature Matching for Text Generation
[DL輪読会]Adversarial Feature Matching for Text Generation[DL輪読会]Adversarial Feature Matching for Text Generation
[DL輪読会]Adversarial Feature Matching for Text Generation
Deep Learning JP
 
自然方策勾配法の基礎と応用
自然方策勾配法の基礎と応用自然方策勾配法の基礎と応用
自然方策勾配法の基礎と応用
Ryo Iwaki
 
Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks
남주 김
 
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
Deep Learning JP
 
三次元表現まとめ（深層学習を中心に）
三次元表現まとめ（深層学習を中心に）三次元表現まとめ（深層学習を中心に）
三次元表現まとめ（深層学習を中心に）
Tomohiro Motoda
 
クラシックな機械学習入門：付録：よく使う線形代数の公式
クラシックな機械学習入門：付録：よく使う線形代数の公式クラシックな機械学習入門：付録：よく使う線形代数の公式
クラシックな機械学習入門：付録：よく使う線形代数の公式
Hiroshi Nakagawa
 
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Yuya Takashina
 
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
Koichi Hamada
 
自己教師学習（Self-Supervised Learning）
自己教師学習（Self-Supervised Learning）自己教師学習（Self-Supervised Learning）
自己教師学習（Self-Supervised Learning）
cvpaper. challenge
 
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
yukihiro domae
 
ウィナーフィルタと適応フィルタ
ウィナーフィルタと適応フィルタウィナーフィルタと適応フィルタ
ウィナーフィルタと適応フィルタ
Toshihisa Tanaka
 
PyTorch, PixyzによるGenerative Query Networkの実装
PyTorch, PixyzによるGenerative Query Networkの実装PyTorch, PixyzによるGenerative Query Networkの実装
PyTorch, PixyzによるGenerative Query Networkの実装
Shohei Taniguchi
 
Neural networks for Graph Data NeurIPS2018読み会@PFN
Neural networks for Graph Data NeurIPS2018読み会@PFNNeural networks for Graph Data NeurIPS2018読み会@PFN
Neural networks for Graph Data NeurIPS2018読み会@PFN
emakryo
 
基底変換、固有値・固有ベクトル、そしてその先
基底変換、固有値・固有ベクトル、そしてその先基底変換、固有値・固有ベクトル、そしてその先
基底変換、固有値・固有ベクトル、そしてその先
Taketo Sano
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Jason Tsai
 
ユーザーサイド情報検索システム
ユーザーサイド情報検索システムユーザーサイド情報検索システム
ユーザーサイド情報検索システム
joisino
 
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
Deep Learning JP
 
[DL Hacks]tensorflow/privacy 使ってみた
[DL Hacks]tensorflow/privacy 使ってみた[DL Hacks]tensorflow/privacy 使ってみた
[DL Hacks]tensorflow/privacy 使ってみた
Deep Learning JP
 
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
Deep Learning JP
 
[Dl輪読会]dl hacks輪読
[Dl輪読会]dl hacks輪読[Dl輪読会]dl hacks輪読
[Dl輪読会]dl hacks輪読
Deep Learning JP
 
[Tf2017] day4 jwkang_pub
[Tf2017] day4 jwkang_pub[Tf2017] day4 jwkang_pub
[Tf2017] day4 jwkang_pub
Jaewook. Kang
 
Tensorflow for Deep Learning(SK Planet)
Tensorflow for Deep Learning(SK Planet)Tensorflow for Deep Learning(SK Planet)
Tensorflow for Deep Learning(SK Planet)
Tae Young Lee
 

More Related Content

What's hot

[DL輪読会]Adversarial Feature Matching for Text Generation
[DL輪読会]Adversarial Feature Matching for Text Generation[DL輪読会]Adversarial Feature Matching for Text Generation
[DL輪読会]Adversarial Feature Matching for Text Generation
Deep Learning JP
 
自然方策勾配法の基礎と応用
自然方策勾配法の基礎と応用自然方策勾配法の基礎と応用
自然方策勾配法の基礎と応用
Ryo Iwaki
 
Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks
남주 김
 
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
Deep Learning JP
 
三次元表現まとめ（深層学習を中心に）
三次元表現まとめ（深層学習を中心に）三次元表現まとめ（深層学習を中心に）
三次元表現まとめ（深層学習を中心に）
Tomohiro Motoda
 
クラシックな機械学習入門：付録：よく使う線形代数の公式
クラシックな機械学習入門：付録：よく使う線形代数の公式クラシックな機械学習入門：付録：よく使う線形代数の公式
クラシックな機械学習入門：付録：よく使う線形代数の公式
Hiroshi Nakagawa
 
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Yuya Takashina
 
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
Koichi Hamada
 
自己教師学習（Self-Supervised Learning）
自己教師学習（Self-Supervised Learning）自己教師学習（Self-Supervised Learning）
自己教師学習（Self-Supervised Learning）
cvpaper. challenge
 
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
yukihiro domae
 
ウィナーフィルタと適応フィルタ
ウィナーフィルタと適応フィルタウィナーフィルタと適応フィルタ
ウィナーフィルタと適応フィルタ
Toshihisa Tanaka
 
PyTorch, PixyzによるGenerative Query Networkの実装
PyTorch, PixyzによるGenerative Query Networkの実装PyTorch, PixyzによるGenerative Query Networkの実装
PyTorch, PixyzによるGenerative Query Networkの実装
Shohei Taniguchi
 
Neural networks for Graph Data NeurIPS2018読み会@PFN
Neural networks for Graph Data NeurIPS2018読み会@PFNNeural networks for Graph Data NeurIPS2018読み会@PFN
Neural networks for Graph Data NeurIPS2018読み会@PFN
emakryo
 
基底変換、固有値・固有ベクトル、そしてその先
基底変換、固有値・固有ベクトル、そしてその先基底変換、固有値・固有ベクトル、そしてその先
基底変換、固有値・固有ベクトル、そしてその先
Taketo Sano
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Jason Tsai
 
ユーザーサイド情報検索システム
ユーザーサイド情報検索システムユーザーサイド情報検索システム
ユーザーサイド情報検索システム
joisino
 
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
Deep Learning JP
 
[DL Hacks]tensorflow/privacy 使ってみた
[DL Hacks]tensorflow/privacy 使ってみた[DL Hacks]tensorflow/privacy 使ってみた
[DL Hacks]tensorflow/privacy 使ってみた
Deep Learning JP
 
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
Deep Learning JP
 
[Dl輪読会]dl hacks輪読
[Dl輪読会]dl hacks輪読[Dl輪読会]dl hacks輪読
[Dl輪読会]dl hacks輪読
Deep Learning JP
 

What's hot (20)

[DL輪読会]Adversarial Feature Matching for Text Generation
[DL輪読会]Adversarial Feature Matching for Text Generation[DL輪読会]Adversarial Feature Matching for Text Generation
[DL輪読会]Adversarial Feature Matching for Text Generation
 
自然方策勾配法の基礎と応用
自然方策勾配法の基礎と応用自然方策勾配法の基礎と応用
自然方策勾配法の基礎と応用
 
Generative adversarial networks
Generative adversarial networksGenerative adversarial networks
Generative adversarial networks
 
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
 
三次元表現まとめ（深層学習を中心に）
三次元表現まとめ（深層学習を中心に）三次元表現まとめ（深層学習を中心に）
三次元表現まとめ（深層学習を中心に）
 
クラシックな機械学習入門：付録：よく使う線形代数の公式
クラシックな機械学習入門：付録：よく使う線形代数の公式クラシックな機械学習入門：付録：よく使う線形代数の公式
クラシックな機械学習入門：付録：よく使う線形代数の公式
 
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
Estimating Mutual Information for Discrete‐Continuous Mixtures 離散・連続混合の相互情報量の推定
 
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
[データマイニング+WEB勉強会][R勉強会] はじめてでもわかる R言語によるクラスター分析 - 似ているものをグループ化する-
 
自己教師学習（Self-Supervised Learning）
自己教師学習（Self-Supervised Learning）自己教師学習（Self-Supervised Learning）
自己教師学習（Self-Supervised Learning）
 
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recog...
 
ウィナーフィルタと適応フィルタ
ウィナーフィルタと適応フィルタウィナーフィルタと適応フィルタ
ウィナーフィルタと適応フィルタ
 
PyTorch, PixyzによるGenerative Query Networkの実装
PyTorch, PixyzによるGenerative Query Networkの実装PyTorch, PixyzによるGenerative Query Networkの実装
PyTorch, PixyzによるGenerative Query Networkの実装
 
Neural networks for Graph Data NeurIPS2018読み会@PFN
Neural networks for Graph Data NeurIPS2018読み会@PFNNeural networks for Graph Data NeurIPS2018読み会@PFN
Neural networks for Graph Data NeurIPS2018読み会@PFN
 
基底変換、固有値・固有ベクトル、そしてその先
基底変換、固有値・固有ベクトル、そしてその先基底変換、固有値・固有ベクトル、そしてその先
基底変換、固有値・固有ベクトル、そしてその先
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
ユーザーサイド情報検索システム
ユーザーサイド情報検索システムユーザーサイド情報検索システム
ユーザーサイド情報検索システム
 
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
[DL輪読会]Dream to Control: Learning Behaviors by Latent Imagination
 
[DL Hacks]tensorflow/privacy 使ってみた
[DL Hacks]tensorflow/privacy 使ってみた[DL Hacks]tensorflow/privacy 使ってみた
[DL Hacks]tensorflow/privacy 使ってみた
 
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
[DL輪読会]3D Human Pose Estimation @ CVPR’19 / ICCV’19
 
[Dl輪読会]dl hacks輪読
[Dl輪読会]dl hacks輪読[Dl輪読会]dl hacks輪読
[Dl輪読会]dl hacks輪読
 

Similar to Graph attention network - deep learning paper review

[Tf2017] day4 jwkang_pub
[Tf2017] day4 jwkang_pub[Tf2017] day4 jwkang_pub
[Tf2017] day4 jwkang_pub
Jaewook. Kang
 
Tensorflow for Deep Learning(SK Planet)
Tensorflow for Deep Learning(SK Planet)Tensorflow for Deep Learning(SK Planet)
Tensorflow for Deep Learning(SK Planet)
Tae Young Lee
 
Naive ML Overview
Naive ML OverviewNaive ML Overview
Naive ML Overview
Chul Ju Hong
 
판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수
판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수
판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수
Amazon Web Services Korea
 
Anomaly detection practive_using_deep_learning
Anomaly detection practive_using_deep_learningAnomaly detection practive_using_deep_learning
Anomaly detection practive_using_deep_learning
도형 임
 
기계학습을 이용한 숫자인식기 제작
기계학습을 이용한 숫자인식기 제작기계학습을 이용한 숫자인식기 제작
기계학습을 이용한 숫자인식기 제작
Do Hoerin
 
Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks I
Sang Jun Lee
 
EveryBody Tensorflow module2 GIST Jan 2018 Korean
EveryBody Tensorflow module2 GIST Jan 2018 KoreanEveryBody Tensorflow module2 GIST Jan 2018 Korean
EveryBody Tensorflow module2 GIST Jan 2018 Korean
Jaewook. Kang
 
딥뉴럴넷 클러스터링 실패기
딥뉴럴넷 클러스터링 실패기딥뉴럴넷 클러스터링 실패기
딥뉴럴넷 클러스터링 실패기
Myeongju Kim
 
객체지향 개념 (쫌 아는체 하기)
객체지향 개념 (쫌 아는체 하기)객체지향 개념 (쫌 아는체 하기)
객체지향 개념 (쫌 아는체 하기)
Seung-June Lee
 
Just java
Just javaJust java
Just java
Jong Wook Kim
 
딥러닝으로 구현한 이상거래탐지시스템
딥러닝으로 구현한 이상거래탐지시스템딥러닝으로 구현한 이상거래탐지시스템
딥러닝으로 구현한 이상거래탐지시스템
ChoDae
 
인공지능 맛보기
인공지능 맛보기인공지능 맛보기
인공지능 맛보기
Park JoongSoo
 
검색엔진에 적용된 딥러닝 모델 방법론
검색엔진에 적용된 딥러닝 모델 방법론검색엔진에 적용된 딥러닝 모델 방법론
검색엔진에 적용된 딥러닝 모델 방법론
Tae Young Lee
 
september report in korean
september report in koreanseptember report in korean
september report in koreannao takatoshi
 
Openface
OpenfaceOpenface
Openface
jaeho kang
 
Anomaly Detection with GANs
Anomaly Detection with GANsAnomaly Detection with GANs
Anomaly Detection with GANs
홍배 김
 
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
Gyubin Son
 
[Tf2017] day3 jwkang_pub
[Tf2017] day3 jwkang_pub[Tf2017] day3 jwkang_pub
[Tf2017] day3 jwkang_pub
Jaewook. Kang
 

Similar to Graph attention network - deep learning paper review (20)

[Tf2017] day4 jwkang_pub
[Tf2017] day4 jwkang_pub[Tf2017] day4 jwkang_pub
[Tf2017] day4 jwkang_pub
 
Tensorflow for Deep Learning(SK Planet)
Tensorflow for Deep Learning(SK Planet)Tensorflow for Deep Learning(SK Planet)
Tensorflow for Deep Learning(SK Planet)
 
Naive ML Overview
Naive ML OverviewNaive ML Overview
Naive ML Overview
 
판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수
판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수
판교 개발자 데이 – AWS 인공지능 서비스를 활용하여 스마트 애플리케이션 개발하기 – 박철수
 
Anomaly detection practive_using_deep_learning
Anomaly detection practive_using_deep_learningAnomaly detection practive_using_deep_learning
Anomaly detection practive_using_deep_learning
 
기계학습을 이용한 숫자인식기 제작
기계학습을 이용한 숫자인식기 제작기계학습을 이용한 숫자인식기 제작
기계학습을 이용한 숫자인식기 제작
 
Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks I
 
EveryBody Tensorflow module2 GIST Jan 2018 Korean
EveryBody Tensorflow module2 GIST Jan 2018 KoreanEveryBody Tensorflow module2 GIST Jan 2018 Korean
EveryBody Tensorflow module2 GIST Jan 2018 Korean
 
딥뉴럴넷 클러스터링 실패기
딥뉴럴넷 클러스터링 실패기딥뉴럴넷 클러스터링 실패기
딥뉴럴넷 클러스터링 실패기
 
객체지향 개념 (쫌 아는체 하기)
객체지향 개념 (쫌 아는체 하기)객체지향 개념 (쫌 아는체 하기)
객체지향 개념 (쫌 아는체 하기)
 
Just java
Just javaJust java
Just java
 
Mahout
MahoutMahout
Mahout
 
딥러닝으로 구현한 이상거래탐지시스템
딥러닝으로 구현한 이상거래탐지시스템딥러닝으로 구현한 이상거래탐지시스템
딥러닝으로 구현한 이상거래탐지시스템
 
인공지능 맛보기
인공지능 맛보기인공지능 맛보기
인공지능 맛보기
 
검색엔진에 적용된 딥러닝 모델 방법론
검색엔진에 적용된 딥러닝 모델 방법론검색엔진에 적용된 딥러닝 모델 방법론
검색엔진에 적용된 딥러닝 모델 방법론
 
september report in korean
september report in koreanseptember report in korean
september report in korean
 
Openface
OpenfaceOpenface
Openface
 
Anomaly Detection with GANs
Anomaly Detection with GANsAnomaly Detection with GANs
Anomaly Detection with GANs
 
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
[paper review] 손규빈 - Eye in the sky & 3D human pose estimation in video with ...
 
[Tf2017] day3 jwkang_pub
[Tf2017] day3 jwkang_pub[Tf2017] day3 jwkang_pub
[Tf2017] day3 jwkang_pub
 

More from taeseon ryu

VoxelNet
VoxelNetVoxelNet
VoxelNet
taeseon ryu
 
OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...
taeseon ryu
 
3D Gaussian Splatting
3D Gaussian Splatting3D Gaussian Splatting
3D Gaussian Splatting
taeseon ryu
 
JetsonTX2 Python
JetsonTX2 Python JetsonTX2 Python
JetsonTX2 Python
taeseon ryu
 
Hyperbolic Image Embedding.pptx
Hyperbolic Image Embedding.pptxHyperbolic Image Embedding.pptx
Hyperbolic Image Embedding.pptx
taeseon ryu
 
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
taeseon ryu
 
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdfLLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
taeseon ryu
 
YOLO V6
YOLO V6YOLO V6
YOLO V6
taeseon ryu
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories
taeseon ryu
 
RL_UpsideDown
RL_UpsideDownRL_UpsideDown
RL_UpsideDown
taeseon ryu
 
Packed Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation ExtractionPacked Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation Extraction
taeseon ryu
 
MOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement LearningMOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement Learning
taeseon ryu
 
Scaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language ModelsScaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language Models
taeseon ryu
 
Visual prompt tuning
Visual prompt tuningVisual prompt tuning
Visual prompt tuning
taeseon ryu
 
mPLUG
mPLUGmPLUG
mPLUG
taeseon ryu
 
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdfvariBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
taeseon ryu
 
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdfReinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
taeseon ryu
 
The Forward-Forward Algorithm
The Forward-Forward AlgorithmThe Forward-Forward Algorithm
The Forward-Forward Algorithm
taeseon ryu
 
Towards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural NetworksTowards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural Networks
taeseon ryu
 
BRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive SummarizationBRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive Summarization
taeseon ryu
 

More from taeseon ryu (20)

VoxelNet
VoxelNetVoxelNet
VoxelNet
 
OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...
 
3D Gaussian Splatting
3D Gaussian Splatting3D Gaussian Splatting
3D Gaussian Splatting
 
JetsonTX2 Python
JetsonTX2 Python JetsonTX2 Python
JetsonTX2 Python
 
Hyperbolic Image Embedding.pptx
Hyperbolic Image Embedding.pptxHyperbolic Image Embedding.pptx
Hyperbolic Image Embedding.pptx
 
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
 
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdfLLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
 
YOLO V6
YOLO V6YOLO V6
YOLO V6
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories
 
RL_UpsideDown
RL_UpsideDownRL_UpsideDown
RL_UpsideDown
 
Packed Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation ExtractionPacked Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation Extraction
 
MOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement LearningMOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement Learning
 
Scaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language ModelsScaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language Models
 
Visual prompt tuning
Visual prompt tuningVisual prompt tuning
Visual prompt tuning
 
mPLUG
mPLUGmPLUG
mPLUG
 
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdfvariBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
 
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdfReinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
 
The Forward-Forward Algorithm
The Forward-Forward AlgorithmThe Forward-Forward Algorithm
The Forward-Forward Algorithm
 
Towards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural NetworksTowards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural Networks
 
BRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive SummarizationBRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive Summarization
 

Graph attention network - deep learning paper review

  • 1. Graph Attention Network 딥러닝 논문읽기 “자연어처리팀” 팀원 :주정헌(발표자), 김은희, 황소현, 신동진, 문경언 2021.02.21
  • 2. Contents • Attention-mechanism • Graph Neural Network • Graph Attention Network • Inductive-learning & Transductive-learning • Experiments • Q & A
  • 4. Attention-mechanism Key Similar function Value Similarity Amazon 0.31 AWS, Amazon prime 0.35 Google 0.28 Youtube, Android 0.20 Facebook 0.46 Instagram, Whatsapp 0.53 Tesla 0.19 Model S, Model X 0.11 Nvidia 0.04 Geforce, ARM 0.15 Apple Query Query * Key Similar function * Value Output = 0.35 + 0.20 + 0.53 + 0.11 + 0.15 Attention score == Similar function
  • 5. Attention-mechanism • Document classification based RNN Attention : Query와 Key의 유사도를 구한 후 value와 weighted sum KEY, VALUE : RNN의 hidden-state QUERY : 학습될 문맥벡터(context vector) input key value Attention score document classes x1 h1 h1 a1 = h1 * c Feature = a1 + a2 + a3 + a4 + a5 class1 x2 h2 h2 a2 = h2 * c class2 x3 h3 h3 a3 = h3 * c class3 x4 h4 h4 a4 = h4 * c class4 x5 h5 h5 a5 = h5 * c class5 [0.12, 0.35, 0.041, 0.05, 0.711] Query : c RNN Feature 는 문서의 특성을 갖고 있다고 여긴다 !! RNN Similarity function???
  • 6. Attention-mechanism Similarity function (Alignment model) Alignment는 벡터간의 유사도를 구하는 방법을 나타냄 Additive / concat = Bahdanua attention(2014), GAT Scaled-dot product = Transformer(2017) Others = Luong(2015) et al 출처 : https://www.youtube.com/watch?v=NSjpECvEf0Y (강현규, 김성범)
  • 7. Graph Neural Network • Node(vertex) : 데이터 그 자체의 특성값 • Edge : 데이터들 사이의 관계를 나타내는 값
  • 8. Graph Neural Network (부록) • Language modeling NLP 분야에서는 문장 혹은 단어들 간의 시퀀스(관계) 를 수치화하여 다차원으로 표현될 수 있는 값을 Euclidean space에 표현할 수 있게 한다. 이를 언어모델링이라 표현하고 이렇게 나온 값들을 Representation 혹은 Embedding이라 부른다.
  • 9. Graph Neural Network CORA PPI(protein-protein interaction)
  • 10. Graph Neural Network Tesla Apple Nvidia Google Amazon Facebook IOS IPHONE IPAD MAC React pytorch Facebook instagram MODEL Neural link spacex bitcoin geforce arm cuda tegra aws sagemaker kinddle Amazon go Android pixel youtube Chrome
  • 11. Graph Neural Network GNN Layer Graph data Adjacency Matrix 0 1 0 1 0 1 1 0 1 0 1 0 0 1 0 1 1 1 1 0 1 0 1 0 0 1 1 1 0 1 1 0 1 0 1 0 1:apple, 2:google, 3:facebook 4: tesla, 5:nvidia, 6:amazon
  • 12. Graph Neural Network • Aggregation a1 = aggregate(. , ) 인접 노드의 임베딩 벡터를 요약한다. • Combine h1 = combine(a1, ) h1 = • Readout aws sagemaker kinddle Amazon go Android pixel youtube Chrome IOS IPHONE IPAD MAC h1 h2 h3 h4 h1 h2 h3 h4
  • 13. Graph Neural Network 출처 : https://www.youtube.com/watch?v=NSjpECvEf0Y (강현규, 김성범)
  • 14. Graph Attention Network 1) Nodes 들 간의 linear transform 2) Nodes들 간의 Attention 3) Activation -> Softmax -> Attention score 4) Hidden-states와 Aij를 weighted sum 5) Self-attention을 이용한 multi-head attention 6) 새로운 hidden-state로 target node의 feature 학습
  • 15. Graph Attention Network IOS IPHONE IPAD MAC React pytorch Facebook instagram MODEL Neural link spacex bitcoin geforce arm cuda tegra aws sagemaker kinddle Amazon go Android pixel youtube Chrome 1-layered Feed-Forward Leaky-relu m a12 m a14 m a16 a21 0 a23 0 a25 0 0 a32 0 a34 a35 a36 a41 m a43 m a45 m m a52 a53 a54 m a56 a61 m a63 m a65 m
  • 16. Inductive-learning & Transductive-learning • 귀납법(Induction) 개별적 사실을 근거로 확률적으로 일반화된 추론 방식 1) 참새는 날 수 있다. 2) 독수리는 날 수 있다. - 모든 새는 날 수 있다.(일반화된 추론) • 연역법(Deduction) 기존 전제(혹은 가설)이 참이면 결론도 반드시(necessarily) 참인 추론 방식 1) 소크라테스는 사람이다. 2) 모든 사람은 죽는다. - 소크라테스는 죽는다.(일반화된 사실)
  • 18. Inductive-learning & Transductive-learning W Input graph Prediction labele d unlabel ed Original dataset mask Test set ⇒ Accuracy = 67% Min-cut
  • 20. Experiments l Inductive learning (GraphSAGE) -Hamilton et al., 2017 ü GraphSAGE-GCN ü Graph Convolution Network ü GraphSAGE-mean ü 자신 node와 이웃 node의 평균값 사용 ü GraphSAGE-LSTM ü 이웃 node를 램덤으로 섞은 후 LSTM 방식으로 aggregating ü GraphSAGE-pool ü Max/mean을 통해서 이웃을 pooling 하는 방법
  • 22. Experiments Explainable model Color = 노드 분류 Edge = 멀티헤드 어텐션 평균값 1개 노드를 updat하는 과정을 알려면 해당 edges의 노드들의 클래스를 관찰할 수 있다. t-SNE 고차원 -> 저차원으로 임베딩 시키는 알고리즘 1. High dimentsion에서 i,j가 이웃 node일 확률 2. low dimentsion에서 i,j가 이웃 node일 확률 3. t-SNE 목적함수 (KLD) -> minimize with sgd
  • 23.
  • 24. Reference • GAT : https://arxiv.org/abs/1710.10903 (논문 원문) • Youtube : https://www.youtube.com/watch?v=NSjpECvEf0Y (고려대학교 산업공학과 김성범 교수님) • Books : http://cv.jbnu.ac.kr/index.php?mid=ml (전북대학교 오일석 교수님)