Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Semi-Supervised Classification with Graph Convolutional Networks @ICLR2017読み会

23,414 views

Published on

ICLR2017@DeNA(https://connpass.com/event/57631/)の論文紹介資料

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Semi-Supervised Classification with Graph Convolutional Networks @ICLR2017読み会

  1. 1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. June 17, 2017 ICLR @DeNA Eiji Sekiya AI Research and Development Gr. AI System Dept. 
 DeNA Co., Ltd. 1 h"ps://openreview.net/pdf?id=SJU4ayYgl
  2. 2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • • (@eratostennis) • • Master • DeNA (2014) • : (~2016) • Hadoop, Vertica • : (2016~) • • GameAI 2
  3. 3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Abstract • Semi-supervised learning on graph-structured data • Localized first-order approximation of spectral graph • : • • 3
  4. 4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Agenda • Introduction • Related Work • Fast Approximate Convolutions on Graphs • Semi-Supervised Node Classification • Experiments • Results • Conclusion 4
  5. 5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Relation to Weisfeiler-Lehman Algorithm • WL-1 • Hash Neural Network 5 hash
  6. 6. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Node Embeddings with Random Weights 6 output 2
  7. 7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Semi-Supervised Node Embeddings • 4 1 7 : h"p://tkipf.github.io/graph-convoluDonal-networks/
  8. 8. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Related Work • Graph-Based Semi-Supervised Learning • Graph Laplacian regularization • Label Propagation (Zhu et al., 2003) • Manifold Regularization (Belkin et al., 2006) • Graph embedding-based • DeepWalk (Perozzi et al., 2014) • Planetoid (Yang et al., 2016) 8
  9. 9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Related Work (DeepWalk: Perozzi et al., 2014) • • SkipGram embedding • Random Walk 9
  10. 10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Related Work • Graph-Based Semi-Supervised Learning • Graph Laplacian regularization • Label Propagation (Zhu et al., 2003) • Manifold Regularization (Belkin et al., 2006) • Graph embedding-based • DeepWalk (Perozzi et al., 2014) • Planetoid (Yang et al., 2016) • Neural Networks on Graphs ( ) • Spectral graph convolutional neural network (Bruna et al., 2014) • Fast localized convolution (Defferrard et al., 2016) 10
  11. 11. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Graph-Based Semi-Supervised Learning • (citation network ) (documents) • • • • encode • 11 Neural Networks on Graph
  12. 12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Multi-layer Graph Convolutional Network (GCN) • • • • 12 Contribution
  13. 13. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Spectral Graph Convolutions • Convolution with filter • : • • …(Hammond at el., 2011) • • ( ) 13
  14. 14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Spectral Graph Convolutions (Defferrad at el., 2016) • Graph Convolution ( , ) • K K • : • 14
  15. 15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Layer-Wise Linear Model (Contribution) • K=1 … • k k • • Renormalization Trick ( ) • Input C, F … • 15
  16. 16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Semi-Supervised Node Classification • Forward Model ( ) • Cross-Entropy: 16
  17. 17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Experiments (Dataset) • (Citeseer, Cora, Pubmed) • Sparse bag-of-words feature vectors for each document • a list of citation links between documents • (NELL) • A set of entities connected with directed, labeled edges • Random Graph • A dataset with N nodes we create a random graph assigning 2N edges uniformly at random 17
  18. 18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Experiments (Setup) • • 2 GCN • • Prediction Accuracy • 1000 • • 500 • Dropout rate, L2 regularization factor, Number of hidden units • Cora Citeseer Pubmed 18
  19. 19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Experiments (Baseline) • Baseline • Label Propagation (LP) • Semi-supervised embedding (SemiEmb) • Manifold regularization (ManiReg) • Skip-gram based graph embeddings (DeepWalk) • Iterative classification algorithm (ICA) • Planetoid 19
  20. 20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Results (Semi-Supervised Node Classification) • Results for all other baseline methods are taken from the Planetoid paper • Environment • The same hardware as planetoid 20 Training Dme in seconds unDl convergence
  21. 21. • Compare different variants of our proposed per-layer propagation model on the citation network datasets Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Results (Evaluation Propagation Model) 21
  22. 22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Results (Training Time per Epoch) • The mean training time per epoch for 100 epochs on simulated random graphs 22
  23. 23. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Experiments on Model Depth • Residual connections • 2, 3 • 7 … • residual connections • 23
  24. 24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Discussion • Keypoints • Overcome both limitations • Graph-Laplacian regularization • Edges encode mere similarity of nodes • Skip-gram based methods • Limited by the fact that they are based on a multi-step pipeline which difficult to optimize • Findings • Propagation from neighboring nodes improves classification performance • Proposed renormalized propagation model offers both improved efficiency 24
  25. 25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Future Work • Memory Requirement • Full-batch gradient descent • Grows linearly in the size of the dataset • Mini-batch stochastic gradient descent can alleviate this issue • Directed edges and edge features • Limited to undirected graph • NELL show it is possible to handle • By representing the original directed graph as an undirected bipartite graph • Limiting assumptions • Locality • dependence on the Kth-order neighborhood for a GCN with K layers • Equal importance of self-connections vs. edges to neighboring nodes 25
  26. 26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Reference • Graph Semi-Supervised Learning • Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In International Conference on Machine Learning (ICML), volume 3, pp. 912–919, 2003. • Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. Journal of machine learning research (JMLR), 7(Nov):2399–2434, 2006. • JasonWeston,Fre ́de ́ricRatle,HosseinMobahi,andRonanCollobert. Deeplearningviasemi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639–655. Springer, 2012. • Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems (NIPS), pp. 3111–3119, 2013. • Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701–710. ACM, 2014. • Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning (ICML), 2016. 26
  27. 27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Reference • Dataset & Pre-Processing • Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93, 2008. • Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr, and Tom M. Mitchell. Toward an architecture for never-ending language learning. In AAAI, volume 5, pp. 3, 2010. • Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning (ICML), 2016. • Website • http://tkipf.github.io/graph-convolutional-networks/ • Implementation • https://github.com/tkipf/gcn 27

×