Recommended
More Related Content
Similar to KDD17Tutorial_final (1).pdf
Similar to KDD17Tutorial_final (1).pdf (20)
Recently uploaded
Recently uploaded (20)
KDD17Tutorial_final (1).pdf
- 1. Learning Representations of Large-Scale Networks! Jian Tang1, Cheng Li2, Qiaozhu Mei2 1HEC Montréal & Montréal Ins=tute of Learning Algorithms (MILA) 2School of Informa=on, University of Michigan 1!
- 2. 2! • • •
- 3. 3
- 5. 5 Coauthor Network Information retrieval Machine learning Data mining - Q.Mei, D.Cai, D.Zhang, and C.Zhai, Topic Modeling with Hitting Time, WWW 2008
- 6. 6 • • • - Graph from Jerry Zhu’s tutorial in ICML 07
- 7. 7 - from Lada Adamic’s course
- 8. 8
- 9. 9
- 10. 10!
- 11. : 11!
- 12. 12! ! Text representation, e.g., word and document representation, …! …! Deep learning has been attracting increasing! attention …! A future direction of deep learning is to integrate! unlabeled data …! The Skip-gram model is quite effective and efficient …! …! degree network edge node word document classification text embedding •
- 13. 13! ….! ….! ….! ….!….! ….! ! Heatmaps Network Diagrams Scatter Plots ….! !
- 14. 14!
- 15. 15!
- 16. , 16!
- 17. , 17!
- 18. 18! ! ! ui ∈ Rd
- 19. 19!
- 20. 20! O = 1 2 wij ( ! ui (i, j)∈E ∑ − ! uj )2 = tr(UT LU) U =[ ! u1, ! u2,", ! uN ] L = D −W Dii = wij j ∑ Lu = λDu
- 21. 21! (w1,w2, !,wT ) ! vi ∈ Rd
- 22. 22!
- 23. 23!
- 24. 24!
- 25. 25!
- 26. 26! p1(vi,vj ) = exp( ! ui T ! uj ) exp( ! um T ! un ) (m,n)∈V×V ∑ p̂1(vi,vj ) = wij wmn (m,n)∈E ∑ O1 = KL(p̂1, p1) = − wij log p1(vi,vj ) (i, j)∈E ∑ ! ui
- 27. 27! p̂2 (vj | vi ) = wij wik k∈V ∑ p2 (vj | vi ) = exp( ! u'i T ! uj ) exp( ! u'k T ! ui ) k∈V ∑ O2 = KL(p̂2 (⋅| vi ), p2 (⋅| vi )) i ∑ = − wij log p2 (vj | vi ) (i, j)∈E ∑
- 28. 28! ∂O2 ∂ ! ui = wij ∂log p̂2 (vj | vi ) ∂ ! ui
- 29. 29!
- 30. 30! p(vj | vi ) = exp( ! u'i T ! uj ) exp( ! u'k T ! ui ) k∈V ∑
- 31. 31! !
- 32. 32!
- 33. 33!
- 34. 34!
- 35. 35!
- 36. 36! • •
- 37. 37! • •
- 38. 38!
- 39. 39!
- 40. degree network edge node word document classification text embedding ! Text representation, e.g., word and document representation, …! …! Deep learning has been attracting increasing! attention …! A future direction of deep learning is to integrate! unlabeled data …! The Skip-gram model is quite effective and efficient …! Information networks encode the relationships! between the data objects …! text information network word … classification doc_1 doc_2 doc_3 doc_4 … … 40! •
- 41. • • • 41!
- 42. 42! Accuracy!
- 43. 43! Accuracy!
- 44. 44!
- 45. 45!
- 46. • • . . , • • 46!
- 47. 47!
- 48. 48!
- 49. 49!
- 50. • 50!
- 51. • • • 51!
- 52. 52!
- 53. 53!
- 54. 54! • • •
- 55. 55!
- 56. 56!
- 57. ! 57!
- 58. 58!
- 59. Text representation, e.g., word and document representation, …! …! Deep learning has been attracting increasing …! A future direction of deep learning is to integrate …! The Skip-gram model is quite effective and efficient …! Information networks encode the relationships! label! document! label label null null null degree network edge node word document classification text embedding text information network word … classification doc_1 doc_2 doc_3 doc_4 … … text information network word … classification label_2 label_1 label_3 … … 59!
- 60. 60!
- 61. ≈ 61!
- 62. X ∈ RN×D 62!
- 64. 64!
- 65. 65!
- 67. 67!
- 68. 68!
- 69. 69!
- 70. 70!
- 71. 71!
- 72. • 72!
- 73. 73!
- 74. 74!
- 75. 75!
- 76. 76!
- 77. 77!
- 78. 78!
- 79. LargeVis! t-SNE! Random projection trees ! 79!
- 80. p(eij =1) = 1 1+ || ! yi − ! yj ||2 p(eij = wij ) = p(eij =1) wij 80!
- 81. O = p(eij = wij ) (i, j)∈E ∏ (1− p(eij = wij ) (i, j)∈E ∏ )γ 81!
- 83. LargeVis! t-SNE (optimal parameters)! t-SNE (default parameters)! ≈ 83!
- 84. 84!
- 85. 85!
- 86. 86!
- 87. 87!
- 88. 88!
- 89. 89!
- 90. 90!
- 91. 91!
- 92. 92!
- 93. 7 93!
- 95. 95!
- 96. ! 96!
- 97. ! 97!
- 98. ! 98!
- 99. ! 99!
- 100. ! 100!
- 101. ! Efficient graphlet kernels for large graph comparison F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 Figure 2: All graphlets of size 4 We now consider size 4 graphlets. Modulo isomorphism there are 11 graphlets of size 4 (see Figure 2). Let us denote these graphlets Fi and their counts |Fi|, i 2 1, 2, . . . , 11. As in the previous case, we will first count all graphlets which contain at least one edge. Assume we want to count subgraphs containing edge (v1, v2). As before, for v2 there are |N(v1)| choices and for each pair (v1, v2) we have 4 cases for the third node v3: v3 2 N(v1) N(v2), v3 2 N(v1) N(v2), dataset size classes MUTAG 188 2 (125 vs. PTC 344 2 (192 vs. Enzyme 600 6 (100 each D & D 1178 2 (691 vs. Table 1: Statistics on cl these graphlets by 2. 5 Experiments In this section, we evaluate th nel and compare it with stat in terms of runtime, scalabi racy. Our baseline compara dom walk kernel of (Gärtne et al., 2004; Vishwanathan e common walks in two graph kernel of (Borgwardt & Krie shortest path lengths in two 101!
- 102. ! 102!
- 103. ! 103!
- 104. ! 104!
- 107. ! 107!
- 108. ! 108!
- 109. ! 109!
- 110. ! Convolutional architecture can be applied! 110!
- 112. ! 112!
- 113. ! ó ó 113!
- 114. ! ó ó ó ó How to assemble by end-to-end learning?! We can adapt deep learning methods developed for text! 114!
- 115. ! GRU GRU GRU GRU GRU GRU B C F ℎ # ℎ $ ℎ % x 3 x 2 x 4 dense layers Attention Size Increment (c) (d) equence Input Output ' sequence & nodes Output! Dense layers! A B C F D F + + A A A A D F + + GRU GRU GRU GRU GRU GRU GRU GRU A B C F ℎ" ℎ# ℎ$ ℎ% x 1 x 3 x 2 x 4 Sample Attentio (b) (c) (d) & nodes ' sequence Sequence Input Node Embedding Bi-directional GRU Output ' sequence & nodes A B C F D F + + A A A A D F + + C D F A GRU GRU GRU GRU GRU GRU GRU GRU A B C F ℎ" ℎ# ℎ$ ℎ% x 1 x 3 x 2 x 4 Sample Attention (a) (b) (c) (d) & nodes ' sequence Sequence Input Node Embedding Bi-directional GRU Output ' sequence & nodes A B C F D F + + A A A A D F + + GRU GRU GRU GRU GRU GRU GRU GRU A B C F ℎ" ℎ# ℎ$ ℎ% x 1 x 3 x 2 x 4 Sample Attentio (b) (c) (d) & nodes ' sequence Sequence Input Node Embedding Bi-directional GRU Output ' sequence & nodes C D E F B A A B C F D F + + A A A A D F + + 𝑇 nodes 𝐾 sequence sampling Sample through random walks! 115!
- 116. ! 116!
- 117. ! 2#" 1" a>en?on" dense" layers" output%unit% λ1"..."λT A>en?on"over"T"nodes" A>en?on" over"K" sequences" with"mini; batch"size"="2" pgeo (1;pgeo)pgeo (1;pgeo)2pgeo ... 1st mini-batch! 2nd mini-batch! Assume attention has geometric distribution.! We learn pgeo to learn the actual #seq! 117!
- 118. ! 118!
- 119. ! p({Hi},{Xi})∝ Φ(Hi, Xi ) i∈V ∏ Ψ(Hi, Hj ) (i, j)∈E ∏ 119! ! ! ! ! ! ! ! ! ! ! ! ! ! Graph label! A hidden variable! A variable for node attribute!
- 120. ! 120!
- 121. ! 121!
- 122. ! 122!
- 123. 123!
- 124. 124!
- 125. 125!
- 126. - 126!
- 127. 127!
- 128. 128!
- 130. 130!
- 131. 131!