NS-CUK Seminar: H.B.Kim, Review on "metapath2vec: Scalable representation learning for heterogeneous networks", KDD 2017

1. Ho-Beom Kim
Network Science Lab
Dept. of Mathematics
The Catholic University of Korea
E-mail: hobeom2001@catholic.ac.kr
2023 / 07 / 24
DONG, Yuxiao; CHAWLA, Nitesh V.; SWAMI, Ananthram.
ACM SIGKDD

2. 2
Introduction
Problem Statements
• Neural network-based learning models can represent latent embeddings that capture the internal relations
of rich, complex data across various modalities, such as image, audio, and language.
• Social and information networks are similarly rich and complex data that encode the dynamics and types of
human interactions, and are similarly amenable to representation learning using neural networks.
• Recent research publications have proposed word2vec-based network representation learning frameworks
• DeepWalk, LINE, node2vec
→ These work has far focused on representation learning for homogeneous networks-representative of
singular type of nodes and relationships.
• A large number of social and information networks are heterogeneous in nature, involving diversity of node
types and/or relationships between nodes

3. 3
Introduction
Problem Statements
• The aforementioned works commonly concentrate more on user behaviors of recent times
→ They are not capable of fully mining older behavior-sequences to accurately estimate their current
interests.
• There two major challenges in sequential recommendation that have not been well-addressed so far
1. User behaviors in long sequences reflect implicit and noisy preference signals
2. User preferences are always drifting over time due to their diversity
 To address these two challenges, they propose a graph-based method with graph convolutional networks
to extract implicit preference signals
 Sequence  Graph
 Design a attentive graph convolutional network
 Dynamic pooling technique

4. 4
Introduction
Contributions
• They approach sequential recommendation from a new perspective by taking into consideration the
implicit-signal behaviors and fast-changing preferences
• They Propose to aggregate implicit signals into explicit ones from user behaviors by designing graph
neural network-based models on constructed item-item interest graphs. Then they design dynamic-pooling
for filtering and reserving activated core preferences for recommendation
• They conduct extensive experiments on two large-scale datasets collected from real-world applications.
The experimental results show significant performance improvements compared with the state-of-the-art
methods of sequential recommendation. Further studies also verify that our method can model long
behavioral sequences effectively and efficiently

5. 5
Introduction
Problem Formulation
• They approach sequential recommendation from a new perspective by taking into consideration the
implicit-signal behaviors and fast-changing preferences
• They Propose to aggregate implicit signals into explicit ones from user behaviors by designing graph
neural network-based models on constructed item-item interest graphs. Then they design dynamic-pooling
for filtering and reserving activated core preferences for recommendation
• They conduct extensive experiments on two large-scale datasets collected from real-world applications.
The experimental results show significant performance improvements compared with the state-of-the-art
methods of sequential recommendation. Further studies also verify that our method can model long
behavioral sequences effectively and efficiently

6. 6
Problem Definition
Definition
A : authors
P : papers
V : venues
O : organizations
→ Nodes
A-A : coauthor
A-P,P-V : publish
O-A : affiliation

7. 7
Problem Definition
Problem1 : Heterogeneous Network Representation Learning
• The task is to learn the d-dimensional latent representations X ∈ ℝ 𝑉 X𝑑
, 𝑑 ≪ 𝑉 that are able to capture
the structural and semantic relations among them
• The output : the low-dimensional matrix X with the vth row-a d-dimensional vector 𝑋𝑣-corresponding to the
representation of node v
• There are different types of nodes in V  their representations are mapped into the same latent space
• The learned node representations can benefit embedding vector of each node can be used as the feature
input of node classification, clustering, and similarity search tasks
• The main challenge of this problem comes from the network heterogeneity, wherein it is difficult to directly
apply homogeneous language and network embedding methods
• The premise of network embedding models is to preserve the proximity between a anode and its
neighborhood.

8. 8
Related Works
PTE
• Fist learns a low dimensional embedding for words through a heterogeneous text network.
• The network encodes different levels of co-occurrence information between words and words, words and
documents, and words and labels.
• The network is embedded into a low dimensional vector space that preserves the second-order proximity
between the vertices in the network.
• Proposed to learn predictive text embeddings in a semi-supervised manner.
• Unlabeled data and labeled information are integrated into a heterogeneous text network which
incorporates different levels of cooccurrence information in text
• Propose PTE, which learns a distributed representation of text through embedding the heterogeneous text
network into a low dimensional space
• PTE uses the average value of word embeddings
• PTE considers the linear relationship between words and labels

9. 9
Related Works
PTE
• Deep neural networks fully leverage labeled information that is available for a task when they learn the
representations of the data
• Most text embedding methods are not able to consider labeled information when learning the
representations
• Previous embedding models using unsupervised text embeddings
→ Generalizable for different tasks but have a weaker predictive power for a particular task
• Text embedding methods are much more efficient, are much easier to tune, and naturally accommodate
unlabeled data

10. 10
Methodology
Heterogeneous Network Embedding: metapath2vec
𝑙𝑜𝑔𝜎 𝑋𝑐𝑡
∙ 𝑋𝑣 +
𝑚=1
𝑀
𝔼𝑢𝑚~𝑝(𝑢) 𝑙𝑜𝑔𝜎(−𝑋𝑢𝑚 ∙ 𝑋𝑣)
𝑝(𝑢) : pre-defined distribution

11. 11
Methodology
Meta-Path-Based Random Walks

12. 12
Methodology
Heterogeneous negative sampling

13. 13
Experimental Setup
Experiments
The number of walks per node : 1000;
The walk length l : 1000;
The vector dimension d: 128 (LINE: 128 for each order);
The neighborhood size k: 7;
The size of negative samples: 5

14. 14
Multi-class venue node classification results in Aminer data
Experiments

15. 15
Multi-class author node classification results in AMiner data
Experiments

16. 16
Parameter sensitivity in multi-class node classification
Experiments

17. 17
Node clustering results in AMiner data
Experiments

18. 18
Parameter sensitivity in clustering
Experiments

19. 19
Case study of similarity search in AMiner Data
Experiments

20. 20
Case study of similarity search in DBIS Data
Experiments

21. 21
2D t-SNE projections of the 128D embeddings of 48 CS venues, three each from 16 sub-fields
Experiments

22. 22
Scalability of metapaath2vec and metapath2vec++
Experiments

23. 23
Conclusions
Conclusions
• To address the network heterogeneity challenge, we propose the metapath2vec and metapath2vec++
methods.
• To leverage this method, we formalize the heterogeneous neighborhood function of a node, enabling the
skip-gram-based maximization of the network probability in the context of multiple types of nodes
• Finally, we achieve effective and efficient optimization by presenting a heterogeneous negative sampling
technique.
Future work
1. Face the challenge of large intermediate output data when sampling a network into a huge pile of paths, and t
identifying and optimizing the sampling space is an important direction
2. As is also the case with all metapath-based heterogeneous network mining methods, metapath2vec and
metapath2vec++ can be further improved by the automatic learning of meaningful meta-paths
3. Extending the models to incorporate the dynamics of evolving heterogeneous networks
4. Generalizing the models for different genres of heterogeneous networks