NS-CUK Journal club: HBKim, Review on "Neural Graph Collaborative Filtering", SIGIR 2019

1. Ho-Beom Kim
Network Science Lab
Dept. of Mathematics
The Catholic University of Korea
E-mail: hobeom2001@catholic.ac.kr
2023 / 07 / 05
WANG, Xiang, et al.
2019, ACM SIGIR

2. 2
Introduction
Problem Statements
• There are two key components in learnable CF models
• Embedding, which transforms users and items to vectorized representations
• Interaction modeling, which reconstructs historical interactions based on the embeddings
• Embedding function lacks an explicit encoding of the crucial collaborative signal, which is latent in
user-item interactions to reveal the behavioral similarity between users (or items)
• Previous models : Matrix Factorization, Neural Collaborative Filtering,,,
• Most existing methods do not consider the user-item interactions
• Most existing methods build the embedding function with the descriptive features

3. 3
Introduction
Running Example
• An illustration of the user-item interaction graph and the high-order connectivity

4. 4
Introduction
Contributions
• They highlight the critical importance of explicitly exploiting the collaborative signal in the embedding
function of model-based CF methods
• They propose NGCF, a new recommendation framework based on graph neural network, which
explicitly encodes the collaborative signal in the form of high-order connectivities by performing
embedding propagation
• They conduct empirical studies on three million-size datasets

5. 5
Method
Architecture

6. 6
Methodology
Embedding Layer
• 𝑒𝑢, 𝑒𝑖 ∈ ℝ𝑑
• 𝑑 : the embedding size
• 𝑒𝑢𝑁
: users embeddings
• 𝑒𝑖𝑀
: item embeddings
• 𝐸 = [𝑒𝑢1
, … , 𝑒𝑢𝑁
, 𝑒𝑖1
, … , 𝑒𝑖𝑀
]

7. 7
Methodology
Embedding Propagation Layers (First-order Propagation)
• Message Construction
• 𝑚𝑢←𝑖 = 𝑓 𝑒𝑖, 𝑒𝑢, 𝑝𝑢𝑖
• 𝑚𝑢←𝑖 : the message embedding
• (the information to be propagated)
• 𝑓(∙)
• the message encoding function
• 𝑚𝑢←𝑖 =
1
|𝑁𝑢||𝑁𝑖|
𝑊1𝑒𝑖 + 𝑊2(𝑒𝑖⨀𝑒𝑢 )
• 𝑊1, 𝑊2 ∈ ℝ𝑑′×𝑑
• Trainable weight matrices
• 𝑑′
: the transformation size

8. 8
Methodology
Embedding Propagation Layers (First-order Propagation)
• Message Construction
• 𝑚𝑢←𝑖 =
1
|𝑁𝑢||𝑁𝑖|
𝑊1𝑒𝑖 + 𝑊2(𝑒𝑖⨀𝑒𝑢 )
• 𝑝𝑢𝑖
→ 1/ 𝑁𝑢 |𝑁𝑖|
• 𝑁𝑢 , |𝑁𝑖|
• Distinct from conventional graph convolution
networks, they additionally encode the
interaction between 𝑒𝑖, 𝑒𝑢 into the message
being passed via ei⨀𝑒𝑢

9. 9
Methodology
Embedding Propagation Layers (First-order Propagation)
• Message Aggregation
• 𝑒𝑢
(1)
= 𝐿𝑒𝑎𝑘𝑦𝑅𝑒𝐿𝑈 𝑚𝑢←𝑢 + 𝑚𝑢←𝑖
• They aggregate the messages propagated
from u’s neighborhood to refine 𝑢’s
representation.
• 𝑒𝑢
(1)
: the representation of user u obtained
after first embedding propagation layer

10. 10
Methodology
Embedding Propagation Layers (High-order Propagation)
• 𝑒𝑢
(𝑙)
= 𝐿𝑒𝑎𝑘𝑦𝑅𝑒𝐿𝑈(𝑚𝑢←𝑢
(𝑙)
+ 𝑖∈𝑁𝑢
𝑚𝑢←𝑖
(𝑙)
)
•
𝑚𝑢←𝑖
(𝑙)
= 𝑝𝑢𝑖 𝑊
1
𝑙
𝑒𝑖
𝑙−1
+ 𝑊
2
𝑙
(𝑒𝑖
𝑙−1
⊙ 𝑒𝑢
𝑙−1
)
𝑚𝑢←𝑢
(𝑙)
= 𝑊
1
𝑙
𝑒𝑖
(𝑙−1)
• 𝑊
1
(𝑙)
, 𝑊
2
(𝑙)
, ∈ ℝ𝑑𝑙×𝑑𝑙−1
• The trainable transformation matrices
• 𝑒𝑖
(𝑙−1)
• The item representation generated from the
previous message-passing steps, memorizing the
messages from its (l-1)-hop neighbors

11. 11
Methodology
Embedding Propagation Layers (High-order Propagation)
• They can stack more embedding propagation layers to explore the high-order connectivity information
• High-order connectivities are crucial to encode the collaborative signal to estimate the relevance score
between a user and item
• Stacking l embedding propagation layers, a user is capable of receiving the messages propagated
from its l-hop neighbors

12. 12
Methodology
Model Prediction
• {eu
1
, ∙∙∙, eu
L
}
• eu
∗
= eu
(0)
∥∙∙∙∥ eu
(𝐿)
• ei
∗
= ei
(0)
∥∙∙∙∥ ei
(𝐿)
• The advantage of using concatenation lies in
its simplicity
• They conduct the inner product to estimate
the user’s preference towards the target item
• 𝑦NGCF 𝑢, 𝑖 = eu
∗ ⊤
ei
∗

13. 13
Methodology
Optimization
• Loss Function : Pairwise BPR Loss
• Considers the relative order between observed and unobserved user-item interactions
• It assumes that the observed interactions should be assigned higher prediction values than unobserved
ones
• 𝐿𝑜𝑠𝑠 = 𝑢,𝑖,𝑗 ∈𝒪 −ln 𝜎 𝑦𝑢𝑖 − 𝑦𝑢𝑗 + 𝜆 ∥ Θ ∥2
2
• 𝒪 = 𝑢, 𝑖, 𝑗 𝑢, 𝑖 ∈ ℛ+, 𝑢, 𝑖 ∈ ℛ− : Pairwise training data
• ℛ+ : observed interactions
• ℛ−
: unobserved interactions

14. 14
Methodology
Optimization
• Message Dropout
• Randomly drops out the outgoing messages with probability 𝑝1
• In the l-th propagation layer, only partial messages contribute to the refined representations
• Endows the representations more robustness against the presence or absence of single connections
between users and items
• Node Dropout
• Conduct node dropout to randomly block a particular node and discard all its outgoing messages
• l-th propagation layer, they randomly drop 𝑀 + 𝑁 𝑝2 nodes of the Laplacian matrix, where 𝑝2 is the
dropout ratio
• Focuses on reducing the influences of particular users or items

15. 15
Discussions
NGCF Generalizes SVD++
• SVD++ can be viewed as a special case of NGCF with no high-order propagation layer
• They set L to one
• In the propagation layer, they disable the transformation matrix and nonlinear activation function
• 𝑦𝑁𝐺𝐶𝐹−𝑆𝑉𝐷 = 𝑒𝑢 + 𝑖′∈𝑁𝑢
𝑝𝑢𝑖′𝑒𝑖′
⊤
(𝑒𝑖 + 𝑢′∈𝑁𝑖
𝑝𝑖𝑢′𝑒𝑖)
• 𝑝𝑢𝑖′ → 1/ |𝑁𝑢|
• 𝑝𝑢′𝑖 → 0
• They can recover SVD++ model
• FISM
• 𝑝𝑢′𝑖 → 0

16. 16
Experiments
Research Questions
• RQ1 : How does NGCF perform as compared with state-of-the-art CF methods?
• RQ2 : How do different hyper-parameter settings affect NGCF?
• RQ3 : How do the representations benefit from the high-order connectivity?
Datasets

17. 17
Experiments
Evaluation Metrics
• recall@𝐾
• 𝑅𝑒𝑐𝑎𝑙𝑙𝑘 ∶
𝑁𝑟𝑠
𝑁𝑟
Evaluation Metrics
• ndcg@𝐾
• 𝐶𝐺𝑘 =
𝐶𝐺𝑘
𝐷𝐶𝐺𝑘
• 𝐷𝐶𝐺𝑘 = 𝑖=1
𝑘 𝑟𝑒𝑙𝑖
log2(𝑖+1)
𝑜𝑟 𝐷𝐶𝐺𝑘 = 𝑖=1
𝑘 2𝑟𝑒𝑙𝑖−1
log2(𝑖+1)

18. 18
Experiments
Overall Performance Comparison

19. 19
Experiments
The sparsity distribution of user groups on different datasets

20. 20
Experiments
Effect of embedding propagation layer numbers (L)

21. 21
Experiments
Effect of graph convolution layers

22. 22
Experiments
Effect of node dropout and message dropout ratios

23. 23
Experiments
Test performance of each epoch of MF and NGCF

24. 24
Experiments
Visualization of the learned t-SNE transformed representations

25. 25
Conclusion And Future Work
Conclusion
• They explicitly incorporated collaborative signal into the embedding function of model-based CF
• They devised a new framework NGCF, which achieves the target by leveraging high-order connectivities
in the user-item integration graph.
• NGCF is based on which they allow the embeddings of users and items interact with each other to harvest
the collaborative signal
Future Work
• They will further improve NGCF by incorporating the attention mechanism to learn variable weights for
neighbors during embedding propagation and for the connectivities of different orders
• They are interested in exploring the adversarial learning on user/item embedding and the graph structure
for enhancing the robustness of NGCF