NS-CUK Joint Journal Club: V.T.Hoang, Review on "NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs", ICLR 2023

1. 2023. 03. 16
Van Thuy Hoang
Network Science Lab
Dept. of Artificial Intelligence
The Catholic University of Korea
E-mail: hoangvanthuy90@gmail.com

2. 2
Program Flow
Ecological Network
Biological Network Social Network
Chemical Network Web Graph
Graphs are everywhere.

3. 3
• The goal:
• Mapping individual nodes to vector points in latent space.
Graph embedding learning

4. 4
• GNN-based approaches provide faster and practical training and
state-of-the-art results on benchmark datasets for downstream
tasks such as node classification
GNN-based models

5. 5
GNN aggregation
 Aggregation from neighbourhoods

6. 6
Problems
 Most of the existing GNN architectures have two fundamental weaknesses
which restrict their learning ability on general graph-structured data:
 Over-smoothing problems
 GNNs fail to be deep enough: Trade-off between features and
structure
 Noise from neighbours
 GNNs seem to be tailor-made to work on homophilic (associative)
graphs

7. 7
Problems
 The existing graph transformers:
 Treat the nodes as independent tokens
 Construct a single sequence of all the node tokens to train the model

8. 8
Contributions
 Training such models on large graphs will cost a huge GPU resources
 NAGphormer:
 Hop2Token
 Attention-based readout function

9. 9
Model framework of NAGphormer
 XXX

10. 10
Generate sequences of all nodes in graphs
 XXX

11. 11
Details
 Transformer layer:
 Readout function:

12. 12
IMPLEMENTATION DETAILS
 Structural encoding
 Besides the attribute information of nodes, the structural information of
nodes is also a crucial feature for graph mining tasks
 the eigenvectors of Laplacian matrix of the graph for capturing the
structural information of nodes

13. 13
EXPERIMENTS
 Graph datasets
 Baselines:
 GNNs
 Graph transformers

14. 14
Results
 Small-scale datasets

15. 15
Results
 Large-scale datasets

16. 16
Ablation study
 Structural encoding

17. 17
Ablation study
 Different readout functions.

18. 18
Pros & Cons
 Pros
 What Hop2Token can do:
 Capture global information (up to K-hop)
 Solve isomophic subgraphs
 Attention layer:
 learning more informative node representations from the multi-hop
neighborhoods
 Cons:
 Expressive: <= 1-d Weisfeiler lehman test
 Fails to capture graph structure
 Noise from neighbourhood