NS-CUK Seminar: J.H.Lee, Review on "Recipe for a General, Powerful, Scalable Graph Transformer", NeurIPS 2022

1. Joo-Ho Lee
School of Computer Science and
Information Engineering, The Catholic
University of Korea
E-mail: jooho414@gmail.com
2023-04-14

2. 1
 Introduction
• Problem statement
• Contributions
 Related Works
 Methodology
 Experiment
 Conclusion

3. 2
Introduction
Problem Statement
• Graph Transformer alleviates the problems of message passing by using global attention (attention across all
nodes).
• To enable global attention, nodes in the graph and its substructures must be well-identified. This has led to the
development of various positional encoding schemes that adjust spectral features and graph features.
• Standard graph attention has a time complexity of $O(N^2)$, which limits the number of nodes in a GTs to a few
hundred to avoid increasing costs.
• While the focus is on the aspect of identifying specific nodes, there is no principled framework for designing GTs.

4. 3
Introduction
Contribution
• Proposed design principles for GPS Graph Transformers (GTs).
• Defined better positional encoding and spectral encoding schemes, and organized them into local, global, and
relative categories.
• Demonstrated that GPS with linear global attention from Performer and BigBird can scale to thousands of nodes
and achieve good results even without explicit edge features in the attention module, while traditional fully-
connected GTs are limited to graphs with a maximum of a few hundred nodes.

5. 4
Methodology
Architecture

6. 5
Methodology
Modular positional and structural encodings

7. 6
Methodology
GPS layers

8. 7
Experiment

9. 8
Experiment

10. 9
Experiment

11. 10
Experiment

12. 11
Experiment

13. 12
Conclusion
• This paper lays the groundwork for a unified architecture of graph neural networks, incorporating modular and
scalable graph Transformers and a better understanding of the significance of positional and structural
encodings in graphs
• Through our ablation studies, they have shown that each module - the Transformer, flexible message-passing,
and advanced positional and structural encodings - played a vital role in the success of GPS in various
benchmarks.