Gapformer is a model that combines graph transformers with graph pooling for efficient node classification in large graphs. It addresses two issues with existing graph transformers: quadratic complexity with number of nodes and noise from distant neighbors. Gapformer uses graph pooling to reduce the number of attended nodes, computing attention over pooled nodes only. Experiments on 13 datasets show Gapformer outperforms other graph neural networks and graph transformers, with reduced computation and memory costs.

1. Joo-Ho Lee
School of Computer Science and Information Engineering,
The Catholic University of Korea
E-mail: jooho414@gmail.com
2023-09-25

2. 1
Introduction
Problem Statement
• Existing GTs are exploited primarily for graph-level tasks (e.g., graph classification) with a small number of
nodes in a graph
• Developing GTs for node classification, where the number of nodes in a graph is relatively large (up to around
one million), remains a challenging proposition for the following two reasons.

3. 2
Introduction
Problem Statement
• First, the quadratic computational complexity 𝑂 𝑛2
of self-attention in vanilla GTs, in regards to the number of
nodes, inhibits their application to node classification in real-world scenarios
• Second, vanilla GTs calculate the full connected attention and aggregate messages from arbitrary nodes,
including numerous irrelevant nodes
• This results in ambiguous attention weights and the aggregation of noise information from incorrectly correlated
nodes

4. 3
Introduction
Problem Statement
• Only a few existing works have attempted to consider GTs for node classification.
• GT-sparse and SAN confine the receptive field of each node to its 1-hop neighboring nodes

5. 4
Introduction
Problem Statement
• As a result, expressiveness is sacrificed when important interactions are multiple hops away, especially in the
large-scale graph correspondingly requiring a large receptive field
• Existing studies neglect unique characteristics of graph data and tend to yield dense attention, causing an
enormous amount of noise messages to be aggregated from irrelevant nodes
• In light of the above analysis, they propose Gapformer, which combines Graph Transformer with Graph Pooling,
to capture long-range dependencies and improve the efficiency of vanilla GTs

6. 5
Introduction
Problem Statement
• In vanilla GTs, self-attention converts nodes into queries and keys/values, after which each query attends to all
the keys
• Specifically, self-attention involves computing the inner product between the query and key vectors to generate
attention scores
• These scores are then used to perform a weighted aggregation of value vectors
• To reduce the complexity of the dense inner product, Gapformer first utilizes graph pooling to group key and
value nodes into a smaller number of pooling nodes

7. 6
Introduction
Problem Statement
• For graph pooling, we propose two types of strategies to compress the original graph efficiently and effectively
1. global graph pooling
2. local graph pooling

8. 7
Introduction
Contribution
• They propose Gapformer, a deeper combination of Transformer and Graph Neural Networks
 Specifically, Gapformer utilizes Graph Pooling to group the attended nodes of each node into pooling
nodes (fewer in number) and computes its attention using only the pooling nodes.
• They conduct extensive experiments to compare Gapformer with 20 GNN and GT baseline models in the node
classification task on 13 real-world graph datasets, including homophilic and heterphilic datasets

9. 8
Methodology
Architecture

10. 9
Methodology
Attention Enhanced with Graph Pooling
𝐐 = 𝐇𝐖𝑄, 𝐊 = 𝐇𝐖𝐾, 𝐕 = 𝐇𝐖𝑉
𝐊𝒮 𝑖 = Pooling 𝐊
𝐕𝒮 𝑖 = Pooling 𝐕
𝒉𝑖 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝛼𝑞𝑖
⊤
, 𝐊𝒮 𝑖 𝐕𝒮 𝑖
⊤
𝑧𝑖 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝛼𝑞𝑖
⊤
, 𝐊𝒩 𝑖 𝐕𝒩 𝑖
⊤
𝒉𝑖
′
= 𝒉𝑖 + 𝛽 ∗ 𝑧𝑖

11. 10
Methodology
Attention Enhanced with Global Graph Pooling (AGP-G)
𝐊𝒮
𝐺𝑙𝑜𝑏𝑎𝑙
= Pooling 𝐇𝐖𝐾, 𝐀
𝐕𝒮
𝐺𝑙𝑜𝑏𝑎𝑙
= Pooling 𝐇𝐖𝑉
, 𝐀

12. 11
Methodology
Attention Enhanced with Global Graph Pooling (AGP-G)
𝐊𝒮 𝑖
𝐿𝑜𝑐𝑎𝑙
= Pooling 𝐊𝒩(𝑖,𝑘)
𝐕𝒮 𝑖
𝐿𝑜𝑐𝑎𝑙
= Pooling 𝐕𝒩(𝑖,𝑘)

13. 12
Methodology
Merits of Gapformer
• Reducing Computational Complexity
• Reducing the Ratio of Noisy Connections
• Handling Long-range Dependency

14. 13
Experiment
Datasets

15. 14
Experiment
Node Classification on homogeneous datasets

16. 15
Experiment
Node Classification on heterophilic datasets

17. 16
Experiment
Comparison of training time and GPU memory costs

18. 17
Experiment
Ablation Study : Components of Graph Transformer architecture

19. 18
Experiment
Ablation Study : Different Pooling Types

20. 19
Experiment
Ablation Study : Different parameters

21. 20
Conclusion
• In this paper, they propose Gapformer, which combines Graph Transformers (GTs) with Graph Pooling for
efficient node classification
• Their Gapformer addresses the two main issues of existing GTs
• potential noises from long-distance neighbors
• the quadratic computational complexity in regards to the number of nodes
• Extensive experiments on 13 graph datasets demonstrate that Gapformer outperforms existing GTs and Graph
Neural Networks
• Despite its competitive performance, Gapformer still has room for improvement
• devising an effective manner to combine the proposed local pooling enhanced attention and global pooling
enhanced attention
• incorporating useful techniques to further enhance the performance on large-scale graph datasets