This document presents Specformer, a novel Transformer-based spectral graph neural network. Specformer uses a Transformer to encode eigenvalues of the graph Laplacian into representations that capture magnitude and relative differences. It then applies self-attention to learn relationships between eigenvalues. Specformer decodes the eigenvalue representations using learnable bases to perform graph convolutions. Experiments show Specformer outperforms other GNNs on node and graph classification tasks and learns meaningful patterns in the graph spectrum.

1. Joo-Ho Lee
School of Computer Science and Information Engineering,
The Catholic University of Korea
E-mail: jooho414@gmail.com
2023-09-18

2. 1
Introduction
Problem Statement
• Depending on how the graph signals (or features) are leveraged, GNNs can be roughly categorized into two
classes, namely spatial GNNs and spectral GNNs
• Although spatial GNNs have achieved impressive performances in many domains, spectral GNNs are
somewhat under-explored

3. 2
Introduction
Problem Statement
• There are a few reasons why spectral GNNs have not been able to catch up.
• First, most existing spectral filters are essentially scalar-to-scalar functions
• In particular, they take a single eigenvalue as input and apply the same filter to all eigenvalues
• This filtering mechanism could ignore the rich information embedded in the spectrum, i.e., the set of eigenvalues

4. 3
Introduction
Problem Statement
• Second, the spectral filters are often approximated via fixed-order (or truncated) orthonormal bases, e.g.,
Chebyshev polynomials and graph wavelets, in order to avoid the costly spectral decomposition of the graph
Laplacian
• Although the orthonormality is a nice property, this truncated approximation is less expressive and may severely
limit the graph representation learning

5. 4
Introduction
Problem Statement
• Therefore, in order to improve spectral GNNs, it is natural to ask
 How can we build expressive spectral filters that can effectively leverage the spectrum of graph Laplacian?
• To answer this question, they first note that eigenvalues of graph Laplacian represent the frequency, i.e., total
variation of the corresponding eigenvectors
• The magnitudes of frequencies thus convey rich information.
• Moreover, the relative difference between two eigenvalues also reflects important frequency information, e.g.,
the spectral gap

6. 5
Introduction
Problem Statement
• To capture both magnitudes of frequency and relative frequency, they propose a Transformer based set-to-set
spectral filter, termed Specformer
• Their Specformer first encodes the range of eigenvalues via positional embedding and then exploits the self-
attention mechanism to learn relative information from the set of eigenvalues
• Relying on the learned representations of eigenvalues, they also design a decoder with a bank of learnable
bases

7. 6
Introduction
Contribution
• They propose a novel Transformer-based set-to-set spectral filter along with learnable bases, called
Specformer, which effectively captures both magnitudes and relative differences of all eigenvalues of the graph
Laplacian
• They show that Specformer is permutation equivariant and can perform non-local graph convolutions, which is
non-trivial to achieve in many spatial GNNs
• Experiments on synthetic datasets show that Specformer learns to better recover the given spectral filters than
other spectral GNNs
• Extensive experiments on various node-level and graph-level benchmarks demonstrate that Specformer
outperforms state-of-the-art GNNs and learns meaningful spectrum patterns

8. 7
Methodology
Architecture

9. 8
Methodology
EIGENVALUE ENCODING
• Eigenvalue encoding function
𝜌 𝜆, 2𝑖 = sin 𝜖𝜆/100002𝑖/𝑑
𝜌 𝜆, 2𝑖 + 1 = cos 𝜖𝜆/100002𝑖/𝑑
• The benefits of 𝜌 𝜆
• It can capture the relative frequency shifts of eigenvalues and provides high-dimension vector
representations
• It has wavelengths from 2𝜋 to 10000 ⋅ 2𝜋, which forms a multi-scale representation for eigenvalues
• It can control the influence of 𝜆 by adjusting the value of 𝜖

10. 9
Methodology
EIGENVALUE ENCODING
• The initial representation of eigenvalues
𝑍 = 𝜆1| 𝜌 𝜆1 , … , 𝜆𝑛 |𝜌(𝜆𝑛)
𝑍 = 𝑀𝐻𝐴 𝐿𝑁 𝑍 + 𝑍
𝑍 = 𝐹𝐹𝑁 𝐿𝑁 𝑍 + 𝑍

11. 10
Methodology
EIGENVALUE DECODING
• Spectral Filters
𝑍𝑚 = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑄𝑊
𝑚
𝑄
, 𝐾𝑊
𝑚
𝐾, 𝑉𝑊
𝑚
𝑉 , 𝜆𝑚 = 𝜙 𝑍𝑚, 𝑊𝜆
• Learning bases
𝑆𝑚 = 𝑈𝑑𝑖𝑎𝑔 𝜆𝑚 𝑈⊤
, 𝑆 = 𝐹𝐹𝑁 𝐼𝑛| 𝑆1 | … ||𝑆𝑀

12. 11
Methodology
GRAPH CONVOLUTION
• Node representation for GCN
𝑋:,𝑖
𝑙−1
= 𝑆:,:,𝑖𝑋:,𝑖
𝑙−1
, 𝑋 𝑙 = 𝜎 𝑋(𝑙−1)𝑊
𝑥
𝑙−1
+ 𝑋 𝑙−1

13. 12
Experiment
Datasets

14. 13
Experiment
Node Classification

15. 14
Experiment
Graph Classification

16. 15
Experiment
Ablation Study on node-level and graph-level task

17. 16
Experiment
Visualization

18. 17
Experiment
Visualization

19. 18
Conclusion
• In this paper, they propose Specformer that leverages Transformer to build a set-to-set spectral filter along with
learnable bases
• Specformer effectively captures magnitudes and relative dependencies of the eigenvalues in a permutation-
equivariant fashion and can perform non-local graph convolution
• Experiments on synthetic and real-world datasets demonstrate that Specformer outperforms various GNNs and
learns meaningful spectrum patterns
• A promising future direction is to improve the efficiency of Specformer through sparsifying the self-attention
matrix of Transformer