Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to An Introduction to Spectral Graph Theory
Similar to An Introduction to Spectral Graph Theory (20)
More from joisino
More from joisino (13)
Recently uploaded
Recently uploaded (20)
An Introduction to Spectral Graph Theory
- 1. 1 KYOTO UNIVERSITY KYOTO UNIVERSITY An Introduction to Spectral Graph Theory Ryoma Sato
- 2. 2 / 47 KYOTO UNIVERSITY Spectral graph theory studies engenvalues Spectral graph theory studies graphs via the lens of eigenvalues and eigenvectors of adjacency matrices It seems magnesium exists λ2 = 11.9 λ3 = 44.7 It seems there are four clusters It seems this is a clique-like graph List of eigenvalues
- 3. 3 / 47 KYOTO UNIVERSITY Spectra connect combinatorics and algebra Why important? Spectral graph theory connects combinatorics and algebra It also connects discrete and continuous mathematics Graph = combinatorial discrete Matrix = algebraic continuous Spectral Graph Theory
- 4. 4 / 47 KYOTO UNIVERSITY Today’s topic: spectral clustering & GNN Today’s topic Basics: Laplacian and cuts Investigate shapes of graphs i.e., investigate the basis of Laplacian * Spectral clustering * Spectral hashing Investigate signals on graphs i.e., investigate representations based on the basis * Graph neural networks
- 5. 5 / 47 KYOTO UNIVERSITY Basics
- 6. 6 / 47 KYOTO UNIVERSITY Laplacian = dI - A We consider undirected d-regular graphs in this presentation The Laplacian matrix of a graph is Example of 3-regular graphs All nodes have 3 neighbors A: adjacency matrix I: identity matrix
- 7. 7 / 47 KYOTO UNIVERSITY xT Lx measures the smoothness of signal x Let be a signal vector on G is a value assigned to node v measures the smoothness of signal x on G: If and are similar for all edges , is small If and are dissimilar for edges , is large
- 8. 8 / 47 KYOTO UNIVERSITY Min-max Theorem tells eigenvalue&vector Min-max Theorem For any symmetric matrix is the minimum eigenvalue is its corresponding eigenvector
- 9. 9 / 47 KYOTO UNIVERSITY Min-max Theorem tells eigenvalue&vector Min-max Theorem For any symmetric matrix is the second minimum eigenvalue is its corresponding eigenvector
- 10. 10 / 47 KYOTO UNIVERSITY Min-max Theorem tells eigenvalue&vector Min-max Theorem For any symmetric matrix Proof Sketch: Do spectral decomposition. is the third minimum eigenvalue is its corresponding eigenvector
- 11. 11 / 47 KYOTO UNIVERSITY Eigen vectors are smooth signal vectors What does the min-max theorem tell? The eigenvector v1 of L associated with the smallest eigenvalue λ1 is the smoothest signal vector It is obvious that v1 = [1, 1, …, 1]T is the smoothest and λ1 = 0 The second eigenvector v2 is the next smoothest signal vector λ2 tells how smooth v2 is. next smoothest = smoothest vector x s.t. xT v1 = 0 = smoothest vector x s.t. x is sum to zero = almost half are positive & others neg Completely smooth xT Lx = 0
- 12. 12 / 47 KYOTO UNIVERSITY Two component graph has zero second eigenvalue Case 1: Two connected components The second eigenvector does not incur any penalties because no edges exist between the components λ1 = 0 v1 = [1, 1, …, 1]T λ2 = 0 v2 = [1, 1, …, 1, -1, -1, …, -1]T The examples here are not regular, but almost similar arguments follow
- 13. 13 / 47 KYOTO UNIVERSITY Small purturbation does not destroy the results Case 2: Two connected components plus an edge An edge between the components causes a slight penalty But the eigenvector is almost the same λ1 = 0 v1 = [1, 1, …, 1]T λ2 = 0.15 v2 = [1, 0.99, …, 0.97, -0.97, -0.98, …, -1]T
- 14. 14 / 47 KYOTO UNIVERSITY Eigenvectors cut the graph with small cut edges Case 3: Four communities The 2nd-4th eigenvectors cuts the graph into two parts The 2nd one cuts the smallest number of edges The 3rd one cuts the second smallest number of edges The 4th one cuts the third smallest number of edges 1st 2nd 3rd 4th
- 15. 15 / 47 KYOTO UNIVERSITY Matrix factorization also explains eigenvectors Another interpretation: matrix factorization Small eigenvalues of L = dI - A corresponds to large eigenvalues of A If then is the best rank-k approximation (cf. PCA) community IDs
- 16. 16 / 47 KYOTO UNIVERSITY Spectra, revisited Now, we can infer the shape of a graph solely from its spectra ↑ This graph has four small eigenvalues → 4 ways to cut the graph → 4 communities ↑ Any cut (other than a constant signal) causes a large penalty λ2 = 11.9 λ3 = 44.7 It seems there are four clusters It seems this is a clique-like graph List of eigenvalues
- 17. 17 / 47 KYOTO UNIVERSITY Spectral Clustering
- 18. 18 / 47 KYOTO UNIVERSITY We cluster nodes into k parts Problem Setting: node clustering In: A graph G An integer k Out: Cluster nodes into k sets We also consider clustering of vectors In: vectors {x1 , x2 , …, xn } An integer k Out: Cluster nodes into k sets In this case, construct kNN graph → solve the graph version Note that the positions are not observable (but can be recovered by eigenvectors as we will see shortly!)
- 19. 19 / 47 KYOTO UNIVERSITY Spectral clustering uses eigenvectors as features Algorithm (Spectral Clustering) 1. Compute eigenvectors v1 , …, vn of Laplacian 2. Let zi = [v1,i , …, vd,i ]T R ∈ d for each node i d is a hyperparamter 3. Apply k-means to {z1 , z2 , …, zn }
- 20. 20 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Input: Eigenvectors: Output: 2nd 3rd
- 21. 21 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Input: Simple k-means fail on non-spherical clusters Simple k-means without spectral clustering
- 22. 22 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Build a kNN graph
- 23. 23 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Extract eigenvectors 2nd 3rd
- 24. 24 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Extract features 2nd 3rd z2 z3
- 25. 25 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Do k-means on spectra 2nd 3rd z2 z3
- 26. 26 / 47 KYOTO UNIVERSITY Eigenvectors are useful features for clustering Pull back the results 2nd 3rd z2 z3 Use the assignment
- 27. 27 / 47 KYOTO UNIVERSITY Graph Signal Processing
- 28. 28 / 47 KYOTO UNIVERSITY Fundamental example: cycle Let’s consider a cycle graph
- 29. 29 / 47 KYOTO UNIVERSITY First eigenvector is constant Let’s consider a cycle graph The first eigenvector is a constant vector as usual 1nd
- 30. 30 / 47 KYOTO UNIVERSITY Second eigenvector is cosine Let’s consider a cycle graph The second eigenvector is cos(2kπ/n) cos is indeed smooth! This can be proved by high school math (addition theorem) 1st 2nd
- 31. 31 / 47 KYOTO UNIVERSITY Following eigenvectors are also trigonometric Let’s consider a cycle graph 1st 2nd 3rd 4th 3rd = sin(2kπ/n) 4th = cos(4kπ/n) and so on...
- 32. 32 / 47 KYOTO UNIVERSITY Take a random signal on the cycle Let’s consider a signal on the cycle graph 1st 2nd 3rd 4th and so on...
- 33. 33 / 47 KYOTO UNIVERSITY Represent the signal by the eigenvector basis Represent the signal by a linear combination of the eigvecs 1st 2nd 3rd 4th × 0.1 × 0.3 × 0.0 × 0.6 and so on...
- 34. 34 / 47 KYOTO UNIVERSITY Eigen representation is Fourier series This is the discrete version of Fourier expansion! 1st 2nd 3rd 4th × 0.1 × 0.3 × 0.0 × 0.6 and so on...
- 35. 35 / 47 KYOTO UNIVERSITY Fourier series for general graphs A case of a general graph 1st 2nd 3rd 4th × 0.53 × 0.03 × 0.54 × 0.28
- 36. 36 / 47 KYOTO UNIVERSITY Fourier series for general graphs A case of a general graph 1st 2nd 3rd 4th × 0.53 × 0.03 × 0.54 × 0.28 How to compute these values?
- 37. 37 / 47 KYOTO UNIVERSITY Fourier transform can be done by vec-mat product Spectral decomposition of Laplacian Fourier transform: Inverse transform: E.g., when y = [0.8, 0.2, 0, 0, …, 0]T , x = 0.8 v1 + 0.2 v2
- 38. 38 / 47 KYOTO UNIVERSITY Low-pass filter for a standard 1D signal Low-pass filter for a 1D signal smooths the signal [0.1, 0.0, 0.3, 0.6, 0.0, -0.4, -0.1, 0.2, ...] Fourier transform Inverse transform [1, 1, 1, 1, 0, 0, 0, 0, ….] Low-pass filter [0.1, 0.0, 0.3, 0.6, 0.0, 0.0, 0.0, 0.0, ...]
- 39. 39 / 47 KYOTO UNIVERSITY Low-pass filter for a graph signal Low-pass filter for a graph signal smooths the signal [0.03, 0.54, 0.53, 0.28, 0.02, 0.03, ...] Fourier transform Inverse transform [1, 1, 1, 1, 0, 0, 0, 0, ….] Low-pass filter [0.03, 0.54, 0.53, 0.28, 0.0, 0.0, ...]
- 40. 40 / 47 KYOTO UNIVERSITY A special low-pass filter f Filter [1, 1, 1, 1, 0, 0, 0, …] strictly cuts out high-freq signals Let’s consider a filter for some real value w E.g., when f can be seen a smooth low-pass filter (in a loose manner) It is guaranteed that f1 >= |fn |
- 41. 41 / 47 KYOTO UNIVERSITY Applying filter f = Mltiplying A, x and w The results of applying f to x can be computed as follows This is just multiplications of adjacency matrix A and x and w Efficiently computable (by message passing) Multiply f Fourier transform Inverse transform f = w(d - λ) V VT = I, L = V Λ VT L = dI - A
- 42. 42 / 47 KYOTO UNIVERSITY Generalizing to vector signals So far, we have considered a scalar value for each node Let’s consider each node has a vector feature Then, the aforementioned filter can be generalized to W diagonal → each signal is handled independently W full → signals are linearly mixed (we usually use this) 1-dim: d-dim:
- 43. 43 / 47 KYOTO UNIVERSITY Graph Convolutional Networks
- 44. 44 / 47 KYOTO UNIVERSITY We classify each node into categories Problem Setting: supervised node classification In: A graph G Feature vectors X Out: Label of each node Y We want a function that transforms X into Y Assumption: Y is smooth on the graph Labels are similar within a cluster
- 45. 45 / 47 KYOTO UNIVERSITY GCN does filtering and node-wise transformation Graph Convolutional Networks Training of W1 & W2 : Do SGD just as standard neural networks Filtering Node-wise non-linear transformation Filtering Node-wise non-linear transformation
- 46. 46 / 47 KYOTO UNIVERSITY Conclusion
- 47. 47 / 47 KYOTO UNIVERSITY Spectral graph theory is useful Spectral graph theory is a useful toolbox for graph analysis If you want to know the shape of a graph, investigate the eigenvalues and eigenvectors If you want to analyze/manipulate graph signals, use representations based on eigenvectors λ2 = 11.9 λ3 = 44.7 It seems there are four clusters List of eigenvalues Low pass