Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Spectral cnn
Similar to Spectral cnn (20)
More from Brian Kim
Recently uploaded
Recently uploaded (20)
Spectral cnn
- 1. Geometric Deep Learning GoingBeyondEuclideanData KoreaUniversity, DepartmentofComputerScience&Radio CommunicationEngineering 김범수 Bumsoo Kim SpectralMethods
- 2. Convolutional Theorem DMIS Presentation 2 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance
- 3. Convolutional Theorem DMIS Presentation 3 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance Grids in 2D image formats : pixels
- 4. Convolutional Theorem DMIS Presentation 4 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance 같은 물체는 translation 이후에도 같은 feature vector로 맵핑이 이루어져야 한다. Invariance를 강화하기 위한 기법으로 가장 대표적인 것이 Image Augmentation. 회전된 feature vector가 같아지게끔 학습한다.
- 5. Convolutional Theorem DMIS Presentation 5 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance 같은 물체에 대해 translation 한 결과는 그 feature vector에 translation을 한 결과와 같다. Equivariance를 강화하기 위한 기법으로 가장 대표적인 것이 Group Convnet & Capsule Net 회전된 feature vector는 둘 다 ‘비행기＇라고 학습한다.
- 6. Convolutional Theorem DMIS Presentation 6 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance Translational structure allows weight sharing Grid based metric allows compactly supported filters Multiscale dyadic clustering allows subsampling input크기와 무관하게 적은 개수의 parameter개수로 학습 stride convolutions & pooling
- 7. Convolutional Theorem DMIS Presentation 7 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance Translational structure allows weight sharing Grid based metric allows compactly supported filters Multiscale dyadic clustering allows subsampling input크기와 무관하게 적은 개수의 parameter개수로 학습 stride convolutions & pooling dyadic refers to a domain with two abstract sets of objects 𝑋𝑋 = 𝑥𝑥1,… , 𝑥𝑥𝑁𝑁 and 𝑦𝑦 = {𝑦𝑦1, … , 𝑦𝑦𝑀𝑀} in which observations 𝑆𝑆 are made for dyads (𝑥𝑥𝑖𝑖, 𝑦𝑦𝑘𝑘).
- 8. Convolutional Theorem DMIS Presentation 8 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance Translational structure allows weight sharing Grid based metric allows compactly supported filters Multiscale dyadic clustering allows subsampling input크기와 무관하게 적은 개수의 parameter개수로 학습 stride convolutions & pooling 3D-MESH SOCIAL NETWORK GRAPH FUNCTION ON MESH (HEAT) POINT CLOUD
- 9. Convolutional Theorem DMIS Presentation 9 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance Translational structure allows weight sharing Grid based metric allows compactly supported filters Multiscale dyadic clustering allows subsampling input크기와 무관하게 적은 개수의 parameter개수로 학습 stride convolutions & pooling Spatial Construction Spectral Construction multiscale clustering SCNN & Coarsening
- 10. Convolutional Theorem DMIS Presentation 10 Why did Convolutional Neural Networks work so well? Coordinates of data representation has grid structure Convolutional Neural Networks Translational Invariance/Equivariance Translational structure allows weight sharing Grid based metric allows compactly supported filters Multiscale dyadic clustering allows subsampling input크기와 무관하게 적은 개수의 parameter개수로 학습 stride convolutions & pooling Spatial Construction Spectral Construction multiscale clustering SCNN & Coarsening
- 11. Spectral Graph Theorem DMIS Presentation 11 Simple Intuition
- 12. Spectral Graph Theorem DMIS Presentation 12 Simple Intuition
- 13. Spectral Graph Theorem DMIS Presentation 13 Simple Intuition Riemannian manifolds Graphs “Eigenvectors of the graph Laplacian converge to Eigenfunction of the Laplace-Beltrami operator on the underlying manifold” Fourier Basis Eigenvectors of Laplacian Increasing magnitudes of the eigenvalues correspond to increasing frequencies of the eigenvectors
- 14. Spectral Graph Theorem DMIS Presentation 14 What is Spectral? Spectrum of matrix representing G Spectral Graph Theory
- 15. Spectral Graph Theorem DMIS Presentation 15 What is Spectral? Spectrum of matrix representing G Spectral Graph Theory Eigenvectors ordered by magnitude of corresponding eigenvalues. Spectrum
- 16. Spectral Graph Theorem DMIS Presentation 16 What is Spectral? Spectrum of matrix representing G Spectral Graph Theory Eigenvectors ordered by magnitude of corresponding eigenvalues. Spectrum Self-adjoint matrix ex) 2 2 − 𝑖𝑖 4 2 + 𝑖𝑖 3 −𝑖𝑖 4 𝑖𝑖 1 2 2 + 𝑖𝑖 4 2 − 𝑖𝑖 3 +𝑖𝑖 4 −𝑖𝑖 1 𝐴𝐴 = ̅𝐴𝐴 = ̅𝐴𝐴𝑇𝑇 = 2 2 − 𝑖𝑖 4 2 + 𝑖𝑖 3 −𝑖𝑖 4 +𝑖𝑖 1 = 𝐴𝐴 에르미트 행렬 실수 대칭 행렬 𝐴𝐴 = 𝐴𝐴 ∗= ̅𝐴𝐴𝑇𝑇
- 17. Spectral Graph Theorem DMIS Presentation 17 What is Spectral? Spectrum of matrix representing G Spectral Graph Theory Eigenvectors ordered by magnitude of corresponding eigenvalues. Spectrum 에르미트 행렬 실수 대칭 행렬 Self-adjoint matrix 𝐴𝐴 = 𝐴𝐴 ∗= ̅𝐴𝐴𝑇𝑇 Eigen values are real Eigen vectors are orthogonal This assures that when moving to a spectrum, the eigen vectors will be an orthogonal basis and its corresponding eigenvalues will be real.
- 18. Spectral Graph Theorem DMIS Presentation 18 What is Spectral? Spectrum of matrix representing G Spectral Graph Theory Eigenvectors ordered by magnitude of corresponding eigenvalues. Spectrum 𝑀𝑀𝑀𝑀 = 𝜆𝜆𝜆𝜆 𝑀𝑀 − 𝜆𝜆𝜆𝜆 𝑥𝑥 = 0 det 𝑀𝑀 − 𝜆𝜆𝜆𝜆 = 0 ( 𝑥𝑥 ≠ 0) As 𝜆𝜆 is the variable in det 𝑀𝑀 − 𝜆𝜆𝜆𝜆 = 0, det 𝑀𝑀 − 𝜆𝜆𝜆𝜆 = 0 is a 𝑛𝑛 degree polynomial. Therefore, there are 𝑛𝑛 eigenvalues 𝜆𝜆1, 𝜆𝜆2, 𝜆𝜆3, … , 𝜆𝜆𝑛𝑛 When 𝜆𝜆1 ≤ 𝜆𝜆2 ≤ 𝜆𝜆3 ≤ … ≤ 𝜆𝜆𝑛𝑛, spectrum = {𝑥𝑥1, 𝑥𝑥2, 𝑥𝑥3, … , 𝑥𝑥𝑛𝑛} The solution for this equation is the eigen value ∴
- 19. Spectral Graph Theorem DMIS Presentation 19 What is Spectral? Combinatorial Laplacian Spectral Construction Graph Laplacian 𝐿𝐿 = 𝐷𝐷 − 𝑊𝑊 𝐿𝐿 = 𝐼𝐼 − 𝐷𝐷− 1 2 𝑊𝑊𝐷𝐷− 1 2
- 20. Spectral Graph Theorem DMIS Presentation 20 What is Spectral? Combinatorial Laplacian Spectral Construction Graph Laplacian 𝐿𝐿 = 𝐷𝐷 − 𝑊𝑊 𝐿𝐿 = 𝐼𝐼 − 𝐷𝐷− 1 2 𝑊𝑊𝐷𝐷− 1 2 SCNN on d-dimensional grid On grid space, frequency and smoothness relative to W are interrelated through Laplacian. Smoothness If is concentrated in the low-end of the spectrum, the corresponding spatial kernel function is smooth; conversely, if the corresponding spatial functions is localized, m is smooth. Therefore, to obtain a smoother kernel function, we constrain the bandwidth of m, enabling us to learn a smaller number of parameters; varying the smoothness of m would control the kernel size.
- 21. Spectral Graph Theorem DMIS Presentation 21 What is Spectral? Combinatorial Laplacian Spectral Construction Graph Laplacian 𝐿𝐿 = 𝐷𝐷 − 𝑊𝑊 𝐿𝐿 = 𝐼𝐼 − 𝐷𝐷− 1 2 𝑊𝑊𝐷𝐷− 1 2 SCNN on d-dimensional grid On grid space, frequency and smoothness relative to W are interrelated through Laplacian. Smoothness
- 22. Spectral Graph Theorem DMIS Presentation 22 What is Spectral? Combinatorial Laplacian Spectral Construction Graph Laplacian 𝐿𝐿 = 𝐷𝐷 − 𝑊𝑊 𝐿𝐿 = 𝐼𝐼 − 𝐷𝐷− 1 2 𝑊𝑊𝐷𝐷− 1 2 SCNN on d-dimensional grid On grid space, frequency and smoothness relative to W are interrelated through Laplacian. Smoothness Eigenvectors of the Laplacian = Fourier vectors Eigenvector of Combinatorial Laplacian L =
- 23. Spectral Graph Theorem DMIS Presentation 23 What is Spectral? Combinatorial Laplacian Spectral Construction Graph Laplacian 𝐿𝐿 = 𝐷𝐷 − 𝑊𝑊 𝐿𝐿 = 𝐼𝐼 − 𝐷𝐷− 1 2 𝑊𝑊𝐷𝐷− 1 2 SCNN on d-dimensional grid On grid space, frequency and smoothness relative to W are interrelated through Laplacian. Smoothness Eigenvectors of the Laplacian = Fourier vectors Eigenvector of Combinatorial Laplacian L = Eigenvalues of the Laplacian = Fourier coefficients of a signal = smoothness
- 24. Spectral Graph Theorem DMIS Presentation 24 What is Spectral? Spectral CNN (SCNN) 𝜀𝜀 : nonlinearity (𝑓𝑓1, … , 𝑓𝑓𝑝𝑝) : input (𝑛𝑛 × 𝑝𝑝) (𝑔𝑔1, … , 𝑔𝑔𝑞𝑞) : output (𝑛𝑛 × 𝑞𝑞) 𝑛𝑛 = |𝑉𝑉| : # of vertices in the graph : (𝑘𝑘 × 𝑘𝑘) diagonal matrix (=filter) Combinatorial Laplacian Spectral Construction Graph Laplacian 𝐿𝐿 = 𝐷𝐷 − 𝑊𝑊 𝐿𝐿 = 𝐼𝐼 − 𝐷𝐷− 1 2 𝑊𝑊𝐷𝐷− 1 2
- 25. Spectral Graph Theorem DMIS Presentation 25 Basic : Convolutional Neural Networks 𝑓𝑓1 𝑓𝑓2 𝑓𝑓3 Γ1,1*𝑓𝑓1[:3,:3] + Γ1,2*𝑓𝑓2[:3,:3] + Γ1,3*𝑓𝑓3[:3,:3] Γ1,3 Γ1,2 Γ1,1
- 26. Spectral Graph Theorem DMIS Presentation 26 Basic : Convolutional Neural Networks Γ2,1 𝑓𝑓1 𝑓𝑓2 𝑓𝑓3 Γ2,3 Γ2,2 Γ2,1*𝑓𝑓1[:3,:3] + Γ2,2*𝑓𝑓2[:3,:3] + Γ2,3*𝑓𝑓3[:3,:3]
- 27. Spectral Graph Theorem DMIS Presentation 27 Basic : Convolutional Neural Networks 𝑓𝑓1 𝑓𝑓2 𝑓𝑓3 Γ3,3 Γ3,2 Γ3,1 Γ3,1*𝑓𝑓1[:3,:3] + Γ3,2*𝑓𝑓2[:3,:3] + Γ3,3*𝑓𝑓3[:3,:3]
- 28. Spectral Graph Theorem DMIS Presentation 28 Basic : Convolutional Neural Networks 𝑓𝑓1 𝑓𝑓2 𝑓𝑓3 Γ𝑝𝑝,3 Γ𝑝𝑝,2 Γ𝑝𝑝,1 Γ𝑝𝑝,1*𝑓𝑓1[:3,:3] + Γ𝑝𝑝,2*𝑓𝑓2[:3,:3] + Γ𝑝𝑝,3*𝑓𝑓3[:3,:3]
- 29. Spectral Graph Theorem DMIS Presentation 29 Spectral Convolution [𝑓𝑓1, 𝑓𝑓2, 𝑓𝑓3,… 𝑓𝑓𝑝𝑝] [𝑔𝑔1, 𝑔𝑔2, 𝑔𝑔3,… 𝑔𝑔𝑞𝑞] (n x p) (n x q)(n x k)(k x k)(k x n)
- 30. Spectral Graph Theorem DMIS Presentation 30 Spectral Convolution [𝑓𝑓1, 𝑓𝑓2, 𝑓𝑓3,… 𝑓𝑓𝑝𝑝] [𝑔𝑔1, 𝑔𝑔2, 𝑔𝑔3,… 𝑔𝑔𝑞𝑞] (n x p) (n x q)(n x k)(k x k)(k x n) 𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽𝐽 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓, uniquely factored as the product of an orthogonal matrix and a symmetric positive definite matrix complexity O n2 → O(n)
- 31. Spectral Graph Theorem DMIS Presentation 31 Spectral Convolution [𝑓𝑓1, 𝑓𝑓2, 𝑓𝑓3,… 𝑓𝑓𝑝𝑝] [𝑔𝑔1, 𝑔𝑔2, 𝑔𝑔3,… 𝑔𝑔𝑞𝑞] (n x p) (n x q)(n x k)(k x k)(k x n) (Φ𝑘𝑘Γ1,1Φ𝑘𝑘 𝑇𝑇 𝑓𝑓1)+ (Φ𝑘𝑘Γ1,2Φ𝑘𝑘 𝑇𝑇 𝑓𝑓2)+ (Φ𝑘𝑘Γ1,3Φ𝑘𝑘 𝑇𝑇 𝑓𝑓3)+ … +(Φ𝑘𝑘Γ1,𝑝𝑝Φ𝑘𝑘 𝑇𝑇 𝑓𝑓𝑝𝑝) 𝑠𝑠1 = 𝑔𝑔1 = 𝜉𝜉( 𝑔𝑔1)
- 32. Spectral Graph Theorem DMIS Presentation 32 Spectral Convolution [𝑓𝑓1, 𝑓𝑓2, 𝑓𝑓3,… 𝑓𝑓𝑝𝑝] [𝑔𝑔1, 𝑔𝑔2, 𝑔𝑔3,… 𝑔𝑔𝑞𝑞] (n x p) (n x q)(n x k)(k x k)(k x n) (Φ𝑘𝑘Γ2,1Φ𝑘𝑘 𝑇𝑇 𝑓𝑓1)+ (Φ𝑘𝑘Γ2,2Φ𝑘𝑘 𝑇𝑇 𝑓𝑓2)+ (Φ𝑘𝑘Γ2,3Φ𝑘𝑘 𝑇𝑇 𝑓𝑓3)+ … +(Φ𝑘𝑘Γ2,𝑝𝑝Φ𝑘𝑘 𝑇𝑇 𝑓𝑓𝑝𝑝) 𝑠𝑠2 = 𝑔𝑔2 = 𝜉𝜉( 𝑠𝑠2)
- 33. Spectral Graph Theorem DMIS Presentation 33 Spectral Convolution [𝑓𝑓1, 𝑓𝑓2, 𝑓𝑓3,… 𝑓𝑓𝑝𝑝] [𝑔𝑔1, 𝑔𝑔2, 𝑔𝑔3,… 𝑔𝑔𝑞𝑞] (n x p) (n x q)(n x k)(k x k)(k x n) (Φ𝑘𝑘Γ3,1Φ𝑘𝑘 𝑇𝑇 𝑓𝑓1)+ (Φ𝑘𝑘Γ3,2Φ𝑘𝑘 𝑇𝑇 𝑓𝑓2)+ (Φ𝑘𝑘Γ3,3Φ𝑘𝑘 𝑇𝑇 𝑓𝑓3)+ … +(Φ𝑘𝑘Γ3,𝑝𝑝Φ𝑘𝑘 𝑇𝑇 𝑓𝑓𝑝𝑝) 𝑠𝑠3 = 𝑔𝑔3 = 𝜉𝜉( 𝑠𝑠3)
- 34. Spectral Graph Theorem DMIS Presentation 34 Spectral Convolution [𝑓𝑓1, 𝑓𝑓2, 𝑓𝑓3,… 𝑓𝑓𝑝𝑝] [𝑔𝑔1, 𝑔𝑔2, 𝑔𝑔3,… 𝑔𝑔𝑞𝑞] (n x p) (n x q)(n x k)(k x k)(k x n) (Φ𝑘𝑘Γ𝑞𝑞,1Φ𝑘𝑘 𝑇𝑇 𝑓𝑓1)+ (Φ𝑘𝑘Γ𝑞𝑞,2Φ𝑘𝑘 𝑇𝑇 𝑓𝑓2)+ (Φ𝑘𝑘Γ𝑞𝑞,3Φ𝑘𝑘 𝑇𝑇 𝑓𝑓3)+ … +(Φ𝑘𝑘Γ𝑞𝑞,𝑝𝑝Φ𝑘𝑘 𝑇𝑇 𝑓𝑓𝑝𝑝) 𝑠𝑠𝑞𝑞 = 𝑔𝑔𝑞𝑞 = 𝜉𝜉( 𝑠𝑠𝑞𝑞)
- 35. Spectral Graph Theorem DMIS Presentation 35 Limitations of Spectral CNN Failure with different basis
- 36. Spectral Graph Theorem DMIS Presentation 36 Spectral Pooling (n x k)(k x k)(k x n) If is concentrated in the low-end of the spectrum, the corresponding spatial kernel function is smooth; conversely, if the corresponding spatial functions is localized, m is smooth. Therefore, to obtain a smoother kernel function, we constrain the bandwidth of m, enabling us to learn a smaller number of parameters; varying the smoothness of m would control the kernel size. Graph Coarsening Pooling & Strided Convolution
- 37. Spectral Graph Theorem DMIS Presentation 37 Spectral Pooling (n x k)(k x k)(k x n) Graph Coarsening Pooling & Strided Convolution Local filters at deeper layers in the spatial construction to be low frequency. Group structure does not commute with the Laplacian
- 38. Spectral Graph Theorem DMIS Presentation 38 Limitations of Spectral Pooling Graph does not have a group structure Group structure does not commute with the Laplacian Individual high frequency eigenvectors become highly unstable. By grouping high frequency eigenvectors in each octave, can obtain stable information (=Wavelet construction)
- 39. Reference DMIS Presentation 39 Convolution & Laplacian : https://www.khanacademy.org/math/differential-equations/laplace-transform/convolution-integral/v/the-convolution-and-the-laplace-transform Equivariance / Invariance : https://www.slideshare.net/ssuser06e0c5/brief-intro-invariance-and-equivariance Spectral Graph Theory : http://www.cs.yale.edu/homes/spielman/561/ Introduction to Spectral Graph Theory : https://www.youtube.com/watch?v=01AqmIU9Su4 SyncSpecCNN : https://arxiv.org/pdf/1612.00606.pdf Convergence of Laplacian Eigenmaps : http://papers.nips.cc/paper/2989-convergence-of-laplacian-eigenmaps.pdf Laplacian Mesh Processing : https://people.eecs.berkeley.edu/~jrs/meshpapers/Sorkine.pdf What is Fourier Transform? : https://www.youtube.com/watch?v=spUNpyF58BY Harmonic Analysis on Graphs and Networks: http://www.gretsi.fr/peyresq14/documents/1-Vandergheynst.pdf