Fabian Pedregosa, profile picture
Uploaded byFabian Pedregosa
PDF, PPTX33,566 views

Random Matrix Theory and Machine Learning - Part 1

This document provides an introduction to random matrix theory and its applications in machine learning. It discusses several classical random matrix ensembles like the Gaussian Orthogonal Ensemble (GOE) and Wishart ensemble. These ensembles are used to model phenomena in fields like number theory, physics, and machine learning. Specifically, the GOE is used to model Hamiltonians of heavy nuclei, while the Wishart ensemble relates to the Hessian of least squares problems. The tutorial will cover applications of random matrix theory to analyzing loss landscapes, numerical algorithms, and the generalization properties of machine learning models.

Embed presentation

Download as PDF, PPTX
Random Matrix Theory for Machine Learning Part 1: Motivating Questions and Building Blocks Fabian Pedregosa1 , Courtney Paquette1,2 , Tom Trogdon3 , Jeffrey Pennington1 1 Google Research , 2 McGill University , 3 University of Washington https://random-matrix-learning.github.io
About this tutorial Objective: • Applications of Random Matrix Theory (RMT) in Machine Learning. • Proof techniques in RMT 1
About this tutorial Objective: • Applications of Random Matrix Theory (RMT) in Machine Learning. • Proof techniques in RMT Structure: 1. Motivating Questions and Building Blocks, Fabian Pedregosa 2. Introduction to Random Matrix Theory, Courtney Paquette 3. Analysis of Numerical Algorithms, Tom Trogdon 4. The Mystery of Generalization: Why Does Deep Learning Work?, Jeffrey Pennington https://random-matrix-learning.github.io 1
What is a Random Matrix? A random matrix is a matrix whose entries are random variables, not necessarily independent. 2
What is a Random Matrix? A random matrix is a matrix whose entries are random variables, not necessarily independent. Example Realization of a random matrix: Z =        1.066 0.908 1.026 −0.294 0.879 0.908 −1.794 0.596 −1.014 −0.103 1.026 0.596 −0.246 0.968 0.750 −0.294 −1.014 0.968 0.184 0.812 0.879 −0.103 0.750 0.812 0.210        2
What is a Random Matrix? A random matrix is a matrix whose entries are random variables, not necessarily independent. Example Realization of a random matrix: Z =        1.066 0.908 1.026 −0.294 0.879 0.908 −1.794 0.596 −1.014 −0.103 1.026 0.596 −0.246 0.968 0.750 −0.294 −1.014 0.968 0.184 0.812 0.879 −0.103 0.750 0.812 0.210        Goal of Random Matrix Theory is to understand their • eigenvalues • eigenvectors • norms • singular values • singular vectors • … 2
Where do Random Matrices Come From?
1928: Eigenvalues of Normal Covariance Matrices John Wishart 0 2 eigenvalues 0.0 0.5 density 3
1928: Eigenvalues of Normal Covariance Matrices John Wishart 0 2 eigenvalues 0.0 0.5 density W = XX⊤ , Xij ∼ N(0, 1) p(λ1, . . . , λN) ∝ e− 1 2 ∑ i λi Y λ (n−p−1)/2 i Y i<j |λi − λj| 3
1955: Random Symmetric Matrices Eugene Wigner Energy levels of heavy nuclei, compared with the random matrix theory prediction. Source: [Rosenzweig and Porter, 1960] 4
Model for high-dimensional phenomena • Number Theory [Montgomery, 1973, Keating, 1993]. • Graph Theory [Erdos and Rényi, 1960]. • Finance [Bouchaud and Potters, 2009]. • Wireless communication [Tulino et al., 2004] • Machine Learning ... Distribution function of gaps between eigenvalues compared with histogram of gaps between ζ zeros. Source: [Odlyzko, 1987] 5
Random Matrices in Machine Learning: Loss Landscape Spin Glass model of the Loss Landscape Early: [Amit et al., 1985, Gardner and Derrida, 1988, Dotsenko, 1995] Late: [Dauphin et al., 2014, Sagun et al., 2014, Choromanska et al., 2015, Baity-Jesi et al., 2018] Loss study through spin-glass model. Scaled test losses for the spin-glass (left) and the neural network (right). Source: Choromanska et al. [2015] The Loss Surfaces of Multilayer Networks. 6
Random Matrices in Machine Learning: Loss Landscape New methods and software1,2,3 to compute Hessian eigenvalues of large models [Ghorbani et al., 2019, Yao et al., 2020, Papyan, 2020] 1 https://github.com/amirgholami/PyHessian 2 https://github.com/google/spectral-density/ 3 https://github.com/deep-lab/DeepnetHessian Source: [Papyan, 2020] 7
Random Matrices in Machine Learning: Loss Landscape New methods and software1,2,3 to compute Hessian eigenvalues of large models [Ghorbani et al., 2019, Yao et al., 2020, Papyan, 2020] 1 https://github.com/amirgholami/PyHessian 2 https://github.com/google/spectral-density/ 3 https://github.com/deep-lab/DeepnetHessian RMT model for the Hessian still an open problem [Liao and Mahoney, 2021, Baskerville et al., 2021] ... Source: [Papyan, 2020] 7
Random Matrices in Machine Learning: Numerical Algorithms Analyze algorithms with random data. • Simplex [Borgwardt, 1987, Smale, 1983, Spielman and Teng, 2004, Vershynin, 2009] etc. • Conjugate Gradient [Deift and Trogdon, 2017, Paquette and Trogdon, 2020] • Acceleration [Pedregosa and Scieur, 2020, Lacotte and Pilanci, 2020] 8
Random Matrices in Machine Learning: Numerical Algorithms Analyze algorithms with random data. • Simplex [Borgwardt, 1987, Smale, 1983, Spielman and Teng, 2004, Vershynin, 2009] etc. • Conjugate Gradient [Deift and Trogdon, 2017, Paquette and Trogdon, 2020] • Acceleration [Pedregosa and Scieur, 2020, Lacotte and Pilanci, 2020] Topic of Part 3 of this tutorial 8
Random Matrices in Machine Learning: Generalization As a model for generalization [Hastie et al., 2019, Mei and Montanari, 2019, Adlam and Pennington, 2020, Liao et al., 2020] Random Matrices can be used to model the double descent generalization curve. Source: [Mei and Montanari, 2019] The generalization error of random features regression: Precise asymptotics and double descent curve 9
Random Matrices in Machine Learning: Generalization As a model for generalization [Hastie et al., 2019, Mei and Montanari, 2019, Adlam and Pennington, 2020, Liao et al., 2020] Random Matrices can be used to model the double descent generalization curve. Source: [Mei and Montanari, 2019] The generalization error of random features regression: Precise asymptotics and double descent curve Part 4 of this tutorial 9
Building Blocks Classical Random Matrix Ensembles
Gaussian Orthogonal Ensemble (GOE) Motivation: Model Hamiltonian heavy nuclei 238 U 10
Gaussian Orthogonal Ensemble (GOE) Motivation: Model Hamiltonian heavy nuclei 238 U • Rotational invariant for any fixed orthogonal matrix O, A law = OT AO . 10
Gaussian Orthogonal Ensemble (GOE) Motivation: Model Hamiltonian heavy nuclei 238 U • Rotational invariant for any fixed orthogonal matrix O, A law = OT AO . • Symmetric matrix. 10
Gaussian Orthogonal Ensemble (GOE) Motivation: Model Hamiltonian heavy nuclei 238 U • Rotational invariant for any fixed orthogonal matrix O, A law = OT AO . • Symmetric matrix. • Independence Entries Aij, i ≤ j are independent. 10
Gaussian Orthogonal Ensemble (GOE) • real n × n matrix 1 √ n        a11 a12 a13 a13 a14 · · · a21 a22 a23 a24 a25 · · · a13 a32 a33 a34 a35 · · · a41 a42 a43 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
Gaussian Orthogonal Ensemble (GOE) • real n × n matrix • N(0, 1) above diagonal 1 √ n        a11 a12 a13 a13 a14 · · · a21 a22 a23 a24 a25 · · · a13 a32 a33 a34 a35 · · · a41 a42 a43 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
Gaussian Orthogonal Ensemble (GOE) • real n × n matrix • N(0, 1) above diagonal • Symmetric 1 √ n        a11 a12 a13 a13 a14 · · · a12 a22 a23 a24 a25 · · · a13 a23 a33 a34 a35 · · · a14 a24 a34 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
Gaussian Orthogonal Ensemble (GOE) • real n × n matrix • N(0, 1) above diagonal • Symmetric • N(0, 2) diagonal 1 √ n        a11 a12 a13 a13 a14 · · · a12 a22 a23 a24 a25 · · · a13 a23 a33 a34 a35 · · · a14 a24 a34 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
Eigenvalues of GOE Python pseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 0.4 density n=100 12
Eigenvalues of GOE Python pseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=500 12
Eigenvalues of GOE Python pseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=1000 12
Eigenvalues of GOE Python pseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=2000 12
Eigenvalues of GOE Python pseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=5000 12
Eigenvalues of GOE Python pseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=10000 12
Empirical Spectral Distribution (ESD) ESD of matrix An = p.d.f. of an eigenvalue chosen uniformly at random µESD = µn def = 1 n n X i=1 δλi(An). λ1 λ2 λ3 λ4 λ5 λ6 λ7 λ8 λ9 λ10 eigenvalues density 13
Wigner Semicircle Law µESD converges as n → ∞ to the semicircular distribution, µSC(x) def = 1 2π p (4 − x2)+ dx . 2 0 2 eigenvalues 0.0 0.2 density µSC Wigner semicircle To know more: [Tao, 2012, Bai and Silverstein, 2010]. 14
Wishart ensemble Wishart • X = random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite 15
Wishart ensemble Wishart • X = random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite • Same non-zero eigenvalues than 1 n XT X 15
Wishart ensemble Wishart • X = random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite • Same non-zero eigenvalues than 1 n XT X • 1 n XT X is Hessian of the least squares problem 1 2n ∥Xw − y∥2 15
Wishart ensemble Wishart • X = random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite • Same non-zero eigenvalues than 1 n XT X • 1 n XT X is Hessian of the least squares problem 1 2n ∥Xw − y∥2 • Parameter r = d n 15
Wishart ensemble Python pseudocode import numpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 1 2 eigenvalues 0.0 0.5 1.0 density n=100 16
Wishart ensemble Python pseudocode import numpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=500 16
Wishart ensemble Python pseudocode import numpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=1000 16
Wishart ensemble Python pseudocode import numpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=2000 16
Wishart ensemble Python pseudocode import numpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=5000 16
Wishart ensemble Python pseudocode import numpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=10000 16
Limit of Wishart matrices Marchenko-Pastur (MP) law [Marčenko and Pastur, 1967] As n, d → ∞, d n → r, µESD converges to the Marchenko-Pastur distribution: µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . with λ− = (1 − √ r)2 , λ+ = (1 + √ r)2 0 2 eigenvalues 0.0 0.5 density Marchenko-Pastur λ − λ + 17
The r = d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 1 2 eigenvalues 0.0 0.5 1.0 density λ − λ + r=0.2 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
The r = d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 2 eigenvalues 0.0 0.5 1.0 density λ − λ + r=0.5 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
The r = d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 2 eigenvalues 0 1 2 density λ − λ + r=0.9 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
The r = d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 2 4 eigenvalues 0 2 4 density λ − λ + r=1.0 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
The r = d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0.0 2.5 5.0 eigenvalues 0.0 0.1 0.2 0.3 density λ − λ + r=2.0 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
The r = d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 5 10 eigenvalues 0.00 0.05 0.10 density λ − λ + r=5.0 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
Universality Covariance matrices W = 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 19
Universality Covariance matrices W = 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 1.0 n = 10 19
Universality Covariance matrices W = 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 50 19
Universality Covariance matrices W = 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 100 19
Universality Covariance matrices W = 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 500 19
Universality Covariance matrices W = 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 1000 19
Universality Universality • Statistics only mildly depend on the lower order moments of distribution of the entries 20
Universality Universality • Statistics only mildly depend on the lower order moments of distribution of the entries Example: Marchenko-Pastur [Marčenko and Pastur, 1967] Let X be a d × n random matrix with i.i.d. entries that verifies E[Xij] = 0, E[X2 ij] = 1, E[X4 ij] < ∞ Universality: As n, d → ∞ with d n → r, the ESD of W = XXT n converges to Marchenko-Pastur(r) 20
May others ... Other matrix ensembles • Ginibre. Let Gn be n × n matrix of i.i.d. N(0, 1), (bilinear games [Domingo-Enrich et al., 2020]) (Circle law) ESD of Gn/ √ n → Unif(disk) . • Uniform probability measure on orthogonal matrices. V ∼ Unif(O(n)), ESD of V → Unif(S1 ) . 21
References i References Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning. PMLR, 2020. URL https://arxiv.org/pdf/2008.06786.pdf. Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Spin-glass models of neural networks. Physical Review A, 1985. URL https://doi.org/10.1103/PhysRevA.32.1007. Z. Bai and J. Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010. Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gérard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Wyart, and Giulio Biroli. Comparing dynamics: Deep neural networks versus glassy systems. In International Conference on Machine Learning. PMLR, 2018. URL https://arxiv.org/pdf/1803.06969.pdf. Nicholas P Baskerville, Diego Granziol, and Jonathan P Keating. Applicability of random matrix theory in deep learning. arXiv preprint arXiv:2102.06740, 2021. URL https://arxiv.org/pdf/2102.06740.pdf. K H Borgwardt. The simplex method: A probabilistic analysis. Springer–Verlag, Berlin, Heidelberg, 1987. URL https://www.springer.com/gp/book/9783540170969.
References ii Jean-Philippe Bouchaud and Marc Potters. Financial applications of random matrix theory: a short review. arXiv preprint arXiv:0910.1205, 2009. URL https://arxiv.org/pdf/0910.1205. Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics. PMLR, 2015. URL https://arxiv.org/pdf/1412.0233.pdf. Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv preprint arXiv:1406.2572, 2014. URL https://arxiv.org/pdf/1406.2572.pdf. P Deift and T Trogdon. Universality for Eigenvalue Algorithms on Sample Covariance Matrices. SIAM Journal on Numerical Analysis, 2017. URL http://arxiv.org/abs/1701.01896. Carles Domingo-Enrich, Fabian Pedregosa, and Damien Scieur. Average-case acceleration for bilinear games and normal matrices. arXiv preprint arXiv:2010.02076, 2020. Viktor Dotsenko. An introduction to the theory of spin glasses and neural networks. World Scientific, 1995. URL https://doi.org/10.1142/2460. Paul Erdos and Alfréd Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 1960. URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.348.530&rep=rep1&type=pdf. Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 1988. URL http://www.phys.ens.fr/~derrida/PAPIERS/1988/optimal-storage.pdf.
References iii Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning. PMLR, 2019. URL https://arxiv.org/pdf/1901.10159.pdf. Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560, 2019. URL https://arxiv.org/pdf/1903.08560.pdf. J Keating. The Riemann zeta-function and quantum chaology. In Quantum chaos. Elsevier, 1993. URL https://arxiv.org/pdf/0708.4223.pdf. Jonathan Lacotte and Mert Pilanci. Optimal randomized first-order methods for least-squares problems. In International Conference on Machine Learning. PMLR, 2020. URL https://arxiv.org/pdf/2002.09488.pdf. Z Liao and M W Mahoney. Hessian Eigenspectra of More Realistic Nonlinear Models. mar 2021. URL http://arxiv.org/abs/2103.01519. Zhenyu Liao, Romain Couillet, and Michael W Mahoney. A random matrix analysis of random fourier features: beyond the gaussian kernel, a precise phase transition, and the corresponding double descent. arXiv preprint arXiv:2006.05013, 2020. URL https://arxiv.org/pdf/2006.05013.pdf. Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1967. URL https://iopscience.iop.org/article/10.1070/SM1967v001n04ABEH001994/meta. Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.05355, 2019. URL https://arxiv.org/pdf/1908.05355.pdf.
References iv Hugh L Montgomery. The pair correlation of zeros of the zeta function. In Proc. Symp. Pure Math, 1973. URL http://www-personal.umich.edu/~hlm/paircor1.pdf. Andrew M Odlyzko. On the distribution of spacings between zeros of the zeta function. Mathematics of Computation, 1987. URL https://doi.org/10.1090/S0025-5718-1987-0866115-0. Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 2020. URL http://jmlr.org/papers/v21/20-933.html. Elliot Paquette and Thomas Trogdon. Universality for the conjugate gradient and minres algorithms on sample covariance matrices. arXiv preprint arXiv:2007.00640, 2020. URL https://arxiv.org/pdf/2007.00640.pdf. Fabian Pedregosa and Damien Scieur. Acceleration through spectral density estimation. In International Conference on Machine Learning. PMLR, 2020. URL https://arxiv.org/pdf/2002.04756.pdf. Norbert Rosenzweig and Charles E. Porter. ”repulsion of energy levels” in complex atomic spectra. Phys. Rev., 1960. URL https://link.aps.org/doi/10.1103/PhysRev.120.1698. Levent Sagun, V Ugur Guney, Gerard Ben Arous, and Yann LeCun. Explorations on high dimensional landscapes. arXiv preprint arXiv:1412.6615, 2014. URL https://arxiv.org/pdf/1412.6615.pdf. S Smale. On the average number of steps of the simplex method of linear programming. Mathematical Programming, 27(3): 241–262, oct 1983. ISSN 0025-5610. doi: 10.1007/BF02591902. URL http://link.springer.com/10.1007/BF02591902.
References v D A Spielman and S-H Teng. Smoothed analysis of algorithms. Journal of the ACM, 51(3):385–463, may 2004. ISSN 00045411. doi: 10.1145/990308.990310. URL http://dl.acm.org/citation.cfm?id=990308.990310. Terence Tao. Topics in random matrix theory. American Mathematical Soc., 2012. Antonia M Tulino, Sergio Verdú, and Sergio Verdu. Random matrix theory and wireless communications. Now Publishers Inc, 2004. URL http://dx.doi.org/10.1561/0100000001. R Vershynin. Beyond Hirsch Conjecture: Walks on Random Polytopes and Smoothed Complexity of the Simplex Method. SIAM Journal on Computing, 39, 2009. URL http://epubs.siam.org/doi/10.1137/070683386. Eugene Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics, 1955. URL https://doi.org/10.1007/978-3-662-02781-3_35. John Wishart. The generalised product moment distribution in samples from a normal multivariate population. Biometrika, 1928. URL https://doi.org/10.2307/2331939. Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020. URL https://github.com/amirgholami/PyHessian.

More Related Content

PDF
Regression lineaire simple
byMehdi Rajawi
 
PDF
COURS SUR LES SERIES TEMPORELLES
byAbdoulaziz ALHASSANE GARBA
 
PDF
生成對抗模式 GAN 的介紹
byYen-lung Tsai
 
PDF
Integral table for electomagnetic
byFathur Rozaq
 
PDF
5.5 Clustering
byBoris Guarisma
 
PPT
Clustering
bykaddinfoo
 
PDF
Integral table
bybags07
 
PDF
Théorie de graphe
byTECOS
 
Regression lineaire simple
byMehdi Rajawi
 
COURS SUR LES SERIES TEMPORELLES
byAbdoulaziz ALHASSANE GARBA
 
生成對抗模式 GAN 的介紹
byYen-lung Tsai
 
Integral table for electomagnetic
byFathur Rozaq
 
5.5 Clustering
byBoris Guarisma
 
Clustering
bykaddinfoo
 
Integral table
bybags07
 
Théorie de graphe
byTECOS
 

What's hot

PPTX
Présentation pfe
byAbdelghafour Zguindou
 
PPTX
Survey of Attention mechanism & Use in Computer Vision
bySwatiNarkhede1
 
PPTX
Neural network (perceptron)
byJeonghun Yoon
 
PDF
Calculus Cheat Sheet All
byMoe Han
 
PDF
Graphes
byHadi Youssef
 
PDF
آموزش مرتبه اجرایی در ساختمان داده و طراحی الگوریتم (مرور – تست کنکور ارشد)
byfaradars
 
PPT
2013 4-Algoritma Design and Analysis Recurrence.ppt
byEriekOrlando
 
PPT
introduction to Numerical Analysis
byGhulam Mehdi Sahito
 
PPTX
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
by홍배 김
 
PPTX
Cours de probabilités chap2.pptx
byHanaeElabbas
 
PDF
Properties of Exponents
bymlynczyk
 
PPTX
2. Fixed Point Iteration.pptx
bysaadhaq6
 
PDF
26 Computational Geometry
byAndres Mendez-Vazquez
 
PDF
4.Sampling and Hilbert Transform
byINDIAN NAVY
 
PDF
Analyse de données marketing : ACP et AFC
byYannig Roth
 
PDF
5.3 Régression logistique
byBoris Guarisma
 
PDF
Lecture123
bybenjoshpp
 
PPTX
Les 10 plus populaires algorithmes du machine learning
byHakim Nasaoui
 
PDF
Integrasi numerik metode riemann
bysttmc
 
PPTX
Aljabar Linear Bab 2 determinan
byHendro Agung Setiawan
 
Présentation pfe
byAbdelghafour Zguindou
 
Survey of Attention mechanism & Use in Computer Vision
bySwatiNarkhede1
 
Neural network (perceptron)
byJeonghun Yoon
 
Calculus Cheat Sheet All
byMoe Han
 
Graphes
byHadi Youssef
 
آموزش مرتبه اجرایی در ساختمان داده و طراحی الگوریتم (مرور – تست کنکور ارشد)
byfaradars
 
2013 4-Algoritma Design and Analysis Recurrence.ppt
byEriekOrlando
 
introduction to Numerical Analysis
byGhulam Mehdi Sahito
 
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
by홍배 김
 
Cours de probabilités chap2.pptx
byHanaeElabbas
 
Properties of Exponents
bymlynczyk
 
2. Fixed Point Iteration.pptx
bysaadhaq6
 
26 Computational Geometry
byAndres Mendez-Vazquez
 
4.Sampling and Hilbert Transform
byINDIAN NAVY
 
Analyse de données marketing : ACP et AFC
byYannig Roth
 
5.3 Régression logistique
byBoris Guarisma
 
Lecture123
bybenjoshpp
 
Les 10 plus populaires algorithmes du machine learning
byHakim Nasaoui
 
Integrasi numerik metode riemann
bysttmc
 
Aljabar Linear Bab 2 determinan
byHendro Agung Setiawan
 

Similar to Random Matrix Theory and Machine Learning - Part 1

PDF
nonlinear_rmt.pdf
byGieTe
 
PPTX
Unit-1 Introduction and Mathematical Preliminaries.pptx
byavinashBajpayee1
 
PDF
Random Matrix Theory and Machine Learning - Part 3
byFabian Pedregosa
 
PPTX
2.mathematics for machine learning
byKONGU ENGINEERING COLLEGE
 
PDF
monopoly grad talk
byMaxwell Gregoire
 
PDF
Machine Learning Module 2 Introduction.pdf
byumdanswers22
 
PDF
Some Equations for MAchine LEarning
byShridharSharma9
 
PPTX
Matrix PPT.pptx______________________________________________________________...
byVikrant168800
 
PDF
Basic math including gradient
byRamesh Kesavan
 
PPTX
Lecture_9_LA_Review.pptx
bySunny432360
 
PDF
Random Matrix Theory and Machine Learning - Part 2
byFabian Pedregosa
 
PDF
Refresher algebra-calculus
bySteve Nouri
 
PDF
Exercises_in_Machine_Learning_1657514028.pdf
byRaidTan
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
PPT
Machine Learning and Statistical Analysis
bybutest
 
nonlinear_rmt.pdf
byGieTe
 
Unit-1 Introduction and Mathematical Preliminaries.pptx
byavinashBajpayee1
 
Random Matrix Theory and Machine Learning - Part 3
byFabian Pedregosa
 
2.mathematics for machine learning
byKONGU ENGINEERING COLLEGE
 
monopoly grad talk
byMaxwell Gregoire
 
Machine Learning Module 2 Introduction.pdf
byumdanswers22
 
Some Equations for MAchine LEarning
byShridharSharma9
 
Matrix PPT.pptx______________________________________________________________...
byVikrant168800
 
Basic math including gradient
byRamesh Kesavan
 
Lecture_9_LA_Review.pptx
bySunny432360
 
Random Matrix Theory and Machine Learning - Part 2
byFabian Pedregosa
 
Refresher algebra-calculus
bySteve Nouri
 
Exercises_in_Machine_Learning_1657514028.pdf
byRaidTan
 
Machine Learning and Statistical Analysis
bybutest
 
Machine Learning and Statistical Analysis
bybutest
 
Machine Learning and Statistical Analysis
bybutest
 
Machine Learning and Statistical Analysis
bybutest
 
Machine Learning and Statistical Analysis
bybutest
 
Machine Learning and Statistical Analysis
bybutest
 
Machine Learning and Statistical Analysis
bybutest
 

More from Fabian Pedregosa

PDF
Random Matrix Theory and Machine Learning - Part 4
byFabian Pedregosa
 
PDF
Average case acceleration through spectral density estimation
byFabian Pedregosa
 
PDF
Adaptive Three Operator Splitting
byFabian Pedregosa
 
PDF
Sufficient decrease is all you need
byFabian Pedregosa
 
PDF
Asynchronous Stochastic Optimization, New Analysis and Algorithms
byFabian Pedregosa
 
PDF
Parallel Optimization in Machine Learning
byFabian Pedregosa
 
PDF
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Opti...
byFabian Pedregosa
 
PDF
Hyperparameter optimization with approximate gradient
byFabian Pedregosa
 
PDF
Lightning: large scale machine learning in python
byFabian Pedregosa
 
PDF
Profiling in Python
byFabian Pedregosa
 
Random Matrix Theory and Machine Learning - Part 4
byFabian Pedregosa
 
Average case acceleration through spectral density estimation
byFabian Pedregosa
 
Adaptive Three Operator Splitting
byFabian Pedregosa
 
Sufficient decrease is all you need
byFabian Pedregosa
 
Asynchronous Stochastic Optimization, New Analysis and Algorithms
byFabian Pedregosa
 
Parallel Optimization in Machine Learning
byFabian Pedregosa
 
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Opti...
byFabian Pedregosa
 
Hyperparameter optimization with approximate gradient
byFabian Pedregosa
 
Lightning: large scale machine learning in python
byFabian Pedregosa
 
Profiling in Python
byFabian Pedregosa
 

Recently uploaded

PDF
Machine Learning Lectures Explainable AI
byZahra Sadeghi
 
PDF
Rapid diagnosis and contamination of seafood and aquaculture products
byGirija90
 
PDF
S. Dharshini ( Msc. Microbiology) Microbial aspects and Metabolic aspects.
bydharshinisampath6
 
PPTX
Acute kidney injury asu teaching lectures
byfathyradwan1991
 
PPTX
Bioremediation anaerobic process, sulphur, nitrate, iron reduction, hydrocarb...
byNandhiniV90
 
PPTX
Celebrating Breakthroughs in Metal Organic Frameworks - A Poster Showcasing t...
byRunjhunDutta
 
PDF
f - Block Handwritten Notes PDF Download - Irfanullah Mehar - Hindi Medium.pdf
byWorld of Wisdom
 
PDF
Machine Learning Lectures - Loss Functions (1)
byZahra Sadeghi
 
PDF
Bioremedation - Dechlorination ,Types and process , advantage and disadvantage
byswathijagadeesan1402
 
PDF
PHYTOREMEDIATION OF HEAVY METALS IN SOIL
byswethaviji2005
 
PDF
Machine Learning Lectures -- Loss Functions (2)
byZahra Sadeghi
 
PDF
Machine Learning Lectures --- Likelihood
byZahra Sadeghi
 
PPTX
Forensic-Ballistics, identification of Firearms
bysoulquimicaanalitica
 
PPTX
SCIENCE 4 Q4W3 PPT Characteristic of Weather
byRuvelynRemitre1
 
PDF
Periodic Table ( आवर्त सारणी ) Chemistry Notes PDF @irfanullah_mehar @world_o...
byWorld of Wisdom
 
PDF
Machine Learning Lectures -- Support Vector Machine (SVM)
byZahra Sadeghi
 
PDF
UNIT I Stereoisomerism (Pharmaceutical Organic Chemistry III)
byssjsphs
 
PDF
BIOREMEDIATION -MICROBIAL LEACHING OF ORES AND ITS PROCESS
byswethaviji2005
 
PPTX
Water treatment BOD and COD , dissolved gases
byNandhiniV90
 
PDF
谷歌留痕技术 [ 𝙩𝙤𝙥 𝟮𝟯𝟯. 𝙘 𝙤𝙢 ]
by6stynggfo
 
Machine Learning Lectures Explainable AI
byZahra Sadeghi
 
Rapid diagnosis and contamination of seafood and aquaculture products
byGirija90
 
S. Dharshini ( Msc. Microbiology) Microbial aspects and Metabolic aspects.
bydharshinisampath6
 
Acute kidney injury asu teaching lectures
byfathyradwan1991
 
Bioremediation anaerobic process, sulphur, nitrate, iron reduction, hydrocarb...
byNandhiniV90
 
Celebrating Breakthroughs in Metal Organic Frameworks - A Poster Showcasing t...
byRunjhunDutta
 
f - Block Handwritten Notes PDF Download - Irfanullah Mehar - Hindi Medium.pdf
byWorld of Wisdom
 
Machine Learning Lectures - Loss Functions (1)
byZahra Sadeghi
 
Bioremedation - Dechlorination ,Types and process , advantage and disadvantage
byswathijagadeesan1402
 
PHYTOREMEDIATION OF HEAVY METALS IN SOIL
byswethaviji2005
 
Machine Learning Lectures -- Loss Functions (2)
byZahra Sadeghi
 
Machine Learning Lectures --- Likelihood
byZahra Sadeghi
 
Forensic-Ballistics, identification of Firearms
bysoulquimicaanalitica
 
SCIENCE 4 Q4W3 PPT Characteristic of Weather
byRuvelynRemitre1
 
Periodic Table ( आवर्त सारणी ) Chemistry Notes PDF @irfanullah_mehar @world_o...
byWorld of Wisdom
 
Machine Learning Lectures -- Support Vector Machine (SVM)
byZahra Sadeghi
 
UNIT I Stereoisomerism (Pharmaceutical Organic Chemistry III)
byssjsphs
 
BIOREMEDIATION -MICROBIAL LEACHING OF ORES AND ITS PROCESS
byswethaviji2005
 
Water treatment BOD and COD , dissolved gases
byNandhiniV90
 
谷歌留痕技术 [ 𝙩𝙤𝙥 𝟮𝟯𝟯. 𝙘 𝙤𝙢 ]
by6stynggfo
 

Random Matrix Theory and Machine Learning - Part 1

  • 1.
    Random Matrix Theoryfor Machine Learning Part 1: Motivating Questions and Building Blocks Fabian Pedregosa1 , Courtney Paquette1,2 , Tom Trogdon3 , Jeffrey Pennington1 1 Google Research , 2 McGill University , 3 University of Washington https://random-matrix-learning.github.io
  • 2.
    About this tutorial Objective: •Applications of Random Matrix Theory (RMT) in Machine Learning. • Proof techniques in RMT 1
  • 3.
    About this tutorial Objective: •Applications of Random Matrix Theory (RMT) in Machine Learning. • Proof techniques in RMT Structure: 1. Motivating Questions and Building Blocks, Fabian Pedregosa 2. Introduction to Random Matrix Theory, Courtney Paquette 3. Analysis of Numerical Algorithms, Tom Trogdon 4. The Mystery of Generalization: Why Does Deep Learning Work?, Jeffrey Pennington https://random-matrix-learning.github.io 1
  • 4.
    What is aRandom Matrix? A random matrix is a matrix whose entries are random variables, not necessarily independent. 2
  • 5.
    What is aRandom Matrix? A random matrix is a matrix whose entries are random variables, not necessarily independent. Example Realization of a random matrix: Z =        1.066 0.908 1.026 −0.294 0.879 0.908 −1.794 0.596 −1.014 −0.103 1.026 0.596 −0.246 0.968 0.750 −0.294 −1.014 0.968 0.184 0.812 0.879 −0.103 0.750 0.812 0.210        2
  • 6.
    What is aRandom Matrix? A random matrix is a matrix whose entries are random variables, not necessarily independent. Example Realization of a random matrix: Z =        1.066 0.908 1.026 −0.294 0.879 0.908 −1.794 0.596 −1.014 −0.103 1.026 0.596 −0.246 0.968 0.750 −0.294 −1.014 0.968 0.184 0.812 0.879 −0.103 0.750 0.812 0.210        Goal of Random Matrix Theory is to understand their • eigenvalues • eigenvectors • norms • singular values • singular vectors • … 2
  • 7.
    Where do RandomMatrices Come From?
  • 8.
    1928: Eigenvalues ofNormal Covariance Matrices John Wishart 0 2 eigenvalues 0.0 0.5 density 3
  • 9.
    1928: Eigenvalues ofNormal Covariance Matrices John Wishart 0 2 eigenvalues 0.0 0.5 density W = XX⊤ , Xij ∼ N(0, 1) p(λ1, . . . , λN) ∝ e− 1 2 ∑ i λi Y λ (n−p−1)/2 i Y i<j |λi − λj| 3
  • 10.
    1955: Random SymmetricMatrices Eugene Wigner Energy levels of heavy nuclei, compared with the random matrix theory prediction. Source: [Rosenzweig and Porter, 1960] 4
  • 11.
    Model for high-dimensionalphenomena • Number Theory [Montgomery, 1973, Keating, 1993]. • Graph Theory [Erdos and Rényi, 1960]. • Finance [Bouchaud and Potters, 2009]. • Wireless communication [Tulino et al., 2004] • Machine Learning ... Distribution function of gaps between eigenvalues compared with histogram of gaps between ζ zeros. Source: [Odlyzko, 1987] 5
  • 12.
    Random Matrices inMachine Learning: Loss Landscape Spin Glass model of the Loss Landscape Early: [Amit et al., 1985, Gardner and Derrida, 1988, Dotsenko, 1995] Late: [Dauphin et al., 2014, Sagun et al., 2014, Choromanska et al., 2015, Baity-Jesi et al., 2018] Loss study through spin-glass model. Scaled test losses for the spin-glass (left) and the neural network (right). Source: Choromanska et al. [2015] The Loss Surfaces of Multilayer Networks. 6
  • 13.
    Random Matrices inMachine Learning: Loss Landscape New methods and software1,2,3 to compute Hessian eigenvalues of large models [Ghorbani et al., 2019, Yao et al., 2020, Papyan, 2020] 1 https://github.com/amirgholami/PyHessian 2 https://github.com/google/spectral-density/ 3 https://github.com/deep-lab/DeepnetHessian Source: [Papyan, 2020] 7
  • 14.
    Random Matrices inMachine Learning: Loss Landscape New methods and software1,2,3 to compute Hessian eigenvalues of large models [Ghorbani et al., 2019, Yao et al., 2020, Papyan, 2020] 1 https://github.com/amirgholami/PyHessian 2 https://github.com/google/spectral-density/ 3 https://github.com/deep-lab/DeepnetHessian RMT model for the Hessian still an open problem [Liao and Mahoney, 2021, Baskerville et al., 2021] ... Source: [Papyan, 2020] 7
  • 15.
    Random Matrices inMachine Learning: Numerical Algorithms Analyze algorithms with random data. • Simplex [Borgwardt, 1987, Smale, 1983, Spielman and Teng, 2004, Vershynin, 2009] etc. • Conjugate Gradient [Deift and Trogdon, 2017, Paquette and Trogdon, 2020] • Acceleration [Pedregosa and Scieur, 2020, Lacotte and Pilanci, 2020] 8
  • 16.
    Random Matrices inMachine Learning: Numerical Algorithms Analyze algorithms with random data. • Simplex [Borgwardt, 1987, Smale, 1983, Spielman and Teng, 2004, Vershynin, 2009] etc. • Conjugate Gradient [Deift and Trogdon, 2017, Paquette and Trogdon, 2020] • Acceleration [Pedregosa and Scieur, 2020, Lacotte and Pilanci, 2020] Topic of Part 3 of this tutorial 8
  • 17.
    Random Matrices inMachine Learning: Generalization As a model for generalization [Hastie et al., 2019, Mei and Montanari, 2019, Adlam and Pennington, 2020, Liao et al., 2020] Random Matrices can be used to model the double descent generalization curve. Source: [Mei and Montanari, 2019] The generalization error of random features regression: Precise asymptotics and double descent curve 9
  • 18.
    Random Matrices inMachine Learning: Generalization As a model for generalization [Hastie et al., 2019, Mei and Montanari, 2019, Adlam and Pennington, 2020, Liao et al., 2020] Random Matrices can be used to model the double descent generalization curve. Source: [Mei and Montanari, 2019] The generalization error of random features regression: Precise asymptotics and double descent curve Part 4 of this tutorial 9
  • 19.
  • 20.
    Gaussian Orthogonal Ensemble(GOE) Motivation: Model Hamiltonian heavy nuclei 238 U 10
  • 21.
    Gaussian Orthogonal Ensemble(GOE) Motivation: Model Hamiltonian heavy nuclei 238 U • Rotational invariant for any fixed orthogonal matrix O, A law = OT AO . 10
  • 22.
    Gaussian Orthogonal Ensemble(GOE) Motivation: Model Hamiltonian heavy nuclei 238 U • Rotational invariant for any fixed orthogonal matrix O, A law = OT AO . • Symmetric matrix. 10
  • 23.
    Gaussian Orthogonal Ensemble(GOE) Motivation: Model Hamiltonian heavy nuclei 238 U • Rotational invariant for any fixed orthogonal matrix O, A law = OT AO . • Symmetric matrix. • Independence Entries Aij, i ≤ j are independent. 10
  • 24.
    Gaussian Orthogonal Ensemble(GOE) • real n × n matrix 1 √ n        a11 a12 a13 a13 a14 · · · a21 a22 a23 a24 a25 · · · a13 a32 a33 a34 a35 · · · a41 a42 a43 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
  • 25.
    Gaussian Orthogonal Ensemble(GOE) • real n × n matrix • N(0, 1) above diagonal 1 √ n        a11 a12 a13 a13 a14 · · · a21 a22 a23 a24 a25 · · · a13 a32 a33 a34 a35 · · · a41 a42 a43 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
  • 26.
    Gaussian Orthogonal Ensemble(GOE) • real n × n matrix • N(0, 1) above diagonal • Symmetric 1 √ n        a11 a12 a13 a13 a14 · · · a12 a22 a23 a24 a25 · · · a13 a23 a33 a34 a35 · · · a14 a24 a34 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
  • 27.
    Gaussian Orthogonal Ensemble(GOE) • real n × n matrix • N(0, 1) above diagonal • Symmetric • N(0, 2) diagonal 1 √ n        a11 a12 a13 a13 a14 · · · a12 a22 a23 a24 a25 · · · a13 a23 a33 a34 a35 · · · a14 a24 a34 a44 a45 · · · · · · · · · · · · · · · · · · · · ·        11
  • 28.
    Eigenvalues of GOE Pythonpseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 0.4 density n=100 12
  • 29.
    Eigenvalues of GOE Pythonpseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=500 12
  • 30.
    Eigenvalues of GOE Pythonpseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=1000 12
  • 31.
    Eigenvalues of GOE Pythonpseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=2000 12
  • 32.
    Eigenvalues of GOE Pythonpseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=5000 12
  • 33.
    Eigenvalues of GOE Pythonpseudocode import numpy as np import matplotlib.pyplot as plt A = np.random.randn(n, n) GOE = (A+A.T)/np.sqrt(2*n) eig = np.linalg.eigvals(GOE) plt.hist(eig) 2 0 2 eigenvalues 0.0 0.2 density n=10000 12
  • 34.
    Empirical Spectral Distribution(ESD) ESD of matrix An = p.d.f. of an eigenvalue chosen uniformly at random µESD = µn def = 1 n n X i=1 δλi(An). λ1 λ2 λ3 λ4 λ5 λ6 λ7 λ8 λ9 λ10 eigenvalues density 13
  • 35.
    Wigner Semicircle Law µESDconverges as n → ∞ to the semicircular distribution, µSC(x) def = 1 2π p (4 − x2)+ dx . 2 0 2 eigenvalues 0.0 0.2 density µSC Wigner semicircle To know more: [Tao, 2012, Bai and Silverstein, 2010]. 14
  • 36.
    Wishart ensemble Wishart • X= random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite 15
  • 37.
    Wishart ensemble Wishart • X= random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite • Same non-zero eigenvalues than 1 n XT X 15
  • 38.
    Wishart ensemble Wishart • X= random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite • Same non-zero eigenvalues than 1 n XT X • 1 n XT X is Hessian of the least squares problem 1 2n ∥Xw − y∥2 15
  • 39.
    Wishart ensemble Wishart • X= random (d × n) matrix with entries i.i.d. N(0, 1) • Wishart (d × d) matrix, W = XXT n Remarks • W is symmetric, positive semi-definite • Same non-zero eigenvalues than 1 n XT X • 1 n XT X is Hessian of the least squares problem 1 2n ∥Xw − y∥2 • Parameter r = d n 15
  • 40.
    Wishart ensemble Python pseudocode importnumpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 1 2 eigenvalues 0.0 0.5 1.0 density n=100 16
  • 41.
    Wishart ensemble Python pseudocode importnumpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=500 16
  • 42.
    Wishart ensemble Python pseudocode importnumpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=1000 16
  • 43.
    Wishart ensemble Python pseudocode importnumpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=2000 16
  • 44.
    Wishart ensemble Python pseudocode importnumpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=5000 16
  • 45.
    Wishart ensemble Python pseudocode importnumpy as np import matplotlib.pyplot as plt r = 1/2 # for example X = np.random.randn(n * r, n) W = np.dot(X, X.T) / n eig = np.linalg.eigvals(W) plt.hist(eig) 0 2 eigenvalues 0.0 0.5 density n=10000 16
  • 46.
    Limit of Wishartmatrices Marchenko-Pastur (MP) law [Marčenko and Pastur, 1967] As n, d → ∞, d n → r, µESD converges to the Marchenko-Pastur distribution: µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . with λ− = (1 − √ r)2 , λ+ = (1 + √ r)2 0 2 eigenvalues 0.0 0.5 density Marchenko-Pastur λ − λ + 17
  • 47.
    The r =d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 1 2 eigenvalues 0.0 0.5 1.0 density λ − λ + r=0.2 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
  • 48.
    The r =d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 2 eigenvalues 0.0 0.5 1.0 density λ − λ + r=0.5 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
  • 49.
    The r =d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 2 eigenvalues 0 1 2 density λ − λ + r=0.9 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
  • 50.
    The r =d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 2 4 eigenvalues 0 2 4 density λ − λ + r=1.0 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
  • 51.
    The r =d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0.0 2.5 5.0 eigenvalues 0.0 0.1 0.2 0.3 density λ − λ + r=2.0 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
  • 52.
    The r =d n parameter • r < 1 =⇒ d < n =⇒ W is product of two fat matrices. • r > 1 =⇒ d > n =⇒ W is product of two thin matrices (rank-deficient). 0 5 10 eigenvalues 0.00 0.05 0.10 density λ − λ + r=5.0 µMP(x) def = (1 − 1 r )+δ0(x) | {z } nonzero if r > 1 + p (λ+ − x)(x − λ−) 2πr x 1x∈[λ−,λ+] dx . 18
  • 53.
    Universality Covariance matrices W= 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 19
  • 54.
    Universality Covariance matrices W= 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 1.0 n = 10 19
  • 55.
    Universality Covariance matrices W= 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 50 19
  • 56.
    Universality Covariance matrices W= 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 100 19
  • 57.
    Universality Covariance matrices W= 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 500 19
  • 58.
    Universality Covariance matrices W= 1 n XX⊤ What happens if we replace the N(0, 1) with a different distribution? • Xij ∼ N(0, 1) • Xij ∼ Rademacher Pr(Xij = −1) = Pr(Xij = 1) = 1 2 0 1 2 3 0.0 0.5 n = 1000 19
  • 59.
    Universality Universality • Statistics onlymildly depend on the lower order moments of distribution of the entries 20
  • 60.
    Universality Universality • Statistics onlymildly depend on the lower order moments of distribution of the entries Example: Marchenko-Pastur [Marčenko and Pastur, 1967] Let X be a d × n random matrix with i.i.d. entries that verifies E[Xij] = 0, E[X2 ij] = 1, E[X4 ij] < ∞ Universality: As n, d → ∞ with d n → r, the ESD of W = XXT n converges to Marchenko-Pastur(r) 20
  • 61.
    May others ... Othermatrix ensembles • Ginibre. Let Gn be n × n matrix of i.i.d. N(0, 1), (bilinear games [Domingo-Enrich et al., 2020]) (Circle law) ESD of Gn/ √ n → Unif(disk) . • Uniform probability measure on orthogonal matrices. V ∼ Unif(O(n)), ESD of V → Unif(S1 ) . 21
  • 62.
    References i References Ben Adlamand Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning. PMLR, 2020. URL https://arxiv.org/pdf/2008.06786.pdf. Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Spin-glass models of neural networks. Physical Review A, 1985. URL https://doi.org/10.1103/PhysRevA.32.1007. Z. Bai and J. Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010. Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gérard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Wyart, and Giulio Biroli. Comparing dynamics: Deep neural networks versus glassy systems. In International Conference on Machine Learning. PMLR, 2018. URL https://arxiv.org/pdf/1803.06969.pdf. Nicholas P Baskerville, Diego Granziol, and Jonathan P Keating. Applicability of random matrix theory in deep learning. arXiv preprint arXiv:2102.06740, 2021. URL https://arxiv.org/pdf/2102.06740.pdf. K H Borgwardt. The simplex method: A probabilistic analysis. Springer–Verlag, Berlin, Heidelberg, 1987. URL https://www.springer.com/gp/book/9783540170969.
  • 63.
    References ii Jean-Philippe Bouchaudand Marc Potters. Financial applications of random matrix theory: a short review. arXiv preprint arXiv:0910.1205, 2009. URL https://arxiv.org/pdf/0910.1205. Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics. PMLR, 2015. URL https://arxiv.org/pdf/1412.0233.pdf. Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv preprint arXiv:1406.2572, 2014. URL https://arxiv.org/pdf/1406.2572.pdf. P Deift and T Trogdon. Universality for Eigenvalue Algorithms on Sample Covariance Matrices. SIAM Journal on Numerical Analysis, 2017. URL http://arxiv.org/abs/1701.01896. Carles Domingo-Enrich, Fabian Pedregosa, and Damien Scieur. Average-case acceleration for bilinear games and normal matrices. arXiv preprint arXiv:2010.02076, 2020. Viktor Dotsenko. An introduction to the theory of spin glasses and neural networks. World Scientific, 1995. URL https://doi.org/10.1142/2460. Paul Erdos and Alfréd Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 1960. URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.348.530&rep=rep1&type=pdf. Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 1988. URL http://www.phys.ens.fr/~derrida/PAPIERS/1988/optimal-storage.pdf.
  • 64.
    References iii Behrooz Ghorbani,Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning. PMLR, 2019. URL https://arxiv.org/pdf/1901.10159.pdf. Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560, 2019. URL https://arxiv.org/pdf/1903.08560.pdf. J Keating. The Riemann zeta-function and quantum chaology. In Quantum chaos. Elsevier, 1993. URL https://arxiv.org/pdf/0708.4223.pdf. Jonathan Lacotte and Mert Pilanci. Optimal randomized first-order methods for least-squares problems. In International Conference on Machine Learning. PMLR, 2020. URL https://arxiv.org/pdf/2002.09488.pdf. Z Liao and M W Mahoney. Hessian Eigenspectra of More Realistic Nonlinear Models. mar 2021. URL http://arxiv.org/abs/2103.01519. Zhenyu Liao, Romain Couillet, and Michael W Mahoney. A random matrix analysis of random fourier features: beyond the gaussian kernel, a precise phase transition, and the corresponding double descent. arXiv preprint arXiv:2006.05013, 2020. URL https://arxiv.org/pdf/2006.05013.pdf. Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1967. URL https://iopscience.iop.org/article/10.1070/SM1967v001n04ABEH001994/meta. Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.05355, 2019. URL https://arxiv.org/pdf/1908.05355.pdf.
  • 65.
    References iv Hugh LMontgomery. The pair correlation of zeros of the zeta function. In Proc. Symp. Pure Math, 1973. URL http://www-personal.umich.edu/~hlm/paircor1.pdf. Andrew M Odlyzko. On the distribution of spacings between zeros of the zeta function. Mathematics of Computation, 1987. URL https://doi.org/10.1090/S0025-5718-1987-0866115-0. Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 2020. URL http://jmlr.org/papers/v21/20-933.html. Elliot Paquette and Thomas Trogdon. Universality for the conjugate gradient and minres algorithms on sample covariance matrices. arXiv preprint arXiv:2007.00640, 2020. URL https://arxiv.org/pdf/2007.00640.pdf. Fabian Pedregosa and Damien Scieur. Acceleration through spectral density estimation. In International Conference on Machine Learning. PMLR, 2020. URL https://arxiv.org/pdf/2002.04756.pdf. Norbert Rosenzweig and Charles E. Porter. ”repulsion of energy levels” in complex atomic spectra. Phys. Rev., 1960. URL https://link.aps.org/doi/10.1103/PhysRev.120.1698. Levent Sagun, V Ugur Guney, Gerard Ben Arous, and Yann LeCun. Explorations on high dimensional landscapes. arXiv preprint arXiv:1412.6615, 2014. URL https://arxiv.org/pdf/1412.6615.pdf. S Smale. On the average number of steps of the simplex method of linear programming. Mathematical Programming, 27(3): 241–262, oct 1983. ISSN 0025-5610. doi: 10.1007/BF02591902. URL http://link.springer.com/10.1007/BF02591902.
  • 66.
    References v D ASpielman and S-H Teng. Smoothed analysis of algorithms. Journal of the ACM, 51(3):385–463, may 2004. ISSN 00045411. doi: 10.1145/990308.990310. URL http://dl.acm.org/citation.cfm?id=990308.990310. Terence Tao. Topics in random matrix theory. American Mathematical Soc., 2012. Antonia M Tulino, Sergio Verdú, and Sergio Verdu. Random matrix theory and wireless communications. Now Publishers Inc, 2004. URL http://dx.doi.org/10.1561/0100000001. R Vershynin. Beyond Hirsch Conjecture: Walks on Random Polytopes and Smoothed Complexity of the Simplex Method. SIAM Journal on Computing, 39, 2009. URL http://epubs.siam.org/doi/10.1137/070683386. Eugene Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics, 1955. URL https://doi.org/10.1007/978-3-662-02781-3_35. John Wishart. The generalised product moment distribution in samples from a normal multivariate population. Biometrika, 1928. URL https://doi.org/10.2307/2331939. Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020. URL https://github.com/amirgholami/PyHessian.