Recommended
Recommended
More Related Content
Similar to High-dimensional dynamics of generalization error in neural networks (Explained)
Similar to High-dimensional dynamics of generalization error in neural networks (Explained) (20)
Recently uploaded
Recently uploaded (20)
High-dimensional dynamics of generalization error in neural networks (Explained)
- 1. High-dimensional dynamics of generalization error in neural networks Hikaru Ibayashi
- 2. Background Deep Neural Network’s Mystery
- 3. Background Deep Neural Network’s Mystery ● Huge amount of parameters ○ Ex.) VGG net has 155 million parameters ● Train data size can be small ○ Ex.) ImageNet has 1.2 million samples ● Still shows high generalization performance
- 4. Background Deep Neural Network’s Mystery ● Huge amount of parameters ○ Ex.) VGG net has 155 million parameters ● Train data size can be small ○ Ex.) ImageNet has 1.2 million samples ● Still shows high generalization performance
- 5. Background Deep Neural Network’s Mystery ● Huge amount of parameters ○ Ex.) VGG net has 155 million parameters ● Train data size can be small ○ Ex.) ImageNet has 1.2 million samples ● Still shows high generalization performance
- 6. Background Chiyuan Zhang et al. Empirically Raised this Mystery in 2017
- 9. Background Labels Shuffled Labels Can Fit Perfectly
- 10. Background Labels Shuffled Labels Can Fit Perfectly Can Fit Perfectly
- 11. Background Labels Shuffled Labels Shuffled Labels Can Fit Perfectly Can Fit Perfectly
- 12. Background Labels Shuffled Labels Shuffled Labels Can Fit Perfectly Can Fit Perfectly Can Fit Perfectly
- 13. Background Labels Shuffled Labels Shuffled Labels Can Fit Perfectly Can Fit Perfectly Can Fit Perfectly & Generalize Well
- 14. Today’s Paper
- 16. Teacher-Student Learning Dynamics Teacher-Student Scenario
- 18. Teacher-Student Learning Dynamics Train DataTrain Data Teacher Net
- 19. Hypothesis Teacher-Student Learning Dynamics Train DataTrain Data Student Net Teacher Net
- 20. Teacher Net Hypothesis follows Teacher-Student Learning Dynamics Train DataTrain Data Student Net
- 21. Hypothesis follows with initial value Teacher-Student Learning Dynamics Train DataTrain Data Student Net Teacher Net
- 22. After simple rotation Teacher-Student Learning Dynamics
- 23. After simple rotation Teacher-Student Learning Dynamics
- 24. After simple rotation Teacher-Student Learning Dynamics
- 25. After simple rotation Test error Teacher-Student Learning Dynamics
- 26. After simple rotation Test error Noise Teacher-Student Learning Dynamics
- 27. After simple rotation Test error Learning Noise Teacher-Student Learning Dynamics
- 28. After simple rotation Test error Learning Overfitting Noise Teacher-Student Learning Dynamics
- 29. How to take expectation over ? Teacher-Student Learning Dynamics
- 30. How to take expectation over ? Teacher-Student Learning Dynamics Eigen values of where
- 31. How to take expectation over ? Teacher-Student Learning Dynamics Marchenko–Pastur distribution from random matrix theory Eigen values of where
- 32. Teacher-Student Learning Dynamics Final result
- 33. Teacher-Student Learning Dynamics (Result) Eigenvalue Spectrum Learning Dynamics
- 34. Teacher-Student Learning Dynamics (Result) Eigenvalue Spectrum Learning Dynamics
- 35. Teacher-Student Learning Dynamics (Result) Eigenvalue Spectrum Learning Dynamics
- 36. Teacher-Student Learning Dynamics (Result) Eigenvalue Spectrum Learning Dynamics
- 37. Teacher-Student Learning Dynamics (Result) Eigenvalue Spectrum Learning Dynamics
- 38. Teacher-Student Learning Dynamics (Result)
- 39. Teacher-Student Learning Dynamics (Result) Optimal stopping time
- 40. Teacher-Student Learning Dynamics (Result) Optimal stopping time Signal noise ratio
- 41. Teacher-Student Learning Dynamics (Result) Optimal stopping time Signal noise ratio Low quality data requires longer training time
- 42. Is Single-Layer-Linear model Reasonable?
- 43. Is Single-Layer-Linear model Reasonable? Linear Single-Layer Teacher-Student
- 44. Is Single-Layer-Linear model Reasonable? Linear Single-Layer Teacher-Student Nonlinear Two-Layers Teacher-Student
- 45. Is Single-Layer-Linear model Reasonable? Linear Single-Layer Teacher-Student Nonlinear Two-Layers Teacher-Student Nonlinear Two-Layers Real Data
- 46. Is Single-Layer-Linear model Reasonable? One Difference
- 47. Is Single-Layer-Linear model Reasonable? One Difference
- 48. Non-Linear Teacher Student (Result) Learning Dynamics
- 49. Non-Linear Teacher Student (Result) Learning Dynamics
- 50. Non-Linear Teacher Student (Result) Test Error vs Hidden units (Result of Linear case)
- 51. Non-Linear MNIST (Result) Test Error vs Hidden units
- 52. Non-Linear MNIST (Result) Overfitting Error vs Hidden units
- 53. Discussion What does this paper imply?
- 54. What does this paper imply? With early stopping, Larger model →Better generalization Discussion
- 55. What does this paper imply? With early stopping, Larger model →Better generalization Deep Learning’s Mystery!! Discussion
- 56. What does this paper imply? With early stopping, Larger model →Better generalization How can this paper explain? Deep Learning’s Mystery!! Discussion
- 57. What does this paper imply? With early stopping, Larger model →Better generalization How can this paper explain? Larger model → Bigger eigenvalues (Marchenko-Pasteur distribution) Deep Learning’s Mystery!! Discussion
- 58. What does this paper imply? With early stopping, Larger model →Better generalization How can this paper explain? Larger model → Bigger eigenvalues (Marchenko-Pasteur distribution) Deep Learning’s Mystery!! Discussion
- 59. What does this paper imply? With early stopping, Larger model →Better generalization How can this paper explain? Larger model → Bigger eigenvalues (Marchenko-Pasteur distribution) Deep Learning’s Mystery!! Discussion