This slide covers basic principles of optimization and summarizes some optimization algorithms. it includes SGD, Momentum, Nesterov momentum, AdaGrad, RMSProp, AdadDelta, RMSProp \w Nestrov Momentum and Adam. presentation: https://www.youtube.com/watch?v=KdOmIODy3eA Reference: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016

1. Deep Learning
Optimization
Rookie’s Seminar, Jan 2018
Jaehyun Jun
Biointelligence Laboratory
Interdisciplinary Program of Neuro Science, Seoul National Univertisy
http://bi.snu.ac.kr

2. Contents
8.1 Pure Optimization
8.2 Challenges
8.3 Basic Algorithms
SGD, Momentum, Nesterov momentum
8.5 Adaptive Learning Rates
AdaGrad, RMSProp, AdaDelta,
RMS Prop w Nesterov momentum, Adam
8.4 & 8.7 Strategies
Reference: Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016
© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr 2

3. Pure Optimization
 Objective: minimizing J is a goal in and of itself.
 We don't know pdata (intractable)
-> we use empirical risk minimization
-> be prone to overfitting
© 2017, SNU Biointelligence Lab., http://bi.snu.ac.kr 3

4. Differs from Pure Optimization
 Types of optimization algorithm
 batch (deterministic) algorithm: use the entire training set
 stochastic (online) algorithm: use a single example
 minibatch (minibatch stochastic) algorithm: use more than one but
less than all
 select randomly -> prevent biased
4© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

5. Challenges
 Ill-Conditioning
 Second-order Tayler series expansion
 Local minima
5© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

6. Challenges
 Cliff and exploding gradients
 Long-term dependencies
-> vanishing and exploding gradient problem
 Poor correspondence between local and global structure
 the gradient of local minima cannot reach a global solution.
6© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

7. Basic Algorithms
 Stochastic Gradient Descent (SGD)
 Obtain an unbiased estimate of the gradient by taking the average
gradient on a mini batch of m examples drawn i.i.d from the data
generating distribution
7© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

8. Basic Algorithms
 Momentum
 The method of momentum is designed to
solve poor conditioning of the Hessian matrix
and variance in SGD
8© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

9. Basic Algorithms
 Nesterov momentum
 Difference is that with Nesterov momentum the gradient is
evaluated after the current velocity is applied
9© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

10. Adaptive Learning Rates
 AdaGrad
 AdaGrad gives larger weights for gradients of rare terms, and
smaller weights for those of common terms.
10© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

11. Adaptive Learning Rates
 RMSProp
 Perform better in the non-convex setting
 Uses an exponentially decaying average to discard history from the
extreme past so that it can converge rapidly after finding a convex
bowl
11© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

12. Adaptive Learning Rates
 RMSProp with Nesterov Momentum
12© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

13. Adaptive Learning Rates
 AdaDelta
 approximate the second-order optimization instead of the first-order
optimization.
13© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

14. Adaptive Learning Rates
 Adam (RMSProp + momentum)
14

15. Optimizers
15Ref: http://shuuki4.github.io/deep%20learning/2016/05/20/Gradient-Descent-Algorithm-Overview.html

16. Strategies
 Initialization
 Point: break symmetry
 random initialization
 large value: strong symmetry breaking effect, but exploding
gradient problem
 random Gram-Schmidt orthogonalization
 Sparse initialization -> strong prior
 initialize with unsupervised model, different task
16© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr
(Glorot and Bengio, 2010)

17. Strategies
 Batch Normalization
 hard to choose an appropriate learning rate
 Adaptive reparameterization
 A second-order term might be very small or very large based on wi
 second-order term:
𝐵𝑁 𝒉, 𝛾, 𝛽 = 𝛽 + 𝛾
𝒉 − 𝐸(𝒉)
𝑉𝑎𝑟 𝒉 + 𝜖
17© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr

18. Strategies
 Batch Normalization
 Advantage
 reduce the problem of coordinating updates across many layers
 mitigate exploding or vanishing gradient problem
 allow higher learning ra
 reduce the strong dependence on initialization
 act as a regularization method
18© 2018, SNU Biointelligence Lab., http://bi.snu.ac.kr