This document discusses techniques for deep learning, including optimization methods like stochastic gradient descent, momentum, AdaGrad, RMSProp, and Adam. It also covers initializing weight values, preventing overfitting through techniques like batch normalization, weight decay, and dropout. Hyperparameter tuning is also addressed, such as adjusting the learning rate, batch size, number of neurons, and selecting hyperparameters through validation data.

1. Interaction Lab. Kumoh National Institute of Technology
Deep Learning from Scratch
chapter 6. Learning-related skills
JaeYeop Jeong

2. ■Intro
■Optimizer
■Initial Value of Weight
■Overcome of Overfitting
■Hyper Parameter
Agenda
Interaction Lab., Kumoh National Institue of Technology 2

3. ■Optimization
 A parameter that reduces the value of the loss function
• Gradient
 SGD
• 𝑊 = 𝑊 − η
𝜎𝐿
𝜎𝑊
Intro(1/4)
Interaction Lab., Kumoh National Institue of Technology 3

4. ■SGD
 𝑓 𝑥, 𝑦 =
1
20
𝑥2
+ 𝑦2
Intro(2/4)
Interaction Lab., Kumoh National Institue of Technology 4

5. ■SGD
 (-7, 2) start
Intro(3/4)
Interaction Lab., Kumoh National Institue of Technology 5

6. ■SGD
Intro(4/4)
Interaction Lab., Kumoh National Institue of Technology 6

7. ■Momentum
 𝑣 ← 𝑎𝑣 − η
𝜎𝐿
𝜎𝑊
 W ← 𝑊 + 𝑣
Optimizer(1/3)
Interaction Lab., Kumoh National Institue of Technology 7

8. ■AdaGrad
 ℎ ← ℎ +
𝜎𝐿
𝜎𝑊
∙
𝜎𝐿
𝜎𝑊
 𝑊 ← 𝑊 − η
1
ℎ
∙
𝜎𝐿
𝜎𝑊
 Learning rate decay
 RMSProp
Optimizer(2/3)
Interaction Lab., Kumoh National Institue of Technology 8

9. ■Adam
 AdaGrad + Momentum
Optimizer(3/3)
Interaction Lab., Kumoh National Institue of Technology 9

10. ■In case of 0
 Bad idea
• All weights are updated equally in backpropagation
• Learning is not working effectively
Initial value of weight(1/11)
Interaction Lab., Kumoh National Institue of Technology 10

11. ■In case of
 Using sigmoid
 Normal distribution with 1 standard deviation
 Gradient vanishing
Initial value of weight(2/11)
Interaction Lab., Kumoh National Institue of Technology 11

12. ■In case of
 Using sigmoid
 Normal distribution with 0.01 standard deviation
 Representation spectrum
Initial value of weight(3/11)
Interaction Lab., Kumoh National Institue of Technology 12

13. ■In case of Xavier value
 Using sigmoid
 Before N node
 Normal distribution with
1
𝑛
standard deviation
Initial value of weight(4/11)
Interaction Lab., Kumoh National Institue of Technology 13

14. ■In case of He value
 Using ReLU
 Before N node
 Normal distribution with
2
𝑛
standard deviation
Initial value of weight(5/11)
Interaction Lab., Kumoh National Institue of Technology 14

15. ■Batch normalization
 Force distribution of activation values
 Learning speed improvement
 Does not depend on initial value
 Suppression of overfitting
Initial value of weight(6/11)
Interaction Lab., Kumoh National Institue of Technology 15

16. ■Batch normalization
 Insert “Batch Norm” layer
• Adjust so that the activation value is properly distribution
Initial value of weight(7/11)
Interaction Lab., Kumoh National Institue of Technology 16

17. ■Batch normalization
Initial value of weight(8/11)
Interaction Lab., Kumoh National Institue of Technology 17
{x1, x2, x3, …, xn} {𝒙1, 𝒙 2, 𝒙 3, …, 𝒙 n}
Mini-batch mean
Mini-batch variance
normalize

18. Initial value of weight(8/11)
Interaction Lab., Kumoh National Institue of Technology 18

19. Initial value of weight(9/11)
Interaction Lab., Kumoh National Institue of Technology 19

20. ■Batch normalization
Initial value of weight(10/11)
Interaction Lab., Kumoh National Institue of Technology 20

21. ■Batch normalization
Initial value of weight(11/11)
Interaction Lab., Kumoh National Institue of Technology 21

22. ■Model with many parameter and high expressiveness
■Little training data
Overcome of Overfitting(1/3)
Interaction Lab., Kumoh National Institue of Technology 22

23. ■Weight decay
 In learning, penalize large weight
 Loss +
1
2
λ𝑊2
 λ: hyper parameter
• If λ is large, penalize weights

1
2
λ𝑊2
→ λ𝑊
Overcome of Overfitting(2/3)
Interaction Lab., Kumoh National Institue of Technology 23

24. ■Dropout
Overcome of Overfitting(3/3)
Interaction Lab., Kumoh National Institue of Technology 24

25. ■Hyper parameter
 Number of neuron
 Batch size
 Learning rate
 Etc…
Hyper parameter(1/3)
Interaction Lab., Kumoh National Institue of Technology 25

26. ■Training data
 Only train
■Test data
 Only test
■Validation data
 Adjust hyper parameter
Hyper parameter(2/3)
Interaction Lab., Kumoh National Institue of Technology 26

27. ■Optimization
 Setting the range of value
 Randomization
 Evaluation after learning with the extracted value
 Repeat and narrow down
Hyper parameter(3/3)
Interaction Lab., Kumoh National Institue of Technology 27

28. Q&A
Interaction Lab., Kumoh National Institue of Technology 28