deep learning from scratch

1. Feedback

2. ■Activation function
 Why use?
• Large hidden layer : complex function
• Small hidden layer : simple function
• Input node : 13, output node : 1
■ hidden layer 1 (Node 4 ) : 13 * 4 + 5 = 57
■ Hidden layer 2 (Node 2) : 13 * 2 + 2 * 2 + 3 = 35
Ch4_Feedback
Interaction Lab., Kumoh National Institue of Technology 2

3. ■Sigmoid function
 ℎ 𝑥 =
1
1+𝑒−𝑥
 Smooth curve, continuous variation
 Return real-valued
 ℎ 1 = 0.731
■ReLU function
 ℎ(𝑥) =
𝑥 (𝑥 ≥ 0)
0 (𝑥 < 0)
 Leakly ReLU, PReLU
Ch4_Feedback
Interaction Lab., Kumoh National Institue of Technology 3

4. ■Sigmoid function
 Gradient vanishing
• Backpropagation
Ch4_Feedback
Interaction Lab., Kumoh National Institue of Technology 4

5. ■HOG(Histogram of Gradient)
 Use image’s local gradient as a feature of the image
Ch5_Feedback
Interaction Lab., Kumoh National Institue of Technology 5

6. ■GD vs SGD
 Gradient Descent
• Compute all the data => 1 h
• Take the best step forward
• 6 step = 6 h
• Sure, but it is too slow
 Stochastic Gradient Descent
• Compute only some data => 5 m
• Take quickly step forward
• 10 step = 50 m
• It is a little lost, but it is going fast
Ch5_Feedback
Interaction Lab., Kumoh National Institue of Technology 6

7. ■Optimizer
Ch5_Feedback
Interaction Lab., Kumoh National Institue of Technology 7

8. Interaction Lab. Kumoh National Institute of Technology
Deep Learning from Scratch
chapter 6. back propagation
JaeYeop Jeong

9. ■Intro
■Computational graph
■Chain rule
■Back propagation
■Implementation of simple layer
■Implementation of activation function layer
■Implementation of Affine/softmax layer
Agenda
Interaction Lab., Kumoh National Institue of Technology 9

10. ■Numerical differentials are simple and easy to implement
 Long time to calculate
■Back propagation
 To calculate the gradient of the weight efficiently
 A formula or Computational graph
Intro
Interaction Lab., Kumoh National Institue of Technology 10

11. ■A graph of the calculation process
 Node, edge
■Q1
 현빈 군은 슈퍼에서 1개에 100원인 사과를 2개 샀습니다. 이때 지불
금액을 구하세요. 단 소비세가 10% 부과됩니다.
Computational graph(1/5)
Interaction Lab., Kumoh National Institue of Technology 11

12. Computational graph(2/5)
Interaction Lab., Kumoh National Institue of Technology 12

13. ■Q2
 현빈 군은 슈퍼에서 사과를 2개, 귤을 3개 샀습니다. 사과는 1개에 100
원, 귤은 1개 150원입니다. 소비세가 10%일 때 지불 금액을 구하세요.
 Construct the Computational graph
 Proceed from left to right with the calculation
Computational graph(3/5)
Interaction Lab., Kumoh National Institue of Technology 13

14. ■Local computation
 A small range directly related to oneself
Computational graph(4/5)
Interaction Lab., Kumoh National Institue of Technology 14
4000 + 200 = 4200

15. ■Why computational graph
 Local computation
 Keep all intermediate calculation results
 Calculate differentials efficiently
• Apple prices : 𝑥, Payment(𝐿) :
𝜕𝐿
𝜕𝑥
Computational graph(5/5)
Interaction Lab., Kumoh National Institue of Technology 15

16. ■Back propagation of computational graph
 Multiply the local differential in the forward and opposite directions
• 𝑦 = 𝑓 𝑥 = 𝑥2
,
𝜕𝑦
𝜕𝑥
= 2𝑥
Chain rule(1/3)
Interaction Lab., Kumoh National Institue of Technology 16
𝑓
𝑥 𝑦
𝐸
𝜕𝑦
𝜕𝑥
𝐸

17. ■𝑧 = 𝑡2
, 𝑡 = 𝑥 + 𝑦
Chain rule(2/3)
Interaction Lab., Kumoh National Institue of Technology 17

18. Chain rule(3/3)
Interaction Lab., Kumoh National Institue of Technology 18
𝜕𝑧
𝜕𝑧
𝜕𝑧
𝜕𝑡
𝜕𝑡
𝜕𝑥
=
𝜕𝑧
𝜕𝑡
𝜕𝑡
𝜕𝑥
=
𝜕𝑧
𝜕𝑥

19. ■Back propagation of add node
 𝑧 = 𝑥 + 𝑦,
𝜕𝑧
𝜕𝑥
= 1,
𝜕𝑧
𝜕𝑦
= 1
Back propagation(1/5)
Interaction Lab., Kumoh National Institue of Technology 19

20. ■Back propagation of add node
 Add node : Send as it is
Back propagation(2/5)
Interaction Lab., Kumoh National Institue of Technology 20

21. ■Back propagation of multiply node
 𝑧 = 𝑥𝑦,
𝜕𝑧
𝜕𝑥
= 𝑦,
𝜕𝑧
𝜕𝑦
= 𝑥
Back propagation(3/5)
Interaction Lab., Kumoh National Institue of Technology 21

22. ■Back propagation of multiply node
 Multiply interchangeable values
• Input of forward propagation
Back propagation(4/5)
Interaction Lab., Kumoh National Institue of Technology 22

23. ■Example
Back propagation(5/5)
Interaction Lab., Kumoh National Institue of Technology 23

24. ■Multiply layer
Implementation of simple layer(1/3)
Interaction Lab., Kumoh National Institue of Technology 24

25. ■Add layer
Implementation of simple layer(2/3)
Interaction Lab., Kumoh National Institue of Technology 25

26. Implementation of simple layer(3/3)
Interaction Lab., Kumoh National Institue of Technology 26

27. ■ReLU layer
 𝑦 =
𝑥 ( 𝑥 > 0)
0 (𝑥 ≤ 0)
𝜕𝑦
𝜕𝑥
=
1 (𝑥 > 0)
0 (𝑥 ≤ 0)
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 27
𝑟𝑒𝑙𝑢
𝑥 𝑦
𝜕𝐿
𝜕𝑦
𝜕𝐿
𝜕𝑦
𝑟𝑒𝑙𝑢
𝑥 𝑦
0 𝜕𝐿
𝜕𝑦
𝑥 > 0
𝑥 ≤ 0

28. ■Sigmoid layer
 𝑦 =
1
1+exp(−𝑥)
 exp 𝑥 → 𝑦 = exp 𝑥
 / → 𝑦 =
1
𝑥
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 28

29. ■Sigmoid layer
 𝑦 =
1
1+exp(−𝑥)
, (1 + exp −𝑥 = 𝑥) 𝑦 =
1
𝑥
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 29

30. ■Sigmoid layer
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 30

31. ■Sigmoid layer
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 31

32. ■Affine layer
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 32

33. ■Batch affine layer
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 33

34. ■Softmax-with-Loss layer
 Softmax layer
 Cross entropy error
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 34

35. ■Softmax-with-Loss layer
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 35

36. ■Softmax-with-Loss layer
 t : (0, 1, 0)
 y : (0.3, 0.2, 0.5) => y – t : (0.3, -0.8, 0.5)
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 36

37. Q&A
Interaction Lab., Kumoh National Institue of Technology 37