The document discusses backpropagation in neural networks. It begins by introducing computational graphs and how they can be used to efficiently calculate gradients in neural networks using the chain rule. It then explains how to perform backpropagation for different common layer types like multiply, ReLU, sigmoid and affine layers by propagating gradients backward through the computational graph. Code implementations are provided for various layer types like activation functions and affine/softmax layers to demonstrate backpropagation in practice.

1. Feedback

2. ■Activation function
 Why use?
• Large hidden layer : complex function
• Small hidden layer : simple function
• Input node : 13, output node : 1
■ hidden layer 1 (Node 4 ) : 13 * 4 + 5 = 57
■ Hidden layer 2 (Node 2) : 13 * 2 + 2 * 2 + 3 = 35
Ch4_Feedback
Interaction Lab., Kumoh National Institue of Technology 2

3. ■Sigmoid function
 ℎ 𝑥 =
1
1+𝑒−𝑥
 Smooth curve, continuous variation
 Return real-valued
 ℎ 1 = 0.731
■ReLU function
 ℎ(𝑥) =
𝑥 (𝑥 ≥ 0)
0 (𝑥 < 0)
 Leakly ReLU, PReLU
Ch4_Feedback
Interaction Lab., Kumoh National Institue of Technology 3

4. ■Sigmoid function
 Gradient vanishing
• Backpropagation
Ch4_Feedback
Interaction Lab., Kumoh National Institue of Technology 4

5. ■HOG(Histogram of Gradient)
 Use image’s local gradient as a feature of the image
Ch5_Feedback
Interaction Lab., Kumoh National Institue of Technology 5

6. ■GD vs SGD
 Gradient Descent
• Compute all the data => 1 h
• Take the best step forward
• 6 step = 6 h
• Sure, but it is too slow
 Stochastic Gradient Descent
• Compute only some data => 5 m
• Take quickly step forward
• 10 step = 50 m
• It is a little lost, but it is going fast
Ch5_Feedback
Interaction Lab., Kumoh National Institue of Technology 6

7. ■Optimizer
Ch5_Feedback
Interaction Lab., Kumoh National Institue of Technology 7

8. Interaction Lab. Kumoh National Institute of Technology
Deep Learning from Scratch
chapter 6. back propagation
JaeYeop Jeong

9. ■Intro
■Computational graph
■Chain rule
■Back propagation
■Implementation of simple layer
■Implementation of activation function layer
■Implementation of Affine/softmax layer
Agenda
Interaction Lab., Kumoh National Institue of Technology 9

10. ■Numerical differentials are simple and easy to implement
 Long time to calculate
■Back propagation
 To calculate the gradient of the weight efficiently
 A formula or Computational graph
Intro
Interaction Lab., Kumoh National Institue of Technology 10

11. ■A graph of the calculation process
 Node, edge
■Q1
 현빈 군은 슈퍼에서 1개에 100원인 사과를 2개 샀습니다. 이때 지불
금액을 구하세요. 단 소비세가 10% 부과됩니다.
Computational graph(1/5)
Interaction Lab., Kumoh National Institue of Technology 11

12. Computational graph(2/5)
Interaction Lab., Kumoh National Institue of Technology 12

13. ■Q2
 현빈 군은 슈퍼에서 사과를 2개, 귤을 3개 샀습니다. 사과는 1개에 100
원, 귤은 1개 150원입니다. 소비세가 10%일 때 지불 금액을 구하세요.
 Construct the Computational graph
 Proceed from left to right with the calculation
Computational graph(3/5)
Interaction Lab., Kumoh National Institue of Technology 13

14. ■Local computation
 A small range directly related to oneself
Computational graph(4/5)
Interaction Lab., Kumoh National Institue of Technology 14
4000 + 200 = 4200

15. ■Why computational graph
 Local computation
 Keep all intermediate calculation results
 Calculate differentials efficiently
• Apple prices : 𝑥, Payment(𝐿) :
𝜕𝐿
𝜕𝑥
Computational graph(5/5)
Interaction Lab., Kumoh National Institue of Technology 15

16. ■Back propagation of computational graph
 Multiply the local differential in the forward and opposite directions
• 𝑦 = 𝑓 𝑥 = 𝑥2
,
𝜕𝑦
𝜕𝑥
= 2𝑥
Chain rule(1/3)
Interaction Lab., Kumoh National Institue of Technology 16
𝑓
𝑥 𝑦
𝐸
𝜕𝑦
𝜕𝑥
𝐸

17. ■𝑧 = 𝑡2
, 𝑡 = 𝑥 + 𝑦
Chain rule(2/3)
Interaction Lab., Kumoh National Institue of Technology 17

18. Chain rule(3/3)
Interaction Lab., Kumoh National Institue of Technology 18
𝜕𝑧
𝜕𝑧
𝜕𝑧
𝜕𝑡
𝜕𝑡
𝜕𝑥
=
𝜕𝑧
𝜕𝑡
𝜕𝑡
𝜕𝑥
=
𝜕𝑧
𝜕𝑥

19. ■Back propagation of add node
 𝑧 = 𝑥 + 𝑦,
𝜕𝑧
𝜕𝑥
= 1,
𝜕𝑧
𝜕𝑦
= 1
Back propagation(1/5)
Interaction Lab., Kumoh National Institue of Technology 19

20. ■Back propagation of add node
 Add node : Send as it is
Back propagation(2/5)
Interaction Lab., Kumoh National Institue of Technology 20

21. ■Back propagation of multiply node
 𝑧 = 𝑥𝑦,
𝜕𝑧
𝜕𝑥
= 𝑦,
𝜕𝑧
𝜕𝑦
= 𝑥
Back propagation(3/5)
Interaction Lab., Kumoh National Institue of Technology 21

22. ■Back propagation of multiply node
 Multiply interchangeable values
• Input of forward propagation
Back propagation(4/5)
Interaction Lab., Kumoh National Institue of Technology 22

23. ■Example
Back propagation(5/5)
Interaction Lab., Kumoh National Institue of Technology 23

24. ■Multiply layer
Implementation of simple layer(1/3)
Interaction Lab., Kumoh National Institue of Technology 24

25. ■Add layer
Implementation of simple layer(2/3)
Interaction Lab., Kumoh National Institue of Technology 25

26. Implementation of simple layer(3/3)
Interaction Lab., Kumoh National Institue of Technology 26

27. ■ReLU layer
 𝑦 =
𝑥 ( 𝑥 > 0)
0 (𝑥 ≤ 0)
𝜕𝑦
𝜕𝑥
=
1 (𝑥 > 0)
0 (𝑥 ≤ 0)
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 27
𝑟𝑒𝑙𝑢
𝑥 𝑦
𝜕𝐿
𝜕𝑦
𝜕𝐿
𝜕𝑦
𝑟𝑒𝑙𝑢
𝑥 𝑦
0 𝜕𝐿
𝜕𝑦
𝑥 > 0
𝑥 ≤ 0

28. ■Sigmoid layer
 𝑦 =
1
1+exp(−𝑥)
 exp 𝑥 → 𝑦 = exp 𝑥
 / → 𝑦 =
1
𝑥
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 28

29. ■Sigmoid layer
 𝑦 =
1
1+exp(−𝑥)
, (1 + exp −𝑥 = 𝑥) 𝑦 =
1
𝑥
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 29

30. ■Sigmoid layer
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 30

31. ■Sigmoid layer
Implementation of activation function layer
Interaction Lab., Kumoh National Institue of Technology 31

32. ■Affine layer
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 32

33. ■Batch affine layer
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 33

34. ■Softmax-with-Loss layer
 Softmax layer
 Cross entropy error
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 34

35. ■Softmax-with-Loss layer
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 35

36. ■Softmax-with-Loss layer
 t : (0, 1, 0)
 y : (0.3, 0.2, 0.5) => y – t : (0.3, -0.8, 0.5)
Implementation of Affine/softmax layer
Interaction Lab., Kumoh National Institue of Technology 36

37. Q&A
Interaction Lab., Kumoh National Institue of Technology 37