The document discusses backpropagation and other differentiation algorithms used in deep learning. It explains that backpropagation uses computational graphs and recursively applies the chain rule of calculus to calculate the gradients of loss with respect to weights in neural networks. This allows for updating weights to minimize loss during training. Computational graphs are constructed to represent operations in neural networks and the chain rule is used to determine how changes to weights in earlier layers affect the loss.

1. Back-Propagation and
Other Differentiation
Algorithms
Deep learning 2017/6/7 ( )
@ss_shopetan

2. • feedforward NN x
( )
•
• Backprop

3. • Backprop NN ?
->No
• Bp NN ?
->No

4. 6.5.1 Computational Graphs
•
• : etc…
• : 1

5. 6.5.1 Computational Graphs
z = xy

6. 6.5.1 Computational Graphs
ˆy = (xT
w + b)

7. 6.5.1 Computational Graphs
H = max(0, XW + b)

8. 6.5.1 Computational Graphs
ˆy = xT
w
X
i
w2
i
(1)
(2)

9. 6.5.1 Computational Graphs
ˆy = xT
w
X
i
w2
i
(1)
(2)

10. 6.5.2 Chain Rule of Calculus
•
•
• z x

11. 6.5.2 Chain Rule of Calculus
• ?
• y x …
• z xi

12. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• Backprop

13. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• backprob
•

14. 6.5.1 Computational Graphs( )
z = xy

15. …
?
6.5.1 Computational Graphs( )

16. …
?
6.5.1 Computational Graphs( )
dz
dx

17. …
6.5.1 Computational Graphs( )
dz
dy

18. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• ?
x = f(w), y = f(x), z = f(y)

19. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
x = f(w), y = f(x), z = f(y)
dz
dw
=
dz
dy
dy
dx
dx
dw

20. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
x = f(w), y = f(x), z = f(y)
dz
dw
=
dz
dy
dy
dx
dx
dw
= f0
(y)f0
(x)f0
(w)

21. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
x = f(w), y = f(x), z = f(y)
dz
dw
=
dz
dy
dy
dx
dx
dw
= f0
(f(f(w)))f0
(f(w))f0
(w)
= f0
(y)f0
(x)f0
(w)

22. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
x = f(w), y = f(x), z = f(y)
dz
dw
=
dz
dy
dy
dx
dx
dw
= f0
(f(f(w)))f0
(f(w))f0
(w)
= f0
(y)f0
(x)f0
(w)
…

23. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
•
(…?)
•
aa
n
f
…

24. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• Backprop

25. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• Backprop
step1

26. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• Backprop
step2
dz
dx
=
dz
dy
dy
dx

27. 6.5.3 Recursively Applying the
Chain Rule to Obtain Backprop
• Backprop
step3
dz
dw
=
dz
dx
dx
dw