A brief study material for teaching mathematics of back propagation. The notations are based on https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf

1. How to denote general computation in neural networks
1
 ：the width of
 ：the th layer.  ：an activation
function

2. A simple example of layers of a neural net
2
 A simple
example of
layers of a
neural net

3. Chain rules and computation graphs
3
 A chain rule without junctions
 A chain rule with a junction
•
•
•
•

4. Chain rules and computation graphs
4
 General chain rules
•
•

5. Partial differentiations of a loss function with respect to parameters

5
A chain rule
without junctions

6. Partial differentiations of a loss function with respect to parameters
 The same computation with respect to
all the parameters
6
•
：Error
•

7. Calculating errors

7
Chain rule
with a junction

8. Computation of errors
 Calculating errors with vectors and matrices
8
Recursive computation

9. Generalizing the computation
9
Can be
generalized
layers with
any size
 

10. Recursiveness of backprop
10
Error function
Backprop


Forward backprop



11. boldsymbol{a}_t = W_{t-1}boldsymbol{o}_{t-1}
boldsymbol{o}_t = boldsymbol{sigma}( boldsymbol{a}_{t}) left(
begin{array}{c}
a_{t, 1}
a_{t, 2}
end{array}
right) = left(
begin{array}{ccc}
w_{1, 1}&w_{1, 2}&w
w_{2, 1}&w_{2, 2} &
end{array}
right)left(
begin{array}{c}
o_{t-1,1}
o_{t-1,2}
o_{t-1,3}
end{array}
right)
left(
begin{array}{c}
a_{t+1, 1}
a_{t+1, 2}

12. frac{partial ell}{partialw_{1, 1}} = frac{partial ell}{partialo_{t, 1}}
frac{partial o_{t, 1}}{partialw_{1, 1}}
= frac{partial ell}{partial o_{t, 1}}frac{partial sigma (a_{t, 1})}{partialw_{1, 1}}
= frac{partial ell}{partial o_{t, 1}}frac{partial sigma (a_{t, 1})}{partiala_{t, 1}}
frac{partial a_{t, 1}}{partialw_{1, 1}}
= frac{partial ell}{partial o_{t, 1}} sigma ' (a_{t, 1}) o_{t-1,1}

13. frac{partial ell}{partialw_{1, 1}} = frac{partial ell}{partialo_{t, 1}} sigm
(a_{t, 1})o_{t-1,1}
left(
begin{array}{ccc}
frac{partial ell}{partialw_{1, 1}} &frac{partial ell}{partial w_{1,2}} &
frac{partial ell}{partialw_{1, 3}}
frac{partial ell}{partialw_{2, 1}} &frac{partial ell}{partial w_{2,
2}} & frac{partial ell}{partialw_{2,3}}
end{array}
right)=left(
begin{array}{ccc}
frac{partial ell}{partialo_{t, 1}} sigma ' (a_{t, 1})o_{t-1,1} &frac{partia
ell}{partial o_{t, 1}} sigma ' (a_{t, 1}) o_{t-1,2} &frac{partial ell}{partial
1}} sigma '(a_{t, 1})o_{t-1,3}
frac{partial ell}{partialo_{t, 2}} sigma ' (a_{t, 2})o_{t-1,1} &frac{partia
ell}{partial o_{t, 2}} sigma ' (a_{t, 2}) o_{t-1,2} &frac{partial ell}{partial
2}} sigma '(a_{t, 2}) o_{t-1,3}
end{array}
right)

14. right)= left(
begin{array}{cc}
sigma '(a_{t, 1}) &0
0 & sigma '(a_{t, 2})
end{array}
right)left(
begin{array}{c}
frac{partial ell}{partialo_{t, 1}}
frac{partial ell}{partialo_{t, 2}}
end{array}
right) left(
begin{array}{ccc}
o_{t-1,1} & o_{t-1,2} & o_{t-1,3}
end{array}
right)
= diag(boldsymbol{sigma}' (boldsymbol{a}_t) )
boldsymbol{delta_t}boldsymbol{o}_{t-1}^T
boldsymbol{delta_t} = frac{partial ell}{partial boldsymbol{o}_t}

15. = frac{partial ell}{partial o_{t+1, 1}} frac{partialo_{t+1, 1}}{parti
1}} frac{partial a_{t+1, 1}}{partialo_{t, 1}}+ cdots +frac{partial e
o_{t+1, 4}} frac{partial o_{t+1, 4}}{partiala_{t+1,4}} frac{partiala
4}}{partialo_{t, 1}}
= frac{partial ell}{partial o_{t+1, 1}} fra
a_{t+1, 1}} frac{partiala_{t+1, 1}}{partia
ell}{partial o_{t+1,4}} frac{partial sigm
4}} frac{partial a_{t+1, 4}}{partialo_{t, 1
= frac{partial ell}{partial o_{t+1, 1}} sigma ' (a_{t+1, 1}) phi_{1,

16. de
de
en
rig
be
ph
ph
ph
ph
en
rig
sig
le
be
de
de
de
de
en
rig