The document provides an overview of back-propagation in neural networks, detailing the process of error calculation and weight adjustment to improve model predictions. It discusses key concepts such as the error function, the sigmoid activation function, and gradient descent. The author concludes that back-propagation can effectively learn weights from data, has linear performance with respect to network layers, and is simple to implement.

1. A Primer on Back-Propagation of Errors
(applied to neural networks)
Auro Tripathy
auro@shatterline.com

2. Auro Tripathy
Outline
• Summary of Forward-Propagation
• The Calculus of Back-propagation
• Summary
2

3. Auro Tripathy
A Feed-Forward Network is a Brain-Inspired Metaphor
3

4. Auro Tripathy
Feed-forward to calculate the error relative to the
desired output
Error-Function (aka Loss-, Cost-, or Objective-Function)
• In the feed-forward path, calculate the error relative to the desired output
• We define a error-function E(X3, Y) as the “penalty” of predicting X3 when the true output is Y.
• The objective is to minimize the error across all the training samples.
• The error/loss E(X3, Y) assigns a numerical score (a scalar) for the network’s output X3 given
the expected output Y.
• The loss is zero only for cases where the neural network’s output is correct.
4

5. Auro Tripathy
Sigmoid Activation Function
The sigmoid activation function
σ(x) = 1/(1 + e−x)
is an S-shaped activation function transforming all
values of x in the range, [0,1]
5
https://en.wikipedia.org/wiki/File:Logistic-curve.svg

6. Auro Tripathy
Gradient Descent
6
Note, in practice, we don’t expect a global minima, as shown here
a
b

7. Auro Tripathy
“Unshackled by the chain-rule”
-Patrick Winston, MIT
7

8. Auro Tripathy
Derivative of the Error E with-respect-to the
Output, X3
8

9. Auro Tripathy
Derivative of the Sigmoid Activation Function
9
P3 X3
For the Sigmoid function, the cool thing is, the derivative of the output, X3
(with respect to the input, P3) is expressed in terms of the output, i.e.,
X3 . (1 - X3)
http://kawahara.ca/wp-content/uploads/derivative_of_sigmoid.jpg

10. Auro Tripathy
Derivative of P3 with-respect-to W3
10

11. Auro Tripathy
Propagate the errors backward and adjust the weights,
w, so the actual output mimics the desired output
11

12. Auro Tripathy
Computations are Localized & Partially Pre-computed in the Previous Layer
12

13. Auro Tripathy
Summary
☑If there’s a representative set of inputs and
outputs, then back-propagation can learn the
the weights.
☑Back-propagation has linear performance
relative to the number of layers.
☑Simple to implement (and test)
13

14. Auro Tripathy
Credits
14
Concepts crystalized from MIT Professor Patrick Winston’s lecture,
https://www.youtube.com/watch?v=q0pm3BrIUFo
auro@shatterline.com