This document summarizes backpropagation and multi-layer feedforward neural networks. It describes how backpropagation can be used to train multi-layer networks by propagating errors backward from the output to adjust weights. The algorithm initializes weights randomly and then iterates over training examples, propagating input forward and error backward to update weights. Cross-validation is used to avoid overfitting by stopping training when validation error stops improving. The document also provides details on the image recognition problems and how the code can be modified to implement different network structures and target outputs.

1. Multi-layer feedforward network
1
1
Artificial Intelligence I1
H1
O1
I2
Backpropagation training of H2
multi-layer feedforward neural networks Ii: Input unit i
I3
Oi: Output unit i
Hi: Hidden unit i
CS 4633/6633 Artificial Intelligence CS 4633/6633 Artificial Intelligence
Training multi-layer networks Backpropagation: The idea
q Supervised learning: A teacher presents input patterns to q The key idea is that the error in the output/activation of a
the network, compares the resulting outputs with the neuron in a hidden layer can be defined as the weighted
desired outputs, and adjusts the network weights to reduce sum of the error of its successor neurons.
the error q Learning involves changing the weights of the network
q For perceptron networks, adjusting weights to reduce error q Weights are changed based partly on the derivative of the
is straightforward because there is only one weight activation function of a neuron. The sigmoid function has a
between each input and output unit simple derivative: if y = 1/(1 + e-x), then dy/dx = y(1-y).
q For multi-layer networks, adjusting weights to reduce error
is more complicated because there are many weights
between each input unit and output unit, and each of these
weights contributes to more than one output unit
CS 4633/6633 Artificial Intelligence CS 4633/6633 Artificial Intelligence
Backpropagation is
Backpropagation: The Algorithm
Initialize network weights to random numbers between -0.5 and 0.5
hill-climbing search
Repeat until error is sufficiently small (each repetition is an epoch) q Backprogagation is search in weight space, where the
number of dimensions of the search space is equal to the
For each training example:
number of weights.
Propagate input through network to compute output
q It is possible for backpropagation to converge to a local
Propagate errors backward through network, as follows:
minimum, although it rarely happens in practice
For each output unit i, calculate the error:
q α is learning rate. If too low, learning is unnecessarly
Errorj ← Oi × (1 - Oi) × (Ti - Oi)
slow. If too high, it does not converge.
For each hidden unit i, calculate the error:
q Momentum: can improve learning time by making the
Errori ← Oi × (1 - Oi) × ∑i (Wi,j × Errorj)
weight adjustment proportional to the amount of the
Update each network weight:
previous weight change
Wi,j ← Wi,j + α × Ii × Errorj
CS 4633/6633 Artificial Intelligence CS 4633/6633 Artificial Intelligence

2. Cross-validation training Generalization
q A statistical approach for avoiding overfitting
q The data set is randomly partitioned into a training set and
a test set. The training set is further partitioned into two
subsets, one for training and one for validation.
Training set
q Performance will always keep improving on the training %
correct
set, due to overfitting. When performance stops improving
Validation set
on the validation set, training is stopped. Then the network
is tested on the test set.
– train = training set
– test1 = validation set for determining convergence
Training time (# of epochs)
– test2 = test set
CS 4633/6633 Artificial Intelligence CS 4633/6633 Artificial Intelligence
Image files Network structure
q Each file contains a 32 x 30 image, with each pixel an ASCII q All networks have three layers (one hidden layer)
character that represents a greyscale intensity value between 0 q For Glickman recognition problem
(black) and 255 (white)
– 960 input units, one for each pixel in image
q attribute-value pairs – 1 output unit, yes or no
– userid (20 possible values) – 4 hidden units
– pose (straight, left, right, up) q For sunglasses recognition problem
– expression (neutral, happy, sad, angry) – same
– eyes (open, sunglasses) q For 1-of-20 face recognition problem
q Problem: learn function from image to attribute-value pair(s) – 20 output units
– map image to yes if Glickman, otherwise no – suggestion: 20 hidden units
– map image to yes if wearing sunglasses, otherwise no q For pose recognition problem
– map image to userid – 4 output units
– map image to pose – suggestion: 6 hidden units
CS 4633/6633 Artificial Intelligence CS 4633/6633 Artificial Intelligence
Changes to code Pointers
q How well a network learns depends on number of hidden
q Sunglasses recognition:
units, learning rate, etc. You are given these.
Change the target output by changing the function
q Backpropagation algorithm may not train a net to achieve
“load_target” in imagenet.c, as follows.
perfect classification accuracy
if (!strcmp(eyes, “sunglasses”)) {
q 1-of-20 face recognition and pose recognition: q To test network w/o training it, set command-line
parameter for epochs to zero
Change number of output and hidden units by changing
arguments to bpnn_create in facetrain.c. Change the q Output lines: epoch number, sum of error over all units, for
evaluation routines “performance_on_imagelist()” and training set: percentage correct and average error, for test
“evaluate_performance” in facetrain.c to reflect the new set1: percentage correct and average error, for test set2:
network structure. Also change the target outputs by percentage correct and average error
changing the function “load_target” in imagenet.c. q Not required to do questions 11 and 12. But consider
drawing learning curves to show results of training.
CS 4633/6633 Artificial Intelligence CS 4633/6633 Artificial Intelligence