This presentation provides an introduction to the artificial neural networks topic, its learning, network architecture, back propagation training algorithm, and its applications.

1. Artificial Neural
Networks
Mohamed Talaat
Neural Networks 1

2. Agenda
• Neural Networks
• Neural Network Learning
• Network Architectures
• Dimensions of a Neural Network
• Backpropagation Training Algorithm
• Over-Training Prevention
• Successful Applications
Neural Networks 2

3. 3
Neural Networks
• A NN is a machine learning approach inspired by the
way in which the brain performs a particular learning
task:
– Knowledge about the learning task is given in the form of
examples.
– Inter neuron connection strengths (weights) are used to
store the acquired information (the training examples).
– During the learning process the weights are modified in
order to model the particular learning task correctly on the
training examples.

4. 4
• Supervised Learning
– Recognizing hand-written digits, pattern recognition,
regression.
– Labeled examples
(input , desired output)
– Neural Network models: perceptron, feed-forward, radial
basis function, support vector machine.
• Unsupervised Learning
– Find similar groups of documents in the web, content
addressable memory, clustering.
– Unlabeled examples
(different realizations of the input alone)
– Neural Network models: self organizing maps, Hopfield
networks.
Learning

5. Neural Networks NN 1 5
Network architectures
• Three different classes of network architectures
– single-layer feed-forward neurons are organized
– multi-layer feed-forward in acyclic layers
– recurrent
• The architecture of a neural network is linked with the
learning algorithm used to train

6. Neural Networks NN 1 6
Single Layer Feed-forward
Input layer
of
source nodes
Output layer
of
neurons

7. Neural Networks NN 1 7
Multi layer feed-forward
Input
layer
Output
layer
Hidden Layer
3-4-2 Network

8. Neural Networks NN 1 8
Recurrent Network with hidden neuron(s): unit
delay operator z-1
implies dynamic system
z-1
z-1
z-1
Recurrent network
input
hidden
output

9. Neural Networks NN 1 9
Neural Network Architectures

10. 10
Real Neurons
• Cell structures
– Cell body
– Dendrites
– Axon
– Synaptic terminals

11. 11
Real Neural Learning
• Synapses change size and strength
with experience.
• Hebbian learning: When two connected
neurons are firing at the same time, the
strength of the synapse between them
increases.
• “Neurons that fire together, wire
together.”

12. Neural Networks NN 1 12
The Artificial Neuron
• The neuron is the basic information processing unit of
a NN. It consists of:
1 A set of synapses or connecting links, each link
characterized by a weight:
W1, W2, …, Wm
2 An adder function (linear combiner) which
computes the weighted sum of
the inputs:
3 Activation function (squashing function) for
limiting the amplitude of the
output of the neuron.
∑=
=
m
1
jjxwu
j
ϕ
)(uy b+= ϕ

13. Neural Networks NN 1 13
The Artificial Neuron
Input
signal
Synaptic
weights
Summing
function
Bias
b
Activation
functionLocal
Field
v
Output
y
x1
x2
xm
w2
wm
w1
 
∑ )(−ϕ

14. Neural Networks NN 1 14
Bias of a Neuron
• Bias b has the effect of applying an affine
transformation to u
v = u + b
• v is the induced field of the neuron
v
u
∑=
=
m
1
jjxwu
j

15. Neural Networks NN 1 15
Bias as extra input
Input
signal
Synaptic
weights
Summing
function
Activation
functionLocal
Field
v
Output
y
x1
x2
xm
w2
wm
w1
 
∑ )(−ϕ
w0
x0 = +1
• Bias is an external parameter of the neuron. Can be
modeled by adding an extra input.
bw
xwv j
m
j
j
=
= ∑=
0
0

16. Neural Networks NN 1 16
Dimensions of a Neural
Network
• Various types of neurons
• Various network architectures
• Various learning algorithms
• Various applications

17. Backpropagation Training
Algorithm
17
•Create the 3-layer network with H hidden units with full
connectivity between layers. Set weights to small random real
values.
•Until all training examples produce the correct value (within ε),
or mean squared error ceases to decrease, or other termination
criteria:
Begin epoch
For each training example, d, do:
Calculate network output for d’s input values
Compute error between current output and correct output for d
Update weights by backpropagating error and using learning rule
End epoch

18. 18
Comments on Training Algorithm
• Not guaranteed to converge to zero training error,
may converge to local optima or oscillate indefinitely.
• However, in practice, does converge to low error for
many large networks on real data.
• Many epochs (thousands) may be required, hours or
days of training for large networks.
• To avoid local-minima problems, run several trials
starting with different random weights (random
restarts).
– Take results of trial with lowest training set error.
– Build a committee of results from multiple trials (possibly
weighting votes by training set accuracy).

19. 19
Over-Training Prevention
• Running too many epochs can result in over-fitting.
• Keep a hold-out validation set and test accuracy on it after
every epoch. Stop training when additional epochs actually
increase validation error.
• To avoid losing training data for validation:
– Use internal 10-fold CV on the training set to compute the average
number of epochs that maximizes generalization accuracy.
– Train final network on complete training set for this many epochs.
error
on training data
on test data
0 # training epochs

20. 20
Determining the Best
Number of Hidden Units
• Too few hidden units prevents the network from
adequately fitting the data.
• Too many hidden units can result in over-fitting.
• Use internal cross-validation to empirically determine
an optimal number of hidden units.
error
on training data
on test data
0 # hidden units

21. 21
Successful Applications
• Text to Speech (NetTalk)
• Fraud detection
• Financial Applications
– HNC (eventually bought by Fair Isaac)
• Chemical Plant Control
– Pavillion Technologies
• Automated Vehicles
• Game Playing
– Neurogammon
• Handwriting recognition

22. Thanks 
Neural Networks 22