Neural networks are composed of simple processing units (neurons) that are interconnected and can learn from data. Natural neural networks in the brain contain billions of neurons that communicate via electrochemical signals. Early artificial neural networks modeled neurons as simple processing units that sum their weighted inputs and use an activation function to determine their output. These networks had limitations in what functions they could represent. The development of multi-layer perceptrons overcame these limitations by introducing hidden layers that increased their computational and representational power.

1. Neural Networks

2. Neural Networks
2

3. Natural Neural Networks
• Signals “move” via electrochemical signals
• The synapses release a chemical transmitter –
the sum of which can cause a threshold to be
reached – causing the neuron to “fire”
• Synapses can be inhibitory or excitatory
3

4. Natural Neural Networks
• We are born with about 100 billion neurons
• A neuron may connect to as many as 100,000
other neurons
4

5. Natural Neural Networks
• Many of their ideas still used today e.g.
– many simple units, “neurons” combine to give
increased computational power
– the idea of a threshold
5

6. Modelling a Neuron
ini j
Wj, iaj • aj :Activation value of unit j
• wj,i :Weight on link from unit j to unit i
• ini :Weighted sum of inputs to unit i
• ai :Activation value of unit i
• g :Activation function
6

7. Activation Functions
• Stept(x) = 1 if x ≥ t, else 0 threshold=t
• Sign(x) = +1 if x ≥ 0, else –1
• Sigmoid(x) = 1/(1+e-x)
7

8. Building a Neural Network
1. “Select Structure”: Design the way that the
neurons are interconnected
2. “Select weights” – decide the strengths with
which the neurons are interconnected
– weights are selected so get a “good match” to
a “training set”
– “training set”: set of inputs and desired
outputs
– often use a “learning algorithm”
8

9. Basic Neural Networks
• Will first look at simplest networks
• “Feed-forward”
– Signals travel in one direction through net
– Net computes a function of the inputs
9

10. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
Neurons in a McCulloch-Pitts network are connected by directed, weighted
paths
10

11. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
If the on weight on a path is positive the path is
excitatory,
otherwise it is inhibitory
11

12. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
The activation of a neuron is binary. That is, the neuron
either fires (activation of one) or does not fire (activation of
zero).
12

13. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
For the network shown here the activation function for unit Y is
f(y_in) = 1, if y_in >= θ else 0
where y_in is the total input signal received
θ is the threshold for Y
13

14. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
Originally, all excitatory connections into a particular neuron have the same
weight, although different weighted connections can be input to different
neurons
Later weights allowed to be arbitrary
14

15. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
Each neuron has a fixed threshold. If
the net input into the neuron is
greater than or equal to the threshold, the neuron fires
15

16. The First Neural Neural Networks
X1 2
2
X2 Y
-1
X3
The threshold is set such that any non-zero inhibitory input will prevent the neuron
from firing
16

17. Building Logic Gates
• Computers are built out of “logic gates”
• Use threshold (step) function for activation
function
– all activation values are 0 (false) or 1 (true)
17

18. The First Neural Neural Networks
1
AND
X1
Y
X1 X2 Y
1 1 1
X2 1
1 0 0
AND Function
0 1 0
0 0 0
Threshold(Y) = 2
18

19. The First Neural Networks
OR
X1 2 X1 X2 Y
Y 1 1 1
1 0 1
X2 2
0 1 1
ANDFunction
OR Function
0 0 0
Threshold(Y) = 2
19

20. Perceptron
• Synonym for Single-Layer,
Feed-Forward Network
• First Studied in the 50’s
• Other networks were known
about but the perceptron
was the only one capable of
learning and thus all research
was concentrated in this area
20

21. Perceptron
• A single weight only affects
one output so we can restrict
our investigations to a model
as shown on the right
• Notation can be simpler, i.e.
O Step0 j
WjIj
21

22. What can perceptrons represent?
AND XOR
Input 1 0 0 1 1 0 0 1 1
Input 2 0 1 0 1 0 1 0 1
Output 0 0 0 1 0 1 1 0
22

23. What can perceptrons represent?
1,1
1,1
0,1
0,1
0,0 1,0
1,0
0,0
AND XOR
• Functions which can be separated in this way are called Linearly Separable
• Only linearly separable functions can be represented by a perceptron
• XOR cannot be represented by a perceptron
23

24. XOR
• XOR is not “linearly separable”
– Cannot be represented by a perceptron
• What can we do instead?
1. Convert to logic gates that can be represented by
perceptrons
2. Chain together the gates
24

25. Single- vs. Multiple-Layers
• Once we chain together the gates then we have “hidden
layers”
– layers that are “hidden” from the output lines
• Have just seen that hidden layers allow us to represent XOR
– Perceptron is single-layer
– Multiple layers increase the representational power, so
e.g. can represent XOR
• Generally useful nets have multiple-layers
– typically 2-4 layers
25