3.
Natural Neural Networks• Signals “move” via electrochemical signals• The synapses release a chemical transmitter – the sum of which can cause a threshold to be reached – causing the neuron to “fire”• Synapses can be inhibitory or excitatory 3
4.
Natural Neural Networks• We are born with about 100 billion neurons• A neuron may connect to as many as 100,000 other neurons 4
5.
Natural Neural Networks• Many of their ideas still used today e.g. – many simple units, “neurons” combine to give increased computational power – the idea of a threshold 5
6.
Modelling a Neuronini j Wj, iaj • aj :Activation value of unit j • wj,i :Weight on link from unit j to unit i • ini :Weighted sum of inputs to unit i • ai :Activation value of unit i • g :Activation function 6
7.
Activation Functions• Stept(x) = 1 if x ≥ t, else 0 threshold=t• Sign(x) = +1 if x ≥ 0, else –1• Sigmoid(x) = 1/(1+e-x) 7
8.
Building a Neural Network1. “Select Structure”: Design the way that the neurons are interconnected2. “Select weights” – decide the strengths with which the neurons are interconnected – weights are selected so get a “good match” to a “training set” – “training set”: set of inputs and desired outputs – often use a “learning algorithm” 8
9.
Basic Neural Networks• Will first look at simplest networks• “Feed-forward” – Signals travel in one direction through net – Net computes a function of the inputs 9
10.
The First Neural Neural Networks X1 2 2 X2 Y -1 X3 Neurons in a McCulloch-Pitts network are connected by directed, weighted paths 10
11.
The First Neural Neural Networks X1 2 2 X2 Y -1 X3 If the on weight on a path is positive the path is excitatory, otherwise it is inhibitory 11
12.
The First Neural Neural Networks X1 2 2 X2 Y -1 X3The activation of a neuron is binary. That is, the neuroneither fires (activation of one) or does not fire (activation ofzero). 12
13.
The First Neural Neural Networks X1 2 2 X2 Y -1 X3For the network shown here the activation function for unit Y is f(y_in) = 1, if y_in >= θ else 0where y_in is the total input signal received θ is the threshold for Y 13
14.
The First Neural Neural NetworksX1 2 2X2 Y -1X3 Originally, all excitatory connections into a particular neuron have the same weight, although different weighted connections can be input to different neurons Later weights allowed to be arbitrary 14
15.
The First Neural Neural Networks X1 2 2 X2 Y -1 X3Each neuron has a fixed threshold. If the net input into the neuron isgreater than or equal to the threshold, the neuron fires 15
16.
The First Neural Neural NetworksX1 2 2X2 Y -1X3The threshold is set such that any non-zero inhibitory input will prevent the neuronfrom firing 16
17.
Building Logic Gates• Computers are built out of “logic gates”• Use threshold (step) function for activation function – all activation values are 0 (false) or 1 (true) 17
18.
The First Neural Neural Networks 1 AND X1 Y X1 X2 Y 1 1 1 X2 1 1 0 0 AND Function 0 1 0 0 0 0 Threshold(Y) = 2 18
19.
The First Neural Networks ORX1 2 X1 X2 Y Y 1 1 1 1 0 1X2 2 0 1 1 ANDFunction OR Function 0 0 0 Threshold(Y) = 2 19
20.
Perceptron • Synonym for Single-Layer, Feed-Forward Network • First Studied in the 50’s • Other networks were known about but the perceptron was the only one capable of learning and thus all research was concentrated in this area 20
21.
Perceptron • A single weight only affects one output so we can restrict our investigations to a model as shown on the right • Notation can be simpler, i.e. O Step0 j WjIj 21
23.
What can perceptrons represent? 1,1 1,1 0,1 0,1 0,0 1,0 1,0 0,0 AND XOR• Functions which can be separated in this way are called Linearly Separable• Only linearly separable functions can be represented by a perceptron• XOR cannot be represented by a perceptron 23
24.
XOR• XOR is not “linearly separable” – Cannot be represented by a perceptron• What can we do instead? 1. Convert to logic gates that can be represented by perceptrons 2. Chain together the gates 24
25.
Single- vs. Multiple-Layers• Once we chain together the gates then we have “hidden layers” – layers that are “hidden” from the output lines• Have just seen that hidden layers allow us to represent XOR – Perceptron is single-layer – Multiple layers increase the representational power, so e.g. can represent XOR• Generally useful nets have multiple-layers – typically 2-4 layers 25
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.
Be the first to comment