Part of multiple talks on the concepts and implementations of neural networks. covers the mains types of neurons and the concepts behind perceptrons and there learning algorithm.

1. Neural Networks
Chris Sharkey
today
@shark2900

2. Vs

3. Types of Neurons
STypesofNeuronsS
Linear
Binary Threshold
Rectifier
Sigmoid
Stochastic Binary
Simple neurons. Computationally limited.
Fixed output upon passing a threshold
Variable output upon passing a threshold
Outputs a smooth bounded function
Outputs a smooth bounded probability function

4. Linear Neuron
• simple and consequently computationally limited
𝑦 = 𝑏 +
𝑖
𝑎
𝑥𝑖 𝑤𝑖
𝑎
output
bias i th input
weight on i th input
sum of all incoming connections with each connection considered
the activity on the input neuron multiplied by the weight on the line

5. Linear Neuron
𝑏 +
𝑖
𝑎
𝑥𝑖 𝑤𝑖
𝑎
y
• plotting the output by the bias +
the weighted activity on the
input lines produces a straight
line that travels thought the
origin

6. Binary Threshold Neurons
• computes a weighted sum of inputs
• sends out a fixed spike of activity if the weighted sum exceeds a threshold
z = 𝑖 𝑥𝑖 𝑤𝑖
𝑎
y
1 𝑖𝑓 𝑧 ≥ 𝜃
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
z = 𝑏 + 𝑖
𝑎
𝑥𝑖 𝑤𝑖
𝑎
y
1 𝑖𝑓 𝑧 ≥ 0
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝜃 = −𝑏

7. Binary Threshold Neurons
• binary output either a spike in
activity or no activity
• spike is like a truth value
threshold weighted input
output
1
0 threshold

8. Rectifier Linear Neurons
• zero as an output or no output until a threshold is passed
• when threshold is passed the output z is equivalent to the output y
z = 𝑏 + 𝑖 𝑥𝑖 𝑤𝑖
𝑎
y
𝑧 𝑖𝑓 𝑧 > 0
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

9. Rectifier Linear Neurons
• allows for the nice properties of
linear systems above zero and
allows for decision making below
at 0
y
z
0

10. Sigmoid Neurons
• give a more real-valued output
• output is a smooth and bounded function of the total input
z = 𝑏 + 𝑖 𝑥𝑖 𝑤𝑖
𝑎
y =
1
1+𝑒−𝑧

11. Sigmoid Neurons
• nice derivatives of the curve
exist
• nice derivatives are
advantageous for easier learning
algorithms
• (more details in next talk)
z
y
.5
0

12. Stochastic Binary Neurons
• same equation as sigmoid or logistic neurons
• treat the output of the logistics as the probability of producing a spike in a short
window of time
y = 𝑏 + 𝑖 𝑥𝑖 𝑤𝑖
𝑎
𝑝(𝑠 = 1) =
1
1+𝑒−𝑧

13. Stochastic Binary Neurons
• use same as logistic units but
are bounded by measures of
probability
z
p
.5
0

14. Question?

15. Perceptrons
• first generation of neural networks
• good first example of a neural network
• binary threshold neurons
• trained binary neurons work as classifiers
• example of ability includes pattern recognition
• popularized by Frank Rosenblatt in the 1960’s 1
x1
x2
b
w1
w2

16. Perceptrons
• learning procedure:
• add an extra component with value 1 to each input vector
• this accounts for the bias values
• pick training cases using any policy that ensures every training case will keep getting
picked. To begin randomly assign weights then using an iterative method adjust
the weights:
‣ if the output unit is correct do not change the weight
‣ if the output unit is incorrect and output is a 0 add the input vector to the weight vector
‣ if the output unit is incorrect and the output is a 1 subtract the input vector from the weight vector
• stop when the set of weights that correctly classifies all training cases are found
• assuming such set of weights exists

17. Perceptrons
• Weight space
an input vector
with correct
answer (=1)
good weight
vector
bad weight
vector
bad weight
vector
an input vector
with correct
answer (=1)
an input vector
with correct
answer (=1)
bad weight
vectorgood weight
vector
an input vector
with correct
answer (=1)

18. Limitation of perceptron
The flaws and advantageous of perceptrons

19. What is next?
types of network architectures

20. Thank you