The document discusses the development of artificial neural networks (ANNs) by Warren McCulloch and Walter Pitts in 1943, introducing concepts such as threshold logic and perceptrons. It explores various learning methods including gradient descent and backpropagation, focusing on reducing error in ANN models through multilayer structures. Additionally, it briefly covers online and offline learning, as well as the evolution of spiking neural networks as a more advanced form of ANN.

1. Artificial Neural Networks
Ildar Nurgaliev

2. Warren McCulloch and Walter Pitts (1943) created a
computational model for neural networks based on
mathematics and algorithms. They called this model
threshold logic.
Neural networks, as used in artificial intelligence, have
traditionally been viewed as simplified models of neural
processing in the brain.
Introduction

3. Biological Network and Neural System

4. Biological Network and Neural System

5. Biological Network and Neural System

6. Perceptron

7. Perceptron

8. Perceptron

9. Perceptron

10. Bias: Activation Weight

11. Perceptron and Boolean Functions
x1 x2 x1 ⋀ x2
-1 -1
-1 1
1 -1
1 1
-1
-1
-1
1
sign(w0 + w1x1 + w2x2) = x1 ⋀ x2
Conjunction

12. Perceptron and Boolean Functions
x1 x2 x1 ⋀ x2
-1 -1
-1 1
1 -1
1 1
-1
-1
-1
1
sign(w0 + w1x1 + w2x2) = x1 ⋀ x2
Conjunction

13. Perceptron and Boolean Functions
x1 x2 x1 ⋀ x2
-1 -1
-1 1
1 -1
1 1
-1
-1
-1
1
sign(w0 + w1x1 + w2x2) = x1 ⋀ x2
w0 = -1
w1 = 1
w2 = 1
Conjunction

14. Perceptron and Boolean Functions
x1 x2 x1 ⋀ x2
-1 -1
-1 1
1 -1
1 1
-1
1
1
1
sign(w0 + w1x1 + w2x2) = x1 ⋁ x2
Disjunction

15. Perceptron and Boolean Functions
x1 x2 x1 ⋀ x2
-1 -1
-1 1
1 -1
1 1
-1
1
1
1
sign(w0 + w1x1 + w2x2) = x1 ⋁ x2
Disjunction
w0 = 1
w1 = 1
w2 = 1

16. Geometric Interpretation
w0

17. Geometric Interpretation
w0

18. Perceptron Learning
Ensemble-Teacher Learning
Perceptron

19. Perceptron Learning
Ensemble-Teacher Learning
Perceptron
x1 x2 x1 ⋀ x2
-1 -1
-1 1
1 -1
1 1
-1
-1
-1
1
sign(w0 + w1x1 + w2x2) = x1 ⋀ x2

20. Ensemble-Teacher Learning
1
1
-1
1
1
1
0.5
0.5
0.5
-1
1

21. Ensemble-Teacher Learning
1
1
-1
1
1
1
0.5
0.5
0.5
-1
1 -1

22. Ensemble-Teacher Learning
1
1
-1
1
1
1
0.5
0.5
0.5
-1
1 -1
↓↓
↓
↓
↓↓
↑

23. Ensemble-Teacher Learning
1
1
-1
1
1
1
0.5
0.5
0.5
-1
1 -1
↓↓
↓
↓
↓↓
↑
Right ans: a
Net ans : y
Direction of learning
d = a - y = -2
Change weight
Δwi = εdxi|wi|

24. Ensemble-Teacher Learning
1
1
-1
1
1
0.5
0.2
0.7
0
-1.5
11 -1
Right ans: a
Net ans : y
Direction of learning
d = a - y = -2
Change weight
Δwi = εdxi|wi|

25. XOR-function
Doesn’t work?
x1 x2 x1 ⊕ x2
-1 -1
-1 1
1 -1
1 1
-1
1
1
-1

26. XOR-function
Doesn’t work?
x1 x2 x1 ⊕ x2
-1 -1
-1 1
1 -1
1 1
-1
1
1
-1
Solution

27. Multilayer Perceptron

28. Multilayer Perceptron
Input layer Hidden layer Output layer

29. Learning as a Function Minimization
Given:
X=(X1...Xk) input vectors, Xi∈Rn
A=(A1...Ak) correct output vectors, Ai∈Rm
(X,A) learning set
W vector which contains all weights
N(W,X) neuron network’s function
Y = N(W,X) neuron network’s response Y∈Rm
D(Y,A) = ∑m
j=1(Y[j]-A[j])2 error function
D(Yi)=D(Y,Ai) error function on i-th example
Ei(W)=Di(N(W,Xi)) network’s error on i-th example
E(W)=∑k
i=1Ei(W) network’s error in whole set
Goal:
Find vector W such that E(W)➝min (learning in the whole set)
Find vector W such that Ei(W)➝min (learning in the particular example)

30. Gradient Descent Method

31. Gradient Descent Method
Algorithm for single variable Algorithm for GDM
1. Initialize x1 with random value from
R
2. i=1
3. xi+1 = xi-દf’(xi)
4. i++
5. if f(xi ) - f(xi+1) > c goto 3
1. Initialize W1 with random value
from Rn
2. i=1
3. Wi+1 = Wi-દ▽f(Wi)
4. W++
5. if f(Wi ) - f(Wi+1 )> c goto 3

32. Backpropagation Method

33. Backpropagation Method
Given:
X=(X1...Xk) input vectors, Xi∈Rn
A=(A1...Ak) correct output vectors, Ai∈Rm
(X,A) learning set
W vector which contains all weights
N(W,X) neuron network’s function
Y = N(W,X) neuron network’s response Y∈Rm
D(Y,A) = ∑m
j=1(Y[j]-A[j])2 error function
D(Yi)=D(Y,Ai) error function on i-th example
Ei(W)=Di(N(W,Xi)) network’s error on i-th example
E(W)=∑k
i=1Ei(W) network’s error in whole set
Goal:
Find vector W such that E(W)➝min (learning in the whole set)
Find vector W such that Ei(W)➝min (learning in the particular example)

34. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2 Goal: decrease function, using gradient
descent in order increase accuracy of
function.

35. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y2- a2)
Calculate partial derivatives

36. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y2- a2)
y1=y1(w01,w11,w21) = f( sssssssssss )
But y1 is also a function. Let’s consider it as function of weights.
Now we are able to calculate its partial derivatives.

37. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y1- a1)
y1=y1(w01,w11,w21) = f( ssssssssss )
= f’(S1) x2
= 2(y1- a1) = 2(y2- a2)
For example

38. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y2- a2)
y1=y1(w01,w11,w21) = f( ssssssssss )
= f’(S1) x2 = 0
Ek(W)=Dk(y1(w01, w11, w21 ), y2 (w02, w12, w22 ))
y2=y2(w02,w12,w22) = f( sssssssssss )
And now we are able to calculate
partial derivative to function Ek on
each weight.

39. Backpropagation Method
Dk(y1,...,yn) = (y1- an)2 +... + (yn- an )2
Same actions in the general case.

40. Backpropagation Method
Dk(y1,...,yn) = (y1- an)2 +... + (yn- an )2
yi = f(Si )
Now calculate functions Si yi and each derivative on wji.

41. Backpropagation Method
Dk(y1,...,yn) = (y1- an)2 +... + (yn- an )2
yi = f(Si )

42. Backpropagation Method
Dk(y1,...,yn) = (y1- an)2 +... + (yn- an )2
yi = f(Si )
Formula to calculate derivative of Ek function on each weight

43. Backpropagation Method
yi = yi(x1 , … , xm ) xj = xj(v0j , … , vrj )
If would be that Dk= Dk(x1 ,.., xm ) it means that

44. Backpropagation Method
yi = yi(x1 , … , xm ) xj = xj(v0j , … , vrj )
If would be that Dk= Dk(x1 ,.., xm ) it means that
We don’t know only that, so let calculate it!

45. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y2- a2)
y1=y1(w01,w11,w21) = f( sssssssssss )
Now let’s consider f function as a function of xi and
calculate its derivative

46. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y2- a2)
y2=y2(w02,w12,w22) = f( sssssssssss )
Now let’s consider f function as a function of xi and
calculate its derivative

47. Backpropagation Method
Dk(y1,y2)=(y1- a1)2 + (y2- a2)2
= 2(y1- a1) = 2(y2- a2)
y1=y1(w01,w11,w21) = f( sssssssssss )
Now we are able to calculate
derivative of Dk function of x1

48. Backpropagation Method
Now do the same actions in the general case.

49. Backpropagation Method

50. Offline Learning
Train the ANN
on example set
Use the ANN
on the real data
But what if the real data is not i.i.d.?

51. Online learning
1. Receive an instance
2. Predict the outcome
3. Obtain the real outcome
Learn one instance at a time
However, in practice it is not always possible
to obtain the real outcome

52. Spiking Neural Network
● Third-generation of ANN models
● Adds the concept of time to the neuron
● One SNN neuron can replace hundreds of
hidden neurons in conventional ANN models
● Requires huge computational power

53. TrueNorth by IBM
1 million neurons
256 million
synapses
Power: <100 mW