Fuzzy Logic and Neural Networks

1. An Introduction to
Deep Learning
Mehrnaz Faraz
Faculty of Electrical Engineering
K. N. Toosi University of Technology
1
In the name of God
Milad Abbasi
Faculty of Electrical Engineering
Sharif University of Technology

2. Contents
• Introduction to Fuzzy
• Introduction to Neural Network
2

3. Introduction to Fuzzy
• Fuzzy: “Difficult to perceive; indistinct or vague”
– Simplicity and flexibility
– Can handle problems with imprecise data
– More readily customizable in natural language terms
3

4. Introduction to Fuzzy
4
Slowest, 𝐴1 Slow, 𝐴2 Fast, 𝐴3 Fastest, 𝐴4
Subset
0
1
Velocity
𝜇 𝐴
Membership Function
𝜇 𝐴 𝑖
𝑥 : 𝑥 → 0, 1 , i = 1, … , 4
Slowest FastestSlow Fast

5. Introduction to Fuzzy
• Example: Automatic Braking System
5
Close?
 Is car close? 0.2 (Not very close)
Brakes: 0.2 (Slight pressure)
 Is car close? 0.8 (Pretty close)
Brakes: 0.8 (Fairly heavy pressure)

6. Introduction to Fuzzy
6
• Well-known Membership Functions:
trimf trapmf
gaussmf sigmf
a b c a b c d
sig
c

7. Introduction to Fuzzy
7
• Linguistic variables:
• Weather is quite cold.
• Height is almost tall.
• Speed is very high.
• Weather, Height and Speed are linguistic variables.
• Cold, Tall and high are linguistic value.

8. Introduction to Fuzzy
• Operations with fuzzy sets:
– Complement operation:
– Fuzzy union operation or fuzzy OR:
– Fuzzy intersection operation or fuzzy AND:
8
)(1)( xx AA
 
A( ) s[ ( ), ( )]A B Bx x x   
A( ) t [ ( ), ( )]A B Bx x x   

9. 1
1 2 30 4
1
1 2 30 4
1 2 30 4
1
( )A
x
( )B
x
( )A B
x 
Introduction to Fuzzy
• Fuzzy union operation (s-norm):
9
Max is a s-norm operator
x
x
x

10. 1
1 2 30 4
1
1 2 30 4
1 2 30 4
1
( )A
x
( )B
x
,
( )A B
x
Introduction to Fuzzy
10
• Fuzzy intersection operation (t-norm):
Min is a t-norm operator
Product is another t-norm operator
(Mamdani Implication)
x
x
x

11. Introduction to Fuzzy
• Other complement operations:
– Sugeno Class:
– Yager Class:
11
 
1
1
a
c a
a





   
1
1 w w
wc a a 
 Aa x
Where:

12. Introduction to Fuzzy
• Other union operation:
– Yager class:
– Drastic sum:
– Algebraic sum:
12
   
1
, min 1, w w w
ws a b a b
 
  
 
 0,w 
 
, 0
, , 0
1 . .
ds
a b
s a b b a
o w


 


 ,ass a b a b ab  
 Aa x  Bb xWhere:

13. Introduction to Fuzzy
• Other intersection operation:
– Yager class:
– Drastic product:
– Algebraic product:
13
      
1
, 1 min 1, 1 1
w w w
wt a b a b
 
     
  
 
, 1
, , 1
0 . .
dp
a b
t a b b a
o w


 


 ,apt a b ab

14. Introduction to Fuzzy
• Example: Assume that we want to evaluate the health of a
person based on his height and weight.
• The input variables are the crisp numbers of the person’s
height and weight.
• Output is percentage of healthiness.
14
Input Output
Fazzifier Defazzifier
Rule Base
Data Base
Inference
Engine
Crisp Number Crisp Number

15. Introduction to Fuzzy
• Step 1: Fuzzification
15
SlimVery Slim Medium Heavy Very Heavy
50 Kg 75 Kg 100 Kg 125 Kg
Weight
𝜇
1
Very Short Short Medium Tall Very Tall
Height
140 cm 160 cm 180 cm 200 cm
𝜇
1
Input Membership Functions:

16. Introduction to Fuzzy
• Step 2: Rules
• Rules reflect experts decisions
• Rules can be redundant
• Rules can be adjusted to match desired
• Rules are tabulated as fuzzy words
• If x is A then y is B
• if 𝑥1is 𝐴1 and/or 𝑥2 is 𝐴2 … and/or 𝑥 𝑛 is 𝐴 𝑛 then y is B
16

17. Introduction to Fuzzy
• Implications:
– Mamdani implication:
17
1 2
1 2
FP FP
IF FP THEN FP
 
   
     1 2
, min ,QMM FP FPx y x y     
     1 2
, ,QMP FP FPx y prod x y     

18. Introduction to Fuzzy
– Godel implication:
– Zadeh implication:
18
   
 
1 2
2
1 ,
. .
FP FP
QG
FP
x y
y o w
 



 

         1 2 1
, max min , , 1QZ FP FP FPx y x y x      

19. Introduction to Fuzzy
• Inference Rules:
– Modus Ponens:
– Modus Tollens:
– Hypothetical Syllogism:
19
X is A
Y is B
If X is A then Y is B



Y is not B
X is not A
If X is A then Y is B



If X is A then Y is B
If X is A then Z is C
If Y is B then Z is C




20. Introduction to Fuzzy
• Rules are tabulated as fuzzy words
– Healthy (H)
– Somewhat healthy (SH)
– Less Healthy (LH)
– Unhealthy (U)
– Rule function f = {U, LH, SH, H}
20
U LH SH H
0.2 0.4 0.6 0.8 1
f
1
Decision
Output Membership Function:

21. Introduction to Fuzzy
21
Very
Slim
slim Medium Heavy
Very
Heavy
Very Short H SH LH U U
Short SH H SH LH U
Medium LH H H LH U
Tall U SH H SH U
Very Tall U LH H SH LH
WeightHeight
• Fuzzy Rule Table:

22. Introduction to Fuzzy
• Step 3: Calculation
• Assume that height = 187 cm and weight = 49 kg
22
SlimVery Slim Medium Heavy Very Heavy
50 Kg 75 Kg 100 Kg 125 Kg
Weight
𝜇
1
Very Short Short Medium Tall Very Tall
Height
140 cm 160 cm 180 cm 200 cm
𝜇
1
0.3
0.7
0.2
0.8

23. Introduction to Fuzzy
23
0.7 0.3 Medium Heavy
Very
Heavy
Very Short H SH LH U U
Short SH H SH LH U
0.8 LH H H LH U
0.2 U SH H SH U
Very Tall U LH H SH LH
WeightHeight
• Fuzzy Rule Table:

24. Introduction to Fuzzy
24
0.7 0.3 Medium Heavy
Very
Heavy
Very
Short
H SH LH U U
Short SH H SH LH U
0.8 0.7 0.3 H LH U
0.2 0.2 0.2 H SH U
Very
Tall
U LH H SH LH
Weight
Height
   0.7, 0.3, 0, 0, 0Weight VS S M H VH      
   0, 0, 0.8, 0.2, 0Height VS S M T VT      

25. Introduction to Fuzzy
• Scaled Fuzzified Decision:
25
   , , , 0.2, 0.7, 0.2, 0.3f U LH SH H 
U
LH
SH H
0.2 0.4 0.6 0.8 1
f
𝟎. 𝟕
𝟎. 𝟑
𝟎. 𝟐
Decision

26. Introduction to Fuzzy
• Defuzzification Methods:
– Centroid:
– Bisector:
26
0
0
( ) ( )
x
A Ax
x dx x dx


  
0
( )
( )
i A i
A i
x x
x
x


 


27. Introduction to Fuzzy
– Middle, Smallest, and Largest of Maximum:
27
( )A
x
x
SOM LOMMOM
• Defuzzification Methods:

28. Introduction to Fuzzy
• Centroid Method:
28
*
l l
l
y w
y
w
 

0.2 0.2 0.7 0.4 0.2 0.6 0.3 0.8
0.4857
0.2 0.7 0.2 0.3
FD
      
 
  
FD= Final Decision
D: Decision
U
LH
SH H
0.2 0.4 0.6 0.8 1
f
𝟎. 𝟕
𝟎. 𝟑
𝟎. 𝟐
Decision
, 1,...,4
l
y l 
lw

29. Introduction to Fuzzy
• Step 4: Final Decision
• Assume that crisp decision index (D) is centroid:
D=0.4857
29
U LH SH H
0.2 0.4 0.6 0.8 1
f
Decision
1
0.75
0.25
0.4857
25% in SH group and 75% in LH group

30. Introduction to Fuzzy
• Fuzzy Extension Principle:
– How far is it from Zanjan to Urmia?
30
Tabriz Shahrekord
Zanjan 0.3 0.9
Shahrekord 1 0
Esfahan 0.95 0.1
x
y
Urmia Ahvaz
Tabriz 0.95 0.1
Shahrekord 0.1 0.9
y
z
( , ) ( , ), ( , )POQ y P Qx z S t x y y z     
Scaled Distance Among Cities:

31. Introduction to Fuzzy
• Assume that t-norm is product, and s-norm is max
31
0.3 0.95
( , ), ( , ) 0.3 0.95 0.285P Qprod Zanjan Tabriz Tabriz Urmia 
 
    
 
 
0.9 0.1
( , ), ( , ) 0.9 0.1 0.09P Qprod Zanjan Shahrekord Shahrekord Urmia 
 
    
 
 
 max 0.285,0.09 0.285
Zanjan is close to Urmia

32. Introduction to Fuzzy
• TSK Fuzzy System:
32
1 1 0 1 1,..., ...l l l l l l
n n n nIf x is C x is C then y c c x c x   
:l
iC Fuzzy sets
:l
ic Constants
 
l l
l
y w
f x
w
 

1,2,...,l M
 
1
l
i
n
l
iC
i
w x

 Output:

33. Introduction to Neural Network
33
Input
Weight
Σ f
Neuron
Output
Activation Function
• Neural Network:

34. Introduction to Neural Network
• Multilayer Perceptron:
34
InputSignal
OutputSignal
Input Layer
First
Hidden
Layer
Second
Hidden
Layer
Output Layer⋮
⋮
⋮⋮
Supervised
Learning
Random
Initialization
Deep NN

35. Introduction to Neural Network
• MLP:
– Fully connected Overfitting
– Suitable for:
• Classification prediction problems
• Regression prediction problems
• Tabular datasets
– Contain data in a columnar format, each column (field)
must have a name and may only contain data of one type
– Try MLPs on:
• Image data (e.g. the pixels of an image can be reduced down
to one long row of data and fed into an MLP)
• Text data
• Time series data
• Other types of data
35
Supervised
Learning

36. Introduction to Neural Network
• Feed Forward:
36
InputSignal
Output Signal
Input Layer
First
Hidden
Layer
Second
Hidden
Layer
Output Layer
Te
⋮⋮ ⋮
⋮ ⋮

37. Introduction to Neural Network
• Back Propagate:
37
InputSignal
Output Signal
Input Layer
First
Hidden
Layer
Second
Hidden
Layer
Output Layer
Te
Training
⋮ ⋮ ⋮
⋮ ⋮

38. Introduction to Neural Network
38
• Simple Neural Network:
𝑥1
𝑥2
𝑥 𝑛0
𝑁𝑒𝑢𝑟𝑜𝑛1
1
𝑁𝑒𝑢𝑟𝑜𝑛2
1
𝑁𝑒𝑢𝑟𝑜𝑛1
2
1
1
Σ
Σ
Σ
𝑤10
1
𝑤11
2
𝑤10
2
𝑤22
1
𝑤21
1
𝑤20
1
𝑤1𝑛0
1
𝑤11
1
𝑤12
1
𝑤12
2
𝑤2𝑛0
1
f
f
f
𝑜1
2
𝑜2
1
𝑜1
1
𝑛𝑒𝑡1
1
𝑛𝑒𝑡1
2
𝑛𝑒𝑡2
1
…
…
⋮

39. Introduction to Neural Network
39
= 𝑤10
1
, 𝑤11
1
, 𝑤12
1
, . . . , 𝑤1𝑛0
1 𝑇
= 𝑤20
1
, 𝑤21
1
, 𝑤22
1
, . . . , 𝑤2𝑛0
1 𝑇
= 𝑤10
2
, 𝑤11
2
, 𝑤12
2 𝑇
𝑛𝑒𝑡1
1
=
𝑖=0
𝑛0
𝑤1𝑖
1
𝑥𝑖
𝑛𝑒𝑡2
1
=
𝑖=0
𝑛0
𝑤2𝑖
1
𝑥𝑖
𝑜1
1
= 𝑓(𝑛𝑒𝑡1
1
𝑜2
1
= 𝑓(𝑛𝑒𝑡2
1
𝑛𝑒𝑡1
2
=
𝑖=0
2
𝑤1𝑖
2
𝑜𝑖
1
𝑜1
2
= 𝑓(𝑛𝑒𝑡1
2
= 𝑜0
1
, 𝑜1
1
, 𝑜2
1 𝑇
𝑜0
1
= 1, 𝑥0 = 1
𝑤1
1
𝑤2
1
𝑤1
2
𝑜1
𝑥 = 𝑥0, 𝑥1, 𝑥2, . . . , 𝑥 𝑛0
𝑇
= 𝑤1
1 𝑇. 𝑥
= 𝑤2
1 𝑇. 𝑥
= 𝑤1
2 𝑇
. 𝑜1
Feed Forward

40. Introduction to Neural Network
40
𝑥1
𝑥2
𝑥 𝑛0
𝑁𝑒𝑢𝑟𝑜𝑛1
1
𝑁𝑒𝑢𝑟𝑜𝑛2
1
𝑁𝑒𝑢𝑟𝑜𝑛1
2
1
1
Σ
Σ
Σ
𝑤10
1
𝒘 𝟏𝟏
𝟐
𝒘 𝟏𝟎
𝟐
𝑤22
1
𝑤21
1
𝑤20
1
𝑤1𝑛0
1
𝑤11
1
𝑤12
1
𝒘 𝟏𝟐
𝟐
𝑤2𝑛0
1
f
f
f
𝒐 𝟏
𝟐
𝑜2
1
𝑜1
1
𝑛𝑒𝑡1
1
𝒏𝒆𝒕 𝟏
𝟐
𝑛𝑒𝑡2
1
…
…
T
e
Back Propagate
'2 1
2 2
1 1
2 2 2 2
1 1 1 1
1e
f o
E E e o net
w e o net w
 
    
   
     
⋮

41. Introduction to Neural Network
41
𝑥1
𝑥2
𝑥 𝑛0
𝑁𝑒𝑢𝑟𝑜𝑛1
1
𝑁𝑒𝑢𝑟𝑜𝑛2
1
𝑁𝑒𝑢𝑟𝑜𝑛1
2
1
1
Σ
Σ
Σ
𝑤10
1
𝒘 𝟏𝟏
𝟐
𝒘 𝟏𝟎
𝟐
𝑤22
1
𝑤21
1
𝑤20
1
𝑤1𝑛0
1
𝑤11
1
𝑤12
1
𝒘 𝟏𝟐
𝟐
𝑤2𝑛0
1
f
f
f
𝒐 𝟏
𝟐
𝑜2
1
𝑜1
1
𝑛𝑒𝑡1
1
𝒏𝒆𝒕 𝟏
𝟐
𝑛𝑒𝑡2
1
…
…
T
e
Back Propagate
'2 2 '1
11
2 2 1 1
1 1 1 1
1 2 2 1 1 1
1 1 1 1 1 1
1e xf w f
E E e o net o net
w e o net o net w
 
      
     
       
⋮

42. Introduction to Neural Network
42
  21
2
i
i
E k e 
   
 
 
1
E k
w k w k
w k


  

Gradient Descent
Exercise: Rewrite the back-propagation equations for a
neural network with 2 outputs.

43. Introduction to Neural Network
• Popular Activation Functions:
– Sigmoid (Logistic):
• Sigmoids saturate and tend to vanish gradient
• Exp() is a bit compute expensive
• Sigmoid outputs are not zero-centered
43
 
1
1 x
x
e
 


   0,1x 

44. Introduction to Neural Network
– Tanh:
• Zero centered
• Tanh saturate and tend to vanish gradient
• Tanh() is a bit compute expensive
44
 
2
2
1
1
x
x
e
f x
e





   1,1f x  

45. Introduction to Neural Network
– ReLU:
• Rectified Linear Unit
• Does not saturate (in range +)
• Very computationally efficient
• Converges faster than sigmoid/tanh
• Not zero-centered output
• Saturate (in range -)
45
   max 0,f x x
   0,f x  

46. Introduction to Neural Network
– LReLU and PReLU:
• Does not saturate
• Computationally efficient
• Converges much faster
• Zero-centered output
46
   max 0.01 ,f x x x
   max ,f x x x
LReLU:
PReLU:

47. Introduction to Neural Network
– ELU:
• Exponential Linear Unit
• Zero-centered output
• Closer to zero mean output
• Robustness to noise compared with LReLU
47
 
  
0
exp 1 0
x x
f x
x x

 
 

48. Introduction to Neural Network
• Properties of Activation Functions:
– Nonlinear
– Continuously differentiable
– Range
– Monotonic
– Smooth
• Bipolar and Unipolar:
– Unipolar Sigmoid
– Bipolar Sigmoid
48
f(net)=
𝟏
𝟏+𝒆−𝒈.𝒏𝒆𝒕
f(net)=
𝟏−𝒆−𝒈.𝒏𝒆𝒕
𝟏+𝒆−𝒈.𝒏𝒆𝒕

49. Introduction to Neural Network
• Flexible Neural Network:
49
        .
1
,
1
s
j
s
j
s
j g k
s s
j j net k
f gnet k k
e



   
 
 
1s s s
g s
E k
g k g k
g k


  

    
   
   
.
.
1
,
1
s
j
s
j
s
j
s
j
g k
s
j g k
net k
s s
j j net k
e
f n g ket k
e





Unipolar Sigmoid:
Bipolar Sigmoid:
Training:

50. Introduction to Neural Network
50
𝑥1
𝑥2
𝑥 𝑛0
𝑁𝑒𝑢𝑟𝑜𝑛1
1
𝑁𝑒𝑢𝑟𝑜𝑛2
1
𝑁𝑒𝑢𝑟𝑜𝑛1
2
1
1
Σ
Σ
Σ
𝑤10
1
𝒘 𝟏𝟏
𝟐
𝒘 𝟏𝟎
𝟐
𝑤22
1
𝑤21
1
𝑤20
1
𝑤1𝑛0
1
𝑤11
1
𝑤12
1
𝒘 𝟏𝟐
𝟐
𝑤2𝑛0
1
f
f
f
𝒐 𝟏
𝟐
𝑜2
1
𝑜1
1
𝑛𝑒𝑡1
1
𝑛𝑒𝑡1
2
𝑛𝑒𝑡2
1
…
…
T
e
Back Propagate
*2
2
1
2 2 2
1
1e
f
E E e o
g e o g

   
  
   
⋮

51. Introduction to Neural Network
51
Unipolar Sigmoid:
Bipolar Sigmoid:
Training:
    
 
   .
2
,
1
ss
j j
s
js
j a k
s s
j j net k
f
a
net k
k
a k
e



      
   
   
.
.
1 1
,
1
s
j
s
j j
s
j
s
n
kn
a k
s
et k
s s
j j s et k
j
j a
e
f net k
a k e
a k



 

   
 
 
1s s s
a s
E k
a k a k
a k


  

• Flexible Neural Network:
  1s
jg k 

52. Introduction to Neural Network
52
𝑥1
𝑥2
𝑥 𝑛0
𝑁𝑒𝑢𝑟𝑜𝑛1
1
𝑁𝑒𝑢𝑟𝑜𝑛2
1
𝑁𝑒𝑢𝑟𝑜𝑛1
2
1
1
Σ
Σ
Σ
𝑤10
1
𝒘 𝟏𝟏
𝟐
𝒘 𝟏𝟎
𝟐
𝑤22
1
𝑤21
1
𝑤20
1
𝑤1𝑛0
1
𝑤11
1
𝑤12
1
𝒘 𝟏𝟐
𝟐
𝑤2𝑛0
1
f
f
f
𝒐 𝟏
𝟐
𝑜2
1
𝑜1
1
𝑛𝑒𝑡1
1
𝒏𝒆𝒕 𝟏
𝟐
𝑛𝑒𝑡2
1
…
…
T
e
Back Propagate
*1
'2 2
11
2 2 1
1 1 1
1 2 2 1 1
1 1 1
1e ff w
E E e o net o
a e o net o a

     
    
     
⋮

53. Introduction to Neural Network
• Radial Basis Function (RBF):
– Similarity between input signal and prototype
53
⋮
𝑥1
𝑥n
𝑥2
𝑤 y
⋮
Gaussian Activation Function
⋮ ⋮

54. Introduction to Neural Network
54
Σ
Σ
Σ
.
.
.
Neuron 1
Neuron m
Neuron j⋮
⋮
Σ y
𝑤1
𝑤 𝑚
𝑤𝑗
𝑛𝑒𝑡1
𝑛𝑒𝑡𝑗
𝑛𝑒𝑡 𝑚
𝑜1
1
𝑜 𝑚
1
𝑜𝑗
1
𝜙1 .
𝜙 𝑚 .
𝜙𝑗 .
−𝑐1
−𝑐𝑗
−𝑐 𝑚
𝑥
• RBF:
⋮

55. Introduction to Neural Network
55
𝑛𝑒𝑡𝑗 = ‖𝑥 − 𝑐𝑗‖ = 𝑥1 − 𝑐1𝑗
2
+ 𝑥2 − 𝑐2𝑗
2
+. . . + 𝑥 𝑛 − 𝑐 𝑛𝑗
2
𝑜𝑗
1
= 𝜙 𝑛𝑒𝑡𝑗
𝜙 𝑛𝑒𝑡𝑗 = 𝑒
−1
2
𝑛𝑒𝑡 𝑗
𝜎 𝑗
2
𝑥 = 𝑥1, 𝑥2, . . . , 𝑥 𝑛
𝑐𝑗 = 𝑐1𝑗, 𝑐2𝑗, . . . , 𝑐 𝑛𝑗
𝜎 = 𝜎1, 𝜎2, . . . , 𝜎 𝑚
′
𝑤 = 𝑤1, 𝑤2, . . . , 𝑤 𝑚
′
𝑜1 = 𝑜1
1
, 𝑜2
1
, . . . , 𝑜 𝑚
1 ′

56. Introduction to Neural Network
• Training of RBF Networks:
–
–
56
𝑐𝑗 𝑘 + 1 = 𝑐𝑗 𝑘 − 𝜂
𝜕𝐸
𝜕𝑐𝑗
𝑘  
      
  
 
 
1
22
1
1
1
j j
j
j
j
j j j
e
w k x k c k
o k
k
oE E e y
k k
c e y o c




   
   
    
𝜎𝑗 𝑘 + 1 = 𝜎𝑗 𝑘 − 𝜂
𝜕𝐸
𝜕𝜎𝑗
𝑘  
   
  
 
 
2
1
3
1
1
1
j
j
j
j
j j j
e
w k net
o k
k
oE E e y
k k
e y o

 


   
   
    
𝒄𝒋 :
𝝈𝒋 :

57. Introduction to Neural Network
• Recurrent Neural Network (RNN):
57
y(k+1)
x(k)
x(k-1)
x(k-2)
𝑧−1
𝑧−1
𝑧−1
𝑧−1
𝑧−1
b
y(k-2)
y(k-1)
y(k)

58. Introduction to Neural Network
• Feedback Types in Recurrent Neural Network:
– Local (A)
– Inter-layer (B)
– Global (C)
58
X(k) y(k)
A
B
C

59. 𝑤𝑥X(k) y(k)
𝑤 𝑟,1
𝑤 𝑟,2
𝑤 𝑟,𝑛1
Introduction to Neural Network
• Local feedback activation:
59
Feed Forward
     
1
x r
i
net k w x k w net k i

     

60. Introduction to Neural Network
• Local output feedback:
60
𝑤𝑥X(k) y(k)
𝑤 𝑟,1
𝑤 𝑟,2
𝑤 𝑟,𝑛1
Feed Forward
     
0 1
x r
j i
net k w x k j w y k i
 
     
    y k f net k

61. Introduction to Neural Network
• The Vanishing Gradient Problem:
– Causes small gradients
– Prevents the weights from updating
61
0.29
0.28999
InputSignal
OutputSignal

62. Introduction to Neural Network
• The Exploding Gradient Problem:
– Causes large gradients
– The weights get away from their optimum value
62
0.29
1.872351
InputSignal
OutputSignal

63. Introduction to Neural Network
63
⋮
• Elman Neural Network:
⋮
⋮⋮
𝑥1
𝑥2
𝑥 𝑛0
𝑥 𝑐1
𝑥 𝑐𝑛1
𝑥 𝑐2
𝑓1
𝑓1
𝑓1
𝑓2
𝑓2
𝑓2
cw
xw yw
𝑜1
1
𝑜 𝑛1
1
⋮
𝑒1
𝑒 𝑛2
𝑒2
Hidden Layer 1
𝑇1
𝑇2
𝑇𝑛2
⋮
⋮
𝑛1 𝑛2

64. Introduction to Neural Network
• Elman Neural Network:
64
Feed Forward
    
        
1 1 1
1 1
1 1
1 1 1 1x c
o k f net k
f w k x k w k o k
   
       
 1
1cx o k  
       2 2 2 2 1
yo K f net k f w o k   

65. Introduction to Neural Network
• Elman Neural Network:
65
Back Propagate
   
2
2
1
1
2
n
j
j
E k e k

 
'2
1
2 2
2 2
1
y y
e f
o
E E e o net
w e o net w


    
   
      
   
 
 
1y y
y
E k
w k w k
w k


  

In the same way for 𝑤 𝑥

66. Introduction to Neural Network
• Jordan Neural Network:
66
⋮⋮
𝑥1
𝑥2
𝑥 𝑛0
𝑥 𝑐1
𝑥 𝑐𝑛2
𝑥 𝑐2
𝑓1
𝑓1
𝑓1
𝑓2
𝑓2
𝑓2
cw
xw yw
𝑜1
1
𝑜 𝑛2
1
𝑦 𝑛2
𝑦2
𝑦1
⋮
⋮
⋮
𝑛1
𝑛2
Output Layer