This document outlines a course on neural networks and fuzzy systems. The course is divided into two parts, with part one focusing on neural networks over 11 weeks, covering topics like perceptrons, multi-layer feedforward networks, and unsupervised learning. Part two focuses on fuzzy systems over 4 weeks, covering fuzzy set theory and fuzzy systems. The document also provides details on concepts like linear separability, decision boundaries, perceptron learning algorithms, and using neural networks to solve problems like AND, OR, and XOR gates.

1. Neural Networks and
Fuzzy Systems
Single layer Perception Classifier
Dr. Tamer Ahmed Farrag
Course No.: 803522-3

2. Course Outline
Part I : Neural Networks (11 weeks)
• Introduction to Machine Learning
• Fundamental Concepts of Artificial Neural Networks
(ANN)
• Single layer Perception Classifier
• Multi-layer Feed forward Networks
• Single layer FeedBack Networks
• Unsupervised learning
Part II : Fuzzy Systems (4 weeks)
• Fuzzy set theory
• Fuzzy Systems
2

3. Building Neural Networks Strategy
• Formulating neural network solutions for particular
problems is a multi-stage process:
1. Understand and specify the problem in terms of inputs and
required outputs
2. Take the simplest form of network you think might be able to
solve your problem
3. Try to find the appropriate connection weights (including neuron
thresholds) so that the network produces the right outputs for
each input in its training data
4. Make sure that the network works on its training data and test its
generalization by checking its performance on new testing data
5. If the network doesn’t perform well enough, go back to stage 3
and try harder.
6. If the network still doesn’t perform well enough, go back to stage
2 and try harder.
7. If the network still doesn’t perform well enough, go back to stage
1 and try harder.
8. Problem solved – or not.
3

4. Decision Hyperplanes and Linear Separability
• If we have two inputs, then the decision boundary that is
a one dimensional straight line in the two dimensional
input space of possible input values.
• In general, A set of points in n-dimensional space are
linearly separable if there is a hyperplane of
(n − 1) dimensions that separates the sets.
• This hyperplane is clearly still linear (i.e., straight or flat
or non-curved) and can still only divide the space into two
regions.
• Problems with input patterns that can be classified using a
single hyperplane are said to be linearly separable.
Problems (such as XOR) which cannot be classified in this
way are said to be non-linearly separable.
4

5. Linear separated vs Non linear separated
binary Classification problems
Linear separated
Non linear separated
Class 2
Class 1
Non linear separated
Non linear separated Non linear separated

6. Decision Boundary for some of Logic Gates
6
outx2x1
000
010
001
111 A
x1
A
x1
x2
x1
outx2x1
000
110
101
111
outx2x1
000
110
101
011
out = 2
out= 1
AND Gate
Linear separated
OR Gate
Linear separated
XOR Gate
Non Linear separated

7. Implementation of Logical NOT, AND, and OR using
McCulloch-Pitts neuron
7
outx2x1
000
010
001
111
outx2x1
000
110
101
111
AND ORNOT
outx1
10
01

8. How to Finding Weights and Threshold?
• Constructing simple networks by hand (e.g., by
trial and error) is one thing. But what about harder
problems?
• How long should we keep looking for a solution?
We need to be able to calculate appropriate
parameter values rather than searching for
solutions by trial and error.
• Each training pattern produces a linear inequality
for the output in terms of the inputs and the
network parameters. These can be used to
compute the weights and thresholds.
8

9. Finding Weights Analytically for the AND Network?
9
x1
x2
W1=?
W2=?
out
T=?
outx2x1
000
010
001
111
equations
0 * w1 + 0 * w2<T
0 w1 + 1 * w2<T
1 * w1 + 0 * w2<T
1 * w1 + 1 * w2≥ T
F(z)=
0 𝑓𝑜𝑟 𝑥 < 𝑇
1 𝑓𝑜𝑟 𝑥 ≥ 𝑇
𝑤ℎ𝑒𝑟𝑒 𝑧 = (𝑥1 ∗ 𝑤1 + 𝑥2 ∗ 𝑤2)
equations
T >0
w2< T
w1 < T
w1 + w2≥ T
Easy to solve
(infinite number of solutions)
For example assume:
T=1.5 , w1=1 , w2=1

10. Finding Weights Analytically for the XOR Network?
10
x1
x2
W1=?
W2=?
out
T=?
outx2x1
000
110
101
011
equations
0 * w1 + 0 * w2<T
0 w1 + 1 * w2≥T
1 * w1 + 0 * w2≥T
1 * w1 + 1 * w2< T
F(z)=
0 𝑓𝑜𝑟 𝑥 < 𝑇
1 𝑓𝑜𝑟 𝑥 ≥ 𝑇
𝑤ℎ𝑒𝑟𝑒 𝑧 = (𝑥1 ∗ 𝑤1 + 𝑥2 ∗ 𝑤2)
equations
T >0 (1)
w2 ≥ T (2)
w1 ≥ T (3)
w1 + w2 < T (4)
No Solution
Why?
How to solve this problem ?

11. Useful Notation
• We often need to deal with ordered sets of numbers, which we write as
vectors, e.g.
x = (x1, x2, x3, …, xn) , y = (y1, y2, y3, …, ym)
• The components xi can be added up to give a scalar (number), e.g.
s = x1 + x2 + x3 + … + xn =
𝑖=1
𝑛
𝑥𝑖
• Two vectors of the same length may be added to give another vector,
e.g.
z = x + y = (x1 + y1, x2 + y2, …, xn + yn)
• Two vectors of the same length may be multiplied to give a scalar, e.g.
p = x.y = x1y1 + x2 y2 + …+ xnyn =
𝑖=1
𝑛
𝑥𝑖 𝑦𝑖
11

12. What is the problem of implementing XOR?
• In the previous slide, Clearly the first, second and
third inequalities are incompatible with the fourth,
so there is in fact no solution.
• We need more complex networks, e.g. that combine
together many simple networks, or use different
activation/thresholding/transfer functions.
• It then becomes much more difficult to determine
all the weights and thresholds by hand. Next, we
will see how a neural network can learn these
parameters.
12

13. Perceptron
• In 1958, Frank Rosenblatt introduced a training algorithm that
provided the first procedure for training a simple ANN: a perceptron.
• Any number of McCulloch-Pitts neurons can be connected together in
any way we like.
• The arrangement that has one layer of input neurons feeding forward
to one output layer of McCulloch-Pitts neurons, with full connectivity, is
known as a Perceptron:
13
𝒊= 𝟎
𝒏
𝒘𝒊 𝒙𝒊 +𝒃
1
0
X1
X2
X3
w1
w2
w3
Output
𝑶𝒖𝒕𝒑𝒖𝒕 = 𝒉𝒂𝒓𝒅𝒍𝒊𝒎
𝒊=𝟎
𝒏
𝒘𝒊 𝒙𝒊 + 𝒃

14. Perceptron
• In 1958, Frank Rosenblatt introduced a training algorithm
that provided the first procedure for training a simple ANN:
a perceptron.
• Any number of McCulloch-Pitts neurons can be connected
together in any way we like.
• The arrangement that has one layer of input neurons
feeding forward to one output layer of McCulloch-Pitts
neurons, with full connectivity, is known as a Perceptron.
• It using hard limiter as the activation function.
• There are a learning rule (algorithm) to adjust the weights
for better results
14

15. Single layer feedforward (Perceptron)
• The following figure showing a perceptron network
with n inputs(organized in one input layer) and m
output (organized in one output layer) no hidden
layers :
15
i j
1
2
3
Wij
n
1
2
3

16. Perceptron
• Simple network:
The following figure showing a perceptron network with
2 input(organized in one input layer) and 1 output
(organized in one output layer) :
• Important Remark: for binary classification problems
we always has single neuron in the output layer
16
b
𝒊= 𝟎
𝒏
𝒘𝒊 𝒙𝒊 +𝒃
1
0
X1
X2
w1
w2
1
output

17. Fixed Increment Perceptron Learning
Algorithm
• The problem: using the perceptron to perform binary
classification task
• The goal: adjust the weights and bias to some values by which
the percentage of the classification error decreased.
• How: by training the perceptron using pre- classified examples
(supervised learning).
• the desired output is the correct output should be generated (D)
• Network calculates the output (Y) which may be wrong and need
to be recalculated after adjust the network parameters.
• Normally , we starts random initial weights and adjust them in
small steps (using the learning algorism) until the required
outputs are produced.
17

18. Fixed Increment Perceptron Learning
Algorithm (cont.)
18
Calculating the new weights:
• Calculate the error (ej)
𝑒𝑗 = 𝐷𝑗 − 𝑌𝑗
• Network changes its weights in proportion to the error:
Δ 𝑤𝑖𝑗 = 𝛼 ∗ 𝑒𝑗 ∗ 𝑥𝑖
• Where  is the learning rate or step size :
1. Used to control the amount of weight adjustment at each step of training.
2. ranges from 0 to 1.
3. determines the rate of learning in each time step.
• The new (adjusted) weight:
𝑤𝑖𝑗
𝑛𝑒𝑤
= 𝑤𝑖𝑗
𝑜𝑙𝑑
+ Δ 𝑤𝑖𝑗
Or , 𝑤𝑖𝑗
𝑛𝑒𝑤
= 𝑤𝑖𝑗
𝑜𝑙𝑑
+ 𝛼 𝑒𝑗 𝑥𝑖
• This rule can be extended to train the bias by noting that a bias is
simply a weight whose input is always 1
𝑏𝑗
𝑛𝑒𝑤
= 𝑏𝑗
𝑜𝑙𝑑
+ 𝛼 𝑒𝑗

19. Fixed Increment Perceptron Learning Algorithm flow chart
19
random initial weights
Calculate the output (Yj) using
Training Examples
Calculate Error
𝑒𝑗 = 𝐷𝑗 − 𝑌𝑗
Error =0
End
Update Weights and biases
Yes
No

20. Perceptron
• The perceptron is a linear classifier. (Why?)
• The Perceptron algorithm rule is guaranteed to
converge to a weight vector that correctly
classifies the examples provided the training
examples are linearly separable.
• To get the correct weights, many training epochs
are used with suitable learning rate α
• So, it can’t be used to solve Non linear separated
(such as XOR problem)
20

21. Example: AND problem
• The AND problem is linear separated problem Has
2 inputs(X1 , X2) and one output (out)
21
outx2x1
000
010
001
111
x1
x2
out
bias
w1
w2
1

22. Example of perceptron learning: the logical operation AND
22
Assume α =1 ,initial values : ( b=1 , w1=0.3 , w2=-1)
Final
bias
Final weights
Error
(e)
Actual
Output
(Y)
bias
Initial weights
desired
output (D)
input
epoch
w2w1w2w1x2X1
0-10.3-111-10.3000
1
0-10.3000-10.3010
-1-1-0.7-110-10.3001
000.310-1-1-0.7111
-100.3-11000.3000
2
-100.300-100.3010
-100.300-100.3001
011.310-100.3111
-111.3-11011.3000
-201.3-11-111.30103
-201.300-201.3001
-112.310-201.3111
.
.
.
.
-312.300-312.3000
-312.300-312.30107
-312.300-312.3001
-312.301-312.3111

23. More Examples
• Try another initial values and learning rate
• Try another linear separated functions such as OR ,
NAND, NOR.
• What do you notice?
23

24. Example: Trucks Classification problems
24
• Consider the simple example of classifying trucks given their masses and
lengths
• How do we construct a neural network that can classify any Lorry and
Van?
Mass Length Class
10 6 Lorry
20 5 Lorry
5 4 Van
2 5 Van
2 5 Van
3 6 Lorry
10 7 Lorry
15 8 Lorry
5 9 Lorry
0
1
2
3
4
5
6
7
8
9
10
0510152025

25. Solution
• The trucks classification problem has the folowing
features:
• It is is a binary classification problem ( 2 category lorry
or van)
• As shown, it is linear separated problem
• Has 2 inputs(mass , length)
• Has one output (Class)
• So, It can be solved by single layer perceptron as
shown:
Check: TrunkExample.ipynb
25
mass
length
class
bias
w1
w2
1

26. Overcome Perceptron the limitations
• To overcome the limitations of single layer
networks, multi-layer feed-forward networks can
be used, which not only have input and output
units, but also have hidden units that are neither
input nor output units.
• Using Non-linear Activation functions (e.g.
sigmoid).
• Using advanced learning algorithms (e.g. Gradian
descent, backpropagation).
26