Neural Networks References: Data mining techniques by arun k. pujari MRS.SOWMYA JYOTHI SDMCBM MANGALORE

1. Neural Networks
REFERENCES: DATA MINING TECHNIQUES BY ARUN K. PUJARI
MRS.SOWMYA JYOTHI
SDMCBM
MANGALORE

2. Neural network are different paradigm for computing,
which draws its inspiration from neuroscience.
The human brain consists of a network of neurons,
each of which is made up of a number of nerve fibres
called dendrites, connected to the cell body where the
cell nucleus is located.
The axon is a long, single fibre that originates from the
cell body that branches near its end into a number of
strands.
At single axon typically makes synapses with other
neurons.
The transmission process is a complex chemical process
which effectively increases or decreases the electrical
potential within the cell body of the receiving neuron.

3. Neuron vs. Node

4. Artifical neurons are highly simplified model of biological
neurons.
Artifical neural network are densly interconnected networks
PEs together with a rule to adjust the strength of the
connections between the units in response to externally
supplied data.
The network has 2 binary input, I0 and I1 and one binary
output Y.
W0 and W1 are the connection strengths of input 1 and input
2 respectively.
Thus the total input received at the processing unit is given by
W0I0+W1I1-Wb
Where Wb is the threshold.
The output Y takes on the value 1,if W0I0+W1I1-Wb >0 and,
otherwise, it is 0 if W0I0+W1I1-Wb≤0

5. But the model, known as perceptron, was far
from a true model of a biological neuron as,
for a start, the, biological neuron’s output is a
continuous function rather than a step function.
This model also has a limited computational
capability as it represents only a linear-
separation.

6. There have been many improvements on this simple model and
much architecture has been presented in recently.
The threshold function or the step function is replaced by more
continuous functions called activation functions.
For this particular node n, weighted inputs(denoted Wi,
i=1………n) are combined via a combination function that consists
of a simple summation.
A transfer function then calculates a corresponding value the
result yielding a single output, usually between 0 and 1. together,
the combination function and the transfer function make up the
activation function of the node.
Three common transfer function are the sigmoid,linear and
hyperbolic functions. The sigmoid function is very widely used and
it produces values between 0 and 1 for any input from the
combination function.

7. The Neuron
Neural Networks NN 1 7
Input
Summing
function
Activation
function
Output
y
x1
x2
xm
w2
wm
w1
 
 )
(


8. Individual nodes are linked together in different
ways to create neural networks.
In a feed-forward network, the connections
between layers are unidirectional from input to
output.
two different architectures of the feed-forward
network, multi-layer Perceptron and Radial-
Basis Function.

9. MULTI-LAYER PERCEPTRON(MLP)
MLP is a development from the simple perceptron
in which extra hidden layers are added
More than one hidden layer can be used.
The network topology is a constrained to be
feedforward,ie.,loop free.
Connections are allowed from the input layer to
the first hidden layer; the to second and so on,
until the last hidden layer is the output layer.

10. Multi layer feed-forward
Neural Networks NN 1 10
Input
layer
Output
layer
Hidden Layer
3-4-2 Network

11. Single Layer Feed-forward
Neural Networks NN 1 11
Input layer
of
source nodes
Output layer
of
neurons

12. RADIAL BASIS FUNCTION NETWORKS
Radial Basis Function (RBF) networks
feedforward, but have only one hidden layer.
Like MLP, RBF nets can learn arbitrary mappings;
the primary difference is in the hidden layer.
RBF hidden layer units have a receptive field
;that is, a particular input value at which they
have a maximal output.

13. LEARNING IN NN
In order to fit a particular ANN to a particular
problem, it must be trained to generate a correct
response for a given set of inputs.
1. Unsupervised training may be used when a clear
link between input data sets and the targets
output values does not exist.
2. Supervised training involves providing an ANN
with specified input and output values and
allowing it to iteratively reach a solution.
MLP and RBF employ the supervised learning.

14. PERCEPTRON LEARNING RULE
This is the first learning scheme of neural computing.
The weights are changed by an amount proportional to the
difference between the desired output and the actual output.
if W is the weight vector and
Wi is the change in the ith weight
learning rate parameter is used to decide the magnitude of
change.
If learning rate is high, the change in the weight is bigger at every
step. The rule is given by
Wi=ξ(D-Y).Ii
Where ξ is the learning rate
D is the desired output and
Y is the actual output.

15. TRAINING IN MLP
The multi-layer perceptron overcomes the above shortcoming of the
single layer perceptron.
but learning in MLP is not trival.
The idea is to carry out the computation layer wise,moving in the
forward direction.
The weight adjustment can be done layer wise by moving in a
backward direction.
For the nodes in the output layer, it is easy to compute the error as
we know the actual outcome and desired result.
For the nodes in the hidden layers ,since we donot know the desired
result,we propogate the error computed in the last layer backward.
This standard method used in training MLPs is called the back
propogation algorithm.

16. The learning steps consist of the
1. Forward pass
The output and the error at the output
units are calculated.
2. Backward pass
The output unit error is used to alter
weights on the output units. Then the error at
the hidden nodes is calculated and weights on
the hidden nodes are altered using these values

17. TRAINING RBF NETWORKS
The RBF design involves deciding on their centres and
the sharpness of their Gaussians.
The centres and SDs(standard deviation) are decided first
by examining the vectors in training data.
RBF networks are trained in a similar way as MLP.
The output layer weights are trained using the delta
rule.
MLP is the most widely applied neural network
technique.
RBF have advantage that one can add extra units with
their centers near parts of the input, which are difficult to
classify.

18. UNSUPERVISED LEARNING
Simple perceptron, MLP and RBF networks
are unsupervised networks.
In unsupervised mode, the network
adapts purely in response to its inputs.
Such networks can learn to pick structures
in their input.
One of the most popular models in the
unsupervised framework is the self-
organizing map(SOM).
.

19. COMPETITIVE LEARNING
Competitive learning or winner takes all may be
regarded as the basis of a number of unsupervised
learning strategies.
A competitive learning consists of k units with
weight vectors of equal dimension to the input
data. The unit with the closest weight vector is
termed as the winner of the selection process. This
learning strategy is generally implemented by
gradually reducing the difference between the
weight vector and input vector.
The actual amount of reduction at each learning
step may be guided by means of the so -called
learning rate

20. Supervised vs. Unsupervised Learning
Supervised learning (classification)
◦ Supervision: The training data (observations, measurements,
etc.) are accompanied by labels indicating the class of the
observations
◦ New data is classified based on the training set
Unsupervised learning (clustering)
◦ The class labels of training data is unknown
◦ Given a set of measurements, observations, etc. with the aim
of establishing the existence of classes or clusters in the data
20

21. KOHONEN’S SOM
The self- organizing map(SOM) was a neural network model developed by
Teuvo Kohonen during 1979-82.
SOM is one of the most widely used unsupervised NN models and employs
competitive learning steps.
It consist of the input units, each of which is fully connected to a set of
output units.
The input units, after receiving the input patterns X, propogate them as they
are onto the output units. Each of the output units k is assigned a weight
vector wk. During the learning step, the unit c corresponding to the highest
activity level with respect to a randomly-selected input pattern X, is adapted in
a such a way that it exhibits an even higher activity level at a future
presentation of X. During the learning steps of SOM , a set of units around
the winner is tuned towards the currently presented input pattern enabling a
spatial arrangement of the input patterns, such that similar inputs are
mapped onto regions close to each other in the grid of output units. Thus,
the training process of SOM results in a topological organization of the input
patterns. It is ,in some sense related to k-means clustering.

22. SOM is one of the most widely used unsupervised NN models and
employs competitive learning steps.
It consist of the input units, each of which is fully connected to a set of
output units.
The input units, after receiving the input patterns X, propogate them as
they are onto the output units. Each of the output units k is assigned a
weight vector wk. During the learning step, the unit c corresponding to
the highest activity level with respect to a randomly-selected input
pattern X, is adapted in a such a way that it exhibits an even higher
activity level at a future presentation of X. During the learning steps of
SOM , a set of units around the winner is tuned towards the currently
presented input pattern enabling a spatial arrangement of the input
patterns, such that similar inputs are mapped onto regions close to
each other in the grid of output units. Thus, the training process of SOM
results in a topological organization of the input patterns. It is ,in some
sense related to k-means clustering.

23. Self-Organizing Maps (Kohonen Maps)
November 24, 2009
INTRODUCTION TO COGNITIVE SCIENCE
LECTURE 21: SELF-ORGANIZING MAPS
23
Topology-conserving mapping can be achieved by
SOMs:
• Two layers: input layer and output (map) layer
• Input and output layers are completely connected.
• Output neurons are interconnected within a defined
neighborhood.
• A topology (neighborhood relation) is defined on
the output layer.

24. APPLICATIONS OF NEURAL NETWORKS
Neural networks are used in a very large number of
applications. Neural networks are being used in
Investment analysis: To predict the movement of
stocks,currencies etc..,from previous data. There.they are
replacing earlier simpler linear models.
Monitoring: Networks have been used to monitor the state
of aircraft engines. By monitoring vibration levels and
sound, as early warning of engine problems can be given.
Marketing: Neural networks have been used to improve
marketing mailshots. One technique is to run a test
mailshot.and look at the patern of returns from this. The
idea is to find predictive mapping from the data known
about clients to how they have responded. This mapping is
then used to direct further mailshots.

25. Additional Points: A self-organizing map (SOM) or self-organizing feature map
(SOFM) is a type of artificial neural network (ANN) that is trained using
unsupervised learning to produce a low-dimensional (typically two-dimensional),
discretized representation of the input space of the training samples, called a map.
Self-organizing maps are different from other artificial neural networks in the sense
that they use a neighborhood function to preserve the topological properties of the
input space.
A self-organizing map consists of components called nodes or neurons. Associated
with each node is a weight vector of the same dimension as the input data vectors
and a position in the map space. The usual arrangement of nodes is a two-
dimensional regular spacing in a hexagonal or rectangular grid. The self-organizing
map describes a mapping from a higher-dimensional input space to a lower-
dimensional map space.
The Kohonen Self-Organizing Feature Map (SOFM or SOM) is a clustering and data
visualization technique based on a neural network viewpoint. As with other types of
centroid-based clustering, the goal of SOM is to find a set of centroids (reference or
codebook vector in SOM terminology) and to assign each object in the data set to the
centroid that provides the best approximation of that object