SlideShare a Scribd company logo
1 of 32
CCS355
NEURAL NETWORKS & DEEP LEARNING
UNIT – 2 NOTES
BE
III YEAR – VI SEM (R21)
(2023-2024)
Prepared
By
Asst.Prof.M.Gokilavani
Department of Information Technology
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
UNIT II - ASSOCIATIVE MEMORY AND UNSUPERVISED LEARNING NETWORKS
Training Algorithms for Pattern Association-Auto associative Memory Network-Hetero associative
Memory Network-Bidirectional Associative Memory (BAM)-Hopfield Networks-Iterative Auto
associative Memory Networks-Temporal Associative Memory Network-Fixed Weight Competitive Nets-
Kohonen Self-Organizing Feature Maps-Learning Vector Quantization-Counter propagation Networks-
Adaptive Resonance Theory Network.
1. INTRODUCTION:
 An associative memory network can store a set of patterns as memories.
 When the associative memory is being presented with a key pattern, it responds by producing one of
the stored patterns, which closely resembles or relates to the key pattern.
 Thus, the recall is through association of the key pattern, with the help of information memorized.
 These types of memories are also called as content-addressable memories (CAM).
 The CAM can also be viewed as associating data to address, i.e.; for every data in the memory there
is a corresponding unique address.
 Also, it can be viewed as data correlate. Here input data is correlated with that of the stored data in
the CAM.
 It should be noted that the stored patterns must be unique, i.e., different patterns in each location.
 If the same pattern exists in more than one location in the CAM, then, even though the correlation is
correct, the address is noted to be ambiguous. Associative memory makes a parallel search within a
stored data file.
 The concept behind this search is to Output any one or all stored items which match the given search
argument.
TRAINING ALGORITHMS FOR PATTERN ASSOCIATION:
There are two algorithms developed for training of pattern association nets.
i. Hebb Rule
ii. Outer Products Rule
i. Hebb Rule: The Hebb rule is widely used for finding the weights of an associative memory
neural network. The training vector pairs here are denoted as s: t. the weights are updated until
there is no weight change.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
2. Outer Products Rule: Outer products rule is a method for finding weights of an associative net.
ASSOCIATIVE MEMORY:
 Associative memory is also known as content addressable memory (CAM) or associative
storage or associative array.
 It is a special type of memory that is optimized for performing searches through data, as
opposed to providing a simple direct access to the data based on the address.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 It can store the set of patterns as memories when the associative memory is being presented
with a key pattern, it responds by producing one of the stored pattern which closely resembles
or relates to the key pattern.
 It can be viewed as data correlation here. Input data is correlated with that of stored data in
the CAM.
There two types of associative memories
i. Auto Associative Memory Network
ii. Hetero Associative memory Network
2. AUTO ASSOCIATIVE MEMORY NETWORK:
 An auto-associative memory recovers a previously stored pattern that most closely relates to the
current pattern. It is also known as an auto-associative correlate.
 In the auto associative memory network, the training input vector and training output vector are
the same.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
AUTO ASSOCIATIVE MEMORY ALGORITHM:
2. HETERO ASSOCIATIVE MEMORY NETWORK:
 In a hetero-associate memory, the training input and the target output vectors are different.
 The weights are determined in a way that the network can store a set of pattern associations. The
association here is a pair of training input target output vector pairs (s (p), t(p)), with p = 1,2,…p.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 Each vector s(p) has n components and each vector t(p) has m components. The determination
of weights is done either by using Hebb rule or delta rule.
 The net finds an appropriate output vector, which corresponds to an input vector x, that may be
either one of the stored patterns or a new pattern.
HETERO ASSOCIATIVE MEMORY ALGORITHM
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
4. BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM):
 Bidirectional associative memory (BAM), first proposed by Bart Kosko in the year 1988.
 The BAM network performs forward and backward associative searches for stored stimulus
responses.
 The BAM is a recurrent hetero associative pattern-marching network that encodes binary or
bipolar patterns using Hebbian learning rule.
 It associates patterns, say from set A to patterns from set B and vice versa is also performed.
 BAM neural nets can respond to input from either layers (input layer and output layer).
BAM ARCHITECTURE:
 The architecture of BAM network consists of two layers of neurons which are connected by
directed weighted pare interconnections.
 The network dynamics involve two layers of interaction. The BAM network iterates by sending
the signals back and forth between the two layers until all the neurons reach equilibrium.
 The weights associated with the network are bidirectional. Thus, BAM can respond to the inputs
in either layer.
 Figure shows a BAM network consisting of n units in X layer and m units in Y layer. The layers
can be connected in both directions (bidirectional) with the result the weight matrix sent from
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
the X layer to the Y layer is W and the weight matrix for signals sent from the Y layer to the X
layer is WT
. Thus, the Weight matrix is calculated in both directions.
Determination of Weights:
Let the input vectors be denoted by s (p) and target vectors by t (p). p = 1... P. Then the weight
matrix to store a set of input and target vectors,
Where,
can be determined by Hebb rule training a1gorithm. In case of input vectors being binary, the
weight matrix W = {wij} is given by
When the input vectors are bipolar, the weight matrix W = {wij} can be defined as
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
TESTING ALGORITHM FOR DISCRETE BIDIRECTIONAL ASSOCIATIVE MEMORY:
5. HOPFIELD NETWORKS:
 Hopfield neural network was proposed by John J. Hopfield in 1982. It is an auto-associative fully
interconnected single layer feedback network. It is a symmetrically weighted network (i.e., Wij =
Wji).
 The Hopfield network is commonly used for auto-association and optimization tasks.
The Hopfield network is of two types
 Discrete Hopfield Network
 Continuous Hopfield Network
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
i. DISCRETE HOPFIELD NETWORK: When this is operated in discrete line fashion it is called as
discrete Hopfield network. The network takes two-valued inputs: binary (0, 1) or bipolar (+1, -1); the use
of bipolar inputs makes the analysis easier. The network has symmetrical weights with no self-
connections, i.e.,
Architecture of Discrete Hopfield Network
 The Hopfield's model consists of processing elements with two outputs, one inverting and the other
non-inverting.
 The outputs from each processing element are fed back to the input of other processing elements but
not to itself.
Training Algorithm of Discrete Hopfield Network
 During training of discrete Hopfield network, weights will be updated. As we know that we can have
the binary input vectors as well as bipolar input vectors.
 Let the input vectors be denoted by s (p), p = 1... P. Then the weight matrix W to store a set of input
vectors, where,
 In case of input vectors being binary, the weight matrix W = {wij} is given by
When the input vectors are bipolar, the weight matrix W = {wij} can be defined as
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
II. CONTINUOUS HOPFIELD NETWORK
 Continuous network has time as a continuous variable, and can be used for associative memory
problems or optimization problems like traveling salesman problem.
 The nodes of this network have a continuous, graded output rather than a two state binary output.
Thus, the energy of the network decreases continuously with time.
The output is defined as:
Where,
 vi = output from the continuous Hopfield network
 ui = internal activity of a node in continuous Hopfield network.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
Energy Function
 The Hopfield networks have an energy function associated with them. It either diminishes or
remains unchanged on update (feedback) after every iteration.
 The energy function for a continuous Hopfield network is defined as:
To determine if the network will converge to a stable configuration, we see if the energy function
reaches its minimum by:
The network is bound to converge if the activity of each neuron time is given by the
following differential equation:
DIFFERENT BETWEEN AUTO ASSOCIATIVE MEMORY AND HETERO ASSOCIATIVE
MEMORY:
S.NO AUTO ASSOCIATIVE MEMORY HETERO ASSOCIATIVE MEMORY
1. The input and output of S and T are same The input and output of S and T are
different.
2. Recalls a memory of same modality as the
one that evoked it.
Recalls a memory of different in character
from output.
3. An Auto Associative memory retrieves the
same pattern.
An Auto Associative memory retrieves
stored pattern.
4. Example: Color correction, Color
Consistency.
Example: Space transforms: Fourier,
Dimensionality Reduction: PCA.
6. FIXED WEIGHT COMPETITIVE NETS: (Unsupervised Learning)
 During training process also the weights remains fixed in these competitive networks.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 The idea of competition is used among neurons for enhancement of contrast in their activation
functions. In this, two networks:
 Maxnet
 Hamming networks
i. MAXNET
 Maxnet network was developed by Lippmann in 1987.
 The Maxnet serves as a sub net for picking the node whose input is larger.
 All the nodes present in this subnet are fully interconnected and there exist symmetrical weights
in all these weighted interconnections.
Architecture of Maxnet
 The architecture of Maxnet is a fixed symmetrical weights are present over the weighted
interconnections.
 The weights between the neurons are inhibitory and fixed. The Maxnet with this structure can
be used as a subnet to select a particular node whose net input is the largest.
Testing Algorithm of Maxnet
The Maxnet uses the following activation function:
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
Testing algorithm
Step 0: Initial weights and initial activations are set. The weight is set as [0 < ε < 1/m], where "m" is the
total number of nodes. Let
Step 1: Perform Steps 2-4, when stopping condition is false.
Step 2: Update the activations of each node. For j = 1 to m,
Step 3: Save the activations obtained for use in the next iteration. For j = 1 to m,
Step 4: Finally, test the stopping condition for convergence of the network. The following is the stopping
condition: If more than one node has a nonzero activation, continue; else stop.
II. HAMMING NETWORK
 The Hamming network is a two-layer feed forward neural network for classification of binary
bipolar n-tuple input vectors using minimum Hamming distance denoted as DH(Lippmann,
1987).
 The first layer is the input layer for the n-tuple input vectors. The second layer (also called the
memory layer) stores p memory patterns.
 A p-class Hamming network has p output neurons in this layer.
 The strongest response of a neuron is indicative of the minimum Hamming distance between the
stored pattern and the input vector.
Hamming Distance
Hamming distance of two vectors, x and y of dimension n
x.y = a - d
Where:
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
a is number of bits in agreement in x & y(No.of Similarities bits in x & y),
d is number of bits different in x and y (No.of Dissimilarities bits in x & y).
The value "a - d" is the Hamming distance existing between two vectors.
Since, the total number of components is n, we have,
From the above equation, it is clearly understood that the weights can be set to one-half the exemplar
vector and bias can be set initially to n/2.
Testing Algorithm of Hamming Network:
Step 0: Initialize the weights. For i = 1 to n and j = 1 to m,
Initialize the bias for storing the "m" exemplar vectors. For j = 1 to m,
Step 1: Perform Steps 2-4 for each input vector x.
Step 2: Calculate the net input to each unit Yj, i.e.,
Step 3: Initialize the activations for Maxnet, i.e.,
Step 4: Maxnet is found to iterate for finding the exemplar that best matches the input patterns.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
7. KOHONEN SELF-ORGANIZING FEATURE MAPS:
 Self-Organizing Feature Maps (SOM) was developed by Dr. Teuvo Kohonen in 1982. Kohonen
Self-Organizing feature map (KSOM) refers to a neural network, which is trained using
competitive learning.
 Basic competitive learning implies that the competition process takes place before the cycle of
learning.
 The competition process suggests that some criteria select a winning processing element.
 After the winning processing element is selected, its weight vector is adjusted according to the
used learning law.
 Feature mapping is a process which converts the patterns of arbitrary dimensionality into a
response of one or two dimensions array of neurons.
 The network performing such a mapping is called feature map.
 The reason for reducing the higher dimensionality, the ability to preserve the neighbor topology.
Training Algorithm
Step 0: Initialize the weights with Random values and the learning rate
Step 1: Perform Steps 2-8 when stopping condition is false.
Step 2: Perform Steps 3-5 for each input vector x.
Step 3: Compute the square of the Euclidean distance, i.e., for each j = i to m,
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
Step 4: Find the winning unit index J, so that D (J) is minimum.
Step 5: For all units j within a specific neighborhood of J and for all i, calculate the new weights:
Step 6: Update the learning rare a using the formula (t is timestamp)
Step 7: Reduce radius of topological neighborhood at specified time intervals.
Step 8: Test for stopping condition of the network.
8. LEARNING VECTOR QUANTIZATION:
 In 1980, Finnish Professor Kohonen discovered that some areas of the brain develop structures
with different areas, each of them with a high sensitive for a specific input pattern.
 It is based on competition among neural units based on a principle called winner-takes-all.
 Learning Vector Quantization (LVQ) is a prototype-based supervised classification algorithm.
 A prototype is an early sample, model, or release of a product built to test a concept or process.
 One or more prototypes are used to represent each class in the dataset. New (unknown) data
points are then assigned the class of the prototype that is nearest to them. In order for "nearest"
to make sense, a distance measure has to be defined.
 There is no limitation on how many prototypes can be used per class, the only requirement being
that there is at least one prototype for each class.
 LVQ is a special case of an artificial neural network and it applies a winner-take-all Hebbian
learning-based approach.
 With a small difference, it is similar to Self-Organizing Maps (SOM) algorithm. SOM and LVQ
were invented by Teuvo Kohonen.
 LVQ system is represented by prototypes W=(W1....,Wn). In winner-take-all training algorithms,
the winner is moved closer if it correctly classifies the data point or moved away if it classifies
the data point incorrectly.
 An advantage of LVQ is that it creates prototypes that are easy to interpret for experts in the
respective application domain.
Training Algorithm
Step 0: Initialize the reference vectors.
This can be done using the following steps.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
From the given set of training vectors, take the first "m" (number of clusters) training vectors
and use them as weight vectors, the remaining vectors can be used for training.
Assign the initial weights and classifications randomly.
K-means clustering method.
Set initial learning rate α
Step l: Perform Steps 2-6 if the stopping condition is false.
Step 2: Perform Steps 3-4 for each training input vector x
Step 3: Calculate the Euclidean distance; for i = 1 to n, j = 1 to m,
Find the winning unit index J, when D (J) is minimum
Step 4: Update the weights on the winning unit, Wj using the following conditions.
Step 5: Reduce the learning rate α
Step 6: Test for the stopping condition of the training process. (The stopping conditions may be fixed
number of epochs or if learning rare has reduced to a negligible value.)
9. COUNTER PROPAGATION NETWORKS:
 Counter propagation network (CPN) were proposed by Hecht Nielsen in 1987.They are
multilayer network based on the combinations of the input, output, and clustering layers.
 The application of counter propagation net are data compression, function approximation and
pattern association.
 The counter propagation network is basically constructed from an instar-outstar model.
 This model is three layer neural network that performs input-output data mapping, producing an
output vector y in response to input vector x, on the basis of competitive learning.
 The three layer in an instar-out star model are the input layer, the hidden (competitive) layer and
the output layer.
 There are two stages involved in the training process of a counter propagation net.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 The input vector are clustered in the first stage.
 In the second stage of training, the weights from the cluster layer units to the output units are
tuned to obtain the desired response.
There are two types of counter propagation network:
i. Full counter propagation network
ii. Forward-only counter propagation network
I. FULL COUNTER PROPAGATION NETWORK:
 Full CPN efficiently represents a large number of vector pair x: y by adaptively constructing a
look-up-table.
 The full CPN works best if the inverse function exists and is continuous.
 The vector x and y propagate through the network in a counter flow manner to yield output vector
x* and y*.
Architecture of Full Counter propagation Network
 The four major components of the instar-out star model are the input layer, the instar, the
competitive layer and the out star.
 For each node in the input layer there is an input value xi. All the instar are grouped into a layer
called the competitive layer.
 Each of the instar responds maximally to a group of input vectors in a different region of space.
 An out star model is found to have all the nodes in the output layer and a single node in the
competitive layer. The out star looks like the fan-out of a node.
Training Algorithm for Full Counter propagation Network:
Step 0: Set the initial weights and the initial learning rare.
Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training.
Step 2: For each of the training input vector pair x: y presented, perform Steps 3-5.
Step 3: Make the X-input layer activations to vector X. Make the Y-input layer activations to vector Y.
Step 4: Find the winning cluster unit. If dot product method is used, find the cluster unit Zj with target
net input: for j = 1 to p.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
If Euclidean distance method is used, find the cluster unit Zj whose squared distance from input vectors
is the smallest
If there occurs a tie in case of selection of winner unit, the unit with the smallest index is the winner.
Take the winner unit index as J.
Step 5: Update the weights over the calculated winner unit Zj
Step 6: Reduce the learning rates α and β
Step 7: Test stopping condition for phase-I training.
Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training.
Step 9: Perform Steps 10-13 for each training input pair x: y. Here α and β are small constant values.
Step 10: Make the X-input layer activations to vector x. Make the Y-input layer activations to vector y.
Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J.
Step 12: Update the weights entering into unit ZJ
Step 13: Update the weights from unit Zj to the output layers.
Step 14: Reduce the learning rates a and b.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
Step 15: Test stopping condition for phase-II training.
II. FORWARD-ONLY COUNTER PROPAGATION NETWORK:
 A simplified version of full CPN is the forward-only CPN.
 Forward-only CPN uses only the x vector to form the cluster on the Kohonen units during phase
I training.
 In case of forward-only CPN, first input vectors are presented to the input units.
 First, the weights between the input layer and cluster layer are trained.
 Then the weights between the cluster layer and output layer are trained.
 This is a specific competitive network, with target known.
Architecture of forward-only CPN
 It consists of three layers: input layer, cluster layer and output layer.
 Its architecture resembles the back-propagation network, but in CPN there exists
interconnections between the units in the cluster layer.
Training Algorithm for Forward-only Counter propagation network:
Step 0: Initial the weights and learning rare.
Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training.
Step 2: Perform Steps 3-5 for each of training input X
Step 3: Set the X-input layer activations to vector X.
Step 4: Compute the winning cluster unit (J). If dot product method is used, find the cluster unit zj with
the largest net input.
If Euclidean distance method is used, find the cluster unit Zj whose squared distance from input patterns
is the smallest
If there exists a tie in the selection of winner unit, the unit with the smallest index is chosen as the
winner.
Step 5: Perform weight updating for unit Zj. For i= 1 to n,
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
Step 6: Reduce the learning rates α
Step 7: Test stopping condition for phase-I training.
Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training.
Step 9: Perform Steps 10-13 for each training input Pair x: y.
Step 10: Set X-input layer activations to vector X. Sec Y-output layer activations to vector Y.
Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J.
Step 12: Update the weights entering into unit ZJ,
Step 13: Update the weights from unit Zj to the output layers.
Step 14: Reduce the learning rates β.
Step 15: Test stopping condition for phase-II training.
10. ADAPTIVE RESONANCE THEORY NETWORK.
 The Adaptive Resonance Theory (ART) was incorporated as a hypothesis for human cognitive
data handling.
 The hypothesis has prompted neural models for pattern recognition and unsupervised learning.
ART system has been utilized to clarify different types of cognitive and brain data.
 The Adaptive Resonance Theory addresses the stability-plasticity (stability can be defined as
the nature of memorizing the learning and plasticity refers to the fact that they are flexible to
gain new information) dilemma of a system that asks how learning can proceed in response to
huge input patterns and simultaneously not to lose the stability for irrelevant patterns.
 Other than that, the stability-elasticity dilemma is concerned about how a system can adapt new
data while keeping what was learned before.
 For such a task, a feedback mechanism is included among the ART neural network layers.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 In this neural network, the data in the form of processing elements output reflects back and ahead
among layers. If an appropriate pattern is build-up, the resonance is reached, then adaption can
occur during this period.
 It can be defined as the formal analysis of how to overcome the learning instability accomplished
by a competitive learning model, let to the presentation of an expended hypothesis,
called adaptive resonance theory (ART).
 This formal investigation indicated that a specific type of top-down learned feedback and
matching mechanism could significantly overcome the instability issue.
 It was understood that top-down attentional mechanisms, which had prior been found through
an investigation of connections among cognitive and reinforcement mechanisms, had similar
characteristics as these code-stabilizing mechanisms.
 In other words, once it was perceived how to solve the instability issue formally, it also turned
out to be certain that one did not need to develop any quantitatively new mechanism to do so.
 One only needed to make sure to incorporate previously discovered attentional mechanisms.
These additional mechanisms empower code learning to self- stabilize in response to an
essentially arbitrary input system.
 Grossberg presented the basic principles of the adaptive resonance theory.
 A category of ART called ART1 has been described as an arrangement of ordinary differential
equations by carpenter and Grossberg. These theorems can predict both the order of search as
the function of the learning history of the system and the input patterns.
 ART1 is an unsupervised learning model primarily designed for recognizing binary patterns.
 It comprises an attentional subsystem, an orienting subsystem, a vigilance parameter, and a reset
module, as given in the figure given below.
 The vigilance parameter has a huge effect on the system. High vigilance produces higher detailed
memories.
 The ART1 attentional comprises of two competitive networks, comparison field layer L1 and
the recognition field layer L2, two control gains, Gain1 and Gain2, and two short-term memory
(STM) stages S1 and S2.
 Long term memory (LTM) follows somewhere in the range of S1 and S2 multiply the signal in
these pathways.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 Gains control empowers L1 and L2 to recognize the current stages of the running cycle. STM
reset wave prevents active L2 cells when mismatches between bottom-up and top-down signals
happen at L1.
 The comparison layer gets the binary external input passing it to the recognition layer liable for
coordinating it to a classification category.
 This outcome is given back to the comparison layer to find out when the category coordinates
the input vector.
 If there is a match, then a new input vector is read, and the cycle begins once again. If there is a
mismatch, then the orienting system is in charge of preventing the previous category from getting
a new category match in the recognition layer.
 The given two gains control the activity of the recognition and the comparison layer,
respectively.
 The reset wave specifically and enduringly prevents active L2 cell until the current is stopped.
The offset of the input pattern ends its processing L1 and triggers the offset of Gain2. Gain2
offset causes consistent decay of STM at L2 and thereby prepares L2 to encode the next input
pattern without bias.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
ART1 Implementation process:
 ART1 is a self-organizing neural network having input and output neurons mutually couple using
bottom-up and top-down adaptive weights that perform recognition.
 To start our methodology, the system is first trained as per the adaptive resonance theory by
inputting reference pattern data under the type of 5*5 matrix into the neurons for clustering
within the output neurons.
 Next, the maximum number of nodes in L2 is defined following by the vigilance parameter. The
inputted pattern enrolled itself as short term memory activity over a field of nodes L1.
 Combining and separating pathways from L1 to coding field L2, each weighted by an adaptive
long-term memory track, transform into a net signal vector T.
 Internal competitive dynamics at L2 further transform T, creating a compressed code or content
addressable memory.
 With strong competition, activation is concentrated at the L2 node that gets the maximal L1 →
L2 signal.
 The primary objective of this work is divided into four phases as follows Comparison,
recognition, search, and learning.
Advantage of adaptive learning theory (ART):
 It can be coordinated and utilized with different techniques to give more precise outcomes.
 It doesn't ensure stability in forming clusters.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 It can be used in different fields such as face recognition, embedded system, and robotics, target
recognition, medical diagnosis, signature verification, etc.
 It shows stability and is not disturbed by a wide range of inputs provided to inputs.
 It has got benefits over competitive learning. The competitive learning cant include new clusters
when considered necessary.
Application of ART:
 ART stands for Adaptive Resonance Theory. ART neural networks used for fast, stable learning
and prediction have been applied in different areas.
 The application incorporates target recognition, face recognition, medical diagnosis, signature
verification, mobile control robot.
i. Target recognition:
 Fuzzy ARTMAP neural network can be used for automatic classification of targets depend on
their radar range profiles.
 Tests on synthetic data show the fuzzy ARTMAP can result in substantial savings in memory
requirements when related to k nearest neighbor (kNN) classifiers.
 The utilization of multi wavelength profiles mainly improves the performance of both kinds of
classifiers.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
ii. Medical diagnosis:
 Medical databases present huge numbers of challenges found in general information
management settings where speed, use, efficiency, and accuracy are the prime concerns.
 A direct objective of improved computer-assisted medicine is to help to deliver intensive care in
situations that may be less than ideal.
 Working with these issues has stimulated several ART architecture developments, including
ARTMAP-IC.
iii. Signature verification:
 Automatic signature verification is a well-known and active area of research with various
applications such as bank check confirmation, ATM access, etc.
 The training of the network is finished using ART1 that uses global features as input vector and
the verification and recognition phase uses a two-step process.
 In the initial step, the input vector is coordinated with the stored reference vector, which was
used as a training set, and in the second step, cluster formation takes place.
iv. Mobile control robot:
 Nowadays, we perceive a wide range of robotic devices. It is still a field of research in their
program part, called artificial intelligence.
 The human brain is an interesting subject as a model for such an intelligent system. Inspired by
the structure of the human brain, an artificial neural emerges.
 Similar to the brain, the artificial neural network contains numerous simple computational units,
neurons that are interconnected mutually to allow the transfer of the signal from the neurons to
neurons.
 Artificial neural networks are used to solve different issues with good outcomes compared to
other decision algorithms.
Limitations of ART:
Some ART networks are contradictory as they rely on the order of the training data, or upon the learning
rate.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
2 MARKS QUESTIONS AND ANSWERS
1. What is recall?
Ans: If the input vector are uncorrelated, the Hebb rule will produce the correct weight and the
response of the net when tested with one of the training vectors will be perfect recall.
2. Explain Learning vector Quantization.
Ans: LVQ is adaptive data classification method. It is based on training data with desired class
information. LVQ uses unsupervised data clustering techniques to preprocessing the data set
and obtain cluster center.
3. What is meant by associative memory?
Ans: Associative memory is also known as content addressable memory (CAM) or associative
storage or associative array. It is a special type of memory that is optimized for performing
searches through data, as opposed to providing a simple direct access to the data based on the
address. It can store the set of patterns as memories when the associative memory is being
presented with a key pattern, it responds by producing one of the stored pattern which closely
resembles or relates to the key pattern. It can be viewed as data correlation here. Input data
is correlated with that of stored data in the CAM.
4. Define auto associative memory.
Ans: An auto-associative memory network, also known as a recurrent neural network, is
a type of associative memory that is used to recall a pattern from partial or degraded inputs.
In an auto-associative network, the output of the network is fed back into the input, allowing
the network to learn and remember the patterns it has been trained on. This type of memory
network is commonly used in applications such as speech and image recognition, where the
input data may be incomplete or noisy.
5. What is Hebbian Learning?
Ans: Hebb rule is the simplest and most common method of determining weights for an associative
memory neural net. It can be us with patterns are represented as either binary or bipolar vectors.
6. What is Bidirectional Associative memory (BAM)?
Ans: Bidirectional Associative Memory (BAM) is a type of artificial neural network designed for
storing and retrieving heterogeneous pattern pairs. It plays a crucial role in various applications,
such as password authentication, neural network models, and cognitive management.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
7. List the problems of BAM Network.
Ans:
 Storage capacity of the BAM: In the BAM, stored number of associations should not be
exceeded the number of neurons in the smaller layer.
 Incorrect convergence: Always the closest association may not be produced by BAM.
8. What is content addressable memory?
Ans: Content-addressable memory (CAM) is a special type of computer memory used in certain very-
high-speed searching applications.
 It is also known as associative memory or associative storage and compares input search data
against a table of stored data, and returns the address of matching data.
9. What are the delta rule for pattern association?
Ans: The delta rule is typically applied to the case in which pairs of pat- terns, consisting of an input
pattern and a target output pattern, are to be associated.
 When an input pattern is presented to an input layer of units, the appropriate output pattern will
appear on the output layer of units.
10. What is continuous BAM?
Ans: Continuous BAM transforms input smoothly and continuously into output in the range [0, 1]
using the logistic sigmoid function as the activation function for all units.
11. Which are the rules used in Hebb’s law?
Ans:
i. If two neurons on either side of a connection are activated synchronously, then the weight of
that connection is increased.
ii. If two neurons on either side of a connection are activated asynchronously, then the weight of
that connection is decreased.
12. What do you mean counter Propagation network?
Ans: Counter propagation Network is supervised neural network that can be used for multimodal
processing, but is not trained using the back propagation rule is the counter propagation network.
This network has been specifically developed to provide bidirectional mapping between input and
output training patterns.
13. What is Hopfield model?
Ans: The Hopfield model is a single layered recurrent network. Like the associative memory, it is
usually initialized with appropriate weights instead of being trained.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
14. Define Self-Organizing Map.
Ans: Self Organizing Map (or Kohonen Map or SOM) is a type of Artificial Neural Network
which is also inspired by biological models of neural systems from the 1970s.
 It follows an unsupervised learning approach and trained its network through a competitive
learning algorithm.
 SOM is used for clustering and mapping (or dimensionality reduction) techniques to map
multidimensional data onto lower-dimensional which allows people to reduce complex
problems for easy interpretation.
15. What is principle goal of the self-organizing map?
Ans: The principal goal of the Self Organizing Map (SOM) is to transform an incoming signal pattern
of arbitrary dimension into a one or two dimensional discrete map and to perform this
transformation adaptively in a topologically ordered fashion.
16. List the stags of the SOM algorithm.
Ans:
i. Initialization: Choose random values for the initial weight vectors wj.
ii. Sampling: Draw a sample training input vector x form the input space.
iii. Matching: Find the winner neuron I(x) with weight vector closest to input vector.
iv. Updating: Apply the weight update equation.
v. Continuation: Keep returning to step 2 until the feature map stops changing.
17. How does counter Propagation nets are trained?
Ans: Counter Propagation nets are trained in two stages:
i. First Stage: The input vectors are clustered. The Clusters that are formed may be based on
either the dot product metric or the Euclidean norm metric.
ii. Second stage: The weight from the cluster units to the output units are adapted to produce
the desired response.
18. List the possible drawback of counter propagation networks.
Ans:
 Training a counter Propagation network has the same difficulty associate with training a
Kohonen network.
CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani
 Counter Propagation networks tend to be larger than back propagation networks. If a certain
number of mapping are to be learned, the middle layer must have that many number of neurons.
19. How forward only differs from full counter propagation nets?
Ans:
 In full counter propagation only the X vectors to form the clusters on the Kohonen units during
the first stage of training.
 The original presentation of forward only counter propagation used the Euclidean distance
between the input vector and the weight vector for the Kohonen unit.
20. What is forward only counter propagation?
Ans:
 Is a simplified version of the full counter propagation.
 Are intended to approximate y=f(x) function that is not necessarily invertible.
 It may be used if the mapping from x to y is well define, but the mapping from y to x is not.
21. Define plasticity.
Ans: The ability of a net to respond to learn a new pattern equally well at any stage of learning called
plasticity.
22. List the components of ART1.
Ans: Components are as follows:
 The Short term memory layer (F1).
 The recognition layer (F2): It contains the long term memory of the system.
 Vigilance parameter (p): A parameter that control the generality of the memory. Larger p
means more detailed memories. Smaller p produces more general memories.

More Related Content

What's hot

what is neural network....???
what is neural network....???what is neural network....???
what is neural network....???
Adii Shah
 
Hebbian Learning
Hebbian LearningHebbian Learning
Hebbian Learning
ESCOM
 
NEURAL NETWORKS
NEURAL NETWORKSNEURAL NETWORKS
NEURAL NETWORKS
ESCOM
 
lecture07.ppt
lecture07.pptlecture07.ppt
lecture07.ppt
butest
 
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
Simplilearn
 

What's hot (20)

Neural Networks: Self-Organizing Maps (SOM)
Neural Networks:  Self-Organizing Maps (SOM)Neural Networks:  Self-Organizing Maps (SOM)
Neural Networks: Self-Organizing Maps (SOM)
 
Neural network & its applications
Neural network & its applications Neural network & its applications
Neural network & its applications
 
Feedforward neural network
Feedforward neural networkFeedforward neural network
Feedforward neural network
 
Deep Learning Tutorial
Deep Learning TutorialDeep Learning Tutorial
Deep Learning Tutorial
 
Neural Networks: Rosenblatt's Perceptron
Neural Networks: Rosenblatt's PerceptronNeural Networks: Rosenblatt's Perceptron
Neural Networks: Rosenblatt's Perceptron
 
Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)Neural Networks: Radial Bases Functions (RBF)
Neural Networks: Radial Bases Functions (RBF)
 
14 Machine Learning Single Layer Perceptron
14 Machine Learning Single Layer Perceptron14 Machine Learning Single Layer Perceptron
14 Machine Learning Single Layer Perceptron
 
what is neural network....???
what is neural network....???what is neural network....???
what is neural network....???
 
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
 
Introduction Of Artificial neural network
Introduction Of Artificial neural networkIntroduction Of Artificial neural network
Introduction Of Artificial neural network
 
Artifical Neural Network and its applications
Artifical Neural Network and its applicationsArtifical Neural Network and its applications
Artifical Neural Network and its applications
 
Neural networks
Neural networksNeural networks
Neural networks
 
Deep Learning - RNN and CNN
Deep Learning - RNN and CNNDeep Learning - RNN and CNN
Deep Learning - RNN and CNN
 
Artificial neural network for machine learning
Artificial neural network for machine learningArtificial neural network for machine learning
Artificial neural network for machine learning
 
Hebbian Learning
Hebbian LearningHebbian Learning
Hebbian Learning
 
Neural Networks: Introducton
Neural Networks: IntroductonNeural Networks: Introducton
Neural Networks: Introducton
 
NEURAL NETWORKS
NEURAL NETWORKSNEURAL NETWORKS
NEURAL NETWORKS
 
lecture07.ppt
lecture07.pptlecture07.ppt
lecture07.ppt
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural network
 
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
Artificial Neural Network | Deep Neural Network Explained | Artificial Neural...
 

Similar to CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf

ANNs have been widely used in various domains for: Pattern recognition Funct...
ANNs have been widely used in various domains for: Pattern recognition  Funct...ANNs have been widely used in various domains for: Pattern recognition  Funct...
ANNs have been widely used in various domains for: Pattern recognition Funct...
vijaym148
 

Similar to CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf (20)

Intro to Deep learning - Autoencoders
Intro to Deep learning - Autoencoders Intro to Deep learning - Autoencoders
Intro to Deep learning - Autoencoders
 
Unit iii update
Unit iii updateUnit iii update
Unit iii update
 
NEURALNETWORKS_DM_SOWMYAJYOTHI.pdf
NEURALNETWORKS_DM_SOWMYAJYOTHI.pdfNEURALNETWORKS_DM_SOWMYAJYOTHI.pdf
NEURALNETWORKS_DM_SOWMYAJYOTHI.pdf
 
Ann
Ann Ann
Ann
 
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
 
N ns 1
N ns 1N ns 1
N ns 1
 
Artificial neural networks
Artificial neural networks Artificial neural networks
Artificial neural networks
 
Artificial Neural Network for machine learning
Artificial Neural Network for machine learningArtificial Neural Network for machine learning
Artificial Neural Network for machine learning
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Artificial Neural Networks ppt.pptx for final sem cse
Artificial Neural Networks  ppt.pptx for final sem cseArtificial Neural Networks  ppt.pptx for final sem cse
Artificial Neural Networks ppt.pptx for final sem cse
 
Anfis (1)
Anfis (1)Anfis (1)
Anfis (1)
 
Artificial Neuron network
Artificial Neuron network Artificial Neuron network
Artificial Neuron network
 
ANNs have been widely used in various domains for: Pattern recognition Funct...
ANNs have been widely used in various domains for: Pattern recognition  Funct...ANNs have been widely used in various domains for: Pattern recognition  Funct...
ANNs have been widely used in various domains for: Pattern recognition Funct...
 
APPLIED MACHINE LEARNING
APPLIED MACHINE LEARNINGAPPLIED MACHINE LEARNING
APPLIED MACHINE LEARNING
 
Image Recognition With the Help of Auto-Associative Neural Network
Image Recognition With the Help of Auto-Associative Neural NetworkImage Recognition With the Help of Auto-Associative Neural Network
Image Recognition With the Help of Auto-Associative Neural Network
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
On The Application of Hyperbolic Activation Function in Computing the Acceler...
On The Application of Hyperbolic Activation Function in Computing the Acceler...On The Application of Hyperbolic Activation Function in Computing the Acceler...
On The Application of Hyperbolic Activation Function in Computing the Acceler...
 
Mathematical Foundation of Discrete time Hopfield Networks
Mathematical Foundation of Discrete time Hopfield NetworksMathematical Foundation of Discrete time Hopfield Networks
Mathematical Foundation of Discrete time Hopfield Networks
 
Artificial neural networks seminar presentation using MSWord.
Artificial neural networks seminar presentation using MSWord.Artificial neural networks seminar presentation using MSWord.
Artificial neural networks seminar presentation using MSWord.
 
Artificial neural network by arpit_sharma
Artificial neural network by arpit_sharmaArtificial neural network by arpit_sharma
Artificial neural network by arpit_sharma
 

More from Asst.prof M.Gokilavani

More from Asst.prof M.Gokilavani (20)

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete RecordCCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
 
IT8073_Information Security_UNIT I _.pdf
IT8073_Information Security_UNIT I _.pdfIT8073_Information Security_UNIT I _.pdf
IT8073_Information Security_UNIT I _.pdf
 
IT8073 _Information Security _UNIT I Full notes
IT8073 _Information Security _UNIT I Full notesIT8073 _Information Security _UNIT I Full notes
IT8073 _Information Security _UNIT I Full notes
 
GE3151 PSPP UNIT IV QUESTION BANK.docx.pdf
GE3151 PSPP UNIT IV QUESTION BANK.docx.pdfGE3151 PSPP UNIT IV QUESTION BANK.docx.pdf
GE3151 PSPP UNIT IV QUESTION BANK.docx.pdf
 
GE3151 PSPP UNIT III QUESTION BANK.docx.pdf
GE3151 PSPP UNIT III QUESTION BANK.docx.pdfGE3151 PSPP UNIT III QUESTION BANK.docx.pdf
GE3151 PSPP UNIT III QUESTION BANK.docx.pdf
 
GE3151 UNIT II Study material .pdf
GE3151 UNIT II Study material .pdfGE3151 UNIT II Study material .pdf
GE3151 UNIT II Study material .pdf
 
GE3151 PSPP All unit question bank.pdf
GE3151 PSPP All unit question bank.pdfGE3151 PSPP All unit question bank.pdf
GE3151 PSPP All unit question bank.pdf
 
GE3151_PSPP_All unit _Notes
GE3151_PSPP_All unit _NotesGE3151_PSPP_All unit _Notes
GE3151_PSPP_All unit _Notes
 
GE3151_PSPP_UNIT_5_Notes
GE3151_PSPP_UNIT_5_NotesGE3151_PSPP_UNIT_5_Notes
GE3151_PSPP_UNIT_5_Notes
 
GE3151_PSPP_UNIT_4_Notes
GE3151_PSPP_UNIT_4_NotesGE3151_PSPP_UNIT_4_Notes
GE3151_PSPP_UNIT_4_Notes
 
GE3151_PSPP_UNIT_3_Notes
GE3151_PSPP_UNIT_3_NotesGE3151_PSPP_UNIT_3_Notes
GE3151_PSPP_UNIT_3_Notes
 
GE3151_PSPP_UNIT_2_Notes
GE3151_PSPP_UNIT_2_NotesGE3151_PSPP_UNIT_2_Notes
GE3151_PSPP_UNIT_2_Notes
 
AI3391 Artificial intelligence Unit IV Notes _ merged.pdf
AI3391 Artificial intelligence Unit IV Notes _ merged.pdfAI3391 Artificial intelligence Unit IV Notes _ merged.pdf
AI3391 Artificial intelligence Unit IV Notes _ merged.pdf
 
AI3391 Artificial intelligence Session 29 Forward and backward chaining.pdf
AI3391 Artificial intelligence Session 29 Forward and backward chaining.pdfAI3391 Artificial intelligence Session 29 Forward and backward chaining.pdf
AI3391 Artificial intelligence Session 29 Forward and backward chaining.pdf
 
AI3391 Artificial intelligence Session 28 Resolution.pptx
AI3391 Artificial intelligence Session 28 Resolution.pptxAI3391 Artificial intelligence Session 28 Resolution.pptx
AI3391 Artificial intelligence Session 28 Resolution.pptx
 
AI3391 Artificial intelligence session 27 inference and unification.pptx
AI3391 Artificial intelligence session 27 inference and unification.pptxAI3391 Artificial intelligence session 27 inference and unification.pptx
AI3391 Artificial intelligence session 27 inference and unification.pptx
 
AI3391 Artificial Intelligence Session 26 First order logic.pptx
AI3391 Artificial Intelligence Session 26 First order logic.pptxAI3391 Artificial Intelligence Session 26 First order logic.pptx
AI3391 Artificial Intelligence Session 26 First order logic.pptx
 
AI3391 Artificial Intelligence Session 25 Horn clause.pptx
AI3391 Artificial Intelligence Session 25 Horn clause.pptxAI3391 Artificial Intelligence Session 25 Horn clause.pptx
AI3391 Artificial Intelligence Session 25 Horn clause.pptx
 
AI3391 Artificial Intelligence session 24 knowledge representation.pptx
AI3391 Artificial Intelligence session 24 knowledge representation.pptxAI3391 Artificial Intelligence session 24 knowledge representation.pptx
AI3391 Artificial Intelligence session 24 knowledge representation.pptx
 
AI3391 Artificial Intelligence UNIT III Notes_merged.pdf
AI3391 Artificial Intelligence UNIT III Notes_merged.pdfAI3391 Artificial Intelligence UNIT III Notes_merged.pdf
AI3391 Artificial Intelligence UNIT III Notes_merged.pdf
 

Recently uploaded

Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Lovely Professional University
 
Complex plane, Modulus, Argument, Graphical representation of a complex numbe...
Complex plane, Modulus, Argument, Graphical representation of a complex numbe...Complex plane, Modulus, Argument, Graphical representation of a complex numbe...
Complex plane, Modulus, Argument, Graphical representation of a complex numbe...
MohammadAliNayeem
 
Paint shop management system project report.pdf
Paint shop management system project report.pdfPaint shop management system project report.pdf
Paint shop management system project report.pdf
Kamal Acharya
 
School management system project report.pdf
School management system project report.pdfSchool management system project report.pdf
School management system project report.pdf
Kamal Acharya
 

Recently uploaded (20)

Introduction to Machine Learning Unit-4 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-4 Notes for II-II Mechanical EngineeringIntroduction to Machine Learning Unit-4 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-4 Notes for II-II Mechanical Engineering
 
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
 
Complex plane, Modulus, Argument, Graphical representation of a complex numbe...
Complex plane, Modulus, Argument, Graphical representation of a complex numbe...Complex plane, Modulus, Argument, Graphical representation of a complex numbe...
Complex plane, Modulus, Argument, Graphical representation of a complex numbe...
 
Lect_Z_Transform_Main_digital_image_processing.pptx
Lect_Z_Transform_Main_digital_image_processing.pptxLect_Z_Transform_Main_digital_image_processing.pptx
Lect_Z_Transform_Main_digital_image_processing.pptx
 
Software Engineering - Modelling Concepts + Class Modelling + Building the An...
Software Engineering - Modelling Concepts + Class Modelling + Building the An...Software Engineering - Modelling Concepts + Class Modelling + Building the An...
Software Engineering - Modelling Concepts + Class Modelling + Building the An...
 
Attraction and Repulsion type Moving Iron Instruments.pptx
Attraction and Repulsion type Moving Iron Instruments.pptxAttraction and Repulsion type Moving Iron Instruments.pptx
Attraction and Repulsion type Moving Iron Instruments.pptx
 
Arduino based vehicle speed tracker project
Arduino based vehicle speed tracker projectArduino based vehicle speed tracker project
Arduino based vehicle speed tracker project
 
Online book store management system project.pdf
Online book store management system project.pdfOnline book store management system project.pdf
Online book store management system project.pdf
 
Lab Manual Arduino UNO Microcontrollar.docx
Lab Manual Arduino UNO Microcontrollar.docxLab Manual Arduino UNO Microcontrollar.docx
Lab Manual Arduino UNO Microcontrollar.docx
 
Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2Research Methodolgy & Intellectual Property Rights Series 2
Research Methodolgy & Intellectual Property Rights Series 2
 
Quiz application system project report..pdf
Quiz application system project report..pdfQuiz application system project report..pdf
Quiz application system project report..pdf
 
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
 
E-Commerce Shopping using MERN Stack where different modules are present
E-Commerce Shopping using MERN Stack where different modules are presentE-Commerce Shopping using MERN Stack where different modules are present
E-Commerce Shopping using MERN Stack where different modules are present
 
Natalia Rutkowska - BIM School Course in Kraków
Natalia Rutkowska - BIM School Course in KrakówNatalia Rutkowska - BIM School Course in Kraków
Natalia Rutkowska - BIM School Course in Kraków
 
Theory for How to calculation capacitor bank
Theory for How to calculation capacitor bankTheory for How to calculation capacitor bank
Theory for How to calculation capacitor bank
 
Paint shop management system project report.pdf
Paint shop management system project report.pdfPaint shop management system project report.pdf
Paint shop management system project report.pdf
 
"United Nations Park" Site Visit Report.
"United Nations Park" Site  Visit Report."United Nations Park" Site  Visit Report.
"United Nations Park" Site Visit Report.
 
BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING
BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWINGBRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING
BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING
 
School management system project report.pdf
School management system project report.pdfSchool management system project report.pdf
School management system project report.pdf
 
Dairy management system project report..pdf
Dairy management system project report..pdfDairy management system project report..pdf
Dairy management system project report..pdf
 

CCS355 Neural Network & Deep Learning Unit II Notes with Question bank .pdf

  • 1. CCS355 NEURAL NETWORKS & DEEP LEARNING UNIT – 2 NOTES BE III YEAR – VI SEM (R21) (2023-2024) Prepared By Asst.Prof.M.Gokilavani Department of Information Technology
  • 2.
  • 3. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani UNIT II - ASSOCIATIVE MEMORY AND UNSUPERVISED LEARNING NETWORKS Training Algorithms for Pattern Association-Auto associative Memory Network-Hetero associative Memory Network-Bidirectional Associative Memory (BAM)-Hopfield Networks-Iterative Auto associative Memory Networks-Temporal Associative Memory Network-Fixed Weight Competitive Nets- Kohonen Self-Organizing Feature Maps-Learning Vector Quantization-Counter propagation Networks- Adaptive Resonance Theory Network. 1. INTRODUCTION:  An associative memory network can store a set of patterns as memories.  When the associative memory is being presented with a key pattern, it responds by producing one of the stored patterns, which closely resembles or relates to the key pattern.  Thus, the recall is through association of the key pattern, with the help of information memorized.  These types of memories are also called as content-addressable memories (CAM).  The CAM can also be viewed as associating data to address, i.e.; for every data in the memory there is a corresponding unique address.  Also, it can be viewed as data correlate. Here input data is correlated with that of the stored data in the CAM.  It should be noted that the stored patterns must be unique, i.e., different patterns in each location.  If the same pattern exists in more than one location in the CAM, then, even though the correlation is correct, the address is noted to be ambiguous. Associative memory makes a parallel search within a stored data file.  The concept behind this search is to Output any one or all stored items which match the given search argument. TRAINING ALGORITHMS FOR PATTERN ASSOCIATION: There are two algorithms developed for training of pattern association nets. i. Hebb Rule ii. Outer Products Rule i. Hebb Rule: The Hebb rule is widely used for finding the weights of an associative memory neural network. The training vector pairs here are denoted as s: t. the weights are updated until there is no weight change.
  • 4. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani 2. Outer Products Rule: Outer products rule is a method for finding weights of an associative net. ASSOCIATIVE MEMORY:  Associative memory is also known as content addressable memory (CAM) or associative storage or associative array.  It is a special type of memory that is optimized for performing searches through data, as opposed to providing a simple direct access to the data based on the address.
  • 5. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  It can store the set of patterns as memories when the associative memory is being presented with a key pattern, it responds by producing one of the stored pattern which closely resembles or relates to the key pattern.  It can be viewed as data correlation here. Input data is correlated with that of stored data in the CAM. There two types of associative memories i. Auto Associative Memory Network ii. Hetero Associative memory Network 2. AUTO ASSOCIATIVE MEMORY NETWORK:  An auto-associative memory recovers a previously stored pattern that most closely relates to the current pattern. It is also known as an auto-associative correlate.  In the auto associative memory network, the training input vector and training output vector are the same.
  • 6. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani AUTO ASSOCIATIVE MEMORY ALGORITHM: 2. HETERO ASSOCIATIVE MEMORY NETWORK:  In a hetero-associate memory, the training input and the target output vectors are different.  The weights are determined in a way that the network can store a set of pattern associations. The association here is a pair of training input target output vector pairs (s (p), t(p)), with p = 1,2,…p.
  • 7. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  Each vector s(p) has n components and each vector t(p) has m components. The determination of weights is done either by using Hebb rule or delta rule.  The net finds an appropriate output vector, which corresponds to an input vector x, that may be either one of the stored patterns or a new pattern. HETERO ASSOCIATIVE MEMORY ALGORITHM
  • 8. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani 4. BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM):  Bidirectional associative memory (BAM), first proposed by Bart Kosko in the year 1988.  The BAM network performs forward and backward associative searches for stored stimulus responses.  The BAM is a recurrent hetero associative pattern-marching network that encodes binary or bipolar patterns using Hebbian learning rule.  It associates patterns, say from set A to patterns from set B and vice versa is also performed.  BAM neural nets can respond to input from either layers (input layer and output layer). BAM ARCHITECTURE:  The architecture of BAM network consists of two layers of neurons which are connected by directed weighted pare interconnections.  The network dynamics involve two layers of interaction. The BAM network iterates by sending the signals back and forth between the two layers until all the neurons reach equilibrium.  The weights associated with the network are bidirectional. Thus, BAM can respond to the inputs in either layer.  Figure shows a BAM network consisting of n units in X layer and m units in Y layer. The layers can be connected in both directions (bidirectional) with the result the weight matrix sent from
  • 9. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani the X layer to the Y layer is W and the weight matrix for signals sent from the Y layer to the X layer is WT . Thus, the Weight matrix is calculated in both directions. Determination of Weights: Let the input vectors be denoted by s (p) and target vectors by t (p). p = 1... P. Then the weight matrix to store a set of input and target vectors, Where, can be determined by Hebb rule training a1gorithm. In case of input vectors being binary, the weight matrix W = {wij} is given by When the input vectors are bipolar, the weight matrix W = {wij} can be defined as
  • 10. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani TESTING ALGORITHM FOR DISCRETE BIDIRECTIONAL ASSOCIATIVE MEMORY: 5. HOPFIELD NETWORKS:  Hopfield neural network was proposed by John J. Hopfield in 1982. It is an auto-associative fully interconnected single layer feedback network. It is a symmetrically weighted network (i.e., Wij = Wji).  The Hopfield network is commonly used for auto-association and optimization tasks. The Hopfield network is of two types  Discrete Hopfield Network  Continuous Hopfield Network
  • 11. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani i. DISCRETE HOPFIELD NETWORK: When this is operated in discrete line fashion it is called as discrete Hopfield network. The network takes two-valued inputs: binary (0, 1) or bipolar (+1, -1); the use of bipolar inputs makes the analysis easier. The network has symmetrical weights with no self- connections, i.e., Architecture of Discrete Hopfield Network  The Hopfield's model consists of processing elements with two outputs, one inverting and the other non-inverting.  The outputs from each processing element are fed back to the input of other processing elements but not to itself. Training Algorithm of Discrete Hopfield Network  During training of discrete Hopfield network, weights will be updated. As we know that we can have the binary input vectors as well as bipolar input vectors.  Let the input vectors be denoted by s (p), p = 1... P. Then the weight matrix W to store a set of input vectors, where,  In case of input vectors being binary, the weight matrix W = {wij} is given by When the input vectors are bipolar, the weight matrix W = {wij} can be defined as
  • 12. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani II. CONTINUOUS HOPFIELD NETWORK  Continuous network has time as a continuous variable, and can be used for associative memory problems or optimization problems like traveling salesman problem.  The nodes of this network have a continuous, graded output rather than a two state binary output. Thus, the energy of the network decreases continuously with time. The output is defined as: Where,  vi = output from the continuous Hopfield network  ui = internal activity of a node in continuous Hopfield network.
  • 13. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani Energy Function  The Hopfield networks have an energy function associated with them. It either diminishes or remains unchanged on update (feedback) after every iteration.  The energy function for a continuous Hopfield network is defined as: To determine if the network will converge to a stable configuration, we see if the energy function reaches its minimum by: The network is bound to converge if the activity of each neuron time is given by the following differential equation: DIFFERENT BETWEEN AUTO ASSOCIATIVE MEMORY AND HETERO ASSOCIATIVE MEMORY: S.NO AUTO ASSOCIATIVE MEMORY HETERO ASSOCIATIVE MEMORY 1. The input and output of S and T are same The input and output of S and T are different. 2. Recalls a memory of same modality as the one that evoked it. Recalls a memory of different in character from output. 3. An Auto Associative memory retrieves the same pattern. An Auto Associative memory retrieves stored pattern. 4. Example: Color correction, Color Consistency. Example: Space transforms: Fourier, Dimensionality Reduction: PCA. 6. FIXED WEIGHT COMPETITIVE NETS: (Unsupervised Learning)  During training process also the weights remains fixed in these competitive networks.
  • 14. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  The idea of competition is used among neurons for enhancement of contrast in their activation functions. In this, two networks:  Maxnet  Hamming networks i. MAXNET  Maxnet network was developed by Lippmann in 1987.  The Maxnet serves as a sub net for picking the node whose input is larger.  All the nodes present in this subnet are fully interconnected and there exist symmetrical weights in all these weighted interconnections. Architecture of Maxnet  The architecture of Maxnet is a fixed symmetrical weights are present over the weighted interconnections.  The weights between the neurons are inhibitory and fixed. The Maxnet with this structure can be used as a subnet to select a particular node whose net input is the largest. Testing Algorithm of Maxnet The Maxnet uses the following activation function:
  • 15. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani Testing algorithm Step 0: Initial weights and initial activations are set. The weight is set as [0 < ε < 1/m], where "m" is the total number of nodes. Let Step 1: Perform Steps 2-4, when stopping condition is false. Step 2: Update the activations of each node. For j = 1 to m, Step 3: Save the activations obtained for use in the next iteration. For j = 1 to m, Step 4: Finally, test the stopping condition for convergence of the network. The following is the stopping condition: If more than one node has a nonzero activation, continue; else stop. II. HAMMING NETWORK  The Hamming network is a two-layer feed forward neural network for classification of binary bipolar n-tuple input vectors using minimum Hamming distance denoted as DH(Lippmann, 1987).  The first layer is the input layer for the n-tuple input vectors. The second layer (also called the memory layer) stores p memory patterns.  A p-class Hamming network has p output neurons in this layer.  The strongest response of a neuron is indicative of the minimum Hamming distance between the stored pattern and the input vector. Hamming Distance Hamming distance of two vectors, x and y of dimension n x.y = a - d Where:
  • 16. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani a is number of bits in agreement in x & y(No.of Similarities bits in x & y), d is number of bits different in x and y (No.of Dissimilarities bits in x & y). The value "a - d" is the Hamming distance existing between two vectors. Since, the total number of components is n, we have, From the above equation, it is clearly understood that the weights can be set to one-half the exemplar vector and bias can be set initially to n/2. Testing Algorithm of Hamming Network: Step 0: Initialize the weights. For i = 1 to n and j = 1 to m, Initialize the bias for storing the "m" exemplar vectors. For j = 1 to m, Step 1: Perform Steps 2-4 for each input vector x. Step 2: Calculate the net input to each unit Yj, i.e., Step 3: Initialize the activations for Maxnet, i.e., Step 4: Maxnet is found to iterate for finding the exemplar that best matches the input patterns.
  • 17. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani 7. KOHONEN SELF-ORGANIZING FEATURE MAPS:  Self-Organizing Feature Maps (SOM) was developed by Dr. Teuvo Kohonen in 1982. Kohonen Self-Organizing feature map (KSOM) refers to a neural network, which is trained using competitive learning.  Basic competitive learning implies that the competition process takes place before the cycle of learning.  The competition process suggests that some criteria select a winning processing element.  After the winning processing element is selected, its weight vector is adjusted according to the used learning law.  Feature mapping is a process which converts the patterns of arbitrary dimensionality into a response of one or two dimensions array of neurons.  The network performing such a mapping is called feature map.  The reason for reducing the higher dimensionality, the ability to preserve the neighbor topology. Training Algorithm Step 0: Initialize the weights with Random values and the learning rate Step 1: Perform Steps 2-8 when stopping condition is false. Step 2: Perform Steps 3-5 for each input vector x. Step 3: Compute the square of the Euclidean distance, i.e., for each j = i to m,
  • 18. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani Step 4: Find the winning unit index J, so that D (J) is minimum. Step 5: For all units j within a specific neighborhood of J and for all i, calculate the new weights: Step 6: Update the learning rare a using the formula (t is timestamp) Step 7: Reduce radius of topological neighborhood at specified time intervals. Step 8: Test for stopping condition of the network. 8. LEARNING VECTOR QUANTIZATION:  In 1980, Finnish Professor Kohonen discovered that some areas of the brain develop structures with different areas, each of them with a high sensitive for a specific input pattern.  It is based on competition among neural units based on a principle called winner-takes-all.  Learning Vector Quantization (LVQ) is a prototype-based supervised classification algorithm.  A prototype is an early sample, model, or release of a product built to test a concept or process.  One or more prototypes are used to represent each class in the dataset. New (unknown) data points are then assigned the class of the prototype that is nearest to them. In order for "nearest" to make sense, a distance measure has to be defined.  There is no limitation on how many prototypes can be used per class, the only requirement being that there is at least one prototype for each class.  LVQ is a special case of an artificial neural network and it applies a winner-take-all Hebbian learning-based approach.  With a small difference, it is similar to Self-Organizing Maps (SOM) algorithm. SOM and LVQ were invented by Teuvo Kohonen.  LVQ system is represented by prototypes W=(W1....,Wn). In winner-take-all training algorithms, the winner is moved closer if it correctly classifies the data point or moved away if it classifies the data point incorrectly.  An advantage of LVQ is that it creates prototypes that are easy to interpret for experts in the respective application domain. Training Algorithm Step 0: Initialize the reference vectors. This can be done using the following steps.
  • 19. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani From the given set of training vectors, take the first "m" (number of clusters) training vectors and use them as weight vectors, the remaining vectors can be used for training. Assign the initial weights and classifications randomly. K-means clustering method. Set initial learning rate α Step l: Perform Steps 2-6 if the stopping condition is false. Step 2: Perform Steps 3-4 for each training input vector x Step 3: Calculate the Euclidean distance; for i = 1 to n, j = 1 to m, Find the winning unit index J, when D (J) is minimum Step 4: Update the weights on the winning unit, Wj using the following conditions. Step 5: Reduce the learning rate α Step 6: Test for the stopping condition of the training process. (The stopping conditions may be fixed number of epochs or if learning rare has reduced to a negligible value.) 9. COUNTER PROPAGATION NETWORKS:  Counter propagation network (CPN) were proposed by Hecht Nielsen in 1987.They are multilayer network based on the combinations of the input, output, and clustering layers.  The application of counter propagation net are data compression, function approximation and pattern association.  The counter propagation network is basically constructed from an instar-outstar model.  This model is three layer neural network that performs input-output data mapping, producing an output vector y in response to input vector x, on the basis of competitive learning.  The three layer in an instar-out star model are the input layer, the hidden (competitive) layer and the output layer.  There are two stages involved in the training process of a counter propagation net.
  • 20. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  The input vector are clustered in the first stage.  In the second stage of training, the weights from the cluster layer units to the output units are tuned to obtain the desired response. There are two types of counter propagation network: i. Full counter propagation network ii. Forward-only counter propagation network I. FULL COUNTER PROPAGATION NETWORK:  Full CPN efficiently represents a large number of vector pair x: y by adaptively constructing a look-up-table.  The full CPN works best if the inverse function exists and is continuous.  The vector x and y propagate through the network in a counter flow manner to yield output vector x* and y*. Architecture of Full Counter propagation Network  The four major components of the instar-out star model are the input layer, the instar, the competitive layer and the out star.  For each node in the input layer there is an input value xi. All the instar are grouped into a layer called the competitive layer.  Each of the instar responds maximally to a group of input vectors in a different region of space.  An out star model is found to have all the nodes in the output layer and a single node in the competitive layer. The out star looks like the fan-out of a node. Training Algorithm for Full Counter propagation Network: Step 0: Set the initial weights and the initial learning rare. Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training. Step 2: For each of the training input vector pair x: y presented, perform Steps 3-5. Step 3: Make the X-input layer activations to vector X. Make the Y-input layer activations to vector Y. Step 4: Find the winning cluster unit. If dot product method is used, find the cluster unit Zj with target net input: for j = 1 to p.
  • 21. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani If Euclidean distance method is used, find the cluster unit Zj whose squared distance from input vectors is the smallest If there occurs a tie in case of selection of winner unit, the unit with the smallest index is the winner. Take the winner unit index as J. Step 5: Update the weights over the calculated winner unit Zj Step 6: Reduce the learning rates α and β Step 7: Test stopping condition for phase-I training. Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training. Step 9: Perform Steps 10-13 for each training input pair x: y. Here α and β are small constant values. Step 10: Make the X-input layer activations to vector x. Make the Y-input layer activations to vector y. Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J. Step 12: Update the weights entering into unit ZJ Step 13: Update the weights from unit Zj to the output layers. Step 14: Reduce the learning rates a and b.
  • 22. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani Step 15: Test stopping condition for phase-II training. II. FORWARD-ONLY COUNTER PROPAGATION NETWORK:  A simplified version of full CPN is the forward-only CPN.  Forward-only CPN uses only the x vector to form the cluster on the Kohonen units during phase I training.  In case of forward-only CPN, first input vectors are presented to the input units.  First, the weights between the input layer and cluster layer are trained.  Then the weights between the cluster layer and output layer are trained.  This is a specific competitive network, with target known. Architecture of forward-only CPN  It consists of three layers: input layer, cluster layer and output layer.  Its architecture resembles the back-propagation network, but in CPN there exists interconnections between the units in the cluster layer. Training Algorithm for Forward-only Counter propagation network: Step 0: Initial the weights and learning rare. Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training. Step 2: Perform Steps 3-5 for each of training input X Step 3: Set the X-input layer activations to vector X. Step 4: Compute the winning cluster unit (J). If dot product method is used, find the cluster unit zj with the largest net input. If Euclidean distance method is used, find the cluster unit Zj whose squared distance from input patterns is the smallest If there exists a tie in the selection of winner unit, the unit with the smallest index is chosen as the winner. Step 5: Perform weight updating for unit Zj. For i= 1 to n,
  • 23. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani Step 6: Reduce the learning rates α Step 7: Test stopping condition for phase-I training. Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training. Step 9: Perform Steps 10-13 for each training input Pair x: y. Step 10: Set X-input layer activations to vector X. Sec Y-output layer activations to vector Y. Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J. Step 12: Update the weights entering into unit ZJ, Step 13: Update the weights from unit Zj to the output layers. Step 14: Reduce the learning rates β. Step 15: Test stopping condition for phase-II training. 10. ADAPTIVE RESONANCE THEORY NETWORK.  The Adaptive Resonance Theory (ART) was incorporated as a hypothesis for human cognitive data handling.  The hypothesis has prompted neural models for pattern recognition and unsupervised learning. ART system has been utilized to clarify different types of cognitive and brain data.  The Adaptive Resonance Theory addresses the stability-plasticity (stability can be defined as the nature of memorizing the learning and plasticity refers to the fact that they are flexible to gain new information) dilemma of a system that asks how learning can proceed in response to huge input patterns and simultaneously not to lose the stability for irrelevant patterns.  Other than that, the stability-elasticity dilemma is concerned about how a system can adapt new data while keeping what was learned before.  For such a task, a feedback mechanism is included among the ART neural network layers.
  • 24. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  In this neural network, the data in the form of processing elements output reflects back and ahead among layers. If an appropriate pattern is build-up, the resonance is reached, then adaption can occur during this period.  It can be defined as the formal analysis of how to overcome the learning instability accomplished by a competitive learning model, let to the presentation of an expended hypothesis, called adaptive resonance theory (ART).  This formal investigation indicated that a specific type of top-down learned feedback and matching mechanism could significantly overcome the instability issue.  It was understood that top-down attentional mechanisms, which had prior been found through an investigation of connections among cognitive and reinforcement mechanisms, had similar characteristics as these code-stabilizing mechanisms.  In other words, once it was perceived how to solve the instability issue formally, it also turned out to be certain that one did not need to develop any quantitatively new mechanism to do so.  One only needed to make sure to incorporate previously discovered attentional mechanisms. These additional mechanisms empower code learning to self- stabilize in response to an essentially arbitrary input system.  Grossberg presented the basic principles of the adaptive resonance theory.  A category of ART called ART1 has been described as an arrangement of ordinary differential equations by carpenter and Grossberg. These theorems can predict both the order of search as the function of the learning history of the system and the input patterns.  ART1 is an unsupervised learning model primarily designed for recognizing binary patterns.  It comprises an attentional subsystem, an orienting subsystem, a vigilance parameter, and a reset module, as given in the figure given below.  The vigilance parameter has a huge effect on the system. High vigilance produces higher detailed memories.  The ART1 attentional comprises of two competitive networks, comparison field layer L1 and the recognition field layer L2, two control gains, Gain1 and Gain2, and two short-term memory (STM) stages S1 and S2.  Long term memory (LTM) follows somewhere in the range of S1 and S2 multiply the signal in these pathways.
  • 25. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  Gains control empowers L1 and L2 to recognize the current stages of the running cycle. STM reset wave prevents active L2 cells when mismatches between bottom-up and top-down signals happen at L1.  The comparison layer gets the binary external input passing it to the recognition layer liable for coordinating it to a classification category.  This outcome is given back to the comparison layer to find out when the category coordinates the input vector.  If there is a match, then a new input vector is read, and the cycle begins once again. If there is a mismatch, then the orienting system is in charge of preventing the previous category from getting a new category match in the recognition layer.  The given two gains control the activity of the recognition and the comparison layer, respectively.  The reset wave specifically and enduringly prevents active L2 cell until the current is stopped. The offset of the input pattern ends its processing L1 and triggers the offset of Gain2. Gain2 offset causes consistent decay of STM at L2 and thereby prepares L2 to encode the next input pattern without bias.
  • 26. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani ART1 Implementation process:  ART1 is a self-organizing neural network having input and output neurons mutually couple using bottom-up and top-down adaptive weights that perform recognition.  To start our methodology, the system is first trained as per the adaptive resonance theory by inputting reference pattern data under the type of 5*5 matrix into the neurons for clustering within the output neurons.  Next, the maximum number of nodes in L2 is defined following by the vigilance parameter. The inputted pattern enrolled itself as short term memory activity over a field of nodes L1.  Combining and separating pathways from L1 to coding field L2, each weighted by an adaptive long-term memory track, transform into a net signal vector T.  Internal competitive dynamics at L2 further transform T, creating a compressed code or content addressable memory.  With strong competition, activation is concentrated at the L2 node that gets the maximal L1 → L2 signal.  The primary objective of this work is divided into four phases as follows Comparison, recognition, search, and learning. Advantage of adaptive learning theory (ART):  It can be coordinated and utilized with different techniques to give more precise outcomes.  It doesn't ensure stability in forming clusters.
  • 27. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  It can be used in different fields such as face recognition, embedded system, and robotics, target recognition, medical diagnosis, signature verification, etc.  It shows stability and is not disturbed by a wide range of inputs provided to inputs.  It has got benefits over competitive learning. The competitive learning cant include new clusters when considered necessary. Application of ART:  ART stands for Adaptive Resonance Theory. ART neural networks used for fast, stable learning and prediction have been applied in different areas.  The application incorporates target recognition, face recognition, medical diagnosis, signature verification, mobile control robot. i. Target recognition:  Fuzzy ARTMAP neural network can be used for automatic classification of targets depend on their radar range profiles.  Tests on synthetic data show the fuzzy ARTMAP can result in substantial savings in memory requirements when related to k nearest neighbor (kNN) classifiers.  The utilization of multi wavelength profiles mainly improves the performance of both kinds of classifiers.
  • 28. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani ii. Medical diagnosis:  Medical databases present huge numbers of challenges found in general information management settings where speed, use, efficiency, and accuracy are the prime concerns.  A direct objective of improved computer-assisted medicine is to help to deliver intensive care in situations that may be less than ideal.  Working with these issues has stimulated several ART architecture developments, including ARTMAP-IC. iii. Signature verification:  Automatic signature verification is a well-known and active area of research with various applications such as bank check confirmation, ATM access, etc.  The training of the network is finished using ART1 that uses global features as input vector and the verification and recognition phase uses a two-step process.  In the initial step, the input vector is coordinated with the stored reference vector, which was used as a training set, and in the second step, cluster formation takes place. iv. Mobile control robot:  Nowadays, we perceive a wide range of robotic devices. It is still a field of research in their program part, called artificial intelligence.  The human brain is an interesting subject as a model for such an intelligent system. Inspired by the structure of the human brain, an artificial neural emerges.  Similar to the brain, the artificial neural network contains numerous simple computational units, neurons that are interconnected mutually to allow the transfer of the signal from the neurons to neurons.  Artificial neural networks are used to solve different issues with good outcomes compared to other decision algorithms. Limitations of ART: Some ART networks are contradictory as they rely on the order of the training data, or upon the learning rate.
  • 29. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani 2 MARKS QUESTIONS AND ANSWERS 1. What is recall? Ans: If the input vector are uncorrelated, the Hebb rule will produce the correct weight and the response of the net when tested with one of the training vectors will be perfect recall. 2. Explain Learning vector Quantization. Ans: LVQ is adaptive data classification method. It is based on training data with desired class information. LVQ uses unsupervised data clustering techniques to preprocessing the data set and obtain cluster center. 3. What is meant by associative memory? Ans: Associative memory is also known as content addressable memory (CAM) or associative storage or associative array. It is a special type of memory that is optimized for performing searches through data, as opposed to providing a simple direct access to the data based on the address. It can store the set of patterns as memories when the associative memory is being presented with a key pattern, it responds by producing one of the stored pattern which closely resembles or relates to the key pattern. It can be viewed as data correlation here. Input data is correlated with that of stored data in the CAM. 4. Define auto associative memory. Ans: An auto-associative memory network, also known as a recurrent neural network, is a type of associative memory that is used to recall a pattern from partial or degraded inputs. In an auto-associative network, the output of the network is fed back into the input, allowing the network to learn and remember the patterns it has been trained on. This type of memory network is commonly used in applications such as speech and image recognition, where the input data may be incomplete or noisy. 5. What is Hebbian Learning? Ans: Hebb rule is the simplest and most common method of determining weights for an associative memory neural net. It can be us with patterns are represented as either binary or bipolar vectors. 6. What is Bidirectional Associative memory (BAM)? Ans: Bidirectional Associative Memory (BAM) is a type of artificial neural network designed for storing and retrieving heterogeneous pattern pairs. It plays a crucial role in various applications, such as password authentication, neural network models, and cognitive management.
  • 30. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani 7. List the problems of BAM Network. Ans:  Storage capacity of the BAM: In the BAM, stored number of associations should not be exceeded the number of neurons in the smaller layer.  Incorrect convergence: Always the closest association may not be produced by BAM. 8. What is content addressable memory? Ans: Content-addressable memory (CAM) is a special type of computer memory used in certain very- high-speed searching applications.  It is also known as associative memory or associative storage and compares input search data against a table of stored data, and returns the address of matching data. 9. What are the delta rule for pattern association? Ans: The delta rule is typically applied to the case in which pairs of pat- terns, consisting of an input pattern and a target output pattern, are to be associated.  When an input pattern is presented to an input layer of units, the appropriate output pattern will appear on the output layer of units. 10. What is continuous BAM? Ans: Continuous BAM transforms input smoothly and continuously into output in the range [0, 1] using the logistic sigmoid function as the activation function for all units. 11. Which are the rules used in Hebb’s law? Ans: i. If two neurons on either side of a connection are activated synchronously, then the weight of that connection is increased. ii. If two neurons on either side of a connection are activated asynchronously, then the weight of that connection is decreased. 12. What do you mean counter Propagation network? Ans: Counter propagation Network is supervised neural network that can be used for multimodal processing, but is not trained using the back propagation rule is the counter propagation network. This network has been specifically developed to provide bidirectional mapping between input and output training patterns. 13. What is Hopfield model? Ans: The Hopfield model is a single layered recurrent network. Like the associative memory, it is usually initialized with appropriate weights instead of being trained.
  • 31. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani 14. Define Self-Organizing Map. Ans: Self Organizing Map (or Kohonen Map or SOM) is a type of Artificial Neural Network which is also inspired by biological models of neural systems from the 1970s.  It follows an unsupervised learning approach and trained its network through a competitive learning algorithm.  SOM is used for clustering and mapping (or dimensionality reduction) techniques to map multidimensional data onto lower-dimensional which allows people to reduce complex problems for easy interpretation. 15. What is principle goal of the self-organizing map? Ans: The principal goal of the Self Organizing Map (SOM) is to transform an incoming signal pattern of arbitrary dimension into a one or two dimensional discrete map and to perform this transformation adaptively in a topologically ordered fashion. 16. List the stags of the SOM algorithm. Ans: i. Initialization: Choose random values for the initial weight vectors wj. ii. Sampling: Draw a sample training input vector x form the input space. iii. Matching: Find the winner neuron I(x) with weight vector closest to input vector. iv. Updating: Apply the weight update equation. v. Continuation: Keep returning to step 2 until the feature map stops changing. 17. How does counter Propagation nets are trained? Ans: Counter Propagation nets are trained in two stages: i. First Stage: The input vectors are clustered. The Clusters that are formed may be based on either the dot product metric or the Euclidean norm metric. ii. Second stage: The weight from the cluster units to the output units are adapted to produce the desired response. 18. List the possible drawback of counter propagation networks. Ans:  Training a counter Propagation network has the same difficulty associate with training a Kohonen network.
  • 32. CCS355 NN & DL Department of IT Asst.Prof.M.gokilavani  Counter Propagation networks tend to be larger than back propagation networks. If a certain number of mapping are to be learned, the middle layer must have that many number of neurons. 19. How forward only differs from full counter propagation nets? Ans:  In full counter propagation only the X vectors to form the clusters on the Kohonen units during the first stage of training.  The original presentation of forward only counter propagation used the Euclidean distance between the input vector and the weight vector for the Kohonen unit. 20. What is forward only counter propagation? Ans:  Is a simplified version of the full counter propagation.  Are intended to approximate y=f(x) function that is not necessarily invertible.  It may be used if the mapping from x to y is well define, but the mapping from y to x is not. 21. Define plasticity. Ans: The ability of a net to respond to learn a new pattern equally well at any stage of learning called plasticity. 22. List the components of ART1. Ans: Components are as follows:  The Short term memory layer (F1).  The recognition layer (F2): It contains the long term memory of the system.  Vigilance parameter (p): A parameter that control the generality of the memory. Larger p means more detailed memories. Smaller p produces more general memories.