1. Assignment-1
Name: om nath kushwaha
Roll No: 2K18/CO/409
Artificial Neural Network
The term "Artificial Neural Network" is derived from Biological neural networks that
develop the structure of a human brain. Similar to the human brain that has neurons
interconnected to one another, artificial neural networks also have neurons that are
interconnected to one another in various layers of the networks. These neurons are known as
nodes.
The given figure illustrates the typical diagram of Biological Neural Network.
The typical Artificial Neural Network looks something like the given figure.
2. Dendrites from Biological Neural Network represent inputs in Artificial Neural Networks,
cell nucleus represents Nodes, synapse represents Weights, and Axon represents Output.
An Artificial Neural Network in the field of Artificial intelligence where it attempts to mimic
the network of neurons makes up a human brain so that computers will have an option to
understand things and make decisions in a human-like manner. The artificial neural network
is designed by programming computers to behave simply like interconnected brain cells.
There are around 1000 billion neurons in the human brain. Each neuron has an association
point somewhere in the range of 1,000 and 100,000. In the human brain, data is stored in such
a manner as to be distributed, and we can extract more than one piece of this data when
necessary, from our memory parallelly. We can say that the human brain is made up of
incredibly amazing parallel processors.
We can understand the artificial neural network with an example, consider an example of a
digital logic gate that takes an input and gives an output. "OR" gate, which takes two inputs.
If one or both the inputs are "On," then we get "On" in output. If both the inputs are "Off,"
then we get "Off" in output. Here the output depends upon input. Our brain does not perform
the same task. The outputs to inputs relationship keep changing because of the neurons in our
brain, which are "learning."
The architecture of an artificial neural network:
To understand the concept of the architecture of an artificial neural network, we have to
understand what a neural network consists of. In order to define a neural network that
consists of a large number of artificial neurons, which are termed units arranged in a
sequence of layers. Let’s us look at various types of layers available in an artificial neural
network.
3. Artificial Neural Network primarily consists of three layers:
Input Layer:
As the name suggests, it accepts inputs in several different formats provided by the
programmer.
Hidden Layer:
The hidden layer presents in-between input and output layers. It performs all the calculations
to find hidden features and patterns.
Output Layer:
The input goes through a series of transformations using the hidden layer, which finally
results in output that is conveyed using this layer.
The artificial neural network takes input and computes the weighted sum of the inputs and
includes a bias. This computation is represented in the form of a transfer function.
It determines weighted total is passed as an input to an activation function to produce the
output. Activation functions choose whether a node should fire or not. Only those who are
fired make it to the output layer. There are distinctive activation functions available that can
be applied upon the sort of task we are performing.
How do artificial neural networks work?
4. Artificial Neural Network can be best represented as a weighted directed graph, where the
artificial neurons form the nodes. The association between the neurons outputs and neuron
inputs can be viewed as the directed edges with weights. The Artificial Neural Network
receives the input signal from the external source in the form of a pattern and image in the
form of a vector. These inputs are then mathematically assigned by the notations x(n) for
every n number of inputs.
Afterward, each of the input is multiplied by its corresponding weights ( these weights are the
details utilized by the artificial neural networks to solve a specific problem ). In general
terms, these weights normally represent the strength of the interconnection between neurons
inside the artificial neural network. All the weighted inputs are summarized inside the
computing unit.
If the weighted sum is equal to zero, then bias is added to make the output non-zero or
something else to scale up to the system's response. Bias has the same input, and weight
equals to 1. Here the total of weighted inputs can be in the range of 0 to positive infinity.
Here, to keep the response in the limits of the desired value, a certain maximum value is
benchmarked, and the total of weighted inputs is passed through the activation function.
The activation function refers to the set of transfer functions used to achieve the desired
output. There is a different kind of the activation function, but primarily either linear or non-
linear sets of functions. Some of the commonly used sets of activation functions are the
Binary, linear, and Tan hyperbolic sigmoidal activation functions. Let us take a look at each
of them in details:
Binary:
In binary activation function, the output is either a one or a 0. Here, to accomplish this, there
is a threshold value set up. If the net weighted input of neurons is more than 1, then the final
output of the activation function is returned as one or else the output is returned as 0.
Sigmoidal Hyperbolic:
The Sigmoidal Hyperbola function is generally seen as an "S" shaped curve. Here the tan
hyperbolic function is used to approximate output from the actual net input. The function is
defined as:
5. F(x) = (1/1 + exp(-????x))
Where ???? is considered the Steepness parameter.
Genetic Algorithms
Genetic Algorithms (GAs) are adaptive heuristic search algorithms that belong to the larger
part of evolutionary algorithms. Genetic algorithms are based on the ideas of natural selection
and genetics. These are intelligent exploitation of random search provided with historical data
to direct the search into the region of better performance in solution space. They are
commonly used to generate high-quality solutions for optimization problems and search
problems.
Genetic algorithms simulate the process of natural selection which means those species who
can adapt to changes in their environment are able to survive and reproduce and go to next
generation. In simple words, they simulate “survival of the fittest” among individual of
consecutive generation for solving a problem. Each generation consist of a population of
individuals and each individual represents a point in search space and possible solution. Each
individual is represented as a string of character/integer/float/bits. This string is analogous to
the Chromosome.
Foundation of Genetic Algorithms
Genetic algorithms are based on an analogy with genetic structure and behaviour of
chromosome of the population. Following is the foundation of GAs based on this analogy –
Individual in population compete for resources and mate
Those individuals who are successful (fittest) then mate to create more offspring than others
Genes from “fittest” parent propagate throughout the generation, that is sometimes parents
create offspring which is better than either parent.
Thus, each successive generation is more suited for their environment.
Search space
The population of individuals are maintained within search space. Each individual represent a
solution in search space for given problem. Each individual is coded as a finite length vector
(analogous to chromosome) of components. These variable components are analogous to
Genes. Thus a chromosome (individual) is composed of several genes (variable components).
6. Fitness Score
A Fitness Score is given to each individual which shows the ability of an individual to
“compete”. The individual having optimal fitness score (or near optimal) are sought.
The GAs maintains the population of n individuals (chromosome/solutions) along with their
fitness scores. The individuals having better fitness scores are given more chance to
reproduce than others. The individuals with better fitness scores are selected who mate and
produce better offspring by combining chromosomes of parents. The population size is static
so the room has to be created for new arrivals. So, some individuals die and get replaced by
new arrivals eventually creating new generation when all the mating opportunity of the old
population is exhausted. It is hoped that over successive generations better solutions will
arrive while least fit die.
Each new generation has on average more “better genes” than the individual (solution) of
previous generations. Thus, each new generations have better “partial solutions” than
previous generations. Once the offspring’s produced having no significant difference than
offspring produced by previous populations, the population is converged. The algorithm is
said to be converged to a set of solutions for the problem.
Operators of Genetic Algorithms
Once the initial generation is created, the algorithm evolve the generation using following
operators –
1) Selection Operator: The idea is to give preference to the individuals with good fitness
scores and allow them to pass there genes to the successive generations.
2) Crossover Operator: This represents mating between individuals. Two individuals are
selected using selection operator and crossover sites are chosen randomly. Then the genes at
these crossover sites are exchanged thus creating a completely new individual (offspring).
For example –
3) Mutation Operator: The key idea is to insert random genes in offspring to maintain the
diversity in population to avoid the premature convergence. For example –
7. The whole algorithm can be summarized as –
1) Randomly initialize populations p
2) Determine fitness of population
3) Until convergence repeat:
a) Select parents from population
b) Crossover and generate new population
c) Perform mutation on new population
d) Calculate fitness for new population
Particle Swarm Optimization (PSO)
• PSO is stochastic optimization technique proposed by Kennedy and Eberhart (1995).
• A population-based search method with position of particle is representing solution
and Swarm of particles as searching agent.
• PSO is a robust evolutionary optimization technique based on the movement and
intelligence of swarms.
• PSO find the minimum value for the function.
• The idea is similar to bird flocks searching for food.
➢ Bird = a particle, Food = a solution
➢ pbest = the best solution (fitness) a particle has achieved so far.
➢ gbest = the global best solution of all particles within the swarm
PSO Search Scheme
The basic concept of PSO lies in accelerating each particle toward its pbest and the gbest
locations, with a random weighted acceleration at each time.
PSO uses a number of agents, i.e., particles, that constitute a swarm flying in the search space
looking for the best solution.
Each particle is treated as a point (candidate solution) in a N-dimensional space which adjusts
its “flying” according to its own flying experience as well as the flying experience of other
particles.
8. Each particle tries to modify its position X using the following formula:
X (t+1) = X(t) + V(t+1) (1)
V(t+1) = wV(t) +
c1 ×rand ( ) × ( Xpbest - X(t)) + c2 ×rand ( ) × ( Xgbest - X(t)) (2)
Alpine function
Particle fly and search for the highest peak in the search space
f( x1, ,xD) = sin x1
( ) sin xD
( ) x1 xD
9. PSO Algorithm
The PSO algorithm pseudocode [2] as following:
Input: Randomly initialized position and velocity of Particles:
Xi (0) and Vi (0)
Output: Position of the approximate global minimum X*
1: while terminating condition is not reached do
2: for i = 1 to number of particles do
3: Calculate the fitness function f
4: Update personal best and global best of each particle
5: Update velocity of the particle using Equation 2
6: Update the position of the particle using equation 1
7: end for
8: end while
Hybrid Systems
A Hybrid system is an intelligent system which is framed by combining atleast two intelligent
technologies like Fuzzy Logic, Neural networks, Genetic algorithm, reinforcement Learning,
etc. The combination of different techniques in one computational model make these systems
possess an extended range of capabilities. These systems are capable of reasoning and
learning in an uncertain and imprecise environment. These systems can provide human-like
expertise like domain knowledge, adaptation in noisy environment etc.
Types of Hybrid Systems:
➢ Neuro Fuzzy Hybrid systems
➢ Neuro Genetic Hybrid systems
➢ Fuzzy Genetic Hybrid systems
(A) Neuro Fuzzy Hybrid systems:
Neuro fuzzy system is based on fuzzy system which is trained on the basis of working of
neural network theory. The learning process operates only on the local information and
causes only local changes in the underlying fuzzy system. A neuro-fuzzy system can be seen
as a 3-layer feedforward neural network. The first layer represents input variables, the middle
(hidden) layer represents fuzzy rules and the third layer represents output variables. Fuzzy
sets are encoded as connection weights within the layers of the network, which provides
functionality in processing and training the model.
10. Working flow:
• In input layer, each neuron transmits external crisp signals directly to the next layer.
• Each fuzzification neuron receives a crisp input and determines the degree to which
the input belongs to input fuzzy set.
• Fuzzy rule layer receives neurons that represent fuzzy sets.
• An output neuron, combines all inputs using fuzzy operation UNION.
• Each defuzzification neuron represents single output of neuro-fuzzy system.
Advantages:
• It can handle numeric, linguistic, logic, etc kind of information.
• It can manage imprecise, partial, vague or imperfect information.
• It can resolve conflicts by collaboration and aggregation.
• It has self-learning, self-organizing and self-tuning capabilities.
• It can mimic human decision-making process.
Disadvantages:
• Hard to develop a model from a fuzzy system
• Problems of finding suitable membership values for fuzzy systems
• Neural networks cannot be used if training data is not available.
Applications:
• Student Modelling
• Medical systems
• Traffic control systems
• Forecasting and predictions
(B) Neuro Genetic Hybrid systems:
11. A Neuro Genetic hybrid system is a system that combines Neural networks: which are
capable to learn various tasks from examples, classify objects and establish relation between
them and Genetic algorithm: which serves important search and optimization techniques.
Genetic algorithms can be used to improve the performance of Neural Networks and they can
be used to decide the connection weights of the inputs. These algorithms can also be used for
topology selection and training network.
Working Flow:
• GA repeatedly modifies a population of individual solutions. GA uses three main
types of rules at each step to create the next generation from the current population:
➢ Selection to select the individuals, called parents, that contribute to the
population at the next generation
➢ Crossover to combine two parents to form children for the next
generation
➢ Mutation to apply random changes to individual parents in order to
form children
• GA then sends the new child generation to ANN model as new input parameter.
• Finally, calculating of the fitness by developed ANN model is performed.
Advantages:
• GA is used for topology optimization i.e to select number of hidden layers, number of
hidden nodes and interconnection pattern for ANN.
12. • In GAs, the learning of ANN is formulated as a weight optimization problem, usually
using the inverse mean squared error as a fitness measure.
• Control parameters such as learning rate, momentum rate, tolerance level, etc are also
optimized using GA.
• It can mimic human decision-making process.
Disadvantages:
• Highly complex system.
• Accuracy of the system is dependent on the initial population.
• Maintaintainance costs are very high.
Applications:
• Face recognition
• DNA matching
• Animal and human research
• Behavioural system
(C) Fuzzy Genetic Hybrid systems:
A Fuzzy Genetic Hybrid System is developed to use fuzzy logic based techniques for
improving and modelling Genetic algorithms and vice-versa. Genetic algorithm has proved to
be a robust and efficient tool to perform tasks like generation of fuzzy rule base, generation
of membership function etc.
Three approaches that can be used to develop such system are:
• Michigan Approach
• Pittsburgh Approach
• IRL Approach
13. Working Flow:
• Start with an initial population of solutions that represent first generation.
• Feed each chromosome from the population into the Fuzzy logic controller and
compute performance index.
• Create new generation using evolution operators till some condition is met.
Advantages:
• GAs are used to develop the best set of rules to be used by a fuzzy inference engine
• GAs are used to optimize the choice of membership functions.
• A Fuzzy GA is a directed random search over all discrete fuzzy subsets.
• It can mimic human decision-making process.
Disadvantages:
• Interpretation of results is difficult.
• Difficult to build membership values and rules.
• Takes lots of time to converge.
Applications:
• Mechanical Engineering
• Electrical Engine
• Artificial Intelligence
• Economics
