Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. 5/7/2012 WHAT ARE NEURAL NETWORKS? Neural networks are parallel information processing systems with their architecture inspired by the structure and functioning of the brain A neural network have the ability to learn and generalize. Neural networks can be trained to make classifications and predictions based on historical data Neural nets are included in many data mining products 1
  2. 2. 5/7/2012 Very popular and effective techniques Artificial Neural Network goes by many names, such as connectionism, parallel distributed processing, neuro-computing and natural intelligent systems. It is abbreviated by ANN ANN has a strong similarity to the biological brain. It is composed of interconnected elements called neurons. 3The Biological Neuron  Biological neuron receives inputs from other sources, combines them in some way, performs a generally nonlinear operation on the result, and then output the final result. send Inputs Process the inputs (eyes, ears) Turn the processed inputs into outputs The electrochemical contact between neurons 4 2
  3. 3. 5/7/2012The Artificial Neuron 5Network Layers The neurons are grouped into layers. The input layer. The output layer. Hidden layers between these two layers. 6 3
  4. 4. 5/7/2012Neural Network Architecture In a Neural Network, neurons are grouped into layers. The neurons in each layer are the same type. There are different types of Layers. The Input layer, consists of neurons that receive input from the external environment. The Output layer, consists of neurons that communicate to the user or external environment. The Hidden layer, consists of neurons that ONLY communicate with other layers of the network.  Now that we have a model for an artificial neuron, we can imagine connecting many of then together to form an Artificial Neural Network: Output layer Hidden layer Input layer 4
  5. 5. 5/7/2012 Based on the following assumptions:1. Information processing occurs at many simple processing elements called neurons.2. Signals are passed between neurons over interconnection links.3. Each interconnection link has an associated weight.4. Each neuron applies an activation function to determine its output signal. A Neuron x0 - mk w0 x1 w1  f output y xn wn Input weight weighted Activation vector x vector w sum function  The n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping 5
  6. 6. 5/7/2012What is a neuron?  a (biological) neuron is a node that has many inputs and one output  f  inputs come from other neurons  the inputs are weighted  weights can be both positive and negative  inputs are summed at the node to produce an activation valueThe Artificial Neuron Model In order to simulate neurons on a computer, we need a mathematical model of this node Node j ( a neuron) has n inputs xi , i = 1 to n Each input (connection) is associated with a weight wij The neuron includes a bias denoted by bj The bias has the effect of increasing or deceasing the net input 12 6
  7. 7. 5/7/2012 The net input to node j (net result) ,uj , is the sum of the products of the connection inputs and their weights: n u j   w ijxi  b j i 1Where uj is called the “activation of the neuron”. The output of node j is determined by applying a non- linear transfer function g to the net input: x j  g(u j ) called the “transfer function”. A common choice for the transfer function is the sigmoid: 1 g(u j )  u j 1 e The sigmoid has similar non-linear properties to the transfer function of real neurons:  It accepts inputs varying from – ∞ to ∞  bounded below by 0  bounded above by 1 7
  8. 8. 5/7/2012An Example of multilayer feedforward neural network 8
  9. 9. 5/7/2012Architecture of ANN Feed-Forward Networks The signals travel one way from input to output Feed-Back Networks The signals travel as loops in the network, the output of the network is connected to the input. 17Learning ANN The purpose of the learning function is to modify the variable connection weights causes the network to learn the solution to a problem according to some neural based algorithm. There are tow types of learning  Supervised (Reinforcement) learning.  Unsupervised learning. 18 9
  10. 10. 5/7/2012Supervised learning Means their exist an external help or a teacher. The teacher may be a training set of data. The target is to minimize the error between the desired and the actual output. The process take place as follows: 19  Presenting input and output data to the network, this data is often referred to as the “training set”.  The network processes the inputs and compares its resulting outputs against the desired outputs.  Errors are then propagated back through the system, causing the system to adjust the weights, which are usually randomly set to begin with.  This process occurs over and over as a closer match between the desired and the predicted output. 20 10
  11. 11. 5/7/2012 When no further learning is necessary, the weights are typically frozen for the application. There are many algorithms used to implement the adaptive feedback required to adjust the weights during training. The most common technique is called “Back-Propagation”.Let us suppose that a sufficiently large set ofexamples (training set) is available.Supervised learning:– The network answer to each input pattern isdirectly compared with the desired answer and afeedback is given to the network to correct possibleerrors 11
  12. 12. 5/7/2012Back Propagation (BP) It is an improving performance method in training of multilayered feed forward neural networks. Are used to adjust the weights and biases of networks to minimize the sum squared error of the network which is given by 1 SSE  2  (xt  xt )2 ˆ where xt and xt^ are the desired and predicted output of the tth output node. 23BP Network – Supervised Training Desired output of the training examples Error = difference between actual & desired output Change weight relative to error size Calculate output layer error , then propagate back to previous layer Hidden weights updated Improved performance 24 12
  13. 13. 5/7/2012 Probably the most common type of ANN used today is a multilayer feed forward network trained using back-propagation (BP) Often called a Multilayer Perceptron (MLP) Unsupervised learning The training set consists of input training patterns but not with desired outputs. This method often referred to as self-organization. Therefore, the network is trained without benefit of any teacher. 26 13
  14. 14. 5/7/2012  Applications in Clustering and reducing dimensionality  Learning may be very slow  No help from the outside  No training data, no information available on the desired output  Learning by doing  Used to pick out structure in the input:  Clustering  Compression 27Choosing the network size It seems better to start with a small number of neurons, because:  learning is faster.  it is often enough.  it avoids over-fitting problems. If the number of neurons are too much we will get an over fit. In principle, one hidden layer is sufficient to solve any problem. In practice, it may happen that two hidden layers with a small number of neurons may work better (and/or learn faster) than a network with a single layer. 28 14
  15. 15. 5/7/2012• Too few nodes: Don’t fit the curve very well• Too many nodes: Over parameterization • May fit noise • makes network more difficult to trainThe Learning Rate The learning rate c, which determines by how much we change the weights w at each step. If c is too small, the algorithm will take a long time to converge. Sum-Square Error Error Surface Epoch 30 15
  16. 16. 5/7/2012 If c is too large, the network may not be able to make the fine discriminations possible with a system that learns more slowly. The algorithm diverges. Sum-Square Error Error Surface Epoch 31 Multi-Layer PerceptronOutput vector Errj  O j (1  O j ) Errk w jkOutput nodes k  j   j  (l) Errj wij  wij  (l ) Errj OiHidden nodes Errj  O j (1  O j )(T j  O j ) wij 1 Oj  I 1 e jInput nodes I j   wij Oi   j iInput vector: xi 16
  17. 17. 5/7/2012Applications of ANNs Prediction – weather, stocks, disease, Predicting financial time series Classification – financial risk assessment, image processing Data Association – Text Recognition Data Conceptualization – Customer purchasing habits Filtering – Normalizing telephone signals Optimization 33 Diagnosing medical conditions Identifying clusters in customer databases Identifying fraudulent credit card transactions Hand-written character recognition and many more…. 17
  18. 18. 5/7/2012  Advantages  Adapt to unknown situations  Autonomous learning and generalization  Their most important advantage is in solving problems that are too complex for conventional technologies, problems that do not have an algorithmic solution or for which an algorithmic solution is too complex to be found.  Disadvantages  Not exact  Large complexity of the network structure 35Using a neural network for prediction  Identify input and outputs  Preprocess inputs - often scale to the range [0,1]  Choose an ANN architecture  Train the ANN with a representative set of training examples (usually using BP)  Test the ANN with another set of known examples  often the known data set is divided in to training and test sets. Cross-validation is a more rigorous validation procedure.  Apply the model to unknown input data 18