Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extreme learning machine:Theory and applications

12,197 views

Published on

20120315

Published in: Technology, Education
  • Login to see the comments

Extreme learning machine:Theory and applications

  1. 1. Extreme learning machine:Theory and applicationsG.-B. Huang, Q.-Y. Zhu, and C.-K. SiewNeurocomputing, 2006 Presenter: James Chou 2012/03/15
  2. 2. Outline2  Introduction  Single-hidden layer feed-forward neural networks  Neural Network Mathematical Model  Back Propagation algorithm  ELM Mathematical Model  Performance Evaluation  Conclusion
  3. 3. Introduction3  For past decades, gradient descent based methods have mainly been used in many learning algorithms of feed-forward neural networks.  Traditionally, all the parameters of the feed-forward neural networks need to tune iterative and need a very long time to learn.  When the input weights and the hidden layer biases are randomly assigned, SLFNs (single-hidden layer feed-forward neural networks) can be simply considered as a linear system and the output weights (linking the hidden layer to the output layer) can be computed through simple generalized inverse operation.
  4. 4. Introduction (Cont.)4  Based on this idea, this paper proposes a simple learning algorithm for SLFNs called extreme learning.  Different from traditional learning algorithms the extreme learning algorithm not only provide the smaller training error but also the better performance.
  5. 5. Single-hidden layer feed-forward5 neural networks N Output  F ( i xi   ) i 1 θ is the threshold  F(.) is activation function  Hard Limiter function  1, when x   f ( x)   0, when x     Sigmoid function 1 f ( x)  1  e x
  6. 6. Single-hidden layer feed-forward6 neural networks (Cont.) G() is activation function L is number of hidden layer nodes
  7. 7. Neural Network Mathematical Model7
  8. 8. Neural Network Mathematical Model (Cont.)8 If ε = 0 , mean FL(x) = f(x) = T , T is known target and Cost function = 0
  9. 9. Neural Network Mathematical Model (Cont.)9 
  10. 10. Back Propagation algorithm10  BP algorithm is the classic gradient base algorithm to find the best weight vectors and minimize the cost function. Demo BP algorithm! η is Leaming Rate
  11. 11. ELM Mathematical Model11  H+ is the Moore-Penrose generalized inverse of hidden layer output matrix H.  H+ = (HTH)-1HT
  12. 12. ELM Mathematical Model (Cont.)12 
  13. 13. ELM Mathematical Model (Cont.)13 
  14. 14. Regression of SinC Function15
  15. 15. Regression of SinC Function (Cont.)16  100000 training data with 5-20% noise.  100000 testing data is noise free. Demo  The result of training 50 times in the ELM! following table. Noise TrainingTime_AVG(sec) TrainingRMS_AVG TestingRMS_AVG 5% 0.6462 0.0113 2.201e-04=0.00022 10% 0.6306 0.0224 2.753e-04=0.00027 15% 0.6427 0.0334 8.336e-04=0.00083 20% 0.6452 0.0449 11.541e-04=0.00115
  16. 16. Real-World Regression Problems17
  17. 17. Real-World Regression Problems (Cont.)18
  18. 18. Real-World Regression Problems (Cont.)19
  19. 19. Real-World Regression Problems (Cont.)20
  20. 20. Real-World Very Large Complex Applications21
  21. 21. Real Medical Diagnosis Application: Diabetes22
  22. 22. Protein Sequence Classification23
  23. 23. Conclusion24  Advantages  ELM needs less training time compared to popular BP and SVM/SVR.  The prediction performance of ELM is usually a little better than BP and close to SVM/SVR in many applications.  Only need to turn the parameter L (hidden layer nodes).  Nonlinear activation function still can work in ELM.  Disadvantages  How to find the optimal soluction?  Local minima issue.  Easy Overfitting.

×