Successfully reported this slideshow.
Upcoming SlideShare
×

# Extreme learning machine:Theory and applications

12,197 views

Published on

20120315

Published in: Technology, Education
• Full Name
Comment goes here.

Are you sure you want to Yes No

### Extreme learning machine:Theory and applications

1. 1. Extreme learning machine:Theory and applicationsG.-B. Huang, Q.-Y. Zhu, and C.-K. SiewNeurocomputing, 2006 Presenter: James Chou 2012/03/15
2. 2. Outline2  Introduction  Single-hidden layer feed-forward neural networks  Neural Network Mathematical Model  Back Propagation algorithm  ELM Mathematical Model  Performance Evaluation  Conclusion
3. 3. Introduction3  For past decades, gradient descent based methods have mainly been used in many learning algorithms of feed-forward neural networks.  Traditionally, all the parameters of the feed-forward neural networks need to tune iterative and need a very long time to learn.  When the input weights and the hidden layer biases are randomly assigned, SLFNs (single-hidden layer feed-forward neural networks) can be simply considered as a linear system and the output weights (linking the hidden layer to the output layer) can be computed through simple generalized inverse operation.
4. 4. Introduction (Cont.)4  Based on this idea, this paper proposes a simple learning algorithm for SLFNs called extreme learning.  Different from traditional learning algorithms the extreme learning algorithm not only provide the smaller training error but also the better performance.
5. 5. Single-hidden layer feed-forward5 neural networks N Output  F ( i xi   ) i 1 θ is the threshold  F(．) is activation function  Hard Limiter function  1, when x   f ( x)   0, when x     Sigmoid function 1 f ( x)  1  e x
6. 6. Single-hidden layer feed-forward6 neural networks (Cont.) G() is activation function L is number of hidden layer nodes
7. 7. Neural Network Mathematical Model7
8. 8. Neural Network Mathematical Model (Cont.)8 If ε = 0 , mean FL(x) = f(x) = T , T is known target and Cost function = 0
9. 9. Neural Network Mathematical Model (Cont.)9 
10. 10. Back Propagation algorithm10  BP algorithm is the classic gradient base algorithm to find the best weight vectors and minimize the cost function. Demo BP algorithm! η is Leaming Rate
11. 11. ELM Mathematical Model11  H+ is the Moore-Penrose generalized inverse of hidden layer output matrix H.  H+ = (HTH)-1HT
12. 12. ELM Mathematical Model (Cont.)12 
13. 13. ELM Mathematical Model (Cont.)13 
14. 14. Regression of SinC Function15
15. 15. Regression of SinC Function (Cont.)16  100000 training data with 5-20% noise.  100000 testing data is noise free. Demo  The result of training 50 times in the ELM! following table. Noise TrainingTime_AVG(sec) TrainingRMS_AVG TestingRMS_AVG 5% 0.6462 0.0113 2.201e-04=0.00022 10% 0.6306 0.0224 2.753e-04=0.00027 15% 0.6427 0.0334 8.336e-04=0.00083 20% 0.6452 0.0449 11.541e-04=0.00115
16. 16. Real-World Regression Problems17
17. 17. Real-World Regression Problems (Cont.)18
18. 18. Real-World Regression Problems (Cont.)19
19. 19. Real-World Regression Problems (Cont.)20
20. 20. Real-World Very Large Complex Applications21
21. 21. Real Medical Diagnosis Application: Diabetes22
22. 22. Protein Sequence Classification23
23. 23. Conclusion24  Advantages  ELM needs less training time compared to popular BP and SVM/SVR.  The prediction performance of ELM is usually a little better than BP and close to SVM/SVR in many applications.  Only need to turn the parameter L (hidden layer nodes).  Nonlinear activation function still can work in ELM.  Disadvantages  How to find the optimal soluction?  Local minima issue.  Easy Overfitting.