Workshop on Grand Challenges of Computational Intelligence (Cyprus, September 14, 2012)     Scalability Improvement of  Ge...
Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. ...
Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. ...
Basic Idea of Evolutionary Computation Environment  PopulationIndividual
Basic Idea of Evolutionary Computation Environment              Environment  Population                Population         ...
Basic Idea of Evolutionary Computation Environment             Environment             Environment  Population            ...
Basic Idea of Evolutionary ComputationEnvironment                                     Environment Population              ...
Applications of Evolutionary ComputationDesign of High Speed TrainsEnvironment                      Population            ...
Applications of Evolutionary ComputationDesign of Stock Trading AlgorithmsEnvironment                    Population     In...
Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. ...
Genetics-Based Machine LearningKnowledge Extraction from Numerical Data                                If (condition) then...
Design of Rule-Based SystemsEnvironment                              Population   If ... Then ...   If ... Then ...   If ....
Design of Decision TreesEnvironment                   Population       Individual = Decision Tree (   )
Design of Neural NetworksEnvironment                    Population       Individual = Neural Network (   )
Multi-Objective EvolutionMinimization of Errors and Complexity    Large    Error                                  Initial ...
Multi-Objective EvolutionMinimization of Errors and Complexity    Large    Error    Small            Simple   Complexity  ...
Multi-Objective EvolutionA number of different neural networks    Large    Error    Small            Simple   Complexity  ...
Multi-Objective EvolutionA number of different decision trees    Large    Error    Small            Simple   Complexity   ...
Multi-Objective EvolutionA number of fuzzy rule-based systems LargeErrorSmall         Simple   Complexity   Complicated
Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. ...
Difficulty in Applications to Large DataComputation Load for Fitness EvaluationEnvironment                                ...
Difficulty in Applications to Large DataComputation Load for Fitness EvaluationEnvironment                                ...
Difficulty in Applications to Large DataComputation Load for Fitness EvaluationEnvironment  Population  If ... Then ...  I...
The Main Issue in This PresentationHow to Decrease the Computation LoadEnvironment  Population  If ... Then ...  If ... Th...
Fitness CalculationStandard Non-Parallel ModelEnvironment  Population  If ... Then ...  If ... Then ...  If ... Then ...  ...
Fitness CalculationStandard Non-Parallel ModelEnvironment  Population  If ... Then ...  If ... Then ...  If ... Then ...  ...
Fitness CalculationStandard Non-Parallel ModelEnvironment  Population  If ... Then ...  If ... Then ...  If ... Then ...  ...
Fitness CalculationStandard Non-Parallel ModelEnvironment  Population                    Fitness Evaluation  If ... Then ....
A Popular Approach for Speed-UpParallel Computation of Fitness Evaluation                     Fitness Evaluation   If ... ...
Another Approach for Speed-UpTraining Data Reduction   If ... Then ...   If ... Then ...   If ... Then ...   If ... Then ....
Another Approach for Speed-UpTraining Data Reduction   If ... Then ...   If ... Then ...   If ... Then ...   If ... Then ....
Another Approach for Speed-UpTraining Data Reduction   If ... Then ...   If ... Then ...   If ... Then ...   If ... Then ....
Another Approach for Speed-UpTraining Data Reduction   If ... Then ...   If ... Then ...   If ... Then ...   If ... Then ....
Another Approach for Speed-UpTraining Data Reduction   If ... Then ...   If ... Then ...   If ... Then ...   If ... Then ....
Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPS...
Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPS...
Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPS...
Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPS...
Visual Image of WindowingA population is moving around in the training data.               Training Data = Environment    ...
Visual Image of WindowingA population is moving around in the training data.     Training Data = Environment              ...
Visual Image of WindowingA population is moving around in the training data.     Training Data = Environment              ...
Visual Image of WindowingA population is moving around in the training data.     Training Data = Environment              ...
Visual Image of WindowingA population is moving around in the training data.     Training Data = Environment              ...
Visual Image of WindowingA population is moving around in the training data.       Training Data = Environment            ...
Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule S...
Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule S...
Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule S...
Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule S...
Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule S...
Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule S...
Computation Load of Four Models           EC           CPU        Fitness       EvaluationStandard Non-Parallel Model Comp...
Computation Load of Four Models           EC            EC = Evolutionary Computation           CPU        Fitness        ...
Computation Load of Four Models             EC             CPU         Fitness        EvaluationStandard Non-Parallel Mode...
Computation Load of Four Models             EC                                EC             CPU                          ...
Computation Load of Four Models             EC                                 EC             CPU                         ...
Computation Load of Four Models             EC                                 EC             CPU                         ...
Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. ...
Our Model in Computational Experimentswith Seven Subpopulations and Seven Data Subsets
Our Model in Computational Experimentswith Seven Subpopulations and Seven Data Subsets                 Seven CPUs        S...
Standard Non-Parallel Non-Distributed Modelwith a Single Population and a Single Data Set           Population
Standard Non-Parallel Non-Distributed Modelwith a Single Population and a Single Data Set                              Sin...
Standard Non-Parallel Non-Distributed Modelwith a Single Population and a Single Data Set                                 ...
Comparison of Computation LoadComputation Load on a Single CPU per GenerationStandard Model:               Parallel Distri...
Comparison of Computation LoadComputation Load on a Single CPU per GenerationStandard Model:               Parallel Distri...
Comparison of Computation LoadComputation Load on a Single 1/7 = per GenerationComputation Load ==> 1/7 x CPU 1/49 (about ...
Data Sets in Computational ExperimentsNine Pattern Classification Problems  Name of      Number of   Number of    Number o...
Computation Time for 50,000 GenerationsComputation time was decreased to about 2%  Name of     Standard    Our Model Perce...
Computation Time for 50,000 GenerationsComputation time was decreased to about 2%  Name of     Standard      Our Model Per...
Computation Time for 50,000 GenerationsComputation time was decreased to about 2%  Name of     Standard      Our Model Per...
Test Data Error Rates (Results of 3x10CV)Test data accuracy was improved for six data sets  Name of     Standard    Our Mo...
Test Data Error Rates (Results of 3x10CV)Test data accuracy was improved for six data sets  Name of     Standard    Our Mo...
Q. Why did our model improve the test data accuracy ?A. Because our model improved the search ability.  Data Set          ...
Q. Why did our model improve the search ability ?A. Because our model maintained the diversity.                       Data...
Effects of Rotation and Migration Intervals      Training Data Rotation: Every 100 Generations      Rule Set Migration: Ev...
Effects of Rotation and Migration Intervals(Rotations in the opposite directions)                                         ...
Effects of Rotation and Migration Intervals(Rotations in the opposite directions)                                         ...
Effects of Rotation and Migration Intervals(Rotations in the same direction)                                             R...
Effects of Rotation and Migration Intervals(Rotations in the same direction)                                              ...
Effects of Rotation and Migration Intervals18                                                 1816                        ...
Effects of Rotation and Migration Intervals                                                           Interaction between ...
Effects of Rotation and Migration Intervals                                                           Interaction between ...
Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. ...
Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Int...
Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Int...
Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Int...
Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Int...
Conclusion5. A little bit different model may be also possible for learning  from locally located data bases.Introduction ...
Conclusion6. This model can be also used for learning from changing  data bases.Introduction       CPU                    ...
Conclusion6. This model can be also used for learning from changing  data bases.Introduction       CPU                    ...
Conclusion6. This model can be also used for learning from changing  data bases.Introduction       CPU                    ...
Conclusion6. This model can be also used for learning from changing  data bases.Introduction       CPU                    ...
Conclusion6. This model can be also used for learning from changing  data bases.Introduction       CPU                    ...
Conclusion   Thank you very much!
Upcoming SlideShare
Loading in …5
×

Hisao Ishibuchi: "Scalability Improvement of Genetics-Based Machine Learning to Large Data Sets"

372 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
372
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Hisao Ishibuchi: "Scalability Improvement of Genetics-Based Machine Learning to Large Data Sets"

  1. 1. Workshop on Grand Challenges of Computational Intelligence (Cyprus, September 14, 2012) Scalability Improvement of Genetics-Based Machine Learning to Large Data Sets Hisao Ishibuchi Osaka Prefecture University, Japan
  2. 2. Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. Parallel Distributed Implementation 4. Computation Experiments 5. Conclusion
  3. 3. Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. Parallel Distributed Implementation 4. Computation Experiments 5. Conclusion
  4. 4. Basic Idea of Evolutionary Computation Environment PopulationIndividual
  5. 5. Basic Idea of Evolutionary Computation Environment Environment Population Population (1)Individual Good Individual (1) Natural selection in a tough environment.
  6. 6. Basic Idea of Evolutionary Computation Environment Environment Environment Population Population Population (1) (2)Individual Good Individual New Individual (1) Natural selection in a tough environment. (2) Reproduction of new individuals by crossover and mutation.
  7. 7. Basic Idea of Evolutionary ComputationEnvironment Environment Population PopulationIteration of the generation update many times(1) Natural selection in a tough environment.(2) Reproduction of new individuals by crossover and mutation.
  8. 8. Applications of Evolutionary ComputationDesign of High Speed TrainsEnvironment Population Individual = Design ( )
  9. 9. Applications of Evolutionary ComputationDesign of Stock Trading AlgorithmsEnvironment Population Individual = Trading Algorithm ( )
  10. 10. Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. Parallel Distributed Implementation 4. Computation Experiments 5. Conclusion
  11. 11. Genetics-Based Machine LearningKnowledge Extraction from Numerical Data If (condition) then (result) If (condition) then (result) Evolutionary If (condition) then (result) Training data Computation If (condition) then (result) If (condition) then (result) Design of Rule-Based Systems
  12. 12. Design of Rule-Based SystemsEnvironment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  13. 13. Design of Decision TreesEnvironment Population Individual = Decision Tree ( )
  14. 14. Design of Neural NetworksEnvironment Population Individual = Neural Network ( )
  15. 15. Multi-Objective EvolutionMinimization of Errors and Complexity Large Error Initial Population Small Simple Complexity Complicated
  16. 16. Multi-Objective EvolutionMinimization of Errors and Complexity Large Error Small Simple Complexity Complicated
  17. 17. Multi-Objective EvolutionA number of different neural networks Large Error Small Simple Complexity Complicated
  18. 18. Multi-Objective EvolutionA number of different decision trees Large Error Small Simple Complexity Complicated
  19. 19. Multi-Objective EvolutionA number of fuzzy rule-based systems LargeErrorSmall Simple Complexity Complicated
  20. 20. Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. Parallel Distributed Implementation 4. Computation Experiments 5. Conclusion
  21. 21. Difficulty in Applications to Large DataComputation Load for Fitness EvaluationEnvironment Fitness Evaluation: Population Evaluation of each individual If ... Then ... If ... Then ... using the given training data If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  22. 22. Difficulty in Applications to Large DataComputation Load for Fitness EvaluationEnvironment Fitness Evaluation: Population Evaluation of each individual If ... Then ... If ... Then ... using the given training data If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  23. 23. Difficulty in Applications to Large DataComputation Load for Fitness EvaluationEnvironment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  24. 24. The Main Issue in This PresentationHow to Decrease the Computation LoadEnvironment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  25. 25. Fitness CalculationStandard Non-Parallel ModelEnvironment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  26. 26. Fitness CalculationStandard Non-Parallel ModelEnvironment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  27. 27. Fitness CalculationStandard Non-Parallel ModelEnvironment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  28. 28. Fitness CalculationStandard Non-Parallel ModelEnvironment Population Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Individual = Rule-Based System ( If ... Then ... If ... Then ... ) If ... Then ...
  29. 29. A Popular Approach for Speed-UpParallel Computation of Fitness Evaluation Fitness Evaluation If ... Then ... If ... Then ... CPU 1 If ... Then ... If ... Then ... If ... Then ... If ... Then ... CPU 2 If ... Then ... If ... Then ... If ... Then ... CPU 3 If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... CPU 4 If ... Then ... If ... Then ... If ... Then ... If we use n CPUs, the computation load for each CPU can be 1/n in comparison with the case of a single CPU (e.g., 25% by four CPUs)
  30. 30. Another Approach for Speed-UpTraining Data Reduction If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... Training data Subset If ... Then ... If ... Then ... If we use x% of the training data, the computation load can be reduced to x% in comparison with the use of all the training data.
  31. 31. Another Approach for Speed-UpTraining Data Reduction If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... Training data Subset If ... Then ... If ... Then ... If we use x% of the training data, the computation load can be reduced to x% in comparison with the use of all the training data.
  32. 32. Another Approach for Speed-UpTraining Data Reduction If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... Training data Subset If ... Then ... If ... Then ... If we use x% of the training data, the computation load can be reduced to x% in comparison with the use of all the training data.
  33. 33. Another Approach for Speed-UpTraining Data Reduction If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... Training data Subset If ... Then ... If ... Then ... If we use x% of the training data, the computation load can be reduced to x% in comparison with the use of all the training data.
  34. 34. Another Approach for Speed-UpTraining Data Reduction If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... Training data If ... Then ... If ... Then ... Training data Subset If ... Then ... If ... Then ... Difficulty: How to choose a training data subset The population will overfit to the selected training data subset.
  35. 35. Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPSN 2004. Training data are divided into data subsets.Training data
  36. 36. Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPSN 2004. 1st Generation Fitness Evaluation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...At each generation, a different datasubset (i.e., different window) is used.
  37. 37. Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPSN 2004. Fitness Evaluation 2nd Generation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...At each generation, a different datasubset (i.e., different window) is used.
  38. 38. Idea of Windowing in J. Bacardit et al.: Speeding-upPittsburgh learning classifier systems: Modeling timeand accuracy. PPSN 2004. Fitness Evaluation 3rd Generation If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...At each generation, a different datasubset (i.e., different window) is used.
  39. 39. Visual Image of WindowingA population is moving around in the training data. Training Data = Environment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...
  40. 40. Visual Image of WindowingA population is moving around in the training data. Training Data = Environment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...
  41. 41. Visual Image of WindowingA population is moving around in the training data. Training Data = Environment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...
  42. 42. Visual Image of WindowingA population is moving around in the training data. Training Data = Environment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...
  43. 43. Visual Image of WindowingA population is moving around in the training data. Training Data = Environment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ...
  44. 44. Visual Image of WindowingA population is moving around in the training data. Training Data = Environment Population If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... If ... Then ... After enough evolution with a moving window The population does not overfit to any particular training data subset. The population may have high generalization ability.
  45. 45. Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule Set Migration and Training Data Rotation. TFS (in Press)Non-parallel Non-distributedPopulation CPU Training data
  46. 46. Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule Set Migration and Training Data Rotation. TFS (in Press)Non-parallel Non-distributed Our Parallel Distributed ModelPopulation CPU CPU CPU CPU CPU CPU Training data (1) A population is divided into multiple subpopulations. (as in an island model)
  47. 47. Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule Set Migration and Training Data Rotation. TFS (in Press)Non-parallel Non-distributed Our Parallel Distributed ModelPopulation CPU CPU CPU CPU CPU CPU Training data Training data (1) A population is divided into multiple subpopulations. (2) Training data are also divided into multiple subsets. (as in the windowing method)
  48. 48. Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule Set Migration and Training Data Rotation. TFS (in Press)Non-parallel Non-distributed Our Parallel Distributed ModelPopulation CPU CPU CPU CPU CPU CPU Training data Training data (1) A population is divided into multiple subpopulations. (2) Training data are also divided into multiple subsets. (3) An evolutionary algorithm is locally performed at each CPU. (as in an island model)
  49. 49. Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule Set Migration and Training Data Rotation. TFS (in Press)Non-parallel Non-distributed Our Parallel Distributed ModelPopulation CPU CPU CPU CPU CPU CPU Training data Training data (1) A population is divided into multiple subpopulations. (2) Training data are also divided into multiple subsets. (3) An evolutionary algorithm is locally performed at each CPU. (4) Training data subsets are periodically rotated. (e.g., every 100 generations)
  50. 50. Our Idea: Parallel Distributed ImplementationH. Ishibuchi et al.: Parallel Distributed Hybrid Fuzzy GBML Modelswith Rule Set Migration and Training Data Rotation. TFS (in Press)Non-parallel Non-distributed Our Parallel Distributed ModelPopulation CPU CPU CPU CPU CPU CPU Training data Training data (1) A population is divided into multiple subpopulations. (2) Training data are also divided into multiple subsets. (3) An evolutionary algorithm is locally performed at each CPU. (4) Training data subsets are periodically rotated. (5) Migration is also periodically performed.
  51. 51. Computation Load of Four Models EC CPU Fitness EvaluationStandard Non-Parallel Model Computation Load = EC Part + Fitness Evaluation Part
  52. 52. Computation Load of Four Models EC EC = Evolutionary Computation CPU Fitness EC Evaluation = { Selection, Crossover, Mutation, Generation Update }Standard Non-Parallel Model Computation Load = EC Part + Fitness Evaluation Part
  53. 53. Computation Load of Four Models EC CPU Fitness EvaluationStandard Non-Parallel Model EC CPU CPU CPU CPU Computation Load CPU CPU CPU = EC Part + Fitness Evaluation Part (1/7) Standard Parallel Model (Parallel Fitness Evaluation)
  54. 54. Computation Load of Four Models EC EC CPU CPU Fitness EvaluationStandard Non-Parallel Model Windowing Model (Reduced Training Data Set) EC Computation Load CPU CPU CPU CPU = EC Part CPU CPU CPU + Fitness Evaluation Part (1/7) Standard Parallel Model (Parallel Fitness Evaluation)
  55. 55. Computation Load of Four Models EC EC CPU CPU Fitness EvaluationStandard Non-Parallel Model Windowing Model (Reduced Training Data Set) EC CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU Standard Parallel Model Parallel Distributed Model (Parallel Fitness Evaluation) (Divided Population & Data Set)
  56. 56. Computation Load of Four Models EC EC CPU CPU Fitness EvaluationStandard Non-Parallel Model Windowing Model (Reduced Training Data Set) EC EC Part (1/7) CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU Fitness Evaluation Part (1/7) Standard Parallel Model Parallel Distributed Model (Parallel Fitness Evaluation) (Divided Population & Data Set)
  57. 57. Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. Parallel Distributed Implementation 4. Computation Experiments 5. Conclusion
  58. 58. Our Model in Computational Experimentswith Seven Subpopulations and Seven Data Subsets
  59. 59. Our Model in Computational Experimentswith Seven Subpopulations and Seven Data Subsets Seven CPUs Seven Subpopulations of Size 30 Population Size = 30 x 7 = 210 Seven Data Subsets
  60. 60. Standard Non-Parallel Non-Distributed Modelwith a Single Population and a Single Data Set Population
  61. 61. Standard Non-Parallel Non-Distributed Modelwith a Single Population and a Single Data Set Single CPU Single Population of Size 210 Whole Data Set
  62. 62. Standard Non-Parallel Non-Distributed Modelwith a Single Population and a Single Data Set Single CPU Single Population of Size 210 Whole Data Set Termination Conditions: 50,000 Generations Computation Load: 210 x 50,000 = 10,500,000 Evaluations (more than ten million evaluations)
  63. 63. Comparison of Computation LoadComputation Load on a Single CPU per GenerationStandard Model: Parallel Distributed Model:Evaluation of 210 rule sets Evaluation of 30 rule sets usingusing all the training data one of the seven data subsets.
  64. 64. Comparison of Computation LoadComputation Load on a Single CPU per GenerationStandard Model: Parallel Distributed Model:Evaluation of 210 rule sets Evaluation of 30 rule sets usingusing all the training data one of the seven data subsets. 1/7 1/7
  65. 65. Comparison of Computation LoadComputation Load on a Single 1/7 = per GenerationComputation Load ==> 1/7 x CPU 1/49 (about 2%)Standard Model: Parallel Distributed Model:Evaluation of 210 rule sets Evaluation of 30 rule sets usingusing all the training data one of the seven data subsets. 1/7 1/7
  66. 66. Data Sets in Computational ExperimentsNine Pattern Classification Problems Name of Number of Number of Number of Data Set Patterns Attributes Classes Segment 2,310 19 7 Phoneme 5,404 5 2 Page-blocks 5,472 10 5 Texture 5,500 40 11 Satimage 6,435 36 6 Twonorm 7,400 20 2 Ring 7,400 20 2 PenBased 10,992 16 10 Magic 19,020 10 2
  67. 67. Computation Time for 50,000 GenerationsComputation time was decreased to about 2% Name of Standard Our Model Percentage of B Data Set A minutes B minutes B/A (%) Segment 203.66 4.69 2.30% Phoneme 439.18 13.19 3.00%Page-blocks 204.63 4.74 2.32% Texture 766.61 15.72 2.05% Satimage 658.89 15.38 2.33% Twonorm 856.58 7.84 0.92% Ring 1015.04 22.52 2.22% PenBased 1520.54 35.56 2.34% Magic 771.05 22.58 2.93%
  68. 68. Computation Time for 50,000 GenerationsComputation time was decreased to about 2% Name of Standard Our Model Percentage of B Data Set A minutes B minutes B/A (%) Segment 203.66 4.69 2.30% Why ? Phoneme 439.18 13.19 3.00% Because the population and the trainingPage-blockswere divided into 4.74 subsets. data 204.63 seven 2.32% Texture 766.61 15.72 2.05% 1/7 x 1/7 = 1/49 (about 2%) Satimage 658.89 15.38 2.33% Twonorm 856.58 7.84 0.92% Ring 1015.04 22.52 2.22% PenBased 1520.54 35.56 2.34% Magic 771.05 22.58 2.93%
  69. 69. Computation Time for 50,000 GenerationsComputation time was decreased to about 2% Name of Standard Our Model Percentage of B Data Set A minutes B minutes B/A (%) Segment 203.66 4.69 2.30% Why ? Phoneme 439.18 13.19 3.00% Because the population and the trainingPage-blockswere divided into 4.74 subsets. data 204.63 seven 2.32% Texture 766.61 15.72 2.05% 1/7 x 1/7 = 1/49 (about 2%) Satimage 658.89 15.38 2.33% Twonorm 856.58 7.84 0.92% Ring 1015.04 22.52 2.22% PenBased 1520.54 35.56 2.34% Magic 771.05 22.58 2.93%
  70. 70. Test Data Error Rates (Results of 3x10CV)Test data accuracy was improved for six data sets Name of Standard Our Model Improvement Data Set (A %) (B %) from A: (A - B)% Segment 5.99 5.90 0.09 Phoneme 15.43 15.96 - 0.53Page-blocks 3.81 3.62 0.19 Texture 4.64 4.77 - 0.13 Satimage 15.54 12.96 2.58 Twonorm 7.36 3.39 3.97 Ring 6.73 5.25 1.48 PenBased 3.07 3.30 - 0.23 Magic 15.42 14.89 0.53
  71. 71. Test Data Error Rates (Results of 3x10CV)Test data accuracy was improved for six data sets Name of Standard Our Model Improvement Data Set (A %) (B %) from A: (A - B)% Segment 5.99 5.90 0.09 Phoneme 15.43 15.96 - 0.53Page-blocks 3.81 3.62 0.19 Texture 4.64 4.77 - 0.13 Satimage 15.54 12.96 2.58 Twonorm 7.36 3.39 3.97 Ring 6.73 5.25 1.48 PenBased 3.07 3.30 - 0.23 Magic 15.42 14.89 0.53
  72. 72. Q. Why did our model improve the test data accuracy ?A. Because our model improved the search ability. Data Set Standard Our Model Improvement Satimage 15.54% 12.96% 2.58% Training Data Error Rate (%) Non-Parallel Non-Distributed Our Parallel Distributed Model Number of Generations
  73. 73. Q. Why did our model improve the search ability ?A. Because our model maintained the diversity. Data Set Standard Our Model Improvement Satimage 15.54% 12.96% 2.58% Training Data Error Rate (%) The best and worst error rates in a particular subpopulation at each generation in a single run. Non-Parallel Non-Distributed Model (Best = Worst: No Diversity) Best Training Data Rotation: Every 100 Generations Worst Rule Set Migration: Parallel Distributed Model Every 100 Generations Number of Generations
  74. 74. Effects of Rotation and Migration Intervals Training Data Rotation: Every 100 Generations Rule Set Migration: Every 100 Generations
  75. 75. Effects of Rotation and Migration Intervals(Rotations in the opposite directions) Rule set migration CPU CPU CPU CPU CPU CPU CPU Population Training data rotation Training data No No
  76. 76. Effects of Rotation and Migration Intervals(Rotations in the opposite directions) Rule set migration CPU CPU CPU CPU CPU CPU CPU Population 18 Good Results 16 14 Training data rotation Training data 12 10 20 50 100 200 20 500 50 100 200 ∞ No ∞ No 500
  77. 77. Effects of Rotation and Migration Intervals(Rotations in the same direction) Rule set migration CPU CPU CPU CPU CPU CPU CPU Population Training data rotation Training data No No
  78. 78. Effects of Rotation and Migration Intervals(Rotations in the same direction) Rule set migration CPU CPU CPU CPU CPU CPU CPU Population 18 16 14 Training data rotation Training data 12 10 Good Results 20 50 100 200 20 500 50 100 200 ∞ No ∞ No 500
  79. 79. Effects of Rotation and Migration Intervals18 1816 1614 14 12 12 10 10 20 20 50 50 100 100 200 200 20 20 500 50 500 50 100 100 200 200 ∞ No ∞ No 500 ∞ No ∞ No 500 Opposite Directions Same Direction
  80. 80. Effects of Rotation and Migration Intervals Interaction between rotation and migration18 1816 1614 14 12 12 10 10 20 20 50 50 100 100 200 200 20 20 500 50 500 50 100 100 200 200 ∞ No ∞ No 500 ∞ No ∞ No 500 Opposite Directions Same Direction
  81. 81. Effects of Rotation and Migration Intervals Interaction between rotation and migration18 1816 16 Negative14 14 Effects 12 12 10 10 20 20 50 50 100 100 200 200 20 20 500 50 500 50 100 100 200 200 ∞ No ∞ No 500 ∞ No ∞ No 500 Opposite Directions Same Direction
  82. 82. Contents of This Presentation 1. Basic Idea of Evolutionary ComputationIntroduction Machine Learning 2. Genetics-Based 3. Parallel Distributed Implementation 4. Computation Experiments 5. Conclusion
  83. 83. Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Introduction3. It was shown that the test data accuracy was improved.4. We explained negative effects of the interaction between the training data rotation and the rule set migration.
  84. 84. Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Introduction3. It was shown that the test data accuracy was improved.4. We explained negative effects of the interaction between the training data rotation and the rule set migration.
  85. 85. Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Introduction3. It was shown that the test data accuracy was improved.4. We explained negative effects of the interaction between the training data rotation and the rule set migration.
  86. 86. Conclusion1. We explained our parallel distributed model.2. It was shown that the computation time was decreased to 2%.Introduction3. It was shown that the test data accuracy was improved.4. We explained negative effects of the interaction between the training data rotation and the rule set migration.
  87. 87. Conclusion5. A little bit different model may be also possible for learning from locally located data bases.Introduction CPU CPU University A University C Subpopulation Rotation CPU CPU University B University D
  88. 88. Conclusion6. This model can be also used for learning from changing data bases.Introduction CPU CPU University A University C Subpopulation Rotation CPU CPU University B University D
  89. 89. Conclusion6. This model can be also used for learning from changing data bases.Introduction CPU CPU University A University C Subpopulation Rotation CPU CPU University B University D
  90. 90. Conclusion6. This model can be also used for learning from changing data bases.Introduction CPU CPU University A University C Subpopulation Rotation CPU CPU University B University D
  91. 91. Conclusion6. This model can be also used for learning from changing data bases.Introduction CPU CPU University A University C Subpopulation Rotation CPU CPU University B University D
  92. 92. Conclusion6. This model can be also used for learning from changing data bases.Introduction CPU CPU University A University C Subpopulation Rotation CPU CPU University B University D
  93. 93. Conclusion Thank you very much!

×