1. Comparing the Parallel Automatic Composition of Inductive Applications with Stacking Methods Hidenao Abe & Takahira Yamaguchi Shizuoka University, JAPAN {hidenao,yamaguti}@ks.cs.inf.shizuoka.ac.jp
2.
3. Overview of meta-learning scheme Meta-Learning Executor Meta Knowledge A Better result to the given data set than its done by each single learning algorithm Meta-learning algorithm 学習 学習 学習 Learning Algorithms データセット Data Set
4.
5.
6. Basic Idea of our Constructive Meta-Learning Analysis of Representative Inductive Learning Algorithms De-composition& Organization Search & Composition + C4.5 VS CS AQ15
7. Basic Idea of Constructive Meta-Learning Analysis of Representative Inductive Learning Algorithms CS AQ15 De-composition& Organization Search & Composition +
8. Basic Idea of Constructive Meta-Learning Analysis of Representative Inductive Learning Algorithms Automatic Composition of Inductive Applications De-composition& Organization Search & Composition + Organizing inductive learning methods, control structures and treated objects.
11. Data Type Hierarchy Organization of input/output/reference data types for inductive learning methods classifier-set data-set If-Then rules tree (Decision Tree) network (Neural Net) training data set test data set Objects validation data set
12. Identifying Inductive Learning Methods and Control Structures Generating Training & Validation Data Sets Generating a classifier set Evaluating a classifier set Modifying training and validation data sets Modifying a classifier set Evaluating classifier sets to test data set Modifying a classifier set
13. CAMLET: a Computer Aided Machine Learning Engineering Tool CAMLET Method Repository Data Type Hierarchy Control Structures Compile Go & Test Construction Instantiation Go to the goal accuracy? No Yes Data Sets, Goal Accuracy Inductive application, etc. Refinement User
14. CAMLET with a parallel environment CAMLET Method Repository Data Type Hierarchy Control Structures Compile Go & Test Execution Level Construction Instantiation Go to the goal accuracy? No Yes Data Sets, Goal Accuracy Inductive application, etc. Composition Level Refinement Specifications Data Sets Results User
15. Parallel Executions of Inductive Applications Constructing specs Sending specs Initializing R: receiving , Ex: executing, S: sending Composition Level is necessary one process Execution Level is necessary one or more process elements R R R R Waiting Executing Ex Ex Ex Ex Receiving a result Receiving Ex Ex S Ex Refining the spec Sending refined one Refining & Sending Ex Ex Ex R
16.
17. CAMLET on a parallel GA refinement CAMLET Method Repository Data Type Hierarchy Control Structures Compile Go & Test Execution Level Construction Instantiation Go to the goal accuracy? No Yes Data Sets, Goal Accuracy Inductive application, etc. Composition Level Refinement Specifications Data Sets Results User
18.
19.
20. Statlog common data sets To generate 10-fold cross validation data sets’ pairs, We have used SplitDatasetFilter implemented in WEKA.
21. Setting up of Stacking methods to Statlog common data sets J4.8 (with pruning, C=0.25 ) IBk ( k=5 ) Bagged J4.8 (Sampling rate=55%, 5 Iterations) Bagged J4.8 (Sampling rate=55%, 10 Iterations) Boosted J4.8 (5 Iterations) Boosted J4.8 (10 Iterations) Part (with pruning, C=0.25 ) Naïve Bayes J4.8 (with pruning, C=0.25 ) Naïve Bayes Classification with Linear Regression (with all of features) Meta-level Learning Algorithms Base-level Learning Algorithms
22. Evaluation on accuracies to Statlog common data sets Inductive applications composed by CAMLET have shown us as good performance as that of stacking with Classification via Linear Regression.
23. Setting up of Stacking methods to UCI common data sets J4.8 (with pruning, C=0.25 ) IBk ( k=5 ) Bagged J4.8 (Sampling rate=55%, 10 Iterations) Boosted J4.8 (10 Iterations) Part (with pruning, C=0.25 ) Naïve Bayes J4.8 (with pruning, C=0.25 ) Naïve Bayes Linear Regression (with all of features) Meta-level Learning Algorithms Base-level Learning Algorithms
24. Evaluation on accuracies to UCI common data sets 83.23 CAMLET 81.92 J4.8 83.79 81.07 81.97 Average Max. of 3 NB LR Algorithm
25.
26. Parallel Efficiency of the case study with StatLog data sets Parallel Efficiencies of 10-fold CV data sets Parallel Efficiencies of training-test data sets
31. Stacking: training and prediction Training phase Prediction phase Training Set Training Set (Int.) Test Set (Int.) Training Set Test Set Meta-level Training Set Meta-level classifier Meta-level learning Learning of Base-level classifiers Transformation Repeating with CV Meta-level Test Set Transformation Prediction Results of Preduction Meta-level classifiers Learning of Base-level classifiers ベースレベル 分類器 ベースレベル 分類器 Base-level classifiers ベースレベル 分類器 ベースレベル 分類器 Base-level classifiers
32.
33. Meta-level classifiers (Ex.) The prediction probability of”0” With classifier by algorithm 1 The prediction probability of”1” With classifier by algorithm 2 The prediction probability of”1” With classifier by algorithm 1 The prediction probability of”0” With classifier by algorithm 3 Prediction “ 1” ．．．． ≦0.1 ＞0.1 ≦0.95 ＞0.95 ＞0.9 ≦0.9 Prediction “ 0” Prediction “ 1”
34. Accuracies of 8 base-level learning algorithms to Statlog data sets Stacking Base-level algorithms Accuracy（％） CAMLET Base-level algorithms Accuracy（％）
35. C4.5 Decision Tree without pruning (reproduction of CAMLET) Generating training and validation data sets with a void validation set Generating a classifier set (decision tree) with entropy + information ratio Evaluating a classifier set with set evaluation Evaluating classifier sets to test data set with single classifier set evaluation 1. Select just one control structure 2. Fill each method with specific methods 3. Instantiate this spec. 4. Compile the spec. 5. Execute its executable code 6. Refine the spec. (if needed)
Editor's Notes
Good morning, I’m Hidenao Abe from Shizuoka University. Today I present an evaluation of our proposal, comparing its performance with that of three stacking methods. Our tool has implemented on a parallel environment to search a huge space faster. In our work, inductive application means an adequate learning algorithm to given data set. Its criteria is also given by a user. “ Data set” in our work indicates relational data set, and treated task is classification task. Now, let get into contents of today’s my talk.
First of all, I introduce general overview of meta-learning scheme, and I’ll point out a problem of ordinary meta-learning scheme. Next, I’ll explain our proposal scheme called “Constructive meta-learning”, and it’s parallel execution environment. After implemented our tool called CAMLET, we have done a case study with two kinds of data sets. Finally, I’ll conclude today’s talk.
This slide shows general overview of meta-learning scheme. We can use meta-learning algorithms to get a better result to the input data set than its done by a single learning algorithm. Meta-learning algorithm consists of “meta knowledge” and “its executor”. Ordinary meta-learning only integrate or combine results of base-level learning algorithms with meta-knowledge.
So we call ordinary meta-learning algorithms “selective meta-learning”. This scheme consists of two approaches. One is integration of classifiers, which are learned with different training data sets generated by “Bootstrap Sampling” and weighting ill-classified instances. The former is called bagging. The latter is called boosting. The other is integration of base-level classifiers, which are learned with different algorithms. If the algorithm use simple voting, we call this voting. If the algorithm use a meta-level classifier learned by a meta-level data set, which built from results of base-level classifiers, we call these stacking or cascading. They works well to most data set. However, when no base-level learning algorithm works well to a given data set, they don’t work well at all. This problem comes why they don’t de-compose base-level learning algorithms. To overcome this problem, we have proposed another meta-learning scheme called “constructive meta-learning” scheme.
I’ll explain about our constructive meta-learning based on method repositories. First, I’ll talk about basic idea of our constructive meta-learning. Second, I’ll present an implementation of a method repository and a tool for doing constructive meta-learning. This tool is called CAMLET. The search of CAMLET is very costly, because we can re-construct many inductive applications with the method repository. To search for the best inductive application faster, we have implemented a parallel execution protocol for CAMLET.
This slide shows our basic idea of our constructive meta-learning. This scheme contains two major phases. The former phase is building a method repository. The latter phase is the re-construction of an adequate inductive application to a given data set. To build a method repository, we have analyzed representative learning algorithms such as C4.5, Version Space, Classifier Systems, AQ15 etc.
With this analysis, we identify inductive learning methods, which are some functional parts of inductive learning algorithms. >> Click now!! The similar methods are built a hierarchy with defining input/output/reference and post-method/pre-method. With this method hierarchy, we need define control structures and data types. Control structures are needed for re-constructing inductive learning algorithms with inductive learning methods. And data types are needed for defining the hierarchy of methods. At this point in time, building of method repository and other information resources are completed. >> Click Now!! With this method repository, we can re-construct much more inductive applications than analyzed learning algorithms. The system search these new inductive application including analyzed learning algorithm for the adequate one.
To implement a method repository, we have analyzed these 8 learning algorithms. From this analysis, inductive learning methods have been identified. ----- For example,
This hierarchy is our implemented method repository. Six generic methods are put in top level of the hierarchy. Each generic method is brunch down by its input/output/reference data types, post-method/pre-method and difference of its function. The leaf nodes get into the executable code written by C language.
Data type hierarchy is a organization of data types for the inductive learning methods.
This slide shows us the generic six methods. They are “generating training and validation data sets”, “generating a classifier set”, “evaluating a classifier set”, “modifying training and validation data sets”, “modifying a classifier set” and “evaluating classifier set to test data set”. To re-construct learning algorithms, we have defined 8 control structure with 3 feed back loops to these generic methods. The ストレート path is necessary to all of the control structures.
With the method repository, CAMLET constructs inductive applications to given data set from a user. CAMLET consists of these activities. In construction activity, CAMLET selects just one control structure and fill each generic method with specific one, referring the method repository and the data type hierarchy. Thus a basic specification of inductive application appears. In instantiation, CAMLET make up a specific specification, checking connections of input/output of each methods. Then the specific specification is complied and executed. If the performance of the inductive application go to the goal accuracy, CAMLET output it. If not so, CAMLET refines it with some strategy. This refinement task is assumed one kind of search task on the specification space of inductive application.
CAMLET searches many inductive applications yield by the method repository for the satisfactory one to the given data set. So it needs very large computational cost. We have designed parallel execution of inductive applications. We have separated activities of CAMLET into two parts. One is composition level process. The other is execution level process. To parallelize executions of inductive applications, we have parallelized execution level process.
This slide shows the parallel executions of inductive applications for CAMLET. At initializing phase, composition level process distributes constructed specifications and given data sets. Then execution level processing elements execute inductive applications. When one execution level processing element finishes its execution, it send the execution result. If the result can’t go to the goal, composition level process refines the specification of inductive application, and send refined specification to the execution level processing element. Execution phase to Refining&Sending phase will be repeated until one inductive application going to the goal. We have implemented this execution protocol on the rack mounted parallel machine.
With the parallel execution, we have also implemented search method with GA operations. This figure shows one refinement phase of this GA search. In this model, t+1 generation contains whole t generation. The GA operations execute as these order. First, executed specifications transform to chromosomes. Second, parents are selected by “Tournament Method”. Third, children are generated with random single point crossover and mutation. Then children are transformed to specifications, and executed. Repeating these operations, CAMLET refines specifications of inductive applications.
With the GA search, the complete system overview of CAMLET as shown in this figure.
After implemented the parallel CAMLET, we have done the case study to compare the accuracies of inductive applications composed by CAMLET with stacking methods to two kinds of common data sets. “Accuracy” is also called success rate. Its definition is a percentage of number of correctly predicted instances divided by the number of whole instances in predicted data set. At the same time, we have analyzed parallel efficiencies of parallel execution of inductive applications in CAMLET.
To compare the performances, we have taken 10 Statlog data sets and 32 UCI data sets. We have input goal accuracies to each data set achieved by a stacking method. Then CAMLET has output just one inductive application to each data set. We have compare the accuracy of these inductive applications. To execute inductive applications in parallel, we have taken 10 processing elements.
This table shows us summary of statlog common data sets. Accuracies of four data sets have been evaluated assigned test data set. To the other data sets, we have evaluated them with 10-fold cross validation. These 10-fold cross validation data set pairs generated with SplitDatasetFilter class of Weka.
We have set up three stacking methods to Statlog common data set as shown in this figure. Their base-level learning algorithms contains 6 different algorithms. However, 8 base-level classifiers have been used, because we have set up different iterations to bagging and boosting. Parameters to each algorithm are written in brackets.
This figure shows us the result of three stacking methods and its of inductive applications composed by CAMLET. The solid blue line with X marked shows accuracies of inductive applications by CAMLET. The average accuracies show no significant difference between the accuracies of stacking via linear regression and them of inductive application composed by CAMLET. With this result, CAMLET has as good performance as stacking methods.
We also evaluated stacking methods and CAMLET with UCI common data sets. This slide shows settings of three stacking methods to UCI common data sets. In this evaluation, each meta-level algorithm has been able to use 6 base-level classifiers with 6 base-level algorithms.
This slide is the evaluation with 32 UCI common data sets. The solid green line with circle marked shows accuracies of inductive application composed by CAMLET to each data set to satisfy the max accuracy of three stacking methods. The average of CAMLET is almost same the goal, comparing the other stacking methods.
With the execution result to Statlog data sets, we have analyzed the parallel efficiency of CAMLET. This parallel efficiency means how much efficient than if we didn’t use this parallel execution.
This slide shows each parallel efficiency of Statlog data set. To 10-fold cross validated data sets, parallel efficiencies are satisfactory. Besides, their of training-test data sets are not enough. Especially, the parallel efficiency of Satimage data set is low.
Now, I conclude my talk.
As conclusion, we have implemented a tool for doing constructive meta-learning. This tool shows us as good performance as stacking methods. Thus CAMLET have shown a significant performance without re-constructing the stacking methods. Then we have analyzed parallel efficiencies of parallel execution in CAMLET. It has achieved 5.93 times efficiency. As future work, we are…
![Contents ,[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)