Voting Based Learning Classifier System for Multi-Label Classification
Upcoming SlideShare
Loading in...5
×
 

Voting Based Learning Classifier System for Multi-Label Classification

on

  • 858 views

Kaveh Ahmadi-Abhari, Ali Hamzeh, Sattar Hashemi. "Voting Based Learning Classifier System for Multi-Label Classification". IWLCS, 2011

Kaveh Ahmadi-Abhari, Ali Hamzeh, Sattar Hashemi. "Voting Based Learning Classifier System for Multi-Label Classification". IWLCS, 2011

Statistics

Views

Total Views
858
Views on SlideShare
807
Embed Views
51

Actions

Likes
0
Downloads
14
Comments
0

3 Embeds 51

http://home.dei.polimi.it 47
http://home.deib.polimi.it 3
http://www.docshut.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Voting Based Learning Classifier System for Multi-Label Classification Voting Based Learning Classifier System for Multi-Label Classification Presentation Transcript

    • Voting-Based Learning Classifier Systemfor multi-label classification Kaveh Ahmadi-Abhari (Presenter) Ali Hamzeh Sattar Hashemi IWLCS 2011 – Dublin, Ireland, 13th July 2011
    • Multi-label Classification  Single Label Classification  Exclusive classes: each example belongs to one class  Multi-label Classification  Each instance can belong to more than one classKaveh Ahmadi-Abhari 2 Shiraz University, Soft Computing Group
    • Multi-label Classification Sky People  Single Label Classification  Exclusive classes: each example belongs to one class  Multi-label Classification  Each instance can belong to more than one class SandKaveh Ahmadi-Abhari 3 Shiraz University, Soft Computing Group
    • Current Methods Problem • Transfer problem to a single- Transformation label classification problem Algorithm • Adapt single-label classifiers Adaptation to Solve the problem [Tsoumakas & Katakis, 2007]Kaveh Ahmadi-Abhari 4 Shiraz University, Soft Computing Group
    • Problem Transformation Approaches Ex. Label- set 1a λ1 Copy Transformation 1b λ4 2a λ3 2b λ4 3 λ1 4a λ2 Ex. Label- set 4b λ3 4c λ4 1 {λ1 , λ4 } 2 {λ3 , λ4 } 3 {λ1} 4 {λ2 , λ3 , λ4 } [Tsoumakas et al., 2009]Kaveh Ahmadi-Abhari 5 Shiraz University, Soft Computing Group
    • Algorithm Adaptation Approaches  Multi-label lazy algorithm  ML-kNN [Zhang & Zhou, PRJ07]  Multi-label decision trees  ADTBoost.MH [DeComité et al. MLDM03]  Multi-Label C4.5 [Clare & King, LNCS2168]  Multi-label kernel methods  Rank-SVM [Elisseeff & Weston, NIPS02]  ML-SVM [M.R. Boutell, et al. PR04]  Multi-label text categorization algorithms  BoosTexter [Schapire & Singer, MLJ00]  Maximal Margin Labeling [Kazawa et al., NIPS04]  Probabilistic generative models [McCallum, AAAI99] [Ueda & Saito, NIPS03]  BP-MLL [Zhang & Zhou, TKDE06]Kaveh Ahmadi-Abhari 6 Shiraz University, Soft Computing Group
    • Motivation A lot has been done in terms of classifications using LCSs Most of these studies have been conducted for single-label classification problems Multi-label classification is in its inception [Vallim et al., IWLCS 08]Kaveh Ahmadi-Abhari 7 Shiraz University, Soft Computing Group
    • Voting Based Learning Classifier System How can we guide the discovery mechanism (e.g. evolutionary operators) in LCSs?Kaveh Ahmadi-Abhari 8 Shiraz University, Soft Computing Group
    • Voting Based Learning Classifier System How can we guide the discovery mechanism (e.g. evolutionary operators) in LCSs? Using the prior knowledge gained from past experiencesKaveh Ahmadi-Abhari 9 Shiraz University, Soft Computing Group
    • Voting Based Learning Classifier System How can we guide the discovery mechanism (e.g. evolutionary operators) in LCSs? Using the prior knowledge gained from past experiences Training instances vote their matched rules according to how correct the rule isKaveh Ahmadi-Abhari 10 Shiraz University, Soft Computing Group
    • Voting Based Learning Classifier System How can we guide the discovery mechanism (e.g. evolutionary operators) in LCSs? Using the prior knowledge gained from past experiences Training instances vote their matched rules according to how correct the rule is Fitness measureKaveh Ahmadi-Abhari 11 Shiraz University, Soft Computing Group
    • Voting Defining Rule Types How can the given votes describe the quality of the rules accurately? Define different types for the rules such that each of these types describes the quality status the rule might have.Kaveh Ahmadi-Abhari 12 Shiraz University, Soft Computing Group
    • Rule Types Example: in a single-label classification problem, rule types might be correct or wrong. Each rule might receive a “correct” or “wrong” vote from each matched training instance. A rule receives a combination of “correct” and “wrong” votes from its matched training instancesKaveh Ahmadi-Abhari 13 Shiraz University, Soft Computing Group
    • Votes as Fitness Measure • Given votes • Describe the quality of the rules • Use as a fitness measure for guiding the discovery mechanism. • For example, a rule with more “wrong” votes, should be discovered with a high probability to achieve a meaningful ruleKaveh Ahmadi-Abhari 14 Shiraz University, Soft Computing Group
    • Rules Definition Antecedent / Consequent ###1 / 110 0011 / 001  Antecedent part matches with the feature vector.  Consequent part are the classes predicted by the rule.  One bit for each class in the consequent part.  Value 1 in the bit indicates existence of the respective class.Kaveh Ahmadi-Abhari 15 Shiraz University, Soft Computing Group
    • VLCS Vote Types for Multi-label Problem Correct Wrong Subset Multi-label Vote Types for VLCS Partial SupersetKaveh Ahmadi-Abhari 16 Shiraz University, Soft Computing Group
    • Multi-Label Simple Dataset 000 111 001 1, 4 110 1, 3 010 2, 4 1, 2 101 011 100 Expand from [Vallim et al., GECCO’ 08]Kaveh Ahmadi-Abhari 17 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Correct Rules (C) 111 000 001 1, 4 110 1, 3 00# /1001 2, 4 010 1, 2 101 • Is correct when it matches with: 011 • 000 or 100 • 001Kaveh Ahmadi-Abhari 18 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Wrong Rules (W) 111 000 001 1, 4 110 1, 3 0#0/0010 2, 4 010 1, 2 101 • Is wrong when it matches with: 011 • 000 or 100 • 010Kaveh Ahmadi-Abhari 19 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Subset Rules 111 000 001 1, 4 110 1, 3 #01/1000 2, 4 010 1, 2 101 • Is subset when it matches with: 011 • 001 or 100 • 101Kaveh Ahmadi-Abhari 20 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Subset Rules 111 000 001 1, 4 110 1, 3 #01/1000 2, 4 010 1, 2 101 • Is subset when it matches with: 011 • 001 or 100 • 101 Excepted Classes: 1, 4Kaveh Ahmadi-Abhari 21 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Superset Rules 111 000 001 1, 4 110 1, 3 #00/1101 2, 4 010 1, 2 101 • Is superset when it matches with: 011 • 001 or 100 • 101Kaveh Ahmadi-Abhari 22 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Superset Rules 111 000 001 1, 4 110 1, 3 #00/1101 2, 4 010 1, 2 101 • Is superset when it matches with: 011 • 001 or 100 • 101 Excepted Classes: 1, 4Kaveh Ahmadi-Abhari 23 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Partial-set Rules 111 000 001 1, 4 110 1, 3 #1# / 0110 2, 4 010 1, 2 101 • Is superset when it matches with: 011 • 010 or 100 • 111Kaveh Ahmadi-Abhari 24 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem  Partial-set Rules 111 000 001 1, 4 110 1, 3 #1# / 0110 2, 4 010 1, 2 101 • Is superset when it matches with: 011 • 010 or 100 • 111 Excepted Classes: 2, 4Kaveh Ahmadi-Abhari 25 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem 000  Rules might receive different votes 111 001 during the time 1, 4 110 1, 3 2, 4 010 1, 2 #0# / 1001 101 011 100Kaveh Ahmadi-Abhari 26 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem 000  Rules might receive different votes 111 001 during the time 1, 4 110 1, 3 2, 4 010 1, 2 #0# / 1001 101 011 100 Is correct for instance 000Kaveh Ahmadi-Abhari 27 Shiraz University, Soft Computing Group
    • VLCS Voting Options for Multi-label Problem 000  Rules might receive different votes 111 001 during the time 1, 4 110 1, 3 2, 4 010 1, 2 #0# / 1001 101 011 100 Is correct for Is partial-set instance 000 for instance 101Kaveh Ahmadi-Abhari 28 Shiraz University, Soft Computing Group
    • Using Stored Prior Knowledge Consider a rule that all received votes are superset } Information } The rule is covering an appropriate area of the problem Inference The rule is predicting greater number of classes for the matched input instance The number of the classes the rule predicts should be subtractedKaveh Ahmadi-Abhari 29 Shiraz University, Soft Computing Group
    • Discovery Operators  In the discovery mechanism an evolutionary algorithm with four mutation operators is defined:Kaveh Ahmadi-Abhari 30 Shiraz University, Soft Computing Group
    • Discovery Operators  Mutation operators on rule’s antecedent part Generalize the rule by flipping the 0 MA-G or 1 bits to # Specializes the rule by flipping # MA-S bits to 1 or 0Kaveh Ahmadi-Abhari 31 Shiraz University, Soft Computing Group
    • Discovery Operators  Mutation operators on rule’s consequent part Subtract the number of predicted MC-S classes by flipping 1 bits to 0 Adds more classes to predicted MC-A classes by flipping 0 bits to 1Kaveh Ahmadi-Abhari 32 Shiraz University, Soft Computing Group
    • Which Discovery Operator? The votes each rule has received guide which mutation operator should act.Kaveh Ahmadi-Abhari 33 Shiraz University, Soft Computing Group
    • Which Discovery Operator? The votes each rule has received guide which mutation operator should act. Wrongly Subtract the assigned some number of Superset Rule non-expected predicted classes classes (MC-S)Kaveh Ahmadi-Abhari 34 Shiraz University, Soft Computing Group
    • Which Discovery Operator? Activated Mutation Rule Received Votes Operator Correct MA-G Subset MC-A Superset MC-S Partial-Set MC-A, MC-S Wrong MC-A, MC-S Correct, Subset MA-S Correct, Superset MA-G Correct, Partial-Set MA-S Correct, Wrong MA-S Wrong, Subset MA-S, MC-A Wrong, Partial MA-S Correct, Subset, Wrong MA-S, MA-GKaveh Ahmadi-Abhari 35 Shiraz University, Soft Computing Group
    • Mutation Rate • Mutation operator performs bit flipping using a probability, which is the mutation rate. • The strength of a rule is the amount of reward we predict the system to receive if the rule acts. • The more the strength, the less the mutation rate.Kaveh Ahmadi-Abhari 36 Shiraz University, Soft Computing Group
    • Strength of a Rule  The mean of the rewards the rule gets over time. Reward Function: C rule ∆C expected R = 1− C rule  C expected Alteration of [Vallim et al., GECCO’ 08]Kaveh Ahmadi-Abhari 37 Shiraz University, Soft Computing Group
    • Strength of a Rule  The mean of the rewards the rule gets over time. Reward Function: C rule ∆C expected R = 1− C rule  C expected A ∆B = {x : ( x ∈ A ) ⊕ ( x ∈ B )} Alteration of [Vallim et al., GECCO’ 08]Kaveh Ahmadi-Abhari 38 Shiraz University, Soft Computing Group
    • Rules Rewards Input Expected Selected Received Reward Instance output Rule Vote 0001 1, 2 ###1 / 110 Correct 1 0101 1, 2, 3 ###1 / 110 Subset 0.66 0111 1 ###1 / 110 Superset 0.50 1111 1,3 ###1 / 110 Partial-set 0.33 0011 3 ###1 / 110 Wrong 0Kaveh Ahmadi-Abhari 39 Shiraz University, Soft Computing Group
    • Experimental Results  Data Sets:  Two binary datasets in the bioinformatics domain  [Chan and Freitas, GECCO’ 06 ]  Extracted from [Alves et al., 2009]Kaveh Ahmadi-Abhari 40 Shiraz University, Soft Computing Group
    • Experimental Results  Quality Metrics: Accuracy • Proportion of predicted classes among all predicted or true classes Precision • Proportion of true classes among all predicted classes Recall • Proportion of predicted classes among all true classes [Tsoumakas & Katakis, 2007]Kaveh Ahmadi-Abhari 41 Shiraz University, Soft Computing Group
    • Experimental Results  For the VLCS, we use a 5-fold cross validation in which the training part is used to evaluate the rules using the voting mechanism described above.  Fixed size population  initially are the most general possible rules.  In each generation, each rule is voted by its matched instances  reward is assigned  Defined mutation operators to discover new rules  The combination of the best rules among the parents and the off-springs make the next generation.  We stop the training phase if the mean strength of the rules decreases in a number of consecutive generations.Kaveh Ahmadi-Abhari 42 Shiraz University, Soft Computing Group
    • Experimental Results  [Chan and Freitas, GECCO’ 06 ]  135 instances  152 attributes  Two classes • Each instance could have one or both of the available class labels. Method Accuracy Precision Recall BR 0.89 0.89 0.87 ML-KNN 0.91 0.93 0.91 VLCS 0.89 0.89 0.89Kaveh Ahmadi-Abhari 43 Shiraz University, Soft Computing Group
    • Experimental Results  Extracted from [Alves et al., 2009]  7877 proteins  40 attributes  Six classes • Each instance could have some of the available class labels. Method Accuracy Precision Recall BR 0.78 0.77 0.78 ML-KNN 0.80 0.81 0.80 VLCS 0.81 0.83 0.82Kaveh Ahmadi-Abhari 44 Shiraz University, Soft Computing Group
    • Conclusion Guiding the discovery mechanism with a prior knowledge, such that is used in VLCS, can help us solve applicable problemsKaveh Ahmadi-Abhari 45 Shiraz University, Soft Computing Group
    • Future Work  A representation for dealing with numeric and nominal datasets.  Future studies on scalability and stability of the system is necessary.  Additional studies on system performance in dealing with imbalanced data and noise is also required.  Improving evolutionary operators, guiding mechanism and rule refinement.Kaveh Ahmadi-Abhari 46 Shiraz University, Soft Computing Group
    • Any Question? The most exciting phrase to hear in science, the one that heralds new discoveries is not “Eureka”! (I found it!) but “Thats funny...” - Isaac AsimovKaveh Ahmadi-Abhari 47 Shiraz University, Soft Computing Group