Fighting Knowledge Acquisition Bottleneck  with   Argument Based Machine Learning Martin Mozina, Matej Guid, Jana Krivec, ...
Motivation for Knowledge Acquisition with Argument Based Machine Learning <ul><li>Knowledge Acquisition is a major bottlen...
Combining Machine Learning and Expert Knowledge <ul><li>Expert provides background knowledge for ML </li></ul><ul><li>Expe...
Combining Machine Learning and Expert Knowledge <ul><li>Expert provides background knowledge for ML </li></ul><ul><li>Expe...
Combining Machine Learning and Expert Knowledge <ul><li>Expert provides background knowledge for ML </li></ul><ul><li>Expe...
Definition of  Argument Based Machine Learning <ul><li>Learning with background knowledge: </li></ul><ul><ul><li>INPUT: le...
Argument Based Rule Learning <ul><li>Classic   rule learning :  IF  HairColor = Blond  THEN  CreditApproved = YES </li></u...
Formal definition of  Argumented Examples <ul><li>Argumented Example (A, C, Arguments) : </li></ul><ul><ul><li>A ; attribu...
ABCN2 <ul><li>ABCN2  = extension of CN2 rule learning algorithm  </li></ul><ul><li>(Clark,Niblett 1991) </li></ul><ul><li>...
Interactions between expert and ABML <ul><li>Learn  a hypothesis  with ABML . </li></ul><ul><li>Find the  most critical ex...
Interactions between expert and ABML <ul><li>Learn a hypothesis with ABML . </li></ul><ul><li>Find the  most critical exam...
Knowledge Acquisition of Chess Concepts  used in  a Chess Tutoring Application Case Study: Bad Bishop
The Concept of the Bad Bishop <ul><li>Chess experts in general understand  the concept of bad bishop . </li></ul><ul><li>P...
Data set <ul><li>Data set:  200 middlegame   positions     from real chess   games </li></ul><ul><li>Chess experts’ evalua...
Standard M achine  L earning  M ethods'  P erformance with  CRAFTY 's features only <ul><li>Machine learning methods’  per...
First Critical Example <ul><li>R ules obtained by ABML method ABCN2   failed to classify this example   as    &quot;not ba...
Introducing new attributes into the domain and adding arguments to an example <ul><li>Experts’ explanation could not be de...
C ounter example <ul><li>M ethod   failed to  explain critical example  with given  argument. </li></ul><ul><li>Counter ex...
I mproving  A rguments  with Counter Examples <ul><li>A ttribute  BAD _ PAWNS  was included into the domain. </li></ul><ul...
Assesing “bad” pawns <ul><li>The experts designed  a look-up table  (left) with predefined values for    the pawns that ar...
After the Final Iteration... <ul><li>The whole  process consisted of  8 iterations . </li></ul><ul><ul><li>7 arguments  we...
Classification Accuracy Through Iterations Results on the final dataset Method CA Brier score AUC Decision trees  ( C4.5) ...
Classification Accuracy Through Iterations <ul><li>The accuracies of all  methods improved   by adding  new  attributes . ...
Advantages of ABML for Knowledge Acquisition explain single example easier for experts to articulate knowledge more knowle...
Advantages of ABML for Knowledge Acquisition counter examples detect deficiencies in expert's explanations even more knowl...
Conclusions <ul><li>more knowledge from experts </li></ul><ul><li>time of experts' involvent is decreased </li></ul><ul><l...
Upcoming SlideShare
Loading in …5
×

Fighting Knowledge Acquisition Bottleneck with Argument Based ...

345 views
257 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
345
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Fighting Knowledge Acquisition Bottleneck with Argument Based ...

  1. 1. Fighting Knowledge Acquisition Bottleneck with Argument Based Machine Learning Martin Mozina, Matej Guid, Jana Krivec, Aleksander Sadikov and Ivan Bratko ECAI 2008 Faculty of Computer and Information Science University of Ljubljana, Slovenia
  2. 2. Motivation for Knowledge Acquisition with Argument Based Machine Learning <ul><li>Knowledge Acquisition is a major bottleneck in building knowledge bases. </li></ul><ul><ul><li>domain experts find it hard to articulate their knowledge </li></ul></ul><ul><ul><li>Machine Learning is a potential solution, but has weaknesses </li></ul></ul><ul><li>Machine Learning & Knowledge Acquisition </li></ul><ul><li>Problem: Models are not comprehensible to domain experts </li></ul><ul><ul><li>mostly statistical learning (not symbolic) </li></ul></ul><ul><ul><li>inducing spurious concepts (e.g. overfitting) </li></ul></ul><ul><li>C ombination of domain expert and machine learning would yield best resu lts </li></ul><ul><ul><li>learn symbolic models </li></ul></ul><ul><ul><li>exploit experts’ knowledge in learning </li></ul></ul>
  3. 3. Combining Machine Learning and Expert Knowledge <ul><li>Expert provides background knowledge for ML </li></ul><ul><li>Expert validates and revises induced theory </li></ul><ul><li>Iterative procedure: Experts and ML improve the model in turns </li></ul>IF ... THEN ... IF ... THEN ... ...
  4. 4. Combining Machine Learning and Expert Knowledge <ul><li>Expert provides background knowledge for ML </li></ul><ul><li>Expert validates and revises induced theory </li></ul><ul><li>Iterative procedure: Experts and ML improve the model in turns </li></ul>IF ... THEN ... IF ... THEN ... ...
  5. 5. Combining Machine Learning and Expert Knowledge <ul><li>Expert provides background knowledge for ML </li></ul><ul><li>Expert validates and revises induced theory </li></ul><ul><li>Iterative procedure: Experts and ML improve the model in turns </li></ul>IF ... THEN ... IF ... THEN ... ... ABML
  6. 6. Definition of Argument Based Machine Learning <ul><li>Learning with background knowledge: </li></ul><ul><ul><li>INPUT: learning examples E , background knowledge BK </li></ul></ul><ul><ul><li>OUTPUT: theory T , T and BK explain all e i from E </li></ul></ul><ul><li>Argument Based Machine Learning: </li></ul><ul><ul><li>INPUT: learning examples E , arguments a i given to e i (from E ) </li></ul></ul><ul><ul><li>OUTPUT: theory T , T explains e i with arguments a i </li></ul></ul>BK,T e i T a i e i
  7. 7. Argument Based Rule Learning <ul><li>Classic rule learning : IF HairColor = Blond THEN CreditApproved = YES </li></ul><ul><li>Possible argument : Miss White received credit ( CreditApproved=YES ) because she has a regular job ( RegularJob=YES ). </li></ul><ul><li>AB rule learning (possible rule) : IF RegularJob=YES AND AccountStatus = Positive THEN CreditApproved = YES </li></ul>Name RegularJob Rich AccountStatus HairColor CreditApproved Mr. Bond No Yes Negative Blond Yes Mr. Grey No No Positive Grey No Miss White Yes No Positive Blond Yes Miss Silver Yes Yes Positive Blond Yes Mrs. Brown Yes No Negative Brown No Name RegularJob Rich AccountStatus HairColor CreditApproved Mr. Bond No Yes Negative Blond Yes Mr. Grey No No Positive Grey No Miss White Yes No Positive Blond Yes Miss Silver Yes Yes Positive Blond Yes Mrs. Brown Yes No Negative Brown No Name RegularJob Rich AccountStatus HairColor CreditApproved Mr. Bond No Yes Negative Blond Yes Mr. Grey No No Positive Grey No Miss White Yes No Positive Blond Yes Miss Silver Yes Yes Positive Blond Yes Mrs. Brown Yes No Negative Brown No Name RegularJob Rich AccountStatus HairColor CreditApproved Mr. Bond No Yes Negative Blond Yes Mr. Grey No No Positive Grey No Miss White Yes No Positive Blond Yes Miss Silver Yes Yes Positive Blond Yes Mrs. Brown Yes No Negative Brown No
  8. 8. Formal definition of Argumented Examples <ul><li>Argumented Example (A, C, Arguments) : </li></ul><ul><ul><li>A ; attribute-value vector [e.g. RegularJob=YES,Rich=NO, ...] </li></ul></ul><ul><ul><li>C ; class value [e.g. CreditApproved=YES] </li></ul></ul><ul><ul><li>Arguments ; a set of arguments Arg 1 , ..., Arg n for this example </li></ul></ul><ul><li>Argument Argi : </li></ul><ul><ul><li>Positive argument: C because Reasons </li></ul></ul><ul><ul><li>Negative Argument: C despite Reasons </li></ul></ul><ul><li>Reasons : a conjunction of reasons r 1 , ..., r m </li></ul>
  9. 9. ABCN2 <ul><li>ABCN2 = extension of CN2 rule learning algorithm </li></ul><ul><li>(Clark,Niblett 1991) </li></ul><ul><li>Extensions: </li></ul><ul><ul><li>Argument Based covering : </li></ul></ul><ul><ul><ul><li>All conditions in R are true for E </li></ul></ul></ul><ul><ul><ul><li>R is consistent with at least one positive argument of E . </li></ul></ul></ul><ul><ul><ul><li>R is not consistent with any negative argument of E . </li></ul></ul></ul><ul><ul><li>Evaluation: Extreme Value Correction (Mozina et al. 2006) </li></ul></ul><ul><ul><li>Probabilistic covering (required for Extreme Value Correction) </li></ul></ul>
  10. 10. Interactions between expert and ABML <ul><li>Learn a hypothesis with ABML . </li></ul><ul><li>Find the most critical example . (if none found, stop procedure) </li></ul><ul><li>Expert explains the example . </li></ul><ul><li>Argument is added to the example . </li></ul><ul><li>Return to step 1. </li></ul>Argument <ul><ul><li>What if expert’s explanation </li></ul></ul><ul><ul><li>is not good enough? </li></ul></ul>ABML critical example learn data set
  11. 11. Interactions between expert and ABML <ul><li>Learn a hypothesis with ABML . </li></ul><ul><li>Find the most critical example . (if none found, stop procedure) </li></ul><ul><li>Expert explains the example . </li></ul><ul><li>Argument is added to the example. </li></ul><ul><li>Return to step 1. </li></ul><ul><li>Expert explains example . </li></ul><ul><li>Add argument to example </li></ul><ul><li>Discover counter examples (if none, then stop). </li></ul><ul><li>Expert improves the argument for example . </li></ul><ul><li>Return to step 3. </li></ul><ul><ul><li>What if expert’s explanation </li></ul></ul><ul><ul><li>is not good enough? </li></ul></ul>
  12. 12. Knowledge Acquisition of Chess Concepts used in a Chess Tutoring Application Case Study: Bad Bishop
  13. 13. The Concept of the Bad Bishop <ul><li>Chess experts in general understand the concept of bad bishop . </li></ul><ul><li>Precise formalisation of this concept is difficult . </li></ul><ul><li>Traditional definition ( John Watson, Secrets of Modern Chess Strategy, 1999 ) </li></ul><ul><li>A bishop that is on the same colour of squares as its own pawns is b ad : </li></ul><ul><ul><li>its mobility is restricted by its own pawns , </li></ul></ul><ul><ul><li>it does not defend the squares in front of these pawns . </li></ul></ul><ul><li>Moreover, cen tralisation of these pawns is the main factor in deciding whether the bishop is bad or not . </li></ul>
  14. 14. Data set <ul><li>Data set: 200 middlegame positions from real chess games </li></ul><ul><li>Chess experts’ evaluation of bishops: </li></ul><ul><ul><li>bad: 78 bishops </li></ul></ul><ul><ul><li>not bad: 122 bishops </li></ul></ul><ul><li>CRAFTY ’s positional feature values served as attribute values for learning. </li></ul><ul><li>We randomly selected : </li></ul><ul><ul><li>100 positions for learning </li></ul></ul><ul><ul><li>100 positions for testing </li></ul></ul>wGM Jana Krivec GM Garry Kasparov FM Matej Guid
  15. 15. Standard M achine L earning M ethods' P erformance with CRAFTY 's features only <ul><li>Machine learning methods’ performance on initial dataset </li></ul><ul><li>The results were obtained on test data set. </li></ul><ul><li>The results obtained with CRAFTY ’s positional features only are too inaccurate for commenting purposes … </li></ul><ul><ul><li>additional information for describing bad bishops is necessary. </li></ul></ul>Method CA Brier score AUC Decision trees ( C4.5) 73% 0,49 0,74 Logistic regression 70% 0,43 0,84 Rule learning (CN2) 72% 0,39 0,80
  16. 16. First Critical Example <ul><li>R ules obtained by ABML method ABCN2 failed to classify this example as &quot;not bad&quot; </li></ul><ul><li>The following question was given to the experts: </li></ul><ul><li>“ Why is the black bishop not bad?“ </li></ul><ul><li>The experts used their domain knowledge : </li></ul><ul><li> “ The black bishop is not bad, since its mobility is not seriously restricted by the pawns of both players.” </li></ul>
  17. 17. Introducing new attributes into the domain and adding arguments to an example <ul><li>Experts’ explanation could not be described with current domain attributes . </li></ul><ul><li>T he argument </li></ul><ul><li> “ BISHOP=“not bad” because I MPROVED _ BISHOP _ MOBILITY is high“ </li></ul><ul><li> was added to th e example . </li></ul><ul><li>A new attribute , IM PROVED _ BISHOP _ MOBILITY , was included into the domain : </li></ul><ul><ul><li>the number of squares accessible to the bishop, taking into account only own and opponent ’ s pawn structure </li></ul></ul>
  18. 18. C ounter example <ul><li>M ethod failed to explain critical example with given argument. </li></ul><ul><li>Counter example was presented to experts: </li></ul><ul><li>Experts’ explanation: “ T here are many pawns on the same colour of squares as the black bishop, and some of these pawns occupy the central squares .” </li></ul>Counter example : “bad”, although IMPR OVED_BISHOP_MOBILITY is high. <ul><li>&quot;Why is the “red” bishop bad, comparing to the “green” one?&quot; </li></ul>Critical example : “not bad”, IMPR OVED_BISHOP_MOBILITY is high.
  19. 19. I mproving A rguments with Counter Examples <ul><li>A ttribute BAD _ PAWNS was included into the domain. </li></ul><ul><ul><li>This attribute evaluates pawns that are on the colour of the square of the bishop (&quot;bad&quot; pawns in this sense). </li></ul></ul><ul><li>The argument given to the critical example was extended to “ BISHOP= “not bad” because IMPROVED _ BISHOP _ MOBILITY is high and BAD _ PAWNS is low ” </li></ul><ul><li>W ith this argument the method could not find any counter examples anymore. </li></ul><ul><li>N ew rule : </li></ul><ul><li>if IMPROVED _ BISHOP _ MOBILITY ≥ 4 </li></ul><ul><li>and BAD _ PAWNS ≤ 32 </li></ul><ul><li>then BISHOP = “not bad” class distribution [0,39] </li></ul>
  20. 20. Assesing “bad” pawns <ul><li>The experts designed a look-up table (left) with predefined values for the pawns that are on the color of the square of the bishop in order to assign weights to such pawns. </li></ul>BAD_PAWNS_AHEAD = 16 + 24 + 2 = 42
  21. 21. After the Final Iteration... <ul><li>The whole process consisted of 8 iterations . </li></ul><ul><ul><li>7 arguments were attached to automatically selected critical examples </li></ul></ul><ul><ul><li>5 new attributes were included into the domain </li></ul></ul>Attribute Description BAD_PAWNS pawns on the color of the square of the bishop - weighted according to their squares ( bad pawns )  BAD_PAWNS_AHEAD  bad pawns ahead of the bishop BAD_PAWNS _BLOCK_BISHOP_DIAGONAL bad pawns that block the bishop's (front) diagonals   BLOCKED_BAD_PAWNS  bad pawns blocked by opponent's pawns or pieces   IMPROVED_BISHOP_MOBILITY  n umber of squares accessible to the bishop taking into account only pawns of both opponents 
  22. 22. Classification Accuracy Through Iterations Results on the final dataset Method CA Brier score AUC Decision trees ( C4.5) 89% 0,21 0,86 Logistic regression 88% 0,19 0,96 Rule learning (CN2) 88% 0,19 0,94 ABCN2 95% 0,11 0,97
  23. 23. Classification Accuracy Through Iterations <ul><li>The accuracies of all methods improved by adding new attributes . </li></ul><ul><li>ABCN2 (which also used the arguments) ou tperformed all others . </li></ul>A rguments suggested useful attributes AND lead to even more accurate models.
  24. 24. Advantages of ABML for Knowledge Acquisition explain single example easier for experts to articulate knowledge more knowledge from experts critical examples expert provide only relevant knowledge time of experts' involvent is decreased
  25. 25. Advantages of ABML for Knowledge Acquisition counter examples detect deficiencies in expert's explanations even more knowledge from experts arguments constrain learning hypotheses are consistent with expert knowledge hypotheses comprehensible to expert more accurate hypotheses
  26. 26. Conclusions <ul><li>more knowledge from experts </li></ul><ul><li>time of experts' involvent is decreased </li></ul><ul><li>hypotheses comprehensible to expert </li></ul><ul><li>more accurate hypotheses </li></ul>ABML-based Knowledge Acquisition process provides: A rgument Based Machine Learning enables better knowledge acquisition

×