Anticipation Mappings for Learning Classifier Systems

925 views
873 views

Published on

In this paper, we study the use of anticipation
mappings in learning classifier systems. At first, we enrich the
eXtended Classifier System (XCS) with two types of anticipation
mappings: one based on array of perceptrons array, one
based on neural networks. We apply XCS with anticipation
mappings (XCSAM) to several multistep problems taken from
the literature and compare its anticipatory performance with
that of the Neural Classifier System X-NCS which is based on
a similar approach. Our results show that, although XCSAM is
not a “true” Anticipatory Classifier System like ACS, MACS, or
X-NCS, nevertheless XCSAM can provide accurate anticipatory
predictions while requiring smaller populations than those
needed by X-NCS.

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
925
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Anticipation Mappings for Learning Classifier Systems

    1. 1. Anticipation Mappings for Learning Classifier Systems Larry Bull, Pier Luca Lanzi, Toby O’hara University of the West of England, Bristol, UK Politecnico di Milano, Italy Illinois Genetic Algorithms Laboratory, University of Illinois at Urbana Champaign, USA CEC 2007, September 27th, 2007, Singapore TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A A A
    2. 2. A Brief Look at Classifier Systems If condition C holds in state S , then action A will produce a payoff p , this prediction has an accuracy F
    3. 3. Anticipatory Classifier Systems (ACS) <ul><li>Modify the structure of classifiers and the systems to learn the effect of actions </li></ul><ul><li>Short history </li></ul><ul><ul><li>Rick L. Riolo. 1990 “Lookahead Planning and Latent Learning in a Classifier System.” </li></ul></ul><ul><ul><li>Wolfgang Stolzmann. 1998 “Anticipatory Classier Systems”, GP98 </li></ul></ul><ul><ul><li>Martin V. Butz (2002), “Anticipatory Classifier Systems”, Springer-Verlag </li></ul></ul>If condition C holds in state S t , then action A will produce an effect resulting in state S t+1
    4. 4. Need Anticipations? Compute them! <ul><li>Basic Idea </li></ul><ul><ul><li>Not a system designed for anticipations </li></ul></ul><ul><ul><li>Enrich a known system to learn anticipations </li></ul></ul><ul><ul><li>Anticipatory prediction comes for free </li></ul></ul><ul><ul><li>Maybe less powerful than ACS, but can be applied to any LCS </li></ul></ul>If condition C holds in state S , then action A will produce a payoff p , with an accuracy F If condition C holds in state S , then action A will produce a payoff p , with an accuracy F , and effect is an f (s t , w) <ul><li>First tried by Larry Bull and Toby O’hara (2000) using neural networks to compute an f (s t ,w) </li></ul>
    5. 5. Learning Anticipatory Functions s t s t+1 a t s t a t -> s t+1
    6. 6. (our way to) Anticipations <ul><li>Classifiers has five parameters: </li></ul><ul><ul><li>The condition and the action </li></ul></ul><ul><ul><li>The prediction, the error and the fitness </li></ul></ul><ul><ul><li>The parameter vector w for anticipation </li></ul></ul>If condition C holds in state S , then action A will produce a payoff p , with an accuracy F , and effect is an f (s t , w)
    7. 7. Learning to Anticipate the Effect <ul><li>The next state s t+1 is used to update w of classifiers </li></ul><ul><li>Classifier error is updated according to </li></ul><ul><li>a is the action computed by the classifier </li></ul><ul><ul><li>ε f (x t ,y t ,a) is the error function </li></ul></ul><ul><li>Several error functions, we used the simplest one: 0 if action is correct (a = y t ), 1000 otherwise </li></ul><ul><li>Classifier fitness is updated as in XCS </li></ul>
    8. 8. Anticipatory Prediction <ul><li>Predicting the next state </li></ul><ul><ul><li>Classifiers matching s t that advocate a t </li></ul></ul><ul><ul><li>For each action a in [M], the classification accuracy C(x t ,a) is computed as, </li></ul></ul><ul><li>Learning to predict the next state </li></ul><ul><ul><li>Use current state s t , a t , and s t+1 to train the vector w </li></ul></ul>
    9. 9. What Function? Sigmoid <ul><li>Action is simply a constant: </li></ul><ul><li>Update is performed with a gradient descent: </li></ul>a f xw
    10. 10. What Function? Neural Networks <ul><li>When the problem involves many action we can either </li></ul><ul><ul><li>use an array of simple action function </li></ul></ul><ul><ul><li>use a powerful action function </li></ul></ul><ul><li>Neural Network </li></ul><ul><ul><li>n inputs </li></ul></ul><ul><ul><li>h hidden nodes </li></ul></ul><ul><ul><li>As many outputs as the action encoding needs </li></ul></ul><ul><li>Update is performed using online backpropagation </li></ul>
    11. 11. Woods
    12. 12. Simple Sequential Problems (1) (2) (3)
    13. 13. Anticipation Accuracy (1) accuracy MSE
    14. 14. Anticipation Accuracy (2) accuracy MSE
    15. 15. Anticipation Accuracy (3) accuracy MSE
    16. 16. Alias
    17. 17. Sequential Problems with Aliasing (1) (2) Extension of anticipatory Classifier Systems for problem with noise was developed by Martin Butz, Dave G. Goldberg, and Wolfgang Stolzmann (2000)
    18. 18. Anticipation Accuracy (1) accuracy MSE
    19. 19. Anticipation Accuracy (2) accuracy MSE
    20. 20. We presented a very simple approach to anticipatory behavior Compute the anticipation of the next state based on the previous state and the action performed Very simple, but provide accurate predictions, while requiring smaller populations Simpler than ACSs, probably less powerful Even simple perceptron can be powerful enough Generalizes to real-valued and/or noisy domains
    21. 21. Any Question? Thank you!
    22. 22. Woods1 <ul><li>The Woods1 model is very sparse : there are only 128X17 state-action pairs </li></ul><ul><li>Woods1 is, in practice, more simple than binary sum </li></ul><ul><li>Both perceptron and NN does not exploit the problem sparseness as SVM </li></ul>
    23. 23. Maze 5 <ul><li>Maze5 is still a sparse problem but slightly more complex: it has 288X37 state-action pairs </li></ul><ul><li>Maze5 is, in practice, slight more difficult than binary sum </li></ul><ul><li>Also in this case SVM exploits very effectively the problem sparseness </li></ul>

    ×