Evolving Rules to Solve Problems: The Learning Classifier Systems Way

Evolving Rules to Solve Problems: The Learning Classifier Systems Way Pier Luca Lanzi EPIA 2007, Guimarães , Portugal, September 4th, 2007

Evolving

Early Evolutionary Research 3 Box (1957). Evolutionary operations. Led to simplex methods, Nelder-Mead. Other Evolutionaries: Friedman (1959), Bledsoe (1961), Bremermann (1961) Rechenberg (1964), Schwefel (1965). Evolution Strategies. Fogel, Owens & Walsh (1966). Evolutionary programming. Common view Evolution = Random mutation + Save the best. Pier Luca Lanzi

Early intuitions 4 “There is the genetical or evolutionary search by which a combination of genes is looked for, the criterion being the survival value.” Alan M. Turing, Intelligent Machinery, 1948 “We cannot expect to find a good child-machine at the first attempt. One must experiment with teaching one such machine and see how well it learns. One can then try another and see if it is better or worse. There is an obvious connection between this process and evolution, by the identifications “Structure of the child machine = Hereditary material “Changes of the child machine = Mutations “Natural selection = Judgment of the experimenter” Alan M. Turing, “Computing Machinery and Intelligence” 1950. Pier Luca Lanzi

Meanwhile… in Ann Arbor… 5 Holland (1959). Iterative circuit computers. Holland (1962). Outline for a logical theory of adaptive systems. Role of recombination (Holland 1965) Role of schemata (Holland 1968, 1971) Two-armed bandit (Holland 1973, 1975) First dissertations (Bagley, Rosenberg 1967) Simple Genetic Algorithm(De Jong 1975) Pier Luca Lanzi

What are Genetic Algorithms? 6 Genetic algorithms (GAs) are search algorithms based on the mechanics of natural selection and genetics Two components Natural selection: survival of the fittest Genetics: recombination of structures, variation Underlying methaphor Individuals in a population must be adapted to the environment to survive and reproduce A problem can be viewed as an environment, we evolve a population of solutions to solve it Different individuals are differently adapted To survive a solution must be “adapted” to the problem Pier Luca Lanzi

A Peek into Genetic Algorithms 7 Population Representation A set of candidate The coding of solutions solutions Originally, binary strings Fitness function Operators inspired by Nature Evaluates candidate Selection solutions Recombination Mutation Genetic Algorithm Generate an initial random population Repeat Select promising solutions Create new solutions by applying variation Incorporate new solutions into original population Until stop criterion met Pier Luca Lanzi

Holland’s Vision, Cognitive System One 9 To state, in concrete technical form, a model of a complete mind and its several aspects A cognitive system interacting with an environment Binary detectors and effectors Knowledge = set of classifiers Condition-action rules that recognize a situation and propose an action Payoff reservoir for the system’s needs Payoff distributed through an epochal algorithm Internal memory as message list Genetic search of classifiers Pier Luca Lanzi

What was the goal? 10 1#11:buy ⇒30 0#0#:sell ⇒-2 … A real system with an unknown underlying dynamics Use a classifier system online to generate a behavior that matched the real system. The evolved rules would provide a plausible, human readable, model of the unknown system Pier Luca Lanzi

Holland’s Learning Classifier Systems 11 Explicit representation of the incoming reward Good classifiers are the ones that predict high rewards Credit Assignment using Bucket Brigade Rule Discovery through a genetic algorithm on all the rule base (on the whole solution) Description was vast It did not work right off! Very limited success David E. Goldberg: Computer-aided gas pipeline operation using genetic algorithms and rule learning, PhD thesis. University of Michigan. Ann Arbor, MI. Pier Luca Lanzi

Learning System LS-1 & 12 Pittsburgh Classifier Systems Holland models learning as an adaptation process De Jong models learning as an optimization process Genetic algorithm applied to a population of rule sets 1. t := 0 2. Initialize the population P(t) 3. Evaluate the rules sets in P(t) 4. While the termination condition is not satisfied 5. Begin 6. Select the rule sets in P(t) and generate Ps(t) 7. Recombine and mutate the rule sets in Ps(t) 8. P(t+1) := Ps(t) 9. t := t+1 No apportionment of credit 10. Evaluate the rules sets in P(t) Offline evaluation of rule sets 11. End Pier Luca Lanzi

As time goes by… 13 Genetic algorithms and CS-1 1970’s Research flourishes Success is limited Reinforcement Learning Evolving rules as optimization Machine 1980’s Research follows Holland’s vision Learning Success is still limited Stewart Wilson Robotics applications 1990’s creates XCS First results on classification But the interest fades away Classifier systems finally work 2000’s Large development of models, facetwise theory, and applications Pier Luca Lanzi

Stewart W. Wilson & 14 The XCS Classifier System 1. Simplify the model 2. Go for accurate predictions not high payoffs 3. Apply the genetic algorithm to subproblems not to the whole problem 4. Focus on classifier systems as reinforcement learning with rule-based generalization 5. Use reinforcement learning (Q-learning) to distribute reward Most successfull model developed so far Wilson, S.W.: Classifier Fitness Based on Accuracy. Evolutionary Computation 3(2), 149-175 (1995). Pier Luca Lanzi

The Classifier System Way

Learning Classifier Systems as 16 Reinforcement Learning Methods System stt+1 at rt+1 Environment The goal: maximize the amount of reward received How much future reward when at is performed in st? What is the expected payoff for st and at? Need to compute a value function, Q(st,at)→ payoff Pier Luca Lanzi

17 Define the inputs, the actions, and how the reward is determined HowDefine the expected payoff does reinforcement learning work? Compute a value function Q(st,at) mapping state-action pairs into expected payoffs Pier Luca Lanzi

How does reinforcement learning work? 18 First we define the expected payoff First we define the expected payoff as γ is the discount factor Pier Luca Lanzi

How does reinforcement learning work? 19 Then, Q-learning is an option. At the beginning, is initialized with random values At time t, previous value new estimate incoming reward new estimate Parameters, Discount factor γ The learning rate β The action selection strategy Pier Luca Lanzi

This looks simple… 20 Let’s bring RL to the real world! Reinforcement learning assumes that Q(st,at) is represented as a table But the real world is complex, the number of possible inputs can be huge! We cannot afford an exact Q(st,at) Pier Luca Lanzi

Example: The Mountain Car 21 rt = 0 when goal is reached, -1 otherwise. GOAL Value Function Q(st,at) st = position, acc . acc velocity . ht, . left, no a cc a= r ig t Task: drive an underpowered car up a steep mountain road Pier Luca Lanzi

What are the issues? 22 Exact representation infeasible Approximation mandatory The function is unknown, it is learnt online from experience Pier Luca Lanzi

What are the issues? 23 Learning an unknown payoff function while also trying to approximate it Approximator works on intermediate estimates While also providing information for the learning Convergence is not guaranteed Pier Luca Lanzi

Whats does this have to do with 24 Learning Classifier Systems? They solve reinforcement learning problems Represent the payoff function Q(st, at) as a population of rules, the classifiers Classifiers are evolved while Q(st, at) is learnt online Pier Luca Lanzi

What is a classifier? 25 IF condition C is true for input s THEN the payoff of action A is p Accurate approximations payoff payoff surface for A p General conditions covering large portions Condition of the problem space C(s)=l≤s≤u s l u Pier Luca Lanzi

What types of solutions? 26 Pier Luca Lanzi

How do learning classifier systems work? 27 The main performance cycle Pier Luca Lanzi

How do learning classifier systems work? 28 The main performance cycle The classifiers predict an expected payoff The incoming reward is used to update the rules which helped in getting the reward Any reinforcement learning algorithm can be used to estimate the classifier prediction. Pier Luca Lanzi

How do learning classifier systems work? 29 The main performance cycle Pier Luca Lanzi

30 In principle, any search method may be used I prefer genetic algorithms Where do classifiers come from? because they are representation independent A genetic algorithm select, recombines, mutate existing classifiers to search for better ones Pier Luca Lanzi

What are the good classifiers? 31 What is the classifier fitness? The goal is to approximate a target value function with as few classifiers as possible We wish to have an accurate approximation One possible approach is to define fitness as a function of the classifier prediction accuracy Pier Luca Lanzi

What about getting as 32 few classifiers as possible? The genetic algorithm can take care of this General classifiers apply more often, thus they are reproduced more But since fitness is based on classifiers accuracy only accurate classifiers are likely to be reproduced The genetic algorithm evolves maximally general maximally accurate classifiers Pier Luca Lanzi

How to apply learning classifier systems 33 Environment Determine the inputs, the actions, and how reward is distributed Determine what is the expected payoff that must be maximized st rt at Decide an action selection strategy Set up the parameters β and γ Learning Classifier System Select a representation for conditions, the recombination and the mutation operators Select a reinforcement learning algorithm Setup the parameters, mainly the population size, the parameters for the genetic algorithm, etc. Pier Luca Lanzi

Things can be extremely simple! 34 For instance in supervised classification Environment 1 if the class is correct 0 if the class is not correct example class Select a representation for conditions and the recombination and mutation operators Setup the parameters, mainly the population size, the parameters for the genetic algorithm, etc. Learning Classifier System Pier Luca Lanzi

35 Genetics-Based Accurate Estimates Generalization About Classifiers (Powerful RL) Classifier Representation Pier Luca Lanzi

One Representation, 36 One Principle Data described by 6 variables a1, …, a6 They represents the simple concept “a1=a2 Ç a5=1” A rather typical approach Select a representation Select an algorithm which produces such a representation Apply the algorithm Decision Rules (attribute-value) if (a5 = 1) then class 1 [95.3%] If (a1=3 Æ a2=3) then class = 1 [92.2%] … FOIL Clause 0: is_0(a1,a2,a3,a4,a5,a6) :- a1≠a2, a5≠ 1 Pier Luca Lanzi

Learning Classifier Systems: 37 One Principle Many Representations Ternary rules ####1#:1 if a5<2, class=1 0, 1, # 22####:1 if a1=a2, class=1 Learning Classifier System Genetic Estimates Knowledge Search RL & ML Representation Conditions & Ternary Conditions Attribute-Value Symbolic Prediction Conditions 0, 1, # No need to change the framework Ternary Conditions Attribute-Value Symbolic Just plug-in your favourite representation 0, 1, # Conditions Conditions Pier Luca Lanzi

Better Estimations

What is computed prediction? 39 Replace the prediction p by a parametrized function p(x,w) Which type of approximation? payoff payoff p(x,w)=w0+xw1 landscape of A Which Representation? Condition C(s)=l≤s≤u x l u Pier Luca Lanzi

Same example with computed prediction 40 Again, no need to change the framework Just plug-in your favourite estimator Linear, Polynomial, NNs, SVMs, tile-coding Pier Luca Lanzi

Any Theory?

42 Learning Classifier Systems involve Representation, Reinforcement Learning, & Genetics-based Search Unified theory is impractical Develop facetwise models Pier Luca Lanzi

Facetwise Models for a Theory of 43 Evolution and Learning Prof. David E. Goldberg University of Illinois at Urbana Champaign David Goldberg & Kumara Sastry Genetic Algorithms: The Design of Innovation Springer-Verlag May 2008 Facetwise approach to analysis and design for genetic algorithms In learning classifier systems Separate learning from evolution Simplify the problem by focusing only on relevant aspect Derive facetwise models Applied to model several aspects of evolution E.g., the time to convergence is O(L 2k) Pier Luca Lanzi

What Applications?

45 Computational Models of Cognition Others Complex Traffic controllers Target recognition Adaptive Fighter maneuvering Systems … Autonomous Classification Robotics & Data mining Pier Luca Lanzi

Modeling Cognition

What Applications? 47 Computational Models of Cognition Learning classifier system model certain aspects of cognition Human language learning Perceptual category learning Affect theory Anticipatory and latent learning Learning classifier systems provide good models for animals in experiments in which the subjects must learn internal models to perform as well as they do Martin V. Butz, University of Würzburg, Department of Cognitive Psychology III Cognitive Bodyspaces: Learning and Behavior (COBOSLAB) Wolfgang Stolzmann, Daimler Chrysler Rick R. Riolo, University of Michigan, Center for the Study of Complex Systems Pier Luca Lanzi

References 48 Butz, M.V.: Anticipatory Learning Classifier Systems, Genetic Algorithms and Evolutionary Computation, vol. 4. Springer-Verlag (2000) Riolo, R.L.: Lookahead Planning and Latent Learning in a Classifier System. In: J.A. Meyer, S.W. Wilson (eds.) From Animals to Animats 1. Proceedings of the First International Conferenceon Simulation of Adaptive Behavior (SAB90), pp. 316{326. A Bradford Book. MIT Press (1990) Stolzmann, W. and Butz, M.V. and Hoffman, J. and Goldberg, D.E.: First Cognitive Capabilities in the Anticipatory Classifier System. In: From Animals to Animats: Proceedings of the Sixth International Conference on Simulation of Adaptive Behavior. MIT Press (2000) Pier Luca Lanzi

Computational Economics

What Applications? 50 Computational Economics To models one single agent acting in the market (BW Arthur, JH Holland, B LeBaron) To model many interactive agents each one controlled by its own classifier system. Modeling the behavior of agents trading risk free bonds and risky assets Different trader types modeled by supplying different input information sets to a group of homogenous agents Later extended to a multi-LCS architecture applied to portfolio optimization Technology startup company founded in March 2005 Pier Luca Lanzi

References 51 Sor Ying (Byron) Wong, Sonia Schulenburg: Portfolio allocation using XCS experts in technical analysis, market conditions and options market. GECCO (Companion) 2007: 2965-2972 Sonia Schulenburg, Peter Ross: An Adaptive Agent Based Economic Model. Learning Classifier Systems 1999: 263-282 BW Arthur, J.H. Holland, B. LeBaron, R. Palmer, and P. Tayler: quot;Asset Pricing Under Endogenous Expectations in an Artificial Stock Market,“ in The Economy as an Evolving Complex System II. Edited (with S. Durlauf and D. Lane), Addison-Wesley, 1997. BW Arthur, R. Palmer, J. Holland, B. LeBaron, and P. Taylor: quot;Artificial Economic Life: a Simple Model of a Stockmarket,“ Physica D, 75, 264-274, 1994 Pier Luca Lanzi

Data analysis

What Applications? 53 Classification and Data Mining Bull, L. (ed) Applications of Learning Classifier Systems. Springer. (2004) Bull, L., Bernado Mansilla, E. & Holmes, J. (eds) Learning Classifier Systems in Data Mining. Springer. (2008) Nowadays, by far the most important application domain for LCSs Many models GA-Miner, REGAL, GALE GAssist Performance comparable to state of the art machine learning Human Competitive Results 2007 X Llorà, R Reddy, B Matesic, R Bhargava: Towards Better than Human Capability in Diagnosing Prostate Cancer Using Infrared Spectroscopic Imaging Pier Luca Lanzi

Hyper Heuristics

What Applications? 55 Hyper-Heuristics Ross P., Marin-Blazquez J., Schulenburg S., and Hart E., Learning a Procedure that can Solve Hard Bin-packing Problems: A New GA-Based Approach to Hyper-Heuristics. In Proceedings of GECCO 2003 Bin-packing and timetabling problems Pick a set of non-evolutionary heuristics Use classifier system to learn a solution process not a solution The classifier system learns a sequence of heuristics which should be applied to gradually transform the problem from its initial state to its final solved state. Pier Luca Lanzi

Medical data

What Applications? 57 Epidemiologic Surveillance John H. Holmes Center for Clinical Epidemiology & Biostatistics Department of Biostatistics & Epidemiology University of Pennsylvania - School of Medicine Epidemiologic surveillance data need adaptivity to abrupt changes Readable rules are attractive Performance similar to state of the art machine learning But several important feature-outcome relationships missed by other methods were discovered Similar results were reported by Stewart Wilson for breast cancer data Pier Luca Lanzi

References 58 John H. Holmes, Jennifer A. Sager: Rule Discovery in Epidemiologic Surveillance Data Using EpiXCS: An Evolutionary Computation Approach. AIME 2005: 444-452 John H. Holmes, Dennis R. Durbin, Flaura K. Winston: A New Bootstrapping Method to Improve Classification Performance in Learning Classifier Systems. PPSN 2000: 745-754 John H. Holmes, Dennis R. Durbin, Flaura K. Winston: The learning classifier system: an evolutionary computation approach to knowledge discovery in epidemiologic surveillance. Artificial Intelligence in Medicine 19(1): 53-74 (2000) Pier Luca Lanzi

Autonomous Robotics

What Applications? 60 Autonomous Robotics In the 1990's, a major testbed for learning classifier systems. Marco Dorigo and Marco Colombetti: Robot Shaping An Experiment in Behavior Engineering, 1997 They introduced the concept of robot shaping defined as the incremental training of an autonomous agent. Behavior engineering methodology named BAT: Behavior Analysis and Training. Recently, University of West England applied several learning classifier system models to several robotics problems Pier Luca Lanzi

Artificial Ecosystems

What Applications? 62 Modeling Artificial Ecosystems Jon McCormack, Monash University Eden: an interactive, self-generating, artificial ecosystem. World populated by collections of evolving virtual creatures Creatures move about the environment, Make and listen to sounds, Foraging for food, Encountering predators, Mating with each other. Creatures evolve to fit their landscape Eden has four seasons per year (15mins) Jon McCormack Simple physics for rocks, biomass and sonic animals. Pier Luca Lanzi

References 63 McCormack, J. Impossible Nature The Art of Jon McCormack Published by the Australian Centre for the Moving Image ISBN 1 920805 08 7, ISBN 1 920805 09 5 (DVD) J. McCormack: New Challenges for Evolutionary Music and Art, ACM SIGEVOlution Newsletter, Vol. 1(1), April 2006, pp. 5-11. McCormack, J. 2005, 'On the Evolution of Sonic Ecosystems' in Adamatzky, et al. (eds), Artificial Life Models in Software, Springer, Berlin. McCormack, J. 2003, 'Evolving Sonic Ecosystems', Kybernetes, 32(1/2), pp. 184-202. Pier Luca Lanzi

Chemical & Neuronal Networks

What Applications? 65 Chemical and Neuronal Networks L. Bull, A. Budd, C. Stone, I. Uroukov, B. De Lacy Costello and A. Adamatzky University of the West of England Behaviour of non-linear media controlled automatically through evolutionary learning Unconventional computing realised by such an approach. Learning classifier systems Control a light-sensitive sub-excitable Belousov-Zhabotinski reaction Control the electrical stimulation of cultured neuronal networks Pier Luca Lanzi

What Applications? 66 Chemical and Neuronal Networks To control a light-sensitive sub-excitable BZ reaction, pulses of wave fragments are injected into the checkerboard grid resulting in rich spatio-temporal behaviour Learning classifier system can direct the fragments to an arbitrary position through control of the light intensity within each cell Learning Classifier Systems control the electrical stimulation of cultured neuronal networks such that they display elementary learning, respond to a given input signal in a pre-specified way Results indicate that the learned stimulation protocols identify seemingly fundamental properties of in vitro neuronal networks Pier Luca Lanzi

References 67 Larry Bull, Adam Budd, Christopher Stone, Ivan Uroukov, Ben De Lacy Costello and Andrew Adamatzky: Towards Unconventional Computing through Simulated Evolution: Learning Classifier System Control of Non-Linear Media Artificial Life (to appear) Budd, A., Stone, C., Masere, J., Adamatzky, A., DeLacyCostello, B., Bull, L.: Towards machine learning control of chemical computers. In: A. Adamatzky, C. Teuscher (eds.) From Utopian to Genuine Unconventional Computers, pp. 17-36. Luniver Press Bull, L., Uroukov, I.S.: Initial results from the use of learning classier systems to control n vitro neuronal networks. In: Lipson [189], pp. 369-376 Pier Luca Lanzi

Conclusions

Conclusions 69 Cognitive Modeling Complex Adaptive Systems Machine Learning Reinforcement Learning Metaheuristics … Many blocks to plug-in Several Representations Several RL algorithms Several evolutionary methods … Pier Luca Lanzi

Further Readings 70 Martin V. Butz “Rule-Based Evolutionary Online Learning Systems: A Principled Approach to LCS Analysis and Design” Studies in Fuzziness and Soft Computing, Springer-Verlag 2005 Proceedings of IWLCS from 2000 to 2007 published by Springer Verlag Pier Luca Lanzi

Thank you! 71 Any Question? Pier Luca Lanzi

http://webspace.elet.polimi.it	164
http://www.pierlucalanzi.net	76
http://www.slideshare.net	10

Evolving Rules to Solve Problems: The Learning Classifier Systems Way

by Pier Luca Lanzi on Dec 06, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 250

Statistics

Evolving Rules to Solve Problems: The Learning Classifier Systems Way — Presentation Transcript