Can Evolution Strategies Improve
g p
Learning Guidance in XCS? Design and
Comparison with GA b d XCS
C i ith GA-based
Sergio Morales-Ortigosa
Albert Orriols-Puig
Ester Bernadó-Mansilla
Enginyeria i Arquitectura La Salle
Universitat Ramon Llull
{is09767,aorriols,esterb}@salle.url.edu

Framework
Michigan style
Michigan-style LCSs (Holland, 1976) have reached maturity
Environment
Sensorial Feedback
Action
state
Any Representation:
Classifier 1
Learning
Genetic
production rules, Classifier 2
Classifier
genetic programs, Algorithm
System
perceptrons,
perceptrons Classifier n
SVMs Rule evolution:
Typically, a GA: selection, crossover,
mutation, and replacement
Extended Classifier System - XCS (Wilson, 1995, 1998)
By far, the most influential LCS
Slide 2
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Motivation
Problems with continuous attributes
Interval-based representation (Wilson, 2001)
IF v1 Є [l1, u1] and v2 Є [l2, u2] and … and vn Є [ln, un] THEN classi
[ [ [
They yield competitive results, but we
have little understanding of how they work!
•2-point crossover
Too disruptive?
p
• Mutation: add a random uniform value
Could we use more information?
Could we design better genetic operators?
Not exactly clear the impact of crossover and mutation
Systematic analysis
Creative analysis: propose new operators
i li
Slide 3
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Purpose of the Work
Looking at the continuous optimization realm
Evolution strategies
Real-coded GAs
The purpose of this work is to
Design an XCS based on evolution strategies (ES)
Adapt classifier representation
Design ES mutation and crossover alike for XCS
Analyze the role of Gaussian mutation
Compare whether ES-based XCS outperforms GA-based XCS
Slide 4
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Outline
1.
1 Description of XCS
2. Evolution Strategies in XCS
3. Experimental Methodology
4. Results
5. Conclusions and Further Work
Slide 5
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Description of XCS
Environment
Match Set [M]
Problem
instance
1C A PεF num as ts exp
Selected
3C A PεF num as ts exp
action
5C A PεF num as ts exp
Population [ ]
p [P] 6C A PεF num as ts exp
Match set
… REWARD
generation
1C A PεF num as ts exp
Prediction Array 1000/0
2C A PεF num as ts exp
3C A PεF num as ts exp
c1 c2 … cn
4C A PεF num as ts exp
p
5C A PεF num as ts exp
Random Action
6C A PεF num as ts exp
…
Action Set [A]
1C A PεF num as ts exp
Deletion Classifier
3C A PεF num as ts exp
Selection, Reproduction,
Parameters
5C A PεF num as ts exp
Mutation
6C A PεF num as ts exp Update
…
Genetic Algorithm
G ti Al ith
Slide 6
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Genetic Operators
Selection
Proportionate selection
Tournament selection
Crossover: Offspring
Parents
Two-point crossover
T it
Mutation:
GA-based XCS: Add a uniform random value
GA based
Slide 7
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Outline
1.
1 Description of XCS
2. Evolution Strategies in XCS
3. Experimental Methodology
4. Results
5. Conclusions and Further Work
Slide 8
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

GAs vs ESs Head to Head
Genetic Algorithms
Initially used with binary representation
Key aspects:
yp
GAs process (mix & ensemble) building blocks
Crossover as primary search operator
Mutation as local search operator
Evolution St t i
E l ti Strategies
Initially designed for problems with continuous attributes
Key aspects:
Search focuses little improvement/selection
Gaussian mutation is the search operator
Crossover included afterwards to resemble GAs
Slide 9
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

ES-based XCS
Representation extended with a vector of strategy p
p gy parameters
IF v1 Є [l1, u1] and v2 Є [l2, u2] and … and vn Є [ln, un] THEN classi
(σ1, σ2, …, σn)
The strategy parameters (SP) evolve with the representation
Genetic operators modified to deal with the new rep.
Mutation
Intervals i mutated as:
ui = ui + σ i N (0,1)
li = li + σ i N (0,1)
Strategy parameter vector mutated as:
w ee
where
τ 0 N 0 ( 0 ,1) τ N i ( 0 ,1)
σi = e
'
τ0 = 1/(2n)0.5 and τ = 1/(2n0.5)0.5
e
Slide 10
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

ES-based XCS
Crossover
Discrete/dominant recombination for object parameters
Each variable and SP are randomly selected from one parent
Intermediate recombination for strategy parameters
Calculates the center of mass of the parents
Pushes to the average value
Selection
Fitness proportionate selection
Tournament selection
Truncation selection
T ti l ti
Slide 11
Grup de Recerca en Sistemes Intel·ligents Çan Evolution Strategies Improve Learning Guidance in XCS?

Outline
1.
1 Description of XCS
2. Evolution Strategies in XCS
3. Experimental Methodology
4. Results
5. Conclusions and Further Work
Slide 12
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Experimental Methodology
Analyze the effects of
Selection + mutation (local search)
Selection + mutation + crossover (innovation)
Experiments run on 12 real-world data sets (UCI rep.)
10-fold cross-validation
Slide 13
Grup de Recerca en Sistemes Intel·ligents Çan Evolution Strategies Improve Learning Guidance in XCS?

Experimental Methodology
Results statistically compared by means of
The multicomparison Friedman test
The post-hoc Bonferroni-Dunn test for multiple comparisons
The Wilcoxon signed-ranks t t for pairwise comparisons
Th Wil id k test f ii i
XCS configured as:
#iter=100000, N = 6400, θGA = 50, Pcross = 0.8, Pmut = 0.04,
r_0 = 0.6, m_0 = 0.1
Slide 14
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Outline
1.
1 Description of XCS
2. Evolution Strategies in XCS
3. Experimental Methodology
4. Results
5. Conclusions and Further Work
Slide 15
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Analysis of Selection + Mutation
Test Accuracy
XCS-GA with XCS-ES with XCS-GA with XCS-ES with XCS-ES weighted
XCS-ES with
proportionate proportionate tournament truncation mutation
tournament
According to a post-hoc Bonferroni-Dunn test:
XCS-ES tourn. significantly outperformed XCS-GA with both selection schemes
XCS-ES proportionate significantly outperformed XCS-GA proportionate
Slide 16
Grup de Recerca en Sistemes Intel·ligents Çan Evolution Strategies Improve Learning Guidance in XCS?

Selection + Crossover + Mutation
XCS-GA with XCS-ES with XCS-GA with XCS-ES with
XCS-ES with
pp
proportionate pp
proportionate tournament truncation
tournament
XCS-ES still is the best method
But now, no significant differences
Slide 17
Grup de Recerca en Sistemes Intel·ligents Çan Evolution Strategies Improve Learning Guidance in XCS?

A Cool Example
Domain
XCS-GA with proportionate selection XCS-ES with proportionate selection
Slide 18
Grup de Recerca en Sistemes Intel·ligents Çan Evolution Strategies Improve Learning Guidance in XCS?

Outline
1.
1 Description of XCS
2. Evolution Strategies in XCS
3. Experimental Methodology
4. Results
5. Conclusions and Further Work
Slide 19
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Conclusions
The analysis performed in this paper permitted
To study the discovery component of XCS, especially focusing
on the role of mutation.
Improve XCS to deal with problems with complex boundaries
described by continuous attributes.
y
Two important observations:
Gaussian mutation performs innovation tasks.
When crossover is included XCS-GA does not significantly
outperform XCS ES B still, it wins.
f XCS-ES. But ill i i
The overall work clearly shows the importance of further
y p
researching on GA operators.
Slide 20
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Further Work
XCS ES
XCS-ES is good! But, always?
On average, yes!
Specific problems may not benefit from ES operators
May
M evolution tell me when to use one type of search or
l ti t ll ht t f h
another?
Existing studies on self-adaptation mutation for ternary rules
ii di lf d i if l
Search for evolution signals
Combine different operators
Let classifiers decide which operator to use
Characterize learning domains
Slide 21
Grup de Recerca en Sistemes Intel·ligents Can Evolution Strategies Improve Learning Guidance in XCS?

Can Evolution Strategies Improve
g p
Learning Guidance in XCS? Design and
Comparison with GA b d XCS
C i ith GA-based
Sergio Morales-Ortigosa
Albert Orriols-Puig
Ester Bernadó-Mansilla
Enginyeria i Arquitectura La Salle
Universitat Ramon Llull
{is09767,aorriols,esterb}@salle.url.edu