The document discusses learning classifier systems (LCS) for addressing class imbalance problems in datasets. It aims to enhance the applicability of LCS to knowledge discovery from real-world datasets that often exhibit class imbalance, where one class is represented by significantly fewer examples than other classes. The author proposes adapting parameters of the XCS learning classifier system, such as learning rate and genetic algorithm threshold, based on estimated class imbalance ratios within classifiers' niches in order to minimize bias towards majority classes and better handle small disjuncts representing minority classes.

1. Learning Classifier Systems
for Class Imbalance
Problems
Ester Bernadó-Mansilla
Research Group in Intelligent Systems
Enginyeria i Arquitectura La Salle
Universitat Ramon Llull
Barcelona, Spain

2. Aim
Enhance the applicability of LCSs
to knowledge discovery from datasets
Classification problems
Real-world domains
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

3. Framework
model
LCS
Dataset
+
estimated
performance
• Representativity of the target
• Evolutionary pressures
concept
• Interpretability
• Geometrical complexity
• Domain of applicability
• Class imbalance
• Noise
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

4. Class Imbalance
When one class is represented by a small number of

examples, compared to other class/es.
Usually the class of that describes the circumscribed

concept (positive class) is the minority class
Where?

Rare medical diagnoses

Fraud detection

Oil spills in satellite images

Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

5. Class Imbalance and Classifiers
Is there a bias towards the majority class?

Probably, because…

Most classifier schemes are trained to minimize the global error

As a result

They classify accurately the examples from the majority class

They tend to misclassify the examples of the minority class,

which are often those representing the target concept.
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

6. Measures of Performance
Confusion matrix
Prediction
A B
Actual A true positive (TP) false negative (FN)
B false positive (FP) true negative (TN)
Accuracy = (TP+TN)/(TP+FN+FP+TN)
TN rate = TN / (TN + FP)
TP rate = TP / (FN + TP)
ROC curves
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

7. The Higher Class Imbalance: the
Higher Bias?
Dataset 1 Dataset 2
concept: 15 concept: 15
counterpart: 150 counterpart: 45
ratio: 10:1 ratio: 3:1
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

8. XCS
XCS
class
input Set of
Rules
update
search
Genetic Reinforcement
Algorithms Learning
reward
Environment
Dataset
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

9. Our Approach with XCS
Bounding XCS’s parameters for unbalanced datasets

Online identification of small disjuncts

Adaptation of parameters for the discovery of small

disjuncts
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

10. XCS’s Behavior in Unbalanced
Datasets
Unbalanced 11-multiplexer problem
ir=16:1 ir=32:1 ir=64:1
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

11. XCS’s Population
Most numerous rules, ir=128:1
Classifier P Error F Num
###########:0 1000 0.12 0.98 385
1.2 10-4
###########:1 0.074 0.98 366
estimated estimated too high
high
prediction:
overgeneral error: numerosity
fitness
992.24
classifiers 15.38
7.75
Test examples are classified as belonging to the majority class
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

12. How Imbalance Affects XCS
Classifier’s error

Stability of prediction and error estimates

Occurrence-based reproduction

Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

13. Classifier’s Error in Unbalanced
Datasets
Will an overgeneral classifier be detected as inaccurate if the

imbalance ratio is high?
Bound for inaccurate classifier: !quot;!0
Given the estimated prediction and error:
P = Pc (cl ) Rmax + (1 ! Pc (cl )) Rmin
quot;=| P ! Rmax | Pc (cl )+ | P ! Rmin | (1 ! Pc (cl ))
We derive:
# quot;o p 2 + 2 p ( Rmax # quot;0 )# quot;0 ! 0
where !quot;!0
p =!C / C
For
Rmax = 1000 !0 = 1
we get maximum imbalance ratio:
irmax = 1998
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

14. Prediction and Error Estimates and
Learning Rate
ir=128:1, ###########:0
Error
Prediction
β=0.2
β=0.002
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

15. Occurrence-based Reproduction
Probability of occurrence (pocc)
Given ir=maj/min:
0,6
Classifier poccB poccI
0,5
1/2 1/2
########### :0
probability of occurrence
1/2 1/2
########### :1 0,4
0,3
0000#######:0 1/32
0,2
0001#######:1 1/32 0,1
0
1 2 4 8 16 32 64 128 256
imbalance ratio
22ir
p occB 00001######:1 00000######:0
ir + 1 ###########:0 ###########:1
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

16. Occurrence-based Reproduction
Probability of reproduction (pGA)

1
pGA =
TGA
if Tocc < % GA
#% GA
where TGA $ quot;
!Tocc otherwise
With θGA=20:

GA
Tocc
…
T (# # # # # # # # # # #: 0) ! quot;
GA GA
θGA
Tocc
GA
…
T (0000# # # # # # #: 0) ! T 1
GA occ
θGA
1 Assuming non-overlapping
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

17. Guidelines for Parameter Tuning
Rmax and є0 determine the threshold between negligible noise and

imbalance ratio
β determines the size of the moving window. The window should be

high enough to allow computing examples from both classes:
f min
! =k
f maj
θGA can counterbalance the reproduction opportunities of most frequent

(majority) and least frequent niches (minority):
1
! GA = k '
f min
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

18. XCS with Parameters Tuning
XCS with parameter tuning
XCS with standard settings
ir=16:1 ir=32:1 ir=64:1 ir=64:1 ir=256:1
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

19. XCS Tuning for Real-world Datasets
How we can estimate the niche frequency?

Estimate from the ratio of majority class instances and minority

class instances
Problem:

• This may not be related to the distribution of niches in the feature
space
Take the approach to the small disjuncts problem

Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

20. Online Identification of Small Disjuncts
We search for regions that promote

overgeneral classifiers
Estimate ircl based on the classifier’s

experience on each class:
exp max
ircl =
exp min
Adapt β and θGA according to ircl

ircl = 20 / 4
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

21. Online Parameter Adaptation
ir=256:1
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

22. What about UCS?
Supervised XCS:

Needs less exploration

Avoids XCS’s fitness dilemma

More robust to parameter settings

Overgeneral classifiers also tend to overcome the

population
Their probability of occurrence depends on the imbalance ratio

Partially minimized with fitness sharing

Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

23. What about UCS?
ir=256:1
ir=512:1
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

24. Are LCSs more error-prone to class imbalance
than other classifier schemes?
C4.5 SMO XCS
TP rate Bal2c1 0,00% ± 0,00% 0,00% ± 0,00% 0,00% ± 0,00%
Bal2c2 81,65% ± 6,83% 93,72% ± 4,64% 81,96% ± 6,00%
Bal2c3 81,90% ± 6,04% 93,77% ± 5,59% 83,99% ± 6,88%
bpa 42,95% ± 14,09% 0,00% ± 0,00% 61,38% ± 9,10%
gls2c1 80,00% ± 42,16% 0,00% ± 0,00% 50,00% ± 52,70%
gls2c2 35,00% ± 47,43% 15,00% ± 33,75% 55,00% ± 49,72%
gls2c3 30,00% ± 42,16% 0,00% ± 0,00% 5,00% ± 15,81%
gls2c4 75,00% ± 32,63% 81,67% ± 25,40% 81,67% ± 25,40%
gls2c5 77,14% ± 16,77% 10,00% ± 9,64% 84,29% ± 14,21%
gls2c6 59,82% ± 15,13% 0,00% ± 0,00% 81,79% ± 13,95%
h-s 75,83% ± 13,29% 80,00% ± 7,03% 80,00% ± 9,78%
pim 55,37% ± 13,27% 53,38% ± 6,42% 55,93% ± 9,75%
tao 95,23% ± 2,14% 84,11% ± 6,17% 92,58% ± 5,72%
thy2c1 90,00% ± 16,10% 76,67% ± 22,50% 90,00% ± 16,10%
thy2c2 94,17% ± 12,45% 54,17% ± 24,92% 90,83% ± 14,93%
thy2c3 90,95% ± 10,34% 33,81% ± 21,35% 90,71% ± 8,05%
wav2c1 75,74% ± 4,06% 88,51% ± 3,20% 87,24% ± 3,43%
wav2c2 72,34% ± 3,89% 84,57% ± 4,05% 78,72% ± 2,57%
wab2c3 77,64% ± 2,38% 89,97% ± 3,48% 87,86% ± 3,65%
wbdc 92,95% ± 3,42% 95,42% ± 5,36% 95,83% ± 5,89%
wdbc 92,47% ± 5,09% 94,81% ± 2,71% 93,83% ± 6,37%
wine2c1 89,00% ± 16,63% 100,00% ± 0,00% 100,00% ± 0,00%
wine2c2 95,00% ± 8,05% 98,33% ± 5,27% 98,33% ± 5,27%
wine2c3 90,18% ± 11,70% 97,14% ± 6,02% 98,57% ± 4,52%
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
wpbc 41,00% ± 12,87% 9,50% ± 17,07% 30,50% ± 24,99%

25. How can we Minimize the Effects of
Small Disjuncts?
Resampling the dataset:
 Addresses small
disjuncts
Classical methods:

• Random oversampling
• Random undersampling Assumes that
clusterization will
Heuristic methods:

find small
• Tomek links
disjuncts and
• CNN match classifier’s
• One-sided selection approximation
• Smote
Cluster-based oversampling
 Could XCS
benefit from the
online
Cost-sensitive classifiers

identification of
small disjuncts?
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

26. Domains of Applicability
Should we use some counterbalancing scheme?

Which learning scheme should we use?

Is there a combination of counterbalancing

scheme+learner that beats all others?
How can we know the presence of small

disjuncts?
Are there other complexity factors mixed up with

the small disjuncts problem?
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

27. Domains of Applicability
Resampling/
Learn it! Classifier/
Resampling+classifier
Where are
LCSs
placed?
Dataset Dataset
Suggested
Prediction
characterization
approach
Type of dataset:
Geometrical distribution of classes
Possible presence of small disjuncts
Other complexity factors
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

28. Future Directions
Potential benefit of XCS to discover small disjuncts

…and learn from it online
Further analyze UCS

How do LCSs perform w.r.t. other classifiers for unbalanced

datasets?
Measures for small disjuncts identification

… and other possible complexity factors
What is noise and what is a small disjunct?

In which cases a LCS is applicable?

Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla

29. Learning Classifier Systems
for Class Imbalance
Problems
Ester Bernadó-Mansilla
Research Group in Intelligent Systems
Enginyeria i Arquitectura La Salle
Universitat Ramon Llull
Barcelona, Spain