Scalability of Simple Genetic
Algorithms: Mixing Analysis of
One-Point Crossover
Kumara Sastry, and David E. Goldberg
{ksastry,deg}@uiuc.edu
Illinois Genetic Algorithms Laboratory
University of Illinois at Urbana-Champaign
Urbana, IL 61801
http://www-illigal.ge.uiuc.edu
ISCIS XVII
Seventeenth International Symposium
on Computer and Information Sciences
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 1
Foreword
• Big models
– Accurate, high cost and opaque
• No models
– High error, zero cost, and no guidance
• Little models: Facetwise and dimensional
– Important middle ground
– Quantitative guidance
– Intuition into physical mechanism
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 2
A Simple Genetic Algorithm
• Holland, 1962; Goldberg, 1989
• Principles of natural selection and genetics
• Evolve population of candidate solutions
• Three key operators
– Selection: Propagate good solutions
– Recombination: Combine solutions to create new
ones
– Mutation: Slightly modify candidate solutions
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 3
Genetic Algorithm Design Decomposition
• Goldberg (1991, 2002); Goldberg et al (1992)
1. Know what GAs process—building blocks (BBs).
2. Solve problems that are of bounded BB difficulty.
3. Ensure an adequate supply of raw BBs.
4. Ensure increased market share for superior BBs.
5. Know BB takeover and convergence times.
6. Ensure that BB decisions are well made.
7. Ensure a good mixing of BBs.
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 4
What is Building-Block Mixing?
• Innovation event (Goldberg et al, 1993)
• Combination of BBs
– From different parents to yield better offspring
• Crossover that leads to increase in BBs
Mixing Event
BB #1 BB #2 BB #3 BB #4 BB #1 BB #2 BB #3 BB #4
Parent 1 Child 1
Parent 2 Child 2
Not A Mixing Event
BB #1 BB #2 BB #3 BB #4
BB #1 BB #2 BB #3 BB #4
Parent 1 Child 1
Parent 2 Child 2
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 5
Why Analyze Building-Block Mixing?
• BB mixing:
– Critical for GA success.
– Often overlooked
∗ Success with standard recombination operators
– Need for problem-specific operators
• Facetwise models assume tight linkage
– Incorporate BB mixing into GA dynamics
• Development of mixing models
– Scalability of GAs
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 6
Overview
• Mixing problem
• Literature review
• Assumptions
• BB mixing: Worst case
• BB mixing: Best case
– Two building-block, and m building-block case
– Mixing-time model
– Population-size model
• Conclusions
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 7
Mixing Problem
• Goldberg, Thierens & Deb, 1993
• How well do fixed crossover operators solve
– GA-easy problems
– GA-hard problems
• Mixing Time:
– Individuals in the population have mc BBs
– Number of generations required to obtain individuals
with mc + 1 BBs
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 8
Literature Review
• Linkage equilibrium/distribution models
– Christiansen, 1989; Booker, 1993; Rabani et al, 1998;
Eshelman et al, 1989; Eshelman & Schaffer, 1995;
• Schemata evolution models
– Holland, 1975; De Jong, 1975; Syswerda, 1989; De Jong &
Spears (1990, 1991, 1992);
• All-in-one models
– Bridges & Goldberg, 1987; Vose & Liepens, 1991; Nix &
Vose, 1992; Pr¨gel-Bennett & Shapiro, 1994;
u
• BB mixing models
– Goldberg et al, 1993; Thierens & Goldberg, 1993; Thierens,
1995; Thierens, 1999
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 9
Assumptions
• Non-overlapping population of fixed size
• Generationwise selectorecombinative GAs
• Binary encoding and fixed string length
• Fully deceptive trap (Deb & Goldberg, 1993)
1 1 1 1 0 0 0 0 0 1 0 0 0 0 1 1 1 1 0 1
4 0 1 2 3
1.00
Fitness
0.75
0.50
0.25
0.00
0 1234
# of ones
+ + + + =
1.00 0.75 0.50 0.25 0.00 2.50
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 10
Building-Block Mixing: Worst Case
• Goldberg et al 1993; Thierens & Goldberg,
1993
• Uniform crossover
– One-point crossover on loosely-linked problems
• Mixing time:
2µk 2m
tx = c
npc m5/2
• Population size: Grows exponentially
2µk 2m
n ln n > c ln s
npc m5/2
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 11
Building-Block Mixing: Best Case
• Assumption of tight linkage
– Alleles belonging to a BB are close to one another.
• Mixing of two BBs
m BBs
• Mixing of
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 12
Building-Block Mixing: Two BBs
= 1)
• At most one BB per individual (mc
b# , and #b
• Two mixing events:
#b b#
• Probability of mixing
1 2k − 1 2k−1
1
=2· k · · ≈
pmix .
2k 2k − 1 2k − 1
2
• Mixing time
2k−1 (2k − 1)
1
= .
tx =
(n/2)pc pmix npc
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 13
Building-Block Mixing: Two BBs
• Convergence time (B¨ck, 1993)
a
cc √
k·m
tconv =
I
• Mixing time is greater than convergence time:
– Premature convergence
• Mixing time is less than convergence time:
√
I k
tx < tconv ⇒ n > cx 2 k
pc
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 14
Results: Two BB mixing
4
10
Theory
Experiment
3
10
Population size, n
2
10
1
10
0
10
2 3 4 5 6 7 8 9 10
Building block size, k
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 15
Building-Block Mixing: m BBs
• Two key empirical observations
• Ladder-climbing phenomenon
– Goldberg, Thierens & Deb (1993)
– Mixing level increases one step at a time
• Length-reduction phenomenon
– BBs at the ends get mixed often
– BBs at string ends converge faster
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 16
Ladder-Climbing Phenomenon
0.025
2 BBs
0.02
Proportion of individuals
0.015
3 BBs
4 BBs
0.01
0.005
0
0 2 4 6 8 10 12
Number of generations
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 17
Length-Reduction Phenomenon
0.9
0.8
Proportion of individuals
0.7
0.6
0.5
0.4
0.3
0.2
0.1
20
10
15
8
10
6
5 4
2
0
Number of generations
Building−block number
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 18
Ladder-Climbing + Length-Reduction
Overall effect
















































































































































































=


























+
















































































Ladder−climbing


























Length−reduction phenomenon







phenomenon


















































BBs covered


Converged BBs






BBs that cause mixing














• Bounding case
– Going from mixing level mc = 1 to mc = 2.
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 19
Proportion of Mixing
b##· · ·# b#· · ·##· · ·# b#· · ·#
Recombination ··· ···
#b#· · ·# ##· · ·b#· · ·# ##· · ·b
scenario
2(m − 1) ··· 2(m − i) ··· 2
No. of events
(i−1)k+1 (m−2)k+1
1
··· ···
Mixing Prob. mk−1 mk−1 mk−1
• Probability of mixing
(m − mc ) ((m − mc − 1)k + 3)
2
≈
pmix
3 (m − mc + 1) ((m − mc + 1)k − 1)
• Number of mixing events
1
nx ≈ cx1 · m
2
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 20
Time to Mix
• Time to climb one step of the ladder
2k m
2(m−1)k − 1 nx
≈ cx1 .
tx = −mk
2 (n/2)pc pmix npc
• Total mixing time
– Time to climb all the steps of the mixing ladder
2k m2
tx,tot = mtx = cx1
npc
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 21
Results: Total Mixing Time
90
80
70
60
x,tot
Total mixing time, t
50
40
30
20
10
0
0 10 20 30 40 50 60
Number of BBs, m
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 22
Population-Sizing Model
• Total mixing time
2k m2
tx,tot = mtx = cx1
npc
• Convergence time
cc √
k·m
tconv =
I
tx,tot < tconv
• Innovation success:
√
2k m m
I
√
n > cx1
pc k
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 23
Results: Population Sizing
4
10
3
10
Population size, n
2
10
Theory: k = 3
Experiment: k = 3
Theory: k = 4
Experiment: k = 4
Theory: k = 5
Experiment: k = 5
Theory: k = 6
Experiment: k = 6
1
10
10 20 30 40 50 60 70
Number of building blocks, m
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 24
Population-Sizing Bounds
• BB decision making + BB supply
– Goldberg et al, 1992; Harik et al, 1997;
√ k
n = cn 2 m
• BB mixing based model
√
2k m m
I
√
n = cx1
pc k
• BB decision making and supply governs
– Small problems
– problem size range increases with BB size
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 25
Population Sizing Phase Boundary
40
35
BB mixing governs
population sizing
Number of building blocks, m
(BB−mixing model)
30
25
BB decision + BB supply
20 governs population sizing
(Gambler’s ruin model)
15
10
5 10 15 20 25 30 35 40 45 50
Building block size, k
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 26
Conclusions
• BB mixing of one-point crossover
– Tight linkage:
2
km
, n = O 2k m1.5 .
tx,tot = O 2
n
– Loose linkage:
tx,tot = n = O 2µk 2m .
• Facetwise models are useful
– First preference in understanding complex systems
• Operators that adapt linkage are required
– Use competent genetic algorithms (Goldberg, 2002)
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL

Building-Block Mixing and One-Point Crossover 27
Acknowledgments
• Air Force Office of Scientific Research, Air Force
Materiel Command, USAF, under grant
F49620-00-1-0163.
• National Science Foundation under grant
DMI-9908252.
• Computational Science and Engineering (CSE)
fellowship, University of Illinois at Urbana-Champaign.
ISCIS XVII Illinois Genetic Algorithms Laboratory
Department of General Engineering
K. Sastry, D. E. Goldberg
Seventeenth International Symposium University of Illinois at Urbana−Champaign
on Computer and Information Sciences Urbana, IL 61801. USA.
http://www−illigal.ge.uiuc.edu
October 28−30, 2002, Orlando, FL