Building-Block Supply in Genetic Programming
K. Sastry1, U.-M. O’Reilly2, D. E. Goldberg1, D. Hill3
1Illinois
Genetic Algorithms Laboratory
Department of General Engineering
University of Illinois at Urbana-Champaign
2
Artificial Intelligence Lab
Massachusetts Institute of Technology
3Department of Civil and Environmental Engineering
University of Illinois at Urbana-Champaign
Supported by:
AFOSR F49620-00-0163, F49620-03-1-0129, NSF DMI-9908252, and CSE fellowship

Motivation
Genetic Programming (GP) Theory
Limited attention paid to model population sizing
Correct population sizing is critical to GP success
Valuable tool for GP-theorists and practitioners
Insight on GP mechanisms
Practical guide to set population size
To obtain good solutions in a robust manner
First step towards GP population-sizing
Address building-block supply
Population size that ensures presence of competing schemas

Outline
GA design decomposition
Extensions to GP domain
Building-block (BB) supply in GP
Modeling BB supply
Supply of a single BB
Supply of all competing schemata
Population sizing for BB supply
Future Work & Conclusions

GA Design Decomposition
Goldberg (1991), Goldberg, Deb, & Clark (1992),
Goldberg (2002)
1. Know what GAs are processing—building blocks
2. Know thy BB challengers—BB-wise difficult problems
3. Ensure an adequate supply of raw BBs
4. Ensure increased market share for superior BBs
5. Know BB takeover and convergence time
6. Make decisions well among competing BBs
7. Mix BBs well

Building Blocks
Minimal, sequential, and superior substructures
Holland(1975), Goldberg (2002)
Nearly decomposable problems
Context, representation, content, …
Hierarchical interaction

Problem Difficulty
What is a GP hard problem?
Koza (1992, 1994); Punch et al (1996); Soule et al
(1996); O’Reilly (1998); Daida et al (2001); Daida (2003);
GA hardness (Goldberg, 2002)
Inter-BB difficulty (deception)
Intra-BB difficulty (scaling)
Extra-BB difficulty (noise)
Key factors
Context, Content, and Structure, …
Algorithm enhancement and understanding
Test problems that thwart GP mechanism
Isolate facets of GP difficulty

Building-Block Supply
Population size that ensures the presence of
competing schema
Temporally through genetic operators
Spatially (Initial population)
Holland, 1975; Goldberg, 1989; Reeves, 1993; Goldberg,
Sastry, & Latoza, 2001;
Key factors
BB Structure
Context
Temporal supply
Goldberg, Sastry & Latoza, GECCO 2001

Building Block Growth
Schema theorem
Poli & Langdon (1997); Poli (2001); Poli & McPhee (2003)
Usually overemphasized
Satisfying schema theorem is easy
Goldberg & Sastry (2001)
BB mixing not guaranteed
Goldberg & Sastry, GECCO 2001

Understanding Time
Convergence and Takeover time in GAs
Goldberg & Deb, 1991; Mühlenbein & Schlierkamp-Voosen,
1993; Báck, 1994; Thierens & Goldberg, 1994; Miller &
Goldberg, 1996;
Understanding time in GP
Goldberg & O’Reilly (1998), O’Reilly & Goldberg (1998)
Expression matters
Key factors
Selection method
Varying size
Context …
Thierens & Goldberg, PPSN 1994; Bäck, ICEC 1994

Building-block Decision Making
Stochastic in nature (Holland, 1975)
2-armed bandit problem
De Jong (1975); Goldberg (1989); Goldberg & Rudnick (1992);
Goldberg, Deb & Clark (1992); Harik, Cantú-Paz, Goldberg, &
Miller (1997);
Usually dictates population sizing
Key GP-specific factors
Varying size
context, …
Harik, Cantu-Paz, Goldberg & Miller, ECJ, 7(3), 1999

Building-block Mixing
BB identification & mixing are difficult
Goldberg, Thierens & Deb (1993); Thierens & Goldberg (1993);
Thierens (1997, 1999); Sastry & Goldberg (1998);
Need operators that adapt linkage
GA designs that ensure BB mixing
Messy GA, fmGA, GemGA, LLGA, PMBGAs, …
Key GP-specific factors
ADFs, context, structure, …
Sastry & Goldberg, GECCO 2003.

GP-specific Design Decomposition Steps
GP Workshop:
Design of Reuse, Parameterized topologies (Koza, Streeter,
& Keane, 2003)
Structure and Content time scales (Ryan, 2003; Soule, 2003;
Howard, 2003)
???

Building-Block Supply In GP
Analyzing BB supply
Similar approach as Goldberg, Sastry, & Latoza (2001).
Selectorecombinative GP
Tree representation
Functions (arity = 2), and Terminals
Problem definition: Initial supply of BBs
Consider only spatial approach
At least one copy all competing schemas
Given tree sizes (or size distribution), and BB size

Translating GA Schemas to GP
Tree fragments (O’Reilly & Oppacher, 1995)
No positional anchor
Size, k; Functional nodes, Nf; Terminal nodes, Nt
F F F F F
F T
T T T F F F
T F
Counting tree fragments is not enough
How are tree fragments expressed?
Goldberg & O’Reilly, 1998; O’Reilly & Goldberg, 1998.
Fragments that express partially correct subfunctions

ORDER Expression Mechanism
Goldberg & O’Reilly, 1998
Has some GP properties, and easy to analyze
Primitive set:
Parse program tree inorder
Primitive first encountered is expressed.
Parsed primitives
Expressed primitives

Test Problems: UNITATION & DECEPTION
UNITATION:
# of expressed primitives, Xi
Similar to OneMax in GAs
DECEPTION:
Expressed primitives

Supply of A Single Building Block
Probability that a primitive, Xi, is expressed
Xi is present, and appears before its complement
Probability that a BB of size, k, is expressed
At least one copy of the BB is present

Supply of A Single Building Block
l = 32
At least one copy of the Building block: X1
Empirical Runs: UNITATION, k = 1
Population initialized with full trees
Average of 1000 independent runs

Supply of A Single Building Block
l = 32
l=4
UNITATION
At least one copy of the Building block:
Empirical Runs: DECEPTION, k = 4
Population initialized with full trees
Average of 1000 independent runs

Supply of Competing Schemas
Don’t know BBs apriori
Ensure supply of all competing schemas
At least one copy of competing schemas
Individual schema successes are independent
Can be modeled more accurately

Supply of All Competing Schemas
UNITATION
DECEPTION
At least one copy of competing schemas
UNITATION:
DECEPTION: 2k schemas

Population Sizing for BB Supply
Population size that ensures
At lease one copy of all competing schemas
Tolerate a failure probability, ε = 1/m
Big enough tree sizes: s À l
Same as for GA
Goldberg, Sastry, & Latoza, 2001

Supply Models for Tree Fragments
Single BB supply
Supply of all competing fragments
Population sizing

Future Work
Handle realistic GP expressions
Symbolic regression
Tree fragment expresses a correct subfunction
Other population-sizing facets
Decision making (Goldberg, Deb, & Clark, 1992)
Gambler’s ruin (Harik, Cantu-Paz, Goldberg, & Miller, 1997)
Mixing (Thierens & Goldberg, 1994; Sastry & Goldberg, 2003)
Temporal Supply of BBs
Via recombination and mutation
Automatically defined functions (ADFs)

Summary & Conclusions
Developed facetwise models for BB supply
At least one copy of a BB in the initial population
At least one copy of all schemas
Population sizing dictated by supply
ORDER expression mechanism
Larger population if good BBs are not expressed
Increasing tree size helps
Population sizing is identical to GAs
Average tree size is greater than the problem size
Understanding expression is important
Fragments express subfunctions that improve fitness