Reflections of a Keen Modeler

Some Reflections of a Keen Modeler David E. Goldberg Illinois Genetic Algorithms Laboratory University of Illinois at Urbana-Champaign Urbana, Illinois 61801 [email_address]

A Life in Models

What should I talk about?

Started engineering college in 1971.

Calculus was cool, and differential equations were amazing

Really dug making models with computers.

Bumped into fluid mechanics my junior year at Michigan.

Didn’t understand impact of that collision then.

Started to understand it when I bumped into GAs in 1980.

Bumped into more models as chief scientist of web startup.

Reflections on life in models.

Roadmap

What’s a model?

An economic model of models.

Models in GAs.

From GAs to PMBGAs/EDAs.

Models in PMBGA/EDAs.

Models in the education of the creative genetic algorithmist.

What is a Model?

A model is a system that represents one or more facets of some other system.

Typical model  facet combinations:

Drawing or solid model  geometry.

Prototype  geometry & operation.

Graph  variation of variable with independent variable (time, space, etc.).

Equilibrium equation  select state variables at steady state.

Dynamic equation  variation of select state variables with time.

Simulation  similar to equations, but uses an intermediate artifact to calculate..

Words and Models

Foregoing examples typical engineering models.

Language is also used in modeling.

Many first models are verbal.

Types of verbal models:

Single word or noun phrase.

Description of an object/process.

Feature list.

Dimension list.

Set of engineering specifications.

An Economy of Models

Have many models with different precision-accuracy and different costs.

Can we evaluate model usage rationally?

Consider economics of intellection and spectrum of models.

Fundamental Modeling Tradeoff

Error versus cost of modeling

ε , Error C, Cost of Modeling Engineer/Inventor Scientist/Mathematician

Marginal Analysis

Optimal thinking, when marginal cost of a thought equals marginal benefit to design.

 C =  B

If cost higher than advance in design, thinking is uneconomic.

Science: model is the end.

Engineering; model is instrumental.

Match model complexity to ends required.

A Spectrum of Models The Modeling Spectrum Low Cost/ High Error High Cost/ Low Error Unarticulated Wisdom Articulated Qualitative Model Dimensional Models Facetwise Models Equations of Motion

Disciplines: Mature & Immature

Mature disciplines are mature partially because they have developed sense of models.

GAs circa 1980: Immature discipline.

Few models available.

Hodgepodge of religious beliefs (xover v. mutation, etc.).

Some qualitative notions of what is important.

What Are Models Good For?

Many uses for models:

Description: describe the ways things are (were).

Prediction: describe the ways things will be.

Prescription: describe the way things should be.

Key variables: time and change.

Usually assumes have extant object to model.

Oftentimes assumes extant models.

Crossing Qual-Quant Divide

Needed quantitative understanding of qualitative theories/experience.

How do you cross the qual-quant divide?

Do not expect full equations of motion to provide answers.

Need little models of important tradeoffs.

Models in GAs

Our interest is in models that are

Principled

Quantitative

Not equations of motion

Call them little models , where “little” is a term of approbation.

Little models simplified, not universal.

Where do little models come from?

Sources of Little Models

Dimensional analysis (more in moment)

Construction of model for single facet.

Reduction of equation of motion for one or small number of facets.

Incorporation of qualitative reasoning.

Extremum principle.

Data usage (data-influenced), but empirical fit alone not enough.

Aside on Dimensional Analysis

Common in physics and engineering, but not CS.

Buckingham PI theorem: n dimensional parameters, m dimensions  n – m independent dimensionless parameters.

[ V ] = L/T, [ D ] = L, [ ν ] = L 2 /T n = 3, m = 2, n – m = 1 R = VD / ν , the Reynolds number

One difference: many CS parameters are pure numbers. Must derive time, spatial, or other scale, then renormalize.

Can play the same game for GAs, AI, & AL.

A LM of Selection Alone

Choose truncation selection because it is easy to analyze.

Truncation selection: make s copies each of top 1/ s th of the population.

Want to know time for good individual to “takeover” population: takeover time.

Takeover Time Under Truncation

Let P be proportion of best individuals.

How many more next generation? s

P ( t +1) = s P ( t ) until P ( t )=1.

P ( t ) = s t P (0).

Solve for takeover time t *: time to go from one good guy to all or all but one good guys.

t* = ln n/ ln s

So What?

Who cares about selection alone?

I want to analyze a “real GA.”

How can selection only analysis help me?

Answer: Imagine another characteristic time, the innovation time.

A LM of Innovation Time

Assume using crossover-based innovation with probability p c .

Assume that crossover event gives improvement with probability p i in population of size n.

Can calculate the expected time of arrival of next improvement.

The Innovation Time, t i

Define innovation time as the expected time to create an individual better than one so far.

t i = ( p c p i n ) -1

Model is facetwise, probabilistic & incomplete ( p i unknown).

p i estimated in Goldberg, Deb, & Thierens (1993) and Thierens & Goldberg (1993).

The Race: Integrating 2 Models

Have two facetwise models, but want integrated understanding.

Putting models in t terms gives us an idea.

Consider relative magnitudes of the two times: a dimensional argument.

Consider which is favorable to innovation: a qualitative argument.

Schematic of the Race

Dimensional & Qualitative Argument

This argument results in an integrated little model.

Define innovation number G = t* / t i

G = p c p i n ln n/ ln s.

Want takeover time greater than innovation time or G > 1.

Quantity like Reynolds number in fluids.

A Simple Control Map

Draw success region in GA parameter space.

Control map.

p c > c ( m, n ) ln s.

For p c versus ln s a straight line .

1993 Empirical Result

Easy problems are no problem.

GA has a large “sweet spot”.

A monkey can set cross probability & selection pressure

[Goldberg, Deb, & Theirens, 1993]

Lessons of LMing

Facetwise approach is fast.

Requires some skill in identifying correct facets and in integrating them.

Payoff in understanding is out of proportion to complexity of the effort.

Do not waste time on needless parametric study.

Verify expected behavior and move on.

Facetwise understanding improves pedagogy and ability to explain things simply.

Models in PMBGA/EDAs

Baluja & others recognized key thing that Holland had identified earlier.

Population + genetic operators  Probability distribution over possible structures.

Early PMBGA/EDAs used fixed models.

Later ones built models.

What is going on?

Primary Effect: Good Solutions Current population Selection New population Bayesian network

Secondary: Structural Knowledge

Mental models give us flexibility to think about world without paying big price.

Built models can be used to speed up evolution of good structures:

Parallelism

Time continuation

Hybridization

Evaluation relaxation

Usually think of them independently.

Supermultiplicative Speedup

Speed-Up: Ratio of # function evaluations without efficiency enhancement to that with it.

Only 1-15% individuals need evaluation

Speed-Up: 30–53

Have decision tree & ECGA versions.

Fitness modeling in BOA

Use of Structural/Fitness Surrogates

Parallelism: decomposition of problem for efficient parallelism.

Time continuation: understanding of problem sequencing and parameterization.

Hybridization: What neighborhood operators appropriate when & information for optimal division of labor.

Estimation of algorithm parameters.

Important Frontier PMBGA/EDAs

Extending the notion of model building.

Pervade every aspect of algorithm coordination.

Already movement in that direction.

More still needs to be done.

Education of Creative GAmist

Early 60s-80s pioneering years.

Years where categories genetic algorithms and evolutionary computation were created.

How do we rekindle that creativity moving forward?

How do we educate creative genetic algorithmists of the future?

Crossed qual-quant divide.

Need to go back to qual.

Key Distinction

Modeling of imagined or desired objects versus extant objects.

What can we draw on?

Existing objects that fail in some regard.

Similar or related objects.

Analogically related objects.

Creatively concocted objects.

Takes us back to the tabula rasa problem.

How do we even discuss that which does not exist?

Tabula Rasa: Curse & Blessing of Category Creator

How do we design when we don’t know how to talk about what we are designing?

Let’s start at the human beginnings of conceptual clarity.

Let’s start at the beginning of formal philosophy.

Let’s start with two key techniques from Athens.

What Examples of New Thought?

Clearest examples are from philosophy.

Presocratic  Socrates  Plato  Aristotle.

Mechanisms of the new thought:

Socratic dialectic

Aristotelian data mining

Socrates and Dialectic

Socrates was a pain in the neck.

Walked around Athens asking everyone impossible questions.

Then proved their answers were wrong, but rarely gave an answer himself.

Nonetheless, Socrates’s method was useful.

Conversation trying to probe what things really are (or might be).

Questions were the rights ones. Whitehead’s famous remark.

Socrates (470-399 BCE)

The Probing of Dialectic

Questions directed at the essence of things.

What is the meaning of a common phrase? “What is virtue?”

Answers often betray our lack of knowledge and understanding.

Examine answers critically, often with more questions.

Ask penetrating questions about the answers.

What’s This Got to Do with GAs?

Questions & conversation is at roots of new inventions.

Research on tech visionaries shows that problem finding is the main activity of successful TVs

Spark of insight may come as flash, but dialectic necessary in new product creation.

Three roles of questions:

Probe field needs.

Probe biases or politics of the field..

Probe developmental hurdles.

Tactics of Dialectic

Critical:

Equivocation: use of term in different senses.

Question begging: assuming the conclusion.

Infinite regress: infinite sequence implying incoherence.

Loss of contrast & emptiness: distinction with little or no difference.

Creative:

Definition: Seek essential distinctions.

Analogies: Certain dialectical similarities.

Thought experiments: Hypothetical w/ true premises that does not follow.

http://www.amazon.com/Practice-Philosophy-Handbook-Beginners-3rd/dp/0132308487

Aristotelian Data Mining

Called The Philosopher by some.

Amazing range and scope of work.

Created many of basic categories of college curriculum.

Founded a school the Lyceum.

We have 1/3 his output (2000 pages in 30 books).

Categories and Metaphysics.

Method very modern:

Empirical search for data.

Considered attributes, which he named.

Classified data according to his attributes.

Aristotle (384-322 BCE)

Analysis is More than Pretty Math

Sons & daughters of Newton spoiled by equations of motion.

Put too much faith in sets of equations.

Complexity demands heavy lifting of the facets for understanding.

Range of modeling skills, particularly qualitative reasoning.

Necessary for creative activity that is genetic and evolutionary computation.

Summary

Models generally:

Economy of models.

Modeling spectrum.

Models in GAs w/ emphasis on little models.

Models in PMBGA/EDAs.

Models in education of the creative genetic algorithmist.

Conclusions

Modeling is central to our business.

But simple view of modeling limits progress.

Need sophisticated, flexible approach to advance state of the art most quickly.

Still many opportunities for model advance in PMBGA/EDAs.

Mastery of the spectrum of models from qual to quant, from little to equations of motion holds hope for continued vibrancy and creativity of the field.

More Information