THE DESIGN OF INNOVATION: LESSONS FROM AND FOR COMPETENT GENETIC ALGORITHMS David E. Goldberg Department of General Engineering University of Illinois at Urbana-Champaign © 2002 David E. Goldberg, All Rights Reserved Unless otherwise noted, figures taken from The Design of Innovation: Lessons from and for Competent Genetic Algorithms , D.E. Goldberg, 2002. Reproduced with the permission of Kluwer Academic Publishers, Boston, MA. Innovation & GAs

Innovation This & Innovation That The business world is abuzz with “innovation.” Popular books tell companies how to get it. But little scientific understanding of what it is.

The business world is abuzz with “innovation.”

Popular books tell companies how to get it.

But little scientific understanding of what it is.

Genetic Algorithms are Coming of Age Genetic algorithms usage growing since 1985. Used to design airplanes, compute factory schedules, even compose art and music. GA millionaires have sold their companies. Yet, practice of GAs remains a black art. New problem requires new operators, codings, and tedious trial and error

Genetic algorithms usage growing since 1985.

Used to design airplanes, compute factory schedules, even compose art and music.

GA millionaires have sold their companies.

Yet, practice of GAs remains a black art.

New problem requires new operators, codings, and tedious trial and error

Connect the Dots Wouldn’t it be nice to understand innovation scientifically? GAs that solve problems once and for all? Yes. Yes. Affirmative answer to both questions found in these 8 lectures. Design of competent GAs, GAs that solve problems quickly, reliably, and accurately gives (1) scalable problem solver, & (2) computational theory of innovation.

Wouldn’t it be nice to understand innovation scientifically?

GAs that solve problems once and for all?

Yes. Yes. Affirmative answer to both questions found in these 8 lectures.

Design of competent GAs, GAs that solve problems quickly, reliably, and accurately gives (1) scalable problem solver, & (2) computational theory of innovation.

Overview One Minute Genetic Algorithmist. GAs and Innovation. A lesson from the Wright brothers. The goals of GA design. Seven-facet theory of competent GA design. A menagerie of competent GAs. Four facets of GA efficiency enhancement. A golden age of computational innovation?

One Minute Genetic Algorithmist.

GAs and Innovation.

A lesson from the Wright brothers.

The goals of GA design.

Seven-facet theory of competent GA design.

A menagerie of competent GAs.

Four facets of GA efficiency enhancement.

A golden age of computational innovation?

Who Am I? General Engineering, University of Illinois at Urbana-Champaign, 1990-2002. Engineering Mechanics, University of Alabama, Tuscaloosa, 1984-1990. BS, MS, PhD in Civil Engineering (Hydraulics), 1971-75, 1976, 1980-83. Project Engineer, Marketing Manager, Stoner Associates, Carlisle, PA, 1976-80. Married, father of two boys, live in Champaign, IL.

General Engineering, University of Illinois at Urbana-Champaign, 1990-2002.

Engineering Mechanics, University of Alabama, Tuscaloosa, 1984-1990.

BS, MS, PhD in Civil Engineering (Hydraulics), 1971-75, 1976, 1980-83.

Project Engineer, Marketing Manager, Stoner Associates, Carlisle, PA, 1976-80.

Married, father of two boys, live in Champaign, IL.

Course Text Goldberg, D. E. (2002). The design of innovation: Lessons from and for competent genetic algorithms. Boston, MA: Kluwer Academic Publishers. http://www-doi.ge.uiuc.edu/ May find it useful to refer to the text for details.

Goldberg, D. E. (2002). The design of innovation: Lessons from and for competent genetic algorithms. Boston, MA: Kluwer Academic Publishers.

http://www-doi.ge.uiuc.edu/

May find it useful to refer to the text for details.

Background GA Stuff Course is self-contained. Requires maturity of engineering, science, or math/CS BS degree. Some GA background helpful, but not necessary. Goldberg, D. E. (1989). Genetic algorithms in search, optimization, and machine learning. Reading, MA: Addison-Wesley. Many original papers downloadable from http:\www-illigal.ge.uiuc.edu www.amazon.com

Course is self-contained. Requires maturity of engineering, science, or math/CS BS degree.

Some GA background helpful, but not necessary.

Goldberg, D. E. (1989). Genetic algorithms in search, optimization, and machine learning. Reading, MA: Addison-Wesley.

Many original papers downloadable from http:\www-illigal.ge.uiuc.edu

Today’s Agenda One-minute genetic algorithmist. The innovation intuition. Intriguing applications.

One-minute genetic algorithmist.

The innovation intuition.

Intriguing applications.

The One-Minute Genetic Algorithmist What is a GA? Solutions as chromosomes. Means of evaluating fitness to purpose. Create initial population. Apply selection and genetic operators. Repeat until solution good enough (time runs out).

What is a GA?

Solutions as chromosomes.

Means of evaluating fitness to purpose.

Create initial population.

Apply selection and genetic operators.

Repeat until solution good enough (time runs out).

What is a Genetic Algorithm? A genetic algorithm is a search procedure based on the mechanics of natural selection and genetics. Require two things Survival-of-the-fittest Variation Started in the cybernetics movement of the 1950s/60s. Renewed vigor in the 1980s.

A genetic algorithm is a search procedure based on the mechanics of natural selection and genetics.

Require two things

Survival-of-the-fittest

Variation

Started in the cybernetics movement of the 1950s/60s. Renewed vigor in the 1980s.

Solutions as Chromosomes Code solutions as artificial chromosomes. Possible strings = possible solutions. Examples: Switches: Bit String: 10001 Tour 5 cities: Permutation: 1 2 4 5 3 Rules: If (P/E is high) then sell Program: (- x (* x (+ x 3))) Vector: (4.3 6.9 20.3e12) Manipulate code, not solutions, directly.

Code solutions as artificial chromosomes.

Possible strings = possible solutions.

Examples:

Switches: Bit String: 10001

Tour 5 cities: Permutation: 1 2 4 5 3

Rules: If (P/E is high) then sell

Program: (- x (* x (+ x 3)))

Vector: (4.3 6.9 20.3e12)

Manipulate code, not solutions, directly.

Determining Fitness Something must decide good from bad. Possibilities: Objective function evaluates with computer Human makes decision (subjective function) Co-evolved against predators and prey Specify what wanted, not how obtained.

Something must decide good from bad.

Possibilities:

Objective function evaluates with computer

Human makes decision (subjective function)

Co-evolved against predators and prey

Specify what wanted, not how obtained.

Operators Selection Recombination Mutation

Selection

Recombination

Mutation

Selection Darwinian survival of the fittest. Give more copies to better guys. Ways to do: roulette wheel tournament truncation By itself, pick best.

Darwinian survival of the fittest.

Give more copies to better guys.

Ways to do:

roulette wheel

tournament

truncation

By itself, pick best.

Recombination Combine bits and pieces of good parents. Speculate on, new, possibly better children. By itself, a random shuffle Example, one-point X: Before X After X 11111 11000 00000 00111

Combine bits and pieces of good parents.

Speculate on, new, possibly better children.

By itself, a random shuffle

Example, one-point X:

Mutation Mutation is a random alteration of a string. Change a gene, small movement in the neighborhood. By itself, a random walk. Example Before M After M 11111 11011

Mutation is a random alteration of a string.

Change a gene, small movement in the neighborhood.

By itself, a random walk.

Example

GAs & Innovation: The Fundamental Intuition How do individually uninteresting operators yield interesting behavior? Genetic algorithm power like that of human innovation. Separate selection+mutation from selection+recombination. Different modes or facets of innovation.

How do individually uninteresting operators yield interesting behavior?

Genetic algorithm power like that of human innovation.

Separate selection+mutation from selection+recombination.

Different modes or facets of innovation.

Selection+Mutation=Improvement Total Quality Management: continual improvement; Japanese term: kaizen. Mutation makes local changes; selection accepts the better ones. A resilient and general form of hillclimbing. “ Invention is nothing more than a fine deviation from, or enlargement on, a fine model.” E. Bulwer-Lytton

Total Quality Management: continual improvement; Japanese term: kaizen.

Mutation makes local changes; selection accepts the better ones.

A resilient and general form of hillclimbing.

“ Invention is nothing more than a fine deviation from, or enlargement on, a fine model.” E. Bulwer-Lytton

Selection+Recombination=Innovation Combine notions to form ideas. “ Indeed, it is obvious that invention or discovery, be it in mathematics or anywhere else, takes place by combining ideas.” J. Hadamard “ It takes two to invent anything. The one makes up combinations; the other chooses, recognizes what he wishes and what is important to him in the mass of the things which the former has imparted to him.” P. Valéry

Combine notions to form ideas.

“ Indeed, it is obvious that invention or discovery, be it in mathematics or anywhere else, takes place by combining ideas.” J. Hadamard

“ It takes two to invent anything. The one makes up combinations; the other chooses, recognizes what he wishes and what is important to him in the mass of the things which the former has imparted to him.” P. Valéry

Two Facets of Innovation Selection+mutation = hillclimbing or improvement. Selection+crossover = cross-fertilizing innovation or simply innovation. Both can be useful, especially if done right. GAs as 2-edged sword: Technology: GAs as useful. Science: GAs as model of innovative cognitive processes.

Selection+mutation = hillclimbing or improvement.

Selection+crossover = cross-fertilizing innovation or simply innovation.

Both can be useful, especially if done right.

GAs as 2-edged sword:

Technology: GAs as useful.

Science: GAs as model of innovative cognitive processes.

4 Intriguing Applications GE designs a Boeing 777 jet engine. John Deere schedules a factory. First Quadrant manages portfolio in the financial markets. NMSU catches criminals.

GE designs a Boeing 777 jet engine.

John Deere schedules a factory.

First Quadrant manages portfolio in the financial markets.

NMSU catches criminals.

GE Designs a Jet Engine GA work started in late 80s. Found 2% efficiency increase w/ GA + expert system hybrid. Used in 777 design. GE spins off company, Engineous Systems.

GA work started in late 80s.

Found 2% efficiency increase w/ GA + expert system hybrid.

Used in 777 design.

GE spins off company, Engineous Systems.

A GA Schedules for John Deere Optimax created scheduling program. Schedules planter line and 5 others. Optimax bought by i2 Technologies for $60m. GA technology now integrated into i2 software.

Optimax created scheduling program.

Schedules planter line and 5 others.

Optimax bought by i2 Technologies for $60m.

GA technology now integrated into i2 software.

Portfolio Management with GA First Quadrant manages $28B with daily help from a GA. Chromosomes in financial applications can be decision rules, prediction rules, or portfolio decisions.

First Quadrant manages $28B with daily help from a GA.

Chromosomes in financial applications can be decision rules, prediction rules, or portfolio decisions.

NMSU, the Beauty & the Beast GA replaces criminal sketch artist. NMSU system called Faceprints. Work has continued on sociobiology of beauty. Face at left composite of web votes + GA. http://www-psych.nmsu.edu/~vic/faceprints/female_study.html

GA replaces criminal sketch artist.

NMSU system called Faceprints.

Work has continued on sociobiology of beauty.

Face at left composite of web votes + GA.

GAs Have Special Kind of Appeal Going boldly where optimization has and has not gone. From engineering design, to scheduling production, to financial decision making, to catching criminals and making art. GAs increasingly used across spectrum of human endeavor. Seem special, but is that special nature due to connection with innovation?

Going boldly where optimization has and has not gone.

From engineering design, to scheduling production, to financial decision making, to catching criminals and making art.

GAs increasingly used across spectrum of human endeavor.

Seem special, but is that special nature due to connection with innovation?

Summary One-minute genetic algorithmist. The paradox of uninteresting operators. The fundamental intuition of GAs: the innovation intuition. Two facets of innovation: kaizen or continual improvement cross-fertilizing or selectorecombinative innovation. Breadth of application.

One-minute genetic algorithmist.

The paradox of uninteresting operators.

The fundamental intuition of GAs: the innovation intuition.

Two facets of innovation:

kaizen or continual improvement

cross-fertilizing or selectorecombinative innovation.

Breadth of application.

Conclusions Innovation is important, but mysterious. GAs are intriguing, but first-generation GAs unreliable. Intuitively, GAs and innovation seem connected. Can we design GAs that scale well on hard problems? Maybe we’ll learn something about innovation along the way. Stay tuned.

Innovation is important, but mysterious.

GAs are intriguing, but first-generation GAs unreliable.

Intuitively, GAs and innovation seem connected.

Can we design GAs that scale well on hard problems?

Maybe we’ll learn something about innovation along the way. Stay tuned.

THE DESIGN OF INNOVATION: LESSONS FROM AND FOR COMPETENT GENETIC ALGORITHMS Produced by The Office of Continuing Engineering Education University of Illinois at Urbana-Champaign