Reflections of a Keen Modeler

2,217 views

Published on

OBUPM-2007 presentation by David E. Goldberg

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,217
On SlideShare
0
From Embeds
0
Number of Embeds
34
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Reflections of a Keen Modeler

  1. 1. Some Reflections of a Keen Modeler David E. Goldberg Illinois Genetic Algorithms Laboratory University of Illinois at Urbana-Champaign Urbana, Illinois 61801 [email_address]
  2. 2. A Life in Models <ul><li>What should I talk about? </li></ul><ul><li>Started engineering college in 1971. </li></ul><ul><li>Calculus was cool, and differential equations were amazing </li></ul><ul><li>Really dug making models with computers. </li></ul><ul><li>Bumped into fluid mechanics my junior year at Michigan. </li></ul><ul><li>Didn’t understand impact of that collision then. </li></ul><ul><li>Started to understand it when I bumped into GAs in 1980. </li></ul><ul><li>Bumped into more models as chief scientist of web startup. </li></ul><ul><li>Reflections on life in models. </li></ul>
  3. 3. Roadmap <ul><li>What’s a model? </li></ul><ul><li>An economic model of models. </li></ul><ul><li>Models in GAs. </li></ul><ul><li>From GAs to PMBGAs/EDAs. </li></ul><ul><li>Models in PMBGA/EDAs. </li></ul><ul><li>Models in the education of the creative genetic algorithmist. </li></ul>
  4. 4. What is a Model? <ul><li>A model is a system that represents one or more facets of some other system. </li></ul><ul><li>Typical model  facet combinations: </li></ul><ul><ul><li>Drawing or solid model  geometry. </li></ul></ul><ul><ul><li>Prototype  geometry & operation. </li></ul></ul><ul><ul><li>Graph  variation of variable with independent variable (time, space, etc.). </li></ul></ul><ul><ul><li>Equilibrium equation  select state variables at steady state. </li></ul></ul><ul><ul><li>Dynamic equation  variation of select state variables with time. </li></ul></ul><ul><ul><li>Simulation  similar to equations, but uses an intermediate artifact to calculate.. </li></ul></ul>
  5. 5. Words and Models <ul><li>Foregoing examples typical engineering models. </li></ul><ul><li>Language is also used in modeling. </li></ul><ul><li>Many first models are verbal. </li></ul><ul><li>Types of verbal models: </li></ul><ul><ul><li>Single word or noun phrase. </li></ul></ul><ul><ul><li>Description of an object/process. </li></ul></ul><ul><ul><li>Feature list. </li></ul></ul><ul><ul><li>Dimension list. </li></ul></ul><ul><ul><li>Set of engineering specifications. </li></ul></ul>
  6. 6. An Economy of Models <ul><li>Have many models with different precision-accuracy and different costs. </li></ul><ul><li>Can we evaluate model usage rationally? </li></ul><ul><li>Consider economics of intellection and spectrum of models. </li></ul>
  7. 7. Fundamental Modeling Tradeoff <ul><li>Error versus cost of modeling </li></ul>ε , Error C, Cost of Modeling Engineer/Inventor Scientist/Mathematician
  8. 8. Marginal Analysis <ul><li>Optimal thinking, when marginal cost of a thought equals marginal benefit to design. </li></ul><ul><li> C =  B </li></ul><ul><li>If cost higher than advance in design, thinking is uneconomic. </li></ul><ul><li>Science: model is the end. </li></ul><ul><li>Engineering; model is instrumental. </li></ul><ul><li>Match model complexity to ends required. </li></ul>
  9. 9. A Spectrum of Models The Modeling Spectrum Low Cost/ High Error High Cost/ Low Error Unarticulated Wisdom Articulated Qualitative Model Dimensional Models Facetwise Models Equations of Motion
  10. 10. Disciplines: Mature & Immature <ul><li>Mature disciplines are mature partially because they have developed sense of models. </li></ul><ul><li>GAs circa 1980: Immature discipline. </li></ul><ul><li>Few models available. </li></ul><ul><li>Hodgepodge of religious beliefs (xover v. mutation, etc.). </li></ul><ul><li>Some qualitative notions of what is important. </li></ul>
  11. 11. What Are Models Good For? <ul><li>Many uses for models: </li></ul><ul><ul><li>Description: describe the ways things are (were). </li></ul></ul><ul><ul><li>Prediction: describe the ways things will be. </li></ul></ul><ul><ul><li>Prescription: describe the way things should be. </li></ul></ul><ul><li>Key variables: time and change. </li></ul><ul><li>Usually assumes have extant object to model. </li></ul><ul><li>Oftentimes assumes extant models. </li></ul>
  12. 12. Crossing Qual-Quant Divide <ul><li>Needed quantitative understanding of qualitative theories/experience. </li></ul><ul><li>How do you cross the qual-quant divide? </li></ul><ul><li>Do not expect full equations of motion to provide answers. </li></ul><ul><li>Need little models of important tradeoffs. </li></ul>
  13. 13. Models in GAs <ul><li>Our interest is in models that are </li></ul><ul><ul><li>Principled </li></ul></ul><ul><ul><li>Quantitative </li></ul></ul><ul><ul><li>Not equations of motion </li></ul></ul><ul><li>Call them little models , where “little” is a term of approbation. </li></ul><ul><li>Little models simplified, not universal. </li></ul><ul><li>Where do little models come from? </li></ul>
  14. 14. Sources of Little Models <ul><li>Dimensional analysis (more in moment) </li></ul><ul><li>Construction of model for single facet. </li></ul><ul><li>Reduction of equation of motion for one or small number of facets. </li></ul><ul><li>Incorporation of qualitative reasoning. </li></ul><ul><li>Extremum principle. </li></ul><ul><li>Data usage (data-influenced), but empirical fit alone not enough. </li></ul>
  15. 15. Aside on Dimensional Analysis <ul><li>Common in physics and engineering, but not CS. </li></ul><ul><li>Buckingham PI theorem: n dimensional parameters, m dimensions  n – m independent dimensionless parameters. </li></ul><ul><li>[ V ] = L/T, [ D ] = L, [ ν ] = L 2 /T n = 3, m = 2, n – m = 1 R = VD / ν , the Reynolds number </li></ul><ul><li>One difference: many CS parameters are pure numbers. Must derive time, spatial, or other scale, then renormalize. </li></ul><ul><li>Can play the same game for GAs, AI, & AL. </li></ul>
  16. 16. A LM of Selection Alone <ul><li>Choose truncation selection because it is easy to analyze. </li></ul><ul><li>Truncation selection: make s copies each of top 1/ s th of the population. </li></ul><ul><li>Want to know time for good individual to “takeover” population: takeover time. </li></ul>
  17. 17. Takeover Time Under Truncation <ul><li>Let P be proportion of best individuals. </li></ul><ul><li>How many more next generation? s </li></ul><ul><li>P ( t +1) = s P ( t ) until P ( t )=1. </li></ul><ul><li>P ( t ) = s t P (0). </li></ul><ul><li>Solve for takeover time t *: time to go from one good guy to all or all but one good guys. </li></ul><ul><li>t* = ln n/ ln s </li></ul>
  18. 18. So What? <ul><li>Who cares about selection alone? </li></ul><ul><li>I want to analyze a “real GA.” </li></ul><ul><li>How can selection only analysis help me? </li></ul><ul><li>Answer: Imagine another characteristic time, the innovation time. </li></ul>
  19. 19. A LM of Innovation Time <ul><li>Assume using crossover-based innovation with probability p c . </li></ul><ul><li>Assume that crossover event gives improvement with probability p i in population of size n. </li></ul><ul><li>Can calculate the expected time of arrival of next improvement. </li></ul>
  20. 20. The Innovation Time, t i <ul><li>Define innovation time as the expected time to create an individual better than one so far. </li></ul><ul><li>t i = ( p c p i n ) -1 </li></ul><ul><li>Model is facetwise, probabilistic & incomplete ( p i unknown). </li></ul><ul><li>p i estimated in Goldberg, Deb, & Thierens (1993) and Thierens & Goldberg (1993). </li></ul>
  21. 21. The Race: Integrating 2 Models <ul><li>Have two facetwise models, but want integrated understanding. </li></ul><ul><li>Putting models in t terms gives us an idea. </li></ul><ul><li>Consider relative magnitudes of the two times: a dimensional argument. </li></ul><ul><li>Consider which is favorable to innovation: a qualitative argument. </li></ul>
  22. 22. Schematic of the Race
  23. 23. Dimensional & Qualitative Argument <ul><li>This argument results in an integrated little model. </li></ul><ul><li>Define innovation number G = t* / t i </li></ul><ul><li>G = p c p i n ln n/ ln s. </li></ul><ul><li>Want takeover time greater than innovation time or G > 1. </li></ul><ul><li>Quantity like Reynolds number in fluids. </li></ul>
  24. 24. A Simple Control Map <ul><li>Draw success region in GA parameter space. </li></ul><ul><li>Control map. </li></ul><ul><li>p c > c ( m, n ) ln s. </li></ul><ul><li>For p c versus ln s a straight line . </li></ul>
  25. 25. 1993 Empirical Result <ul><li>Easy problems are no problem. </li></ul><ul><li>GA has a large “sweet spot”. </li></ul><ul><li>A monkey can set cross probability & selection pressure </li></ul>[Goldberg, Deb, & Theirens, 1993]
  26. 26. Lessons of LMing <ul><li>Facetwise approach is fast. </li></ul><ul><li>Requires some skill in identifying correct facets and in integrating them. </li></ul><ul><li>Payoff in understanding is out of proportion to complexity of the effort. </li></ul><ul><li>Do not waste time on needless parametric study. </li></ul><ul><li>Verify expected behavior and move on. </li></ul><ul><li>Facetwise understanding improves pedagogy and ability to explain things simply. </li></ul>
  27. 27. Models in PMBGA/EDAs <ul><li>Baluja & others recognized key thing that Holland had identified earlier. </li></ul><ul><li>Population + genetic operators  Probability distribution over possible structures. </li></ul><ul><li>Early PMBGA/EDAs used fixed models. </li></ul><ul><li>Later ones built models. </li></ul><ul><li>What is going on? </li></ul>
  28. 28. Primary Effect: Good Solutions Current population Selection New population Bayesian network
  29. 29. Secondary: Structural Knowledge <ul><li>Mental models give us flexibility to think about world without paying big price. </li></ul><ul><li>Built models can be used to speed up evolution of good structures: </li></ul><ul><ul><li>Parallelism </li></ul></ul><ul><ul><li>Time continuation </li></ul></ul><ul><ul><li>Hybridization </li></ul></ul><ul><ul><li>Evaluation relaxation </li></ul></ul><ul><li>Usually think of them independently. </li></ul>
  30. 30. Supermultiplicative Speedup <ul><li>Speed-Up: Ratio of # function evaluations without efficiency enhancement to that with it. </li></ul><ul><li>Only 1-15% individuals need evaluation </li></ul><ul><li>Speed-Up: 30–53 </li></ul><ul><li>Have decision tree & ECGA versions. </li></ul>Fitness modeling in BOA
  31. 31. Use of Structural/Fitness Surrogates <ul><li>Parallelism: decomposition of problem for efficient parallelism. </li></ul><ul><li>Time continuation: understanding of problem sequencing and parameterization. </li></ul><ul><li>Hybridization: What neighborhood operators appropriate when & information for optimal division of labor. </li></ul><ul><li>Estimation of algorithm parameters. </li></ul>
  32. 32. Important Frontier PMBGA/EDAs <ul><li>Extending the notion of model building. </li></ul><ul><li>Pervade every aspect of algorithm coordination. </li></ul><ul><li>Already movement in that direction. </li></ul><ul><li>More still needs to be done. </li></ul>
  33. 33. Education of Creative GAmist <ul><li>Early 60s-80s pioneering years. </li></ul><ul><li>Years where categories genetic algorithms and evolutionary computation were created. </li></ul><ul><li>How do we rekindle that creativity moving forward? </li></ul><ul><li>How do we educate creative genetic algorithmists of the future? </li></ul><ul><li>Crossed qual-quant divide. </li></ul><ul><li>Need to go back to qual. </li></ul>
  34. 34. Key Distinction <ul><li>Modeling of imagined or desired objects versus extant objects. </li></ul><ul><li>What can we draw on? </li></ul><ul><ul><li>Existing objects that fail in some regard. </li></ul></ul><ul><ul><li>Similar or related objects. </li></ul></ul><ul><ul><li>Analogically related objects. </li></ul></ul><ul><ul><li>Creatively concocted objects. </li></ul></ul><ul><li>Takes us back to the tabula rasa problem. </li></ul><ul><li>How do we even discuss that which does not exist? </li></ul>
  35. 35. Tabula Rasa: Curse & Blessing of Category Creator <ul><li>How do we design when we don’t know how to talk about what we are designing? </li></ul><ul><li>Let’s start at the human beginnings of conceptual clarity. </li></ul><ul><li>Let’s start at the beginning of formal philosophy. </li></ul><ul><li>Let’s start with two key techniques from Athens. </li></ul>
  36. 36. What Examples of New Thought? <ul><li>Clearest examples are from philosophy. </li></ul><ul><li>Presocratic  Socrates  Plato  Aristotle. </li></ul><ul><li>Mechanisms of the new thought: </li></ul><ul><ul><li>Socratic dialectic </li></ul></ul><ul><ul><li>Aristotelian data mining </li></ul></ul>
  37. 37. Socrates and Dialectic <ul><li>Socrates was a pain in the neck. </li></ul><ul><li>Walked around Athens asking everyone impossible questions. </li></ul><ul><li>Then proved their answers were wrong, but rarely gave an answer himself. </li></ul><ul><li>Nonetheless, Socrates’s method was useful. </li></ul><ul><li>Conversation trying to probe what things really are (or might be). </li></ul><ul><li>Questions were the rights ones. Whitehead’s famous remark. </li></ul>Socrates (470-399 BCE)
  38. 38. The Probing of Dialectic <ul><li>Questions directed at the essence of things. </li></ul><ul><li>What is the meaning of a common phrase? “What is virtue?” </li></ul><ul><li>Answers often betray our lack of knowledge and understanding. </li></ul><ul><li>Examine answers critically, often with more questions. </li></ul><ul><li>Ask penetrating questions about the answers. </li></ul>
  39. 39. What’s This Got to Do with GAs? <ul><li>Questions & conversation is at roots of new inventions. </li></ul><ul><li>Research on tech visionaries shows that problem finding is the main activity of successful TVs </li></ul><ul><li>Spark of insight may come as flash, but dialectic necessary in new product creation. </li></ul><ul><li>Three roles of questions: </li></ul><ul><ul><li>Probe field needs. </li></ul></ul><ul><ul><li>Probe biases or politics of the field.. </li></ul></ul><ul><ul><li>Probe developmental hurdles. </li></ul></ul>
  40. 40. Tactics of Dialectic <ul><li>Critical: </li></ul><ul><ul><li>Equivocation: use of term in different senses. </li></ul></ul><ul><ul><li>Question begging: assuming the conclusion. </li></ul></ul><ul><ul><li>Infinite regress: infinite sequence implying incoherence. </li></ul></ul><ul><ul><li>Loss of contrast & emptiness: distinction with little or no difference. </li></ul></ul><ul><li>Creative: </li></ul><ul><ul><li>Definition: Seek essential distinctions. </li></ul></ul><ul><ul><li>Analogies: Certain dialectical similarities. </li></ul></ul><ul><ul><li>Thought experiments: Hypothetical w/ true premises that does not follow. </li></ul></ul>http://www.amazon.com/Practice-Philosophy-Handbook-Beginners-3rd/dp/0132308487
  41. 41. Aristotelian Data Mining <ul><li>Called The Philosopher by some. </li></ul><ul><li>Amazing range and scope of work. </li></ul><ul><li>Created many of basic categories of college curriculum. </li></ul><ul><li>Founded a school the Lyceum. </li></ul><ul><li>We have 1/3 his output (2000 pages in 30 books). </li></ul><ul><li>Categories and Metaphysics. </li></ul><ul><li>Method very modern: </li></ul><ul><ul><li>Empirical search for data. </li></ul></ul><ul><ul><li>Considered attributes, which he named. </li></ul></ul><ul><ul><li>Classified data according to his attributes. </li></ul></ul>Aristotle (384-322 BCE)
  42. 42. Analysis is More than Pretty Math <ul><li>Sons & daughters of Newton spoiled by equations of motion. </li></ul><ul><li>Put too much faith in sets of equations. </li></ul><ul><li>Complexity demands heavy lifting of the facets for understanding. </li></ul><ul><li>Range of modeling skills, particularly qualitative reasoning. </li></ul><ul><li>Necessary for creative activity that is genetic and evolutionary computation. </li></ul>
  43. 43. Summary <ul><li>Models generally: </li></ul><ul><ul><li>Economy of models. </li></ul></ul><ul><ul><li>Modeling spectrum. </li></ul></ul><ul><li>Models in GAs w/ emphasis on little models. </li></ul><ul><li>Models in PMBGA/EDAs. </li></ul><ul><li>Models in education of the creative genetic algorithmist. </li></ul>
  44. 44. Conclusions <ul><li>Modeling is central to our business. </li></ul><ul><li>But simple view of modeling limits progress. </li></ul><ul><li>Need sophisticated, flexible approach to advance state of the art most quickly. </li></ul><ul><li>Still many opportunities for model advance in PMBGA/EDAs. </li></ul><ul><li>Mastery of the spectrum of models from qual to quant, from little to equations of motion holds hope for continued vibrancy and creativity of the field. </li></ul>
  45. 45. More Information <ul><li>DoI, the book </li></ul><ul><li>TEE, the book. http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470007230.html </li></ul><ul><li>TEE, the blog. www.entrepreneurialengineer.blogspot.com </li></ul><ul><li>TEE, the course. http://online.engr.uiuc.edu/webcourses/ge498tee/index.html </li></ul><ul><li>MTV, the course. http://online.engr.uiuc.edu/webcourses/ge498tv/index.html </li></ul><ul><li>GAs, the course http://online.engr.uiuc.edu/webcourses/ge531/index.html </li></ul><ul><li>2007 Workshop on Philosophy & Engineering (WPE) http://www- illigal.ge.uiuc.edu/wpe </li></ul><ul><li>Illinois Genetic Algorithms Lab http://www-illigal.ge.uiuc.edu/ </li></ul>

×