Tightness of parallel  complexity bounds: the log()  correction and automatic  parallelization.     F. Teytaud, O. Teytau...
Outline   Introduction   Complexity bounds   Branching Factor   Automatic Parallelization   Real-world algorithms   Log()...
Introduction: I like  large   Grid5000 = 5 000 cores (increasing)   Submitting jobs ==> grouping runs    ==>  much bigge...
Introduction: I like  large   Grid5000 = 5 000 cores (increasing)   Submitting jobs ==> grouping runs    ==>  much bigge...
Introduction: I like  large   Grid5000 = 5 000 cores (increasing)   Submitting jobs ==> grouping runs    ==>  much bigge...
Introduction: I like  large   Grid5000 = 5 000 cores (increasing)   Submitting jobs ==> grouping runs    ==>  much bigge...
Outline   Introduction   Complexity bounds   Branching Factor   Automatic Parallelization   Real-world algorithms   Log()...
Complexity bounds               = nb of fitness evaluations for precision                   with probability at least ½  ...
Complexity bounds on the convergence ratio    FR: full ranking (selected points are ranked)    SB: selection-based (select...
Outline   Introduction   Complexity bounds   Branching Factor   Automatic Parallelization   Real-world algorithms   Log()...
Branching factor K (more in Gelly06; Fournier08)Rewrite your evolutionary algorithm as follows:g has values in a finite se...
Outline   Introduction   Complexity bounds   Branching Factor   Automatic Parallelization   Real-world algorithms   Log()...
Automatic parallelization  Teytaud and Teytaud       TRSH 09 is great   13
Automatic parallelization with branching factor 3                 Consider the sequential algorithm.                 (iter...
Automatic parallelization with branching factor 3     Consider the sequential algorithm.     (iteration 2)  Teytaud and Te...
Automatic parallelization with branching factor 3       Consider the sequential algorithm.       (iteration 3)  Teytaud an...
Automatic parallelization with branching factor 3 Parallel version for D=2. Population = union of all pops for 2 iteration...
Outline   Introduction   Complexity bounds   Branching Factor   Automatic Parallelization   Real-world algorithms   Log()...
Real world algorithms Define: Necessary condition for log() speed-up:  - E log( * ) ~ log()  But for many algorithms,  ...
One-fifth rule: E log( * ) = O(1)  Consider e.g.  Or consider e.g.                  In both cases * is lower-bounded    ...
Self-adaptation, cumulative step-size adaptationIn both case, the same result: with parametersdepending on the dimension o...
Outline   Introduction   Complexity bounds   Branching Factor   Automatic Parallelization   Real-world algorithms   Log()...
The starting point of this work  Many algorithms have parameters defined      by handcrafted rules,  Fournier08 shows rate...
Log() corrections  We can change that:  In the discrete case (XPs): automatic         parallelization surprisingly effici...
Conclusion The case of large population size is not well handled by usual algorithms. We proposed      (I) theoretical gua...
Main limitation All this is about a logarithmic speed-up. The computational power is like this ==>                  <== an...
Further work Apply VC-bounds for considering only “reasonnable” branches in the automatic parallelization. Theoretically e...
Upcoming SlideShare
Loading in …5
×

Complexity bounds in parallel optimization

250 views

Published on

@inproceedings{teytaud:inria-00451416,
hal_id = {inria-00451416},
url = {http://hal.inria.fr/inria-00451416},
title = {{Bias and variance in continuous EDA}},
author = {Teytaud, Fabien and Teytaud, Olivier},
abstract = {{Estimation of Distribution Algorithms are based on statistical estimates. We show that when combining classical tools from statistics, namely bias/variance decomposition, reweighting and quasi-randomization, we can strongly improve the convergence rate. All modifications are easy, compliant with most algorithms, and experimentally very efficient in particular in the parallel case (large offsprings).}},
language = {Anglais},
affiliation = {TAO - INRIA Futurs , Laboratoire de Recherche en Informatique - LRI , TAO - INRIA Saclay - Ile de France},
booktitle = {{EA 09}},
address = {Strasbourg, France},
audience = {internationale },
year = {2009},
month = May,
pdf = {http://hal.inria.fr/inria-00451416/PDF/decsigma.pdf},
}

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Complexity bounds in parallel optimization

  1. 1. Tightness of parallel  complexity bounds: the log()  correction and automatic  parallelization. F. Teytaud, O. Teytaud Birmingham, 2009Tao, Inria Saclay Ile-De-France,LRI (Université Paris Sud, France),UMR CNRS 8623, I&A team, Digiteo,Pascal Network of Excellence.
  2. 2. Outline Introduction Complexity bounds Branching Factor Automatic Parallelization Real-world algorithms Log() corrections Teytaud and Teytaud TRSH 09 is great 2
  3. 3. Introduction: I like  large Grid5000 = 5 000 cores (increasing) Submitting jobs ==> grouping runs ==>  much bigger than number of cores. Next generations of computers: tenths, hundreds, thousands of cores. Evolutionary algorithms are population based but they have a bad speed-up. Teytaud and Teytaud TRSH 09 is great 3
  4. 4. Introduction: I like  large Grid5000 = 5 000 cores (increasing) Submitting jobs ==> grouping runs ==>  much bigger than number of cores. Next generations of computers: tenths, hundreds, thousands of cores. Evolutionary algorithms are population based but they have a bad speed-up. Teytaud and Teytaud TRSH 09 is great 4
  5. 5. Introduction: I like  large Grid5000 = 5 000 cores (increasing) Submitting jobs ==> grouping runs ==>  much bigger than number of cores. Next generations of computers: tenths, hundreds, thousands of cores. Evolutionary algorithms are population based but they have a bad speed-up. Teytaud and Teytaud TRSH 09 is great 5
  6. 6. Introduction: I like  large Grid5000 = 5 000 cores (increasing) Submitting jobs ==> grouping runs ==>  much bigger than number of cores. Next generations of computers: tenths, hundreds, thousands of cores. Evolutionary algorithms are population based but they have a bad speed-up. Teytaud and Teytaud TRSH 09 is great 6
  7. 7. Outline Introduction Complexity bounds Branching Factor Automatic Parallelization Real-world algorithms Log() corrections Teytaud and Teytaud TRSH 09 is great 7
  8. 8. Complexity bounds = nb of fitness evaluations for precision  with probability at least ½ Exp ( - Convergence ratio ) = Convergence rate Convergence ratio ~ 1 / computational cost ==> more convenient for speed-ups Teytaud and Teytaud TRSH 09 is great 8
  9. 9. Complexity bounds on the convergence ratio FR: full ranking (selected points are ranked) SB: selection-based (selected points are not ranked) Teytaud and Teytaud TRSH 09 is great 9
  10. 10. Outline Introduction Complexity bounds Branching Factor Automatic Parallelization Real-world algorithms Log() corrections Teytaud and Teytaud TRSH 09 is great 10
  11. 11. Branching factor K (more in Gelly06; Fournier08)Rewrite your evolutionary algorithm as follows:g has values in a finite set of cardinal K: - e.g. subsets of {1,2,...,} of size  (K=! / (!(-)!) )- or ordered subsets (K=! / (-)! ).- ... Teytaud and Teytaud TRSH 09 is great 11
  12. 12. Outline Introduction Complexity bounds Branching Factor Automatic Parallelization Real-world algorithms Log() corrections Teytaud and Teytaud TRSH 09 is great 12
  13. 13. Automatic parallelization Teytaud and Teytaud TRSH 09 is great 13
  14. 14. Automatic parallelization with branching factor 3 Consider the sequential algorithm. (iteration 1) Teytaud and Teytaud TRSH 09 is great 14
  15. 15. Automatic parallelization with branching factor 3 Consider the sequential algorithm. (iteration 2) Teytaud and Teytaud TRSH 09 is great 15
  16. 16. Automatic parallelization with branching factor 3 Consider the sequential algorithm. (iteration 3) Teytaud and Teytaud TRSH 09 is great 16
  17. 17. Automatic parallelization with branching factor 3 Parallel version for D=2. Population = union of all pops for 2 iterations. Teytaud and Teytaud TRSH 09 is great 17
  18. 18. Outline Introduction Complexity bounds Branching Factor Automatic Parallelization Real-world algorithms Log() corrections Teytaud and Teytaud TRSH 09 is great 18
  19. 19. Real world algorithms Define: Necessary condition for log() speed-up: - E log( * ) ~ log() But for many algorithms, - E log( * ) = O(1) ==> constant speed-up Teytaud and Teytaud TRSH 09 is great 19
  20. 20. One-fifth rule: E log( * ) = O(1) Consider e.g. Or consider e.g. In both cases * is lower-bounded independently of  ==> parameters should strongly depend on  ! Teytaud and Teytaud TRSH 09 is great 20
  21. 21. Self-adaptation, cumulative step-size adaptationIn both case, the same result: with parametersdepending on the dimension only (and not depending on ),the speed-up is limited by a constant! Teytaud and Teytaud TRSH 09 is great 21
  22. 22. Outline Introduction Complexity bounds Branching Factor Automatic Parallelization Real-world algorithms Log() corrections Teytaud and Teytaud TRSH 09 is great 22
  23. 23. The starting point of this work Many algorithms have parameters defined by handcrafted rules, Fournier08 shows rates which are reachable by comparison-based algorithms not reached by usual algorithms. Teytaud and Teytaud TRSH 09 is great 23
  24. 24. Log() corrections We can change that: In the discrete case (XPs): automatic parallelization surprisingly efficient. Simple trick in the continuous case - E log( *) should be linear in log() (see papers for details, sorry!) (this provides corrections which work for SA and CSA) Teytaud and Teytaud TRSH 09 is great 24
  25. 25. Conclusion The case of large population size is not well handled by usual algorithms. We proposed (I) theoretical guarantees (II) an automatic parallelization matching the bound, and which works well in the discrete case. (III) a necessary condition for the continuous case, which provides useful hints. Teytaud and Teytaud TRSH 09 is great 25
  26. 26. Main limitation All this is about a logarithmic speed-up. The computational power is like this ==> <== and the result is like that. ==> much better speed-up for noisy optimization. Teytaud and Teytaud TRSH 09 is great 26
  27. 27. Further work Apply VC-bounds for considering only “reasonnable” branches in the automatic parallelization. Theoretically easy, but provides extremely complicated algorithms. Teytaud and Teytaud TRSH 09 is great 27

×