Multi-objective meta-parameter tuning for mono-objective stochastic metaheuristics Johann Dréo THALES Research & Technology

Johann Dréo

THALES Research & Technology

Introduction Multi-objective method Parameter tuning Stochastic metaheuristics Performance profiles http://www.flickr.com/photos/k23/2792398403/ Dreo & Siarry, 2004

Multi-objective method

Parameter tuning

Stochastic metaheuristics

Performance profiles

Stochastic metaheuristics

Examples of stochastic metaheuristics

Parameter setting

Meta-parameter tuning

As a mono-objective problem Parameter setting: Improve performance http://www.flickr.com/photos/sigfrid/223626315/

Parameter setting:

Improve performance

As a multi-objective problem Parameter setting: What is performance ? -> multi-objective problem http://www.flickr.com/photos/jesusdq/345379863/

Parameter setting:

What is performance ?

-> multi-objective problem

Multi-objective problem Performance ? Precision Speed Robustness Precision Speed Stability (← benchmark) http://www.flickr.com/photos/matthewfch/1688409628/

Performance ?

Precision

Speed

Robustness

Precision

Speed

Stability (← benchmark)

Multi-objective problem Performance ? Precision Speed Robustness Precision Speed Stability (← benchmark)

Performance ?

Precision

Speed

Robustness

Precision

Speed

Stability (← benchmark)

Meta-parameter tuning Mono-objective problem Stochastic metaheuristic

Meta-parameter tuning Multi-objective parameter tuning problem Mono-objective problem Stochastic metaheuristic

Meta-parameter tuning Multi-objective parameter tuning problem Mono-objective problem Stochastic metaheuristic Meta-optimizer

Complexity Difficult Easier 1 time Multi-objective parameter tuning problem Mono-objective problem Stochastic metaheuristic Meta-optimizer

Methodology Speed / Precision Median estimation Mono-objective problem Stochastic metaheuristic NSGA-2

Methodology Speed / Precision Median estimation Mono-objective problem Stochastic metaheuristic NSGA-2

Results plots Speed Precision Performance profile / front

Performance profile / front

Some results

Example 2 continuous EDA (CEDA, CHEDA) Sampling density parameter Rosenbrock, 2 dimensions Median estimated with 10 runs 10 000 max eval. NSGA-2 20 iter., 50 indiv. 10 runs 3 days computation + Nelder-Mead Search

2 continuous EDA (CEDA, CHEDA)

Sampling density parameter

Rosenbrock, 2 dimensions

Median estimated with 10 runs

10 000 max eval.

NSGA-2

20 iter., 50 indiv.

10 runs

3 days computation

Example + simulated annealing stable temperature parameter Rosenbrock, 2 dimensions Median estimated with 10 runs 10 000 max eval. NSGA-2 20 iter., 50 indiv. 10 runs 1 day computation

+ simulated annealing

stable temperature parameter

Rosenbrock, 2 dimensions

Median estimated with 10 runs

10 000 max eval.

NSGA-2

20 iter., 50 indiv.

10 runs

1 day computation

Example + genetic algorithm population parameter Rosenbrock, 2 dimensions Median estimated with 10 runs 10 000 max eval. NSGA-2 20 iter., 50 indiv. 10 runs 1 day computation

+ genetic algorithm

population parameter

Rosenbrock, 2 dimensions

Median estimated with 10 runs

10 000 max eval.

NSGA-2

20 iter., 50 indiv.

10 runs

1 day computation

SA JGEN CEDA CHEDA Speed Precision

Behaviour exploration Speed Precision Genetic algorithm Population size

Genetic algorithm

Population size

Performance front Temporal planner, ''Divide & Evolve > CPT'', version ''GOAL'' 2 mutation parameters IPC ''rovers'' problem, instance 06 Median estimated with 10 runs NSGA-2 10 iter., 5 indiv. 30 runs 1 week computation for 1 run

Temporal planner, ''Divide & Evolve > CPT'', version ''GOAL''

2 mutation parameters

IPC ''rovers'' problem, instance 06

Median estimated with 10 runs

NSGA-2

10 iter., 5 indiv.

30 runs

1 week computation for 1 run

Performance front in Parameters space Speed Precision M1 M2

Previous parameters settings

Conclusion

Drawbacks Computation cost Stochastic M.-O. algo. -> supplementary bias http://www.flickr.com/photos/orvaratli/2690949652/

Computation cost

Stochastic M.-O. algo. -> supplementary bias

Drawbacks Computation cost Stochastic M.-O. algo. -> supplementary bias Valid only for: Algorithm implementation Problem instance Stopping criterion Error Time t steps, improvement < ε http://www.flickr.com/photos/orvaratli/2690949652/

Computation cost

Stochastic M.-O. algo. -> supplementary bias

Valid only for:

Algorithm implementation

Problem instance

Stopping criterion

Error

Time

t steps, improvement < ε

Drawbacks Computation cost Stochastic M.-O. algo. -> supplementary bias Valid only for: Algorithm implementation Problem instance Stopping criterion Error Time t steps, improvement < ε Fronts often convex -> aggregations ? No benchmarking http://www.flickr.com/photos/orvaratli/2690949652/

Computation cost

Stochastic M.-O. algo. -> supplementary bias

Valid only for:

Algorithm implementation

Problem instance

Stopping criterion

Error

Time

t steps, improvement < ε

Fronts often convex -> aggregations ?

No benchmarking

Advantages Performance profiles Objectives space Parameters space Quantification of expert knowledge

Performance profiles

Objectives space

Parameters space

Quantification of expert knowledge

Advantages Performance profiles Objectives space Parameters space Quantification of expert knowledge Automatic parameter tuning One step before use N parameters -> 1 parameter More degrees of freedom

Performance profiles

Objectives space

Parameters space

Quantification of expert knowledge

Automatic parameter tuning

One step before use

N parameters -> 1 parameter

More degrees of freedom

Advantages Performance profiles Objectives space Parameters space Quantification of expert knowledge Automatic parameter tuning One step before use N parameters -> 1 parameter More degrees of freedom Algorithms comparison Statistical tests more meaningful

Performance profiles

Objectives space

Parameters space

Quantification of expert knowledge

Automatic parameter tuning

One step before use

N parameters -> 1 parameter

More degrees of freedom

Algorithms comparison

Statistical tests more meaningful

Advantages Performance profiles Objectives space Parameters space Quantification of expert knowledge Automatic parameter tuning One step before use N parameters -> 1 parameter More degrees of freedom Algorithms comparison Statistical tests more meaningful Behaviour understanding

Performance profiles

Objectives space

Parameters space

Quantification of expert knowledge

Automatic parameter tuning

One step before use

N parameters -> 1 parameter

More degrees of freedom

Algorithms comparison

Statistical tests more meaningful

Behaviour understanding

Perspectives Include robustness Include dispersion estimation Include benchmarking Multi-objective SPO, F-Race Regressions in parameters space Performances / parameters Behaviour models? Links? Fitness Landscape / Performance profiles Run time distribution Taillard's significance plots ... http://www.flickr.com/photos/colourcrazy/2065575762/

Include robustness

Include dispersion estimation

Include benchmarking

Multi-objective SPO, F-Race

Regressions in parameters space

Performances / parameters

Behaviour models?

Links?

Fitness Landscape / Performance profiles

Run time distribution

Taillard's significance plots

...

[email_address] http://www.flickr.com/photos/earlg/275371357/