This document provides an overview of optimization capabilities in the free and open source software Scilab. It discusses the Nelder-Mead optimization component recently added to Scilab, which performs nonlinear optimization without requiring gradients. The document also compares Scilab's optimization solvers to those in MATLAB, finding that Scilab has similar functionality for problems like minimization, equation solving and least squares, though some solvers are only in alpha or beta versions. Finally, it outlines the OMD2 project which aims to develop Scilab into a full optimization platform through additional modules for data management, modeling and visualization.
Call Girls in Aiims Metro Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Optimization in Scilab
1. The free and open source software for numerical computation
Optimization with Scilab
June 29th
2011
Michaël BAUDIN & Vincent COUVERT
Scilab Consortium
2. The free and open source software for numerical computationThe free software for numerical computation
Part 1 - What's new in Scilab 5 ?
Focus on the Nelder-Mead component
Part 2 - Optimization in Scilab: Matlab® compatibility
Part 3 - OMD2 project: Scilab Platform Development
Part 4 - Conclusion
What is missing in Scilab ?
Outline
3. The free and open source software for numerical computationThe free software for numerical computation
1. Introduction
1.1 What's in Scilab ?
1.2 What's new in Scilab v5 ?
1.3 What's new on Atoms ?
Part 1 – What's new in Scilab 5 ?
2. The Nelder-Mead Component
2.1 Introduction
2.2 The algorithm
2.3 Test cases
2.4 Conclusions
4. The free and open source software for numerical computation
Objective Bounds Equality Inequalities Size Gradient
Needed
Solver
Linear yes linear linear medium - linpro
Quadradic yes linear linear medium - quapro
Quadratic yes linear linear large - qpsolve
Quadratic yes linear linear medium - qld
Non-Linear yes large yes optim
Non-Linear small no fminsearch
Non-Linear yes small no neldermead
Non-Linear yes small no optim_ga
Non-Linear small no optim_sa
N.Li.Lea.Sq. large optional lsqrsolve
N.Li.Lea.Sq. large optional leastsq
Min-Max yes medium yes optim/nd
Multi-Obj yes small no optim_moga
Semi-Def. lin. (spectral) large no semidef
L.M.I. lin. (spectral) lin. (spectral) large no lmisolve
1.1 What's in Scilab ?
5. The free and open source software for numerical computation
● Genetic Algorithms: nonlinear objective, bounds, global optimization
● Simulated Annealing: nonlinear objective, global optimization
● The Nelder-Mead component: nonlinear objective, unconstrained,
derivative-free, local optimization
● fminsearch: Matlab® compatible
1.2 What's new in Scilab 5 ?
6. The free and open source software for numerical computation
● Optimization Solvers:
– Quapro: linear or quadratic objective, linear constraints (full
matrices),
– SciIpopt: an interface to Ipopt. Nonlinear objective, nonlinear
constraints (beta version),
– Fmincon: nonlinear objective, nonlinear constraints (alpha
version) – Matlab® compatible,
– Other modules: Cobyla, Particle Swarm Optimization,
Optkelley, …
● Test Problems:
– Uncprb: 35 unconstrained optimization problems,
– AMPL: load AMPL problems into Scilab,
– And also: CUTEr.
1.3 What's new on ATOMS ?
7. The free and open source software for numerical computationThe free software for numerical computation
Outline
1. Introduction
1.1 What's in Scilab ?
1.2 What's new in Scilab v5 ?
1.3 What's new on Atoms ?
2. The Nelder-Mead Component
2.1 Introduction
2.2 The algorithm
2.3 Test cases
2.4 Conclusions
8. The free and open source software for numerical computation
John Ashworth Nelder (8 October 1924 – 7 August 2010)
Source: http://www.guardian.co.uk/technology/2010/sep/23/john-nelder-obituary
2. The Nelder-Mead Component
2.1 Introduction
9. The free and open source software for numerical computation
● We are interested in solving the unconstrained continuous
optimization problem:
Minimize f(x)
with unbounded, real, multidimensional, variable x.
● A direct search algorithm:
– Uses only function values (no gradient needed),
– Does not approximate the gradient.
● « A simplex method for function minimization », John Nelder, Roger
Mead, Computer Journal, vol. 7, no 4, 1965, p. 308-313
2. The Nelder-Mead Algorithm
2.1 Introduction
10. The free and open source software for numerical computation
2. The Nelder-Mead Algorithm
2.1 Introduction
Virginia Torczon (1989) writes:
"Margaret Wright has stated that over fifty percent of the calls
received by the support group for the NAG software library
concerned the version of the Nelder-Mead simplex algorithm to be
found in that library."
11. The free and open source software for numerical computation
A simplex: a set of n+1 vertices, in n dimensions.
In 2 dimensions.
In 3 dimensions.
2. The Nelder-Mead Algorithm
2.2 The algorithm
12. The free and open source software for numerical computation
● Steps in the Nelder-Mead algorithm
● Inputs:
– the n+1 vertices v(1), v(2), ..., v(n+1) of a nondegenerate
simplex in n dimensions,
– the associated function values f(1),...,f(n+1),
– the coefficients ρ (reflection), χ (expansion), γ (contraction),
and σ (shrinkage).
● Standard Nelder-Mead: ρ=1, χ=2, γ=1/2, and σ=1/2.
2. The Nelder-Mead Algorithm
2.2 The algorithm
13. The free and open source software for numerical computation
2. The Nelder-Mead Algorithm
2.2 The algorithm
14. The free and open source software for numerical computation
f x1, x2=x1
2
x2
2
−x1 x2
function [ y , index ] = quadratic ( x , index )
y = x(1)^2 + x(2)^2 - x(1) * x(2);
endfunction
nm = neldermead_new ();
nm = neldermead_configure(nm,"-numberofvariables",2);
nm = neldermead_configure(nm,"-function",quadratic);
nm = neldermead_configure(nm,"-x0",[2 2]');
nm = neldermead_search(nm);
xopt = neldermead_get(nm,"-xopt");
nm = neldermead_destroy(nm);
2. The Nelder-Mead Algorithm
2.3 Test cases
15. The free and open source software for numerical computation
2. The Nelder-Mead Algorithm
2.3 Test cases
16. The free and open source software for numerical computation
● Mc Kinnon, « Convergence of the neldermead simplex method to a nonstationary
point ». SIAM J. on Optimization, 1998
● Failure by repeated inside contraction
2. The Nelder-Mead Algorithm
2.3 Test cases
17. The free and open source software for numerical computation
● C. T. Kelley. « Detection and remediation of stagnation in the neldermead algorithm
using a sufficient decrease condition » SIAM J. on Optimization, 1999
● Restart the algorithm...
2. The Nelder-Mead Algorithm
2.3 Test cases
18. The free and open source software for numerical computation
● Some general facts:
– Memory requirement is O(n²)
– Shrink steps are rare
– Generally 1 or 2 function evaluations by iteration
– Convergence is slow. Typical number of iterations is 100n,
where n is the number of dimensions
– Hundreds of iterations are not rare
– Convergence can be even slower when n > 10 (Han &
Neumann, 2006)
– Restart the algorithm when in doubt for convergence (Kelley,
1999)
– Convergence is guaranteed in 1 dimension (Lagarias et al.,
1999)
2. The Nelder-Mead Algorithm
2.4 Conclusions
19. The free and open source software for numerical computation
● We should not use this algorithm just because the gradient is not
required:
– For example, if f is smooth, Quasi-Newton methods (optim)
with numerical derivatives converge much faster.
● We may use this algorithm when:
– No other property of the problem can be used (e.g. non linear
least squares can be solved by lsqrsolve),
– The objective function is nonsmooth or "noisy" (Kelley, 1999),
– We do not need too much accuracy (Torzcon, 1989),
– The number of parameters is moderate (Han & Neumann,
2006).
2. The Nelder-Mead Algorithm
2.4 Conclusions
20. The free and open source software for numerical computationThe free software for numerical computation
1. Introduction
2. Scilab Coverage
3. Overview
Part 3 –
Optimization in Scilab: Matlab® compatibility
21. The free and open source software for numerical computation
● Matlab® has many functions for optimization:
– Minimization,
– Equation solving,
– Datafitting and nonlinear least squares,
– Global optimization.
● Scilab has often similar functions: let's see which ones.
Matlab is a registered trademark of The Mathworks, Inc.
1. Introduction
22. The free and open source software for numerical computation
● For each Matlab® function, we search:
– Scilab function, if available,
– Differences of features, differences of algorithms.
● (*) : Function will be reviewed at the end of the talk,
● For most functions, the match is not 100% identical,
– But some other functions can do it : which ones ?
● We consider only Scilab Industrial Grade solvers:
– Scilab internal modules,
– ATOMS modules,
– Portables on all OS,
– Well documented,
– Tested.
1. Introduction
23. The free and open source software for numerical computation
1. Introduction
Main differences
● Design:
– Matlab®: problem oriented (may be with several solvers),
– Scilab: solver oriented (may be several solvers).
● Function arguments:
– Matlab® nearly always provides common options,
– Scilab is less homogeneous.
● Management of the callbacks/extra-arguments:
– Matlab®: M-file or @
– Scilab: list
24. The free and open source software for numerical computation
● Minimization:
– fminbnd Not 100% identical,
But optim can do it.
– fmincon ATOMS/fmincon (alpha version)
– fminimax Not 100% identical,
but optim/''nd'' is designed for it.
– fminsearch fminsearch 90% identical in Scilab 5.3.2.
fminsearch 99% identical in 5.4.0
2. Scilab Coverage
25. The free and open source software for numerical computation
– fminunc Not 100% identical,
but optim/''qn'' or optim/''gc'' are designed for it.
No sparsity pattern of Hessian in Scilab.
No PCG in optim: L-BFGS instead.
– linprog 100% for full matrices: karmarkar
ATOMS/quapro: linpro
No known solver for sparse matrices (*).
– quadprog 100% for full matrices: qpsolve, qp_solve
ATOMS/quapro: quapro
No known solver for sparse matrices.
2. Scilab Coverage
26. The free and open source software for numerical computation
● Equation Solving:
– fsolve fsolve 100% for full matrices.
No known solver with sparse Jacobian (*).
– fzero No identical function.
But fsolve can do it.
● Least Squares (Curve Fitting):
– lsqcurvefit datafit
– lsqnonlin lsqrsolve (leastsq)
2. Scilab Coverage
27. The free and open source software for numerical computation
● Global Optimization Toolbox:
● Genetic Algorithm Not 100% identical,
But optim_ga is built-in Scilab.
No linear equality and inequality in Scilab,
but bounds are managed.
● Simulated Annealing Not 100% identical,
But optim_sa is built-in Scilab
No bounds in Scilab SA, but user can
customize the neighbour function.
2. Scilab Coverage
28. The free and open source software for numerical computation
Matlab Problem Scilab
bintprog Binary Integer Programming -
fgoalattain Multiobjective goal attainment -
fminbd Single-variable, on interval optim
fmincon Constrained, nonlinear, multivariable ATOMS/fmincon
fminimax Minimax, constrained optim/''nd''
fminsearch Unconstrained, multivariable, derivative-free fminsearch (100%)
fminunc Unconstrained, multivariable optim/''qn'',''gc''
fseminf Semi-infinitely constrained, multivariable,
nonlinear
-
ktrlink Constrained or unconstrained, nonlinear,
multivariable using Knitro
-
linprog Linear programming karmarkar,
ATOMS/quapro
quadprog Quadratic programming qpsolve,
ATOMS/quapro
3. Overview
29. The free and open source software for numerical computation
Matlab Problem Scilab
fsolve Solve systems of nonlinear equations fsolve
fzero Root of continuous function of one variable -
lsqcurvefi t Nonlinear least squares curve fitting datafit
lsqlin Constrained linear least squares -
lsqnonlin Nonlinear least-squares (nonlinear data-fitting) lsqrsolve, leastsq
lsqnonneg Nonnegative least squares -
optimtool GUI to select solvers, options and run problems -
Global Search Solve GlobalSearch problems -
Multi Start Solve MultiStart problems -
Genetic Algorithm Genetic Algorithms optim_ga
Direct Search Pattern Search -
Simulated Annealing Simulated Annealing optim_sa
3. Overview
30. The free and open source software for numerical computationThe free software for numerical computation
Part 4 -
OMD2 project: Scilab Platform Development
1. Overview
2. Modules
2.1 Data Management
2.2 Modeling
2.3 Optimization
31. The free and open source software for numerical computation
● OMD2 / CSDL projects collaboration
● Will be available on Scilab forge:
– http://forge.scilab.org/index.php/p/omd2/
– Private project up to first release.
● Scilab Optimization Platform:
– Batch mode (script edition, large scale execution),
– GUI mode (interactive edition, prototyping).
● Future Scilab external module available through ATOMS.
Overview
32. The free and open source software for numerical computation
● Project management (Save & Load working data as HDF5 files)
● Wrappers:
– Scilab algorithms,
– External tools,
– Proactive.
● Mask complexity for users
● Modules:
– Data Management,
– Modeling,
– Optimization,
– Visualization.
Main functionalities
33. The free and open source software for numerical computation
● Factors / Parameters:
– Load existing Design Of Experiments (Isight .db files, …)
– Generate Design Of Experiments:
● DoE generator wrappers (LHS, …),
● DoE generator settings.
● Responses simulation using:
– External tool (openFOAM, Catia, CCM+, …),
– Scilab function.
● 2-D visualization:
– Factor / Factor,
– Response / Factor.
Data Management Module (1/2)
34. The free and open source software for numerical computation
Data Management Module (2/2)
35. The free and open source software for numerical computation
● Point selection:
– Learning points used for modeling,
– Validation points used to validate model,
– Bad points (simulation issue, …).
● Modeler:
– Selected among modeler wrappers (DACE, Lolimot, …),
– Parameters configuration,
– Multiple model management with best model user selection.
● Visualization:
– 2-D models,
– Cross correlation,
– Sensibility analysis.
Modeling module (1/2)
36. The free and open source software for numerical computation
Modeling module (2/2)
37. The free and open source software for numerical computation
● Responses coefficients values setting
● Optimizer:
– Selection among generic wrappers (optim, fmincon, genetic
algorithms, …),
– Optimizer configuration,
– Enable two chained optimizers.
● Visualization:
– Optimal point,
– Paretos,
– Robustness.
Optimization Module (1/2)
38. The free and open source software for numerical computation
Optimization Module (2/2)
39. The free and open source software for numerical computationThe free software for numerical computation
1. What is missing ?
2. Bibliography
Part 4 - Conclusion
40. The free and open source software for numerical computation
● High Performance Optimization:
– Use BLAS/LAPACK within optim ?
● Sparse Linear Programming:
– Update LIPSOL ?
● Non Linear Programming:
– Improve fmincon ?
● Non Linear Programming Test Cases:
– CUTEr requires a compiler on the test machine,
– Connect the Hock-Schittkowski collection ?
Conclusion
1. What is missing ?
41. The free and open source software for numerical computation
● « Nelder-Mead User's Manual », Michaël Baudin, Consortium Scilab
– DIGITEO, 2010
● « Optimization in Scilab », Baudin, Couvert, Steer, Consortium Scilab
- DIGITEO – INRIA, 2010
● « Optimization with scilab, present and future », Michaël Baudin and
Serge Steer, 2009 IEEE International Workshop on Open Source
Software for Scientific Computation, pp.99-106, 18-20 Sept. 2009
● « Introduction to Optimization with Scilab », Michaël Baudin,
Consortium Scilab – DIGITEO, 2010
● « Unconstrained Optimality Conditions with Scilab », Michaël Baudin,
Consortium Scilab – DIGITEO, 2010
Conclusion
2. Bibliography
42. The free and open source software for numerical computation
Thanks for your attention
www.scilab.org
43. The free and open source software for numerical computation
Some slides you won't see, unless you ask...
Extra-Slides
44. The free and open source software for numerical computation
Some Historical References:
● Spendley, Hext, Himsworth (1962): fixed shape simplex algorithm
● Nelder, Mead (1965): variable shape algorithm
● Box (1965): simplex algo., with constraints
● O'Neill (1971): Fortran 77 implementation.
● Torczon (1989): Multi Directional Search.
● Mc Kinnon (1998): Counter examples of N-M.
● Lagarias, Reeds, Wright, Wright (1998): Proof of convergence in
dimensions 1 and 2 for strictly convex functions.
● Han, Neumann (2006): More counter examples of N-M.
The Nelder-Mead Algorithm
45. The free and open source software for numerical computation
In what softwares N-M can be found ?
● Matlab (fminsearch)
● NAG (E04CBF)
● Numerical Recipes (amoeba)
● IMSL (UMPOL)
● … and Scilab since v5.2.0 in 2009
● … and R after Sébastien Bihorel's port of Scilab's source code.
The Nelder-Mead Algorithm
46. The free and open source software for numerical computation
1. Sort by function value. Order the vertices: f(1) ≤ · · · ≤ f(n) ≤ f(n+1)
2. Calculate centroid. B = (v(1)+...+v(n))/n
3. Reflection. Compute R = (1+ρ)B − ρv(n+1) and evaluate f(R).
4. Expansion. If f(R)<f(1), compute E=(1+ρχ)B − ρχv(n+1) and evaluate f(E).
If f(E)<f(R), accept E, else accept R and goto 1.
5. Accept R. If f(1) ≤ f(R) < f(n), accept R and goto 1.
6. Outside Contraction. If f(n)≤f(R)<f(n+1), compute Co=(1+ργ)B − ργv(n+1)
and evaluate f(Co). If f(Co)<f(R), then accept Co and goto 1 else, goto 8.
7. Inside Contraction. If f(n+1)≤f(R), compute Ci=(1-γ)B +γv(n+1) and
evaluate f(Ci). If f(Ci)<f(n+1), then accept Ci and goto 1 else, goto 8.
8. Shrink. Compute the points v(i)=v(1)+σ(v(i)-v(1)) and evaluate
f(i)=f(x(i)), for i=2,3,...,n+1. Goto 1.
The Nelder-Mead Algorithm
47. The free and open source software for numerical computation
● Lagarias, Reeds, Wright, Wright (1998)
1. In dimension 1, the Nelder-Mead method converges to a
minimizer, and convergence is eventually M-step linear, when
the reflection parameter ρ = 1.
2. In dimension 2, the function values at all simplex vertices in the
standard Nelder-Mead algorithm converge to the same value.
3. In dimension 2, the simplices in the standard Nelder-Mead
algorithm have diameters converging to zero.
● Note that Result 3 does not implies that the simplices converge to a
single point x*.
The Nelder-Mead Algorithm
48. The free and open source software for numerical computation
● Minimization:
– bintprog Solve binary integer programming problems
– fgoalattain Solve multiobjective goal attainment problems
– fminbnd Find minimum of single-variable function on
fixed interval
– fmincon Find minimum of constrained nonlinear
multivariable function
– fminimax Solve minimax constraint problem
– fminsearch Find minimum of unconstrained multivariable
function using derivative-free method
What's in Matlab® ?
49. The free and open source software for numerical computation
– fminunc Find minimum of unconstrained multivariable
function
– fseminf Find minimum of semi-infinitely constrained
multivariable nonlinear function
– ktrlink Find minimum of constrained or unconstrained
nonlinear multivariable function using KNITRO
third-party libraries
– linprog Solve linear programming problems
– quadprog Quadratic programming
What's in Matlab® ?
50. The free and open source software for numerical computation
● Equation Solving:
– fsolve Solve system of nonlinear equations
– fzero Find root of continuous function of one variable
● Least Squares (Curve Fitting):
– lsqcurvefit Solve nonlinear curve-fitting (data-fitting)
problems in least-squares sense
– lsqlin Solve constrained linear least-squares problems
– lsqnonlin Solve nonlinear least-squares problems
– lsqnonneg Solve nonnegative least-squares constraint
problem
What's in Matlab® ?
51. The free and open source software for numerical computation
● Utilities:
– optimtool GUI to select solver, optimization options, and
run problems
– optimget Optimization options values
– optimset Create or edit optimization options structure
What's in Matlab® ?
52. The free and open source software for numerical computation
● Global Optimization Toolbox:
– GlobalSearch Create and solve GlobalSearch
problems
– MultiStart Create and solve MultiStart problems
– Genetic Algorithm Use genetic algorithm and Optimization
Tool, and modify genetic algorithm
options
– Direct Search Use direct search and Optimization Tool,
and modify pattern search options
– Simulated Annealing Use simulated annealing and
Optimization Tool, and modify simulated
annealing options
What's in Matlab® toolboxes ?