This document presents a new 3-level approach to simultaneously select the best surrogate model type, kernel function, and hyper-parameters for approximation models. The approach uses Regional Error Estimation of Surrogates (REES) to evaluate error and select models. It compares a cascaded technique that performs sequential optimization versus a one-step technique. Numerical examples on benchmark problems show the one-step technique reduces maximum and median errors by at least 60% with lower computational cost compared to the cascaded approach. Future work involves applying the one-step method to more complex problems and developing an online platform for collaborative surrogate model selection.
Arduino_CSE ece ppt for working and principal of arduino.ppt
AIAA-SciTech-ModelSelection-2014-Mehmani
1. A Novel Approach to Simultaneous Selection of
Surrogate Models, Constitutive Kernels, and
Hyper-parameter Values
Ali Mehmani*, Souma Chowdhury#, and Achille Messac#
* Syracuse University, Department of Mechanical and Aerospace Engineering
# Mississippi State University, Bagley College of Engineering
10th Multi-Disciplinary Design Optimization Conference
AIAA Science and Technology Forum and Exposition
January 13 – 17, 2014 National Harbor, Maryland
2. Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual system behavior in many
routine engineering analysis and design activities:
2
3. Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual system behavior in many
routine engineering analysis and design activities:
3
Kriging . . .Model Type RBF SVR
Kernel / basis function
Linear Exponential Gaussian Cubic Multiquadric . . .
Hyper-parameter
Correlation parameter Shape parameter . . .
𝒇 𝒙 =
𝒊=𝟏
𝒏
𝒘𝒊 𝝍( 𝒙 − 𝒙𝒊
)
𝝍 𝒓 = (𝒓 𝟐
+ 𝒄 𝟐
) 𝟏/𝟐
𝒓= 𝒙 − 𝒙𝒊
𝒄𝒍𝒐𝒘𝒆𝒓
< 𝒄 < 𝒄 𝒖𝒑𝒑𝒆𝒓
4. Research Objective
Develop a new model selection approach, which
simultaneously select the best model type, kernel function, and
hyper-parameter.
4
Types of model Types of basis/kernel Hyper-parameter(s)
• RBF,
• Kriging,
• E-RBF,
• SVR,
• QRS,
• …
• Linear
• Gaussian
• Multiquadric
• Inverse multiquadric
• Kriging
• …
• Shape parameter in RBF,
• Smoothness and width
parameters in Kriging,
• Kernel parameter in SVM,
• …
5. Presentation Outline
5
• Surrogate model selection
• REES-based Model Selection
• 3-Level model selection
• Regional Error Estimation of Surrogate (REES)
• Numerical Examples
• Concluding Remarks
6. Surrogate model selection
6
• Dimension and nature of sample points,
• Level of a noise,
• Application domain,
• …
Suitable Surrogate
Error measures are used to
select the best surrogate
Experienced-based model selection
Automated model selection
• RMSE,
• Cross-validation,
• REES,
• …
Hyper-parameter selection (Kriging-Guassian) using cross validation and
maximum likelihood estimation (Martin and Simpson)
Model type and basis function selection using cross validation (Viana and Haftka)
Model type selection using leave-one-out cross validation (Drik Gorisson et al.)
7. 3-Level model selection
7
In 3-level model selection, the selection criteria could depend
on the user preference.
Standard surrogate-based analysis
Structural optimization applications
lower median error
lower maximum error
8. 3-Level model selection
8
Median error
Maximum error
Two model selection criteria
evaluated using advanced surrogate error
estimation method presented in REES
Depending on the problem and the available data set, the
median and maximum errors might be
mutually conflicting
mutually promoting
Pareto models
A single optimum model
9. 3-Level model selection
9
To implement a 3-level model selection, two approaches are
proposed:
(i) Cascaded technique, and
(ii) One-Step technique.
10. 3-Level model selection
10
Cascaded technique
For each candidate kernel function, hyper-parameter optimization is
performed to minimize the median and maximum error.
Post hyper-parameter optimizations,
Pareto filter is used to reach the final
Pareto models.
Hyper-parameter optimization is
the process of quantitative search to
find optimum hyper-parameter
value(s).
11. 3-Level model selection
11
Cascaded technique
Solutions of the hyper-parameter optimization in the cascaded
technique for multiquadric basis function of RBF surrogate for
Baranin-hoo function
12. 3-Level model selection
12
One-Step technique
To escape the potentially high computational cost of the cascaded
technique
Subjected to
The three-level model selection could also be performed by solving
a single uniquely formulated mixed integer nonlinear
programming (MINLP) problem.
model type
basis function
hyper-parameter(s)
13. Regional Error Estimation of Surrogate
(REES)
13
The REES method is derived from the hypothesis that the accuracy of
approximation models is related to the amount of data resources
leveraged to train the model.
• Mehmani, A., Chowdhury, S., Zhang, Jie, and Messac, A., “Quantifying Regional Error in
Surrogates by Modeling its Relationship with Sample Density,” 54th
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference,
Paper No. AIAA 2013-1751, Boston, Massachusetts, April 8-11, 2013.
• Mehmani, et. al., “Model Selection based on Generalized-Regional Error Estimation for
Surrogate,” 10th World Congress on Structural and Multidisciplinary Optimization, Paper No.
5447, Orlando, Florida, May 19-24, 2013.
• Mehmani, et. al.., “Regional Error Estimation of Surrogates (REES),” 14th AIAA/ISSMO
Multidisciplinary Analysis and Optimization Conference, Paper No. AIAA 2012-5707,
Indianapolis, Indiana, September 17-19, 2012.
22. 22
It. 3It. 1
MedianofRAEs
Number of Training Points
t1 t2 t3 t4
It. 2
Momed
It. 4
ModelofMedian
Momax Mode of maximum
error distribution at
each iteration
Predicted Median Error
Predicted Maximum Error
REES
23. 23
The effectiveness of the new 3-level model selection method is
investigated by considering the following three candidate surrogates:
The methods are implemented on three benchmark problems and an
engineering design problem are tested.
Gaussian basis function
Multiquadric basis function
Gaussian correlation function
Exponential correlation function
Radial basis kernel function
Sigmoid kernel function
SVR
RBF
Kriging
Model type Kernel function Hyper-parameter
Numerical Examples
24. 24
Numerical Setting
The numerical settings for the implementation of REES-based model selection
for the benchmark problems
The numerical settings for the hyper-parameter optimization
Numerical Examples
24
25. 25
Hyper-parameter optimization of Cascaded technique in different surrogate type and
Kernel functions for Branin-Hoo function with 2 design variables
Numerical Examples
26. 26
Numerical Setting
The numerical settings for One-Step technique
Integer design variables
Numerical Examples
34. 34
Concluding Remarks
A new 3-level model selection approach is developed to select the best
surrogate among available surrogate candidates based on the level of
accuracy.
This approach is based on the model independent error measure given by the
Regional Error Estimation of Surrogates (REES) method.
The preliminary results on problems indicate at least 60% reduction in
maximum and median error values.
(i) model type selection,
(ii) kernel function selection, and
(iii) hyper-parameter selection.
35. 35
Future Work
Implementation of One-Step technique with
larger pool of surrogates with different number of kernels
higher dimensional and more computationally intensive problems
Develop an open online platform called Collaborative Surrogate
Model Selection (COSMOS) to allow users to submit
- training data for identifying an ideal model from existing pool
of surrogate models, and
- their own new surrogate into the pool of surrogate candidates.
If interested in COSMOS please contact me at amehmani@syr.edu
36. Acknowledgement
I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Dr.
Souma Chowdhury for their immense help and
support in this research.
I would also like to thank my friend and colleague
Weiyang Tong for his valuable contributions to this
paper.
Support from the NSF Awards is also
acknowledged.
36
39. 39
A chi-square (χ2) goodness-of-fit criterion is used to select the
type of distribution from a list of candidates such as lognormal,
Gamma, Weibull, logistic, log logistic, inverse Gaussian, and
generalized extreme value distribution.