One of the primary drawbacks plaguing wider acceptance of surrogate models is their low fidelity in general. This issue can be in a large part attributed to the lack of automated model selection techniques, particularly ones that do not make limiting assumptions regarding the choice of model types and kernel types. A novel model selection technique was recently developed to perform optimal model search concurrently at three levels: (i) optimal model type (e.g., RBF), (ii) optimal kernel type (e.g., multiquadric), and (iii) optimal values of hyper-parameters (e.g., shape parameter) that are conventionally kept constant. The error measures to be minimized in this optimal model selection process are determined by the Predictive Estimation of Model Fidelity (PEMF) method, which has been shown to be significantly more accurate than typical cross-validation-based error metrics. In this paper, we make the following important advancements to the PEMF-based model selection framework, now called the Concurrent Surrogate Model Selection or COS- MOS framework: (i) The optimization formulation is modified through binary coding to allow surrogates with differing num- bers of candidate kernels and kernels with differing numbers of hyper-parameters (which was previously not allowed). (ii) A robustness criterion, based on the variance of errors, is added to the existing criteria for model selection. (iii) A larger candidate pool of 16 surrogate-kernel combinations is considered for selection − possibly making COSMOS one of the most comprehensive surrogate model selection framework (in theory and implementation) currently available. The effectiveness of the COSMOS framework is demonstrated by successfully applying it to four benchmark problems (with 2-30 variables) and an airfoil design problem. The optimal model selection results illustrate how diverse models provide important tradeoffs for different problems.

1. Concurrent Surrogate Model Selection (COSMOS)
based on
Predictive Estimation of Model Fidelity
Souma Chowdhury#, Ali Mehmani*, and Achille Messac#
* Syracuse University, Department of Mechanical and Aerospace Engineering
# Mississippi State University, Bagley College of Engineering
The ASME International Design Engineering Technical Conferences (IDETC)
August 17 – 20, 2014, Buffalo, NY

2. Surrogate Modeling
Surrogate models are commonly used for providing a tractable and inexpensive
approximation of the actual system behavior, as an alternative
 To expensive computational simulations (e.g., CFD), or
 To the lack of a physical model in the case of experiment-derived data (e.g., testing
of new metallic alloys).
2
Kriging . . .Model Type RBF SVR
Kernel/Basis function
Linear Exponential Gaussian Cubic Multiquadric . . .
Hyper-Parameter value
Correlation parameter Shape parameter . . .
𝒇 𝒙 =
𝒊=𝟏
𝒏
𝒘𝒊 𝝍( 𝒙 − 𝒙𝒊
)
𝝍 𝒓 = (𝒓 𝟐
+ 𝒄 𝟐
) 𝟏/𝟐
𝒓= 𝒙 − 𝒙𝒊
𝒄𝒍𝒐𝒘𝒆𝒓
< 𝒄 < 𝒄 𝒖𝒑𝒑𝒆𝒓

3. Outline
• Background and Literature
• Research Objectives
• COSMOS Framework
• Predictive Estimation of Model Fidelity (PEMF)
• Numerical Experiments: Results
• Concluding Remarks
3

4. Surrogate Model Selection
4
Model selection based on the quantitative decision-making
techniques. Automated selection can be performed at these levels:
 Intuitive model selection (experience-based selection)
• Development of general guidelines likely not practical due to problem diversity.
• A few candidate surrogates are generally considered.
• In MDO problems, characteristics of disciplinary phenomena may not be evident.
Model selection based on an understanding of the data characteristics
and/or the application constraints.
 Automated model selection

5. Automated Model or Kernel Selection
5
 Error measures are used to select the model type and basis functions*
𝐹∗ = argmin
𝐹∈𝑭
𝜺( 𝑭)
best surrogate model
set of candidate surrogates
surrogate model error
 Popular error measures used for model selection include: (i) split sample,
(ii) cross-validation, (iii) bootstrapping, (iv) Schwarz’s Bayesian information
criterion (BIC), and (v) Akaike’s information criterion (AIC)
Method Model Type Selection Kernel Type Selection
Holena et al., 2011 
Jin et al., 2001 
Gano et al., 2006 
Chen et al., 2004 
Viana et al., 2009  

6. Hyper-parameter Optimization
6
To mitigate the possibility of constructing a suboptimal surrogate model for a
given Kernel function, one must perform hyper-parameter optimization.
• Martin et al. (AIAAJ, 2005) used MLE and cross-validation methods to find the optimum
hyper-parameter value for the Gaussian correlation function in Kriging.
• Mongillo et al. (SIAM, 2011) used MLE and leave-one-out cross-validation methods to select
an optimal shape parameter in a Gaussian RBF.
• Gorissen et al. (JMLR, 2009) used the leave-one-out cross-validation and AIC error measures
in the SUMO Toolbox to select the hyper parameter value(s) through a genetic algorithm.
Shape parameter, σ
RMSE
X
F
Branin-Hoo function:
RBF Multiquadric
model with different
HP values

7. Research Objectives
 The original PEMF-based surrogate model selection method performed
selection at all three levels based on the median and maximum error.
 Models with similar number of kernel choices and kernels with a single
hyper-parameter was considered.
 The objectives of this research is to advance the PEMF-based COSMOS:
1. By introducing additional selection criteria: (i) the variance of the surrogate error and
(ii) the predicted error at a greater number of sample points.
2. By modifying the optimization formulation to allow competition among surrogates with
differing numbers of candidate kernels, and kernels with differing numbers of HPs.
3. By testing the COSMOS framework with a comprehensive set of model types and
constitutive kernel types − 16 surrogate-kernel combinations with 0 to 2 HPs.
7PEMF: Predictive Estimation of Model Fidelity (Mehmani et al., AIAA Scitech 2014)

8. COSMOS Framework
8
Pareto Filter
Generally, any two
selection criteria, based on
user-preference, could be
considered simultaneously

9. COSMOS: MATLAB-based GUI
9COSMOS MATLAB-based GUI: Courtesy of Ali Mehmani

10. COSMOS: Optimization Formulation
 Separate MINLPs are run in parallel for
each HP class (defined by #HPs involved)
 All hyper-parameters (CHP) are scaled to the
range 0 to 1.
 The candidate model-kernel combinations
are integer-coded.
 A single integer variable (TSK) now identifies
the model-kernel type.
 NSGA-II is used to solve the MINLP
problems. 10
Hyper-Parameter
Values
Candidate
Model-Kernel
Combinations
Branin Hoo
Function

11. Surrogate Model Candidates
11

12. Predictive Estimation of Model Fidelity (PEMF)
12
The PEMF method is derived from the hypothesis that the accuracy of
approximation models is related to the amount of data resources
leveraged to train the model.
 PEMF can be perceived as a novel sequential implementation of k-fold
cross-validation, with carefully constructed error measures.
 The PEMF method analyzes the variation of the model error distribution
with increasing number of training points.
 The PEMF method is a model independent approach for surrogate error
quantification, and does not require any additional test points.
 The PEMF method has been shown to be 1-2 orders of magnitude more
accurate in error quantification compared to leave-one-out cross validation.
Mehmani et al., AIAA SDM 2013, AIAA Scitech 2014, and Aviation, SMO 2014

13. PEMF: Approach
13
Median Error Maximum Error
Using Lognormal distribution at every iteration:
Variation of the modal value of the
median/maximum error is represented by
𝐸 = 𝑎𝑛 𝑏
𝐸 = 𝑎𝑒 𝑏𝑛or
Final
Surrogate
Final
Surrogate

14. PEMF Input-Output
14
Model type
Kernel type HP values
I/O Training dataError Metrics
All error values expressed in terms
of relative absolute errors.

15. Numerical Experiments
Problem
Problem Settings Optimization Settings
Dimension Sample Size Population Size Max. Generations
Branin Hoo 2 30 20 50
Hartman-6 6 60 40 50
Dixon & Price 30 60 30 30
Neumaier & Perm 20 50 30 30
Airfoil Design 4 30 20 50
15
 Case 1: Minimize modal value of the median error and modal value of
the maximum error.
 Case 2: Minimize modal value of the median error and variance of the
median error.
 Case 3: Minimize modal value of the median error and the expected
modal value of the median error at 20% more sample points.

16. COSMOS Results: Benchmark Problems
16
0 0.05 0.1 0.15 0.2
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Mode of Median Error, E
med
mo
ModeofMaximumError,E
max
mo
HP-0
HP-1
HP-2
Pareto
0 0.05 0.1 0.15 0.2
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Mode of Median Error, Emed
mo
ModeofMedianErrorat20%moresamples,E
med,
mo
HP-0
HP-1
HP-2
Pareto
0.02 0.04 0.06 0.08 0.1 0.12
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Mode of Median Error, E
med
mo
StandardDeviationofMedianError,E
med

HP-0
HP-1
HP-2
Pareto
Med vs. Max Med vs. Std-dev Med vs. Med-extra
Branin Hoo (2D)
0.2 0.4 0.6 0.8 1 1.2
0.4
0.8
1.2
1.6
2
2.4
Mode of Median Error, E
med
mo
ModeofMaximumError,E
max
mo
HP-0
HP-1
HP-2
Pareto
0.2 0.4 0.6 0.8 1 1.2
0
5
10
15
20
25
30
35
40
Mode of Median Error, E
med
mo
StandardDeviationofMedianError,E
med

HP-0
HP-1
HP-2
Pareto
0.2 0.4 0.6 0.8 1 1.2
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Mode of Median Error, E
med
mo
ModeofMedianErrorat20%moresamples,E
med,
mo
HP-0
HP-1
HP-2
Pareto
Med vs. Max Med vs. Std-dev Med vs. Med-extra
Hartman-6 (6 D)

17. 0.35 0.4 0.45 0.5
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Mode of Median Error, Emed
mo
ModeofMedianErrorat20%moresamples,Emed,
mo
HP-0
HP-1
HP-2
Pareto
0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7
0
0.5
1
1.5
2
2.5
3
3.5
Mode of Median Error, Emed
mo
ModeofMaximumError,Emax
mo
HP-0
HP-1
HP-2
Pareto
0.09 0.1 0.11 0.12 0.13 0.14 0.15
0.06
0.065
0.07
0.075
0.08
0.085
0.09
0.095
0.1
0.105
0.11
Mode of Median Error, E
med
mo
ModeofMedianErrorat20%moresamples,E
med,
mo
HP-0
HP-1
HP-2
Pareto
0.09 0.1 0.11 0.12 0.13 0.14 0.15
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Mode of Median Error, E
med
mo
StandardDeviationofMedianError,E
med

HP-0
HP-1
HP-2
Pareto
0.09 0.1 0.11 0.12 0.13 0.14 0.15
0.25
0.26
0.27
0.28
0.29
0.3
0.31
0.32
0.33
0.34
0.35
Mode of Median Error, E
med
mo
ModeofMaximumError,E
max
mo
HP-0
HP-1
HP-2
Pareto
COSMOS Results: Benchmark Problems
17
Med vs. Max Med vs. Std-dev Med vs. Med-extra
Dixon & Price (30D)
Med vs. Max Med vs. Std-dev Med vs. Med-extra
Neumaier Perm (20D)

18. COSMOS Results: Airfoil Problem
18
0 0.005 0.01 0.015 0.02
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
Mode of Median Error, Emed
mo
ModeofMaximumError,E
max
mo
HP-0
HP-1
HP-2
Pareto
𝐶𝐿
𝐶 𝐷
= 𝑓 𝑥1, 𝑥2, 𝑥3, 𝛼
High-fidelity samples generated
by a Fluent CFD simulation
Med vs. Max Med vs. Std-dev Med vs. Med-extra
0 0.005 0.01 0.015 0.02
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
Mode of Median Error, Emed
mo
StandardDeviationofMedianError,E
med

HP-0
HP-1
HP-2
Pareto
0 0.005 0.01 0.015 0.02 0.025
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
Mode of Median Error, Emed
mo
ModeofMedianErrorat20%moresamples,E
med,
mo
HP-0
HP-1
HP-2
Pareto

19. COSMOS Results: Summary
 Widely different sets of surrogates models were selected as the
optimum set in the five different problems.
 A diverse set of surrogate-kernel combinations are often observed
to provide important trade-offs.
19
The Best Trade-off Models

20. Concluding Remarks
 A new framework, COSMOS, was developed to select surrogate models
based on criteria driven by user preference (e.g., median or max error).
 COSMOS can identify optimal model-kernel combinations from a large
pool of candidates, by using
1. The model-independent error measures given by PEMF, and
2. A novel MINLP formulation.
 On applying COSMOS to a suite of benchmark test problems, we found:
1. Same surrogate-kernel combinations can yield a noticeable spread of best trade-offs
(at different HP values);
2. Diverse surrogate models often constitute the set of best trade-off models.
 These initial tests readily exhibit the need for such frameworks for
automated selection of globally competitive surrogates.
 Future research directions: Application to more complex practical
problems, and a smarter apriori sorting of the model-kernel candidates.
20

21. Questions
and
Comments
21
Thank you

22. COSMOS: MATLAB-based GUI
22
For those interested to contribute models or test problems,
or interested to try out COSMOS,
please contact chowdhury@bagley.msstate.edu

23. Surrogate-Kernel Combinations
23

24. 24
MedianofRAEs
ε = 𝒎𝒆𝒅 |
𝒇𝒊 − 𝒇𝒊
𝒇𝒊
| ,
𝐢 = 𝟏, … , #{𝑿 𝑻𝑬}
Actual modelIntermediate
surrogate model
𝑋 = 𝑋𝑖𝑛 + 𝑋 𝑜𝑢𝑡
𝑋 𝑡
𝑇𝑅 = 𝑋 𝑜𝑢𝑡 + {𝛽 𝑘
}
𝑋 𝑡
𝑇𝐸 = X − {𝑋 𝑡
𝑇𝑅}
kth subset of
inside-region
sample points
Inside and outside sets
Momed
It. 2
Number of Training Points
t1 t2 t3 t4
It. 1
A chi-square, 𝝌 𝟐
,goodness-of-fit criterion
𝜒 2
=
𝑖=1
𝑚
(𝑜𝑖 − 𝑡𝑖)^2
𝑡𝑖
It. 4
Predicted
Median Error
MeanError
No. of Training points
ModeofMedianError
No. of Training points
Branin-Hoo Function (RBF)
It. 3
𝐹 𝑛 𝑡
= 𝑎0(𝑛 𝑡
)−𝑎1
OR
𝐹 𝑛 𝑡
= 𝑎0 𝑒−𝑎1(𝑛 𝑡)

Model Based Systems
Design
Integrative Modeling and
Design of Wind Farms
Energy-Sustainable Smart
Buildings
Reconfigurable Unmanned
Aerial Vehicles (UAV)
Predictive Estimation of Model Fidelity
(PEMF)
We randomly divide the set of sample points into intermediate sets of
1.Training points and
2.Test points

25. Comparison
25

26. Comparing Computational Time
26
One step method requires around 1/7th the time in
searching for optimal models.