The analysis of complex system behavior often demands expensive experiments or computational simula- tions. Surrogate modeling techniques are often used to provide a tractable and inexpensive approximation of such complex system behavior. Owing to the lack of any general guidelines regarding the suitability of different surrogate models for different applications, model selection approach can be helpful to choose the best surrogate technique. This paper investigates the effectiveness of a recently developed method for surrogate error quantification called, Regional Error Estimation of Surrogate (REES), to select the best surrogate model based on the level of accuracy. The REES method is developed based on the concept that the accuracy of the approximation methods is related to the amount of available resources. In the REES method, intermediate surrogates are iteratively constructed over heuristic subsets of the available sample points (i.e., intermediate training points) and tested over the remaining available sample points (i.e., intermediate test points). The statistical mode of the median and the maximum error distributions are selected to represent the overall and maximum error at each iteration. The estimated modes of the median and maximum error distributions are then represented as functions of the number of interme- diate training points using a regression model. The regression models are used to predict the overall and minimum accuracy of the final surrogate. These two error measures are then applied to select the best surrogate. The proposed model selection technique is applied to select the best surrogate among (i) Kriging, (ii) Radial Basis Functions (RBF), (iii) Extended Radial basis Functions (E-RBF), and (iv) Quadratic Response Surface (QRS), for standard test functions and a wind farm power generation func- tion. The REES-based model selection is compared with (i) model-selection based on cross-validation errors and (ii) model-selection based on error estimated on a large set of additional test points; the lat- ter is assumed to provide the correct model selection. The REES-based model selection is found to be significantly more accurate than that based on cross-validation errors.

1. Model Selection based on
Regional Error Estimation of Surrogates (REES)
Ali Mehmani, Souma Chowdhury, Jie Zhang, Weiyang Tong,
and Achille Messac
Syracuse University, Department of Mechanical and Aerospace Engineering
10th World Congress on Structural and Multidisciplinary Optimization
May 19 - 24, 2013, Orlando, FL

2. Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual system behavior in many
routine engineering analysis and design activities:
2

3. Surrogate model: Model selection
3
Statistical model selection approaches provide support
information for users
• to select a best model,
• to select a best kernel function, and
• to determine an optimum model’s parameter.

4. Surrogate model: Model selection
4
Types of model Types of basis/kernel Parameter estimation
•RBF,
•Kriging,
•E-RBF,
•SVR,
•QRS,
• …
•Linear
•Gaussian
•Multiquadric
•Inverse multiquadric
•Kriging
•…
•Shape parameter in RBF,
•Smoothness and width
parameters in Kriging,
•Kernel parameter in SVM,
• …
𝑓 𝑥 =
𝑖=1
𝑛
𝑤𝑖 ψ( 𝑥 − 𝑥 𝑖
)
RBF
ψ 𝑟 = (𝑟2
+ 𝑐2
) 1/2
𝑟= 𝑥 − 𝑥 𝑖
Multiquadric Shape parameter
ψ 𝑟 = (𝑟2
+ 𝒄2
) 1/2

5. Research Objective
• Investigate the effectiveness of a Regional Error Estimation of
Surrogate (REES) to select the best surrogate model based on
the level of accuracy.
5
Overall fidelity information
Minimum fidelity information
REES
e.g., Hybrid surrogate
e.g., Conservative surrogate

6. Presentation Outline
6
• Surrogate model selection
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

7. Presentation Outline
7
• Surrogate model selection
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

8. Surrogate model selection
8
• Size and location of sample points,
• Dimension and level of a noise,
• Application domain,
• …
Suitable Surrogate
Error measures are used to
select the best surrogate
 lack of any general guidelines regarding the suitability
of different surrogate models for different applications
Application-based model selection (Manual selection)
Error-based model selection (Automatic Selection)

9. Surrogate model: Model selection
Types of model Types of basis/kernel Parameter estimation
•RBF,
•Kriging,
•E-RBF,
•SVR,
•QRS,
• …
•Linear
•Gaussian
•Multiquadric
•Inverse multiquadric
•Kriging
•…
•Shape parameter in RBF,
•Smoothness and width
parameters in Kriging,
•Kernel parameter in SVM,
• …
 Split samples (or holdout samples)
 Bootstrapping
 Cross-Validation (Predictive Sum of Square based on k-fold or leave-one-out cv)
 Akaike information criterion (AIC), and
 Schwarz's Bayesian information criterion (BIC)
There exist many parameter/model/kernel selection approaches
9

10. Presentation Outline
10
• Review of surrogate model error measurement methods
• Regional Error Estimation of Surrogate (REES)
• Numerical examples: benchmark and an engineering
design problems

11. REES: Concept
11
Model accuracy Available resources
In general, this concept can be applied to different types of approximation
models;
- Surrogate modeling,
- Finite Element Analysis, and
- ...

12. REES: Methodology
12
 The REES method formulates the variation of error as a
function of the number of training points using
intermediate surrogates.
 This formulation is used to predict the level of error in the
final surrogate.

13. Step 1 : Generation of sample data
The entire set of sample points is represented by 𝑿 .
13
Sample Point
Step 2 : Estimation of the variation of error with sample density
Test Point
Training Point
First Iteration:
Test Point
Training Point
Second Iteration:
Test Point
Training Point
Third Iteration:
Test Point
Training Point
Forth Iteration:
Training Point
Final Surrogate:
REES: Methodology

14.  A position of sample points which are selected as training
points, at each iteration, is critical to the surrogate accuracy.
14
 Intermediate surrogates are iteratively constructed (at each
iteration) over a heuristic subsets of sample points.
REES: Methodology

15. MedianofRAEs
Number of Training Points
It. 1
t1 t2 t3 t4
Momed
Choose number of iterations, Nit
Choose number of combinations, Kt
Define intermediate training and test points
Construct an intermediate surrogate
Estimate Median and Maximum errors
Fit a distribution over all combinations
Determine Momedian and Momaximum errors
FOR t=1,..,Nit
FOR k=1,…, Kt
variation of the error with sample density

16. MedianofRAEs
t1 t2 t3 t4
It. 2It. 1
Momed
variation of the error with sample density
Number of Training Points

17. MedianofRAEs
t1 t2 t3 t4
It. 3It. 2It. 1
Momed
variation of the error with sample density
Number of Training Points

18. MedianofRAEs
t1 t2 t3 t4
It. 3It. 2It. 1 It. 4
Momed
variation of the error with sample density
Number of Training Points

19. Step 3 : Prediction of error in the final surrogate
 The final surrogate model is constructed using the full set of training data.
 Regression models are applied to relate the evaluated error at each iteration to
the size of training points,
 These regression models are called the variation of error with sample density
(VESD).
The regression models are used to predict the level of the error in the final
surrogate.
19
Exponential regression model
Multiplicative regression model
Linear regression model
 In the choice of these functions we assume a smooth monotonic decrease of
the error with the training point density.
REES: Methodology

20. ModeofMedianofRAEs
Number of Training Points
t1 t2 t3 t4
It. 3It. 2It. 1 It. 4
Predicted Overall Error
Momed
Prediction of error in the final surrogate

21. t1 t2 t3 t4
Predicted Overall Error
Momed
Momax
Number of Training Points
Prediction of error in the final surrogate
It. 3It. 2It. 1 It. 4
Mode of maximum
error distribution at
each iteration

22. t1 t2 t3 t4
Predicted Overall Error
Momed
Momax
Predicted Maximum Error
Number of Training Points
Prediction of error in the final surrogate
It. 3It. 2It. 1 It. 4

23. Presentation Outline
23
• Review of surrogate model error measurement methods
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

24. Numerical Examples
24
 The effectiveness of the model selection based on REES is explored
to select the best surrogate between all candidates including
(i) Kriging,
(ii) RBF,
(iii) E-RBF, and
(iv) Quadratic Response Surface (QRS)
on four benchmark problems and an engineering design problem.
 The results of the REES method in selecting the best surrogate are
compared with the model selection based on actual errors evaluated
using additional test points, and model selection based on a
normalized PRESS.

25. Numerical Examples
Branin-Hoo (2 variables) Hartmann (6 variables)
Dixon & Price (18 variables)Dixon & Price (12 variables)
VESD regression models used to predict the overall error

26. Numerical Examples
VESD regression models used to predict the maximum error
Dixon & Price (18 variables)Dixon & Price (12 variables)
Branin-Hoo (2 variables) Hartmann (6 variables)

27. Numerical Examples
Wind Farm Power Generation
27
Surrogates are developed using Kriging, RBF, E-RBF, and QRS to
represent the power generation of an array-like wind farm.
VESD used to predict the overall error VESD used to predict the maximum error

28. Numerical Examples
28
Model Selection based the actual error on additional test points
Set Number of additional test points 𝑵 𝒕𝒆𝒔𝒕
for all surrogate candidates, 𝒋 = 𝟏, … , 𝑱
for 𝒊 = 𝟏, … , 𝑵𝒕𝒆𝒔𝒕
𝑦𝑖 = System (𝑥𝑖)
𝑦𝑖 = Surrogate(𝑥𝑖)
𝑅𝐴𝐸𝑖 =
𝑦𝑖 − 𝑦𝑖
𝑦𝑖
Fit a distribution over all RAEs
Determine the mode of the error distribution as an actual error
Select the best surrogate with the smallest error

29. Numerical Examples
29
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
Model selection based on the overall error estimated using REES
Branin-Hoo ERBF Kriging RBF QRS
Hartmann - 6 QRS Kriging RBF ERBF
Dixon and Price - 12 ERBF QRS Kriging RBF
Dixon and Price - 18 ERBF QRS Kriging RBF
Wind Farm Kriging QRS RBF ERBF
Model selection based on the actual error
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
Branin-Hoo ERBF RBF Kriging QRS
Hartmann - 6 QRS Kriging RBF ERBF
Dixon and Price - 12 ERBF Kriging RBF QRS
Dixon and Price - 18 ERBF Kriging RBF QRS
Wind Farm Kriging RBF ERBF QRS

30. Numerical Examples
30
Model Selection based the Normalized Prediction Sum of Square (PRESS)
Set Number of additional training points 𝑵
for all surrogate candidates, 𝒋 = 𝟏, … , 𝑱
for 𝒊 = 𝟏, … , 𝑵
𝑦𝑖 = Surrogate(𝑥𝑖)
𝑅𝐴𝐸 𝐶𝑉
𝑖 =
𝑦−𝑖
𝑖
− 𝑦𝑖
𝑦𝑖
Fit a distribution over all 𝑅𝐴𝐸 𝐶𝑉
Determine the root mean square of errors (RAEs), n-PRESS error
Select the best surrogate with the smallest n-PRESS error
𝑦−𝑖
𝑖
= Intermediate Surrogate(𝑥𝑖)

31. Numerical Examples
31
Model selection based on the n-PRESS
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
Model selection based on the RMSE on additional test points
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
Wind Farm QRS ERBF Kriging RBF
Dixon and Price - 18 QRS ERBF RBF Kriging
Dixon and Price - 12 QRS ERBF Kriging RBF
Hartmann - 6 Kriging QRS RBF ERBF
Branin-Hoo ERBF Kriging RBF QRS
Wind Farm Kriging ERBF RBF QRS
Dixon and Price - 18 ERBF RBF Kriging QRS
Dixon and Price - 12 ERBF RBF Kriging QRS
Hartmann - 6 RBF Kriging ERBF QRS
Branin-Hoo RBF ERBF Kriging QRS

32. Numerical Examples
32
Model selection based on the maximum error estimated using REES
Function
Surrogate model selection
Rank 1 Rank 2 Rank 3 Rank 4
Function
75th percentile of the RMSE
Rank 1 Rank 2 Rank 3 Rank 4
75th percentile of the 𝐑𝐀𝐄 𝐂𝐕
Rank 1 Rank 2 Rank 3 Rank 4
Model selection based on the 75th percentile of the RMSE and 𝐑𝐀𝐄 𝐂𝐕
Wind Farm Kriging QRS ERBF RBF
Dixon and Price - 18 ERBF QRS Kriging RBF
Dixon and Price - 12 ERBF QRS Kriging RBF
Hartmann - 6 RBF Kriging QRS ERBF
Branin-Hoo ERBF RBF Kriging QRS
Wind Farm Kriging QRS ERBF RBF QRS ERBF Kriging RBF
Dixon and Price - 18 ERBF RBF Kriging QRS QRS ERBF RBF Kriging
QRS ERBF Kriging RBF
QRS RBF Kriging ERBF
QRS Kriging RBF ERBF
Dixon and Price - 12 ERBF RBF Kriging QRS
Hartmann - 6 Kriging RBF QRS ERBF
Branin-Hoo ERBF RBF Kriging QRS

33. Numerical Examples
33
% Success of Model Selection (overall error)
REES PRESS
W considering QRS 100% 0%
W/OUT considering QRS 100% 40%
% Success of Model Selection (maximum error)
REES 𝐑𝐀𝐄 𝐂𝐕
W considering QRS 80% 0%
W/OUT considering QRS 80% 40%
A Summary and Comparison

34. Concluding Remarks
34
 We developed a new model selection approach to select the best
surrogate among available surrogate models based on the level of
accuracy.
 The REES method is defined based on the hypothesis that:
“The accuracy of the approximation model is related to the amount
of available resources”
 The REES method proposes two error measures;
Overall error measure,
Maximum error measure
to assess the confidence level of the surrogate model.
 The preliminary results on benchmark and wind farm power generation
problems indicate that in all of cases the REES method selects the best
surrogate with a higher level of confidence in comparison to the n-
PRESS.

35. Acknowledgement
35
 I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Prof.
Souma Chowdhury for their immense help and
support in this research.
 Support from the NSF Awards is also
acknowledged.

36. 36
Thank you
Questions
and
Comments

37. Numerical Examples
37
Numerical setup for test problems