This paper develops a novel approach to characterize the uncertainty in the accuracy of surrogate models. This technique segregates the design domain based on the level of cross-validation errors; the overall framework is called Domain Segmentation based on Uncertainty in the Surrogate (DSUS). The estimated errors are classified into physically meaningful classes based on the user’s understanding of the system and/or the accuracy requirements for the concerned system analysis. In each class, the distribution of the cross-validation errors is estimated to represent the uncertainty in the surrogate. Support Vector Machine (SVM) is implemented to determine the boundaries between error classes, and to classify any new design (point) into a meaningful class. The DSUS framework is illustrated using two different surrogate modeling methods: (i) the Kriging method, and (ii) the Adaptive Hybrid Functions (AHF). We apply the DSUS framework to a series of standard problems and engineering problems. The results show that the DSUS framework can successfully classify the design domain and quantify the uncertainty (prediction errors) in surrogates. More than 90% of the test points could be accurately classified into its error class. In real life engineering design, where we use predictive models with different levels of fidelity, the knowledge of the level of error and uncertainty at any location inside the design space is uniquely helpful.

1. Domain Segmentation based on Uncertainty in the
Surrogate (DSUS)
Jie Zhang*, Souma Chowdhury* and Achille Messac#
* Rensselaer Polytechnic Institute, Department of Mechanical, Aerospace, and Nuclear Engineering
# Syracuse University, Department of Mechanical and Aerospace Engineering
53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics
and Materials Conference
8th AIAA Multidisciplinary Design Optimization Specialist Conference
April 23 – 26, 2012
Honolulu, Hawaii

2. Introduction
• Since a surrogate model is an approximation to an unknown
function, prediction errors are generally present in the
estimated function values.
• The two major sources of uncertainty in surrogate modeling
are:
• Uncertainty in the observations (when they are noisy), and
• Uncertainty due to finite sample.
• One of the major challenges in surrogate modeling is to
accurately quantify these uncertainties.
2

3. Applications
3
Art
Chemistry
Math Automotive
Biology
Geology Data mining Material Science
Source: Google Images

4. Research Motivation
 Surrogate model can be used with more confidence if
4
we can do two things:
 Quantify the uncertainty in the surrogate model; and
 Characterize how the levels of errors vary in the variable
space.
 Most existing methods to model the uncertainty in
surrogate are model-dependent.

5. Research Objectives
5
 Develop a framework to characterize the uncertainty in
the surrogate, which can
• Characterize the uncertainty during the surrogate modeling process
itself, and
• Be applicable to a majority of interpolative/regression surrogate
models.

6. Why Domain Segmentation?
 In surrogate-based optimization: optimal solutions in regions with smaller
errors are more reliable than solutions in regions with larger errors. The
domain segmentation technique can quantify the uncertainty in the optimal
solutions based on their locations in the design space.
Wind Farm Layout Optimization
 In surrogate-based system analysis: the knowledge of the errors and
uncertainties in the surrogate is helpful for decision making by the
user/designer.
Wind Farm Cost Model
 In surrogate accuracy improvement: the design-domain-based uncertainty
6
information can be used to implement adaptive sampling strategies.

7. Presentation Outline
 Uncertainty in surrogate overview
 Domain Segmentation based on Uncertainty in the Surrogate
(DSUS)
 Overview of the DSUS framework
 Surrogate modeling
 Cross-validation
 Pattern classification
 Numerical examples: results and discussion
 Concluding remarks
7

8. Uncertainty in Surrogate Review
8
 Uncertainty in Surrogates
 Bayesian approach (Kennedy and O’Hagan, Apley et al., Xiong et al.)
 Adding bias using constant or pointwise margins (Picheny)
 Reliability Based Design Optimization (RBDO) (Neufeld et al.)
 Efficient Global Optimization (EGO) (Jones et al.)
 Sequential Kriging Optimization (SKO) (Huang et al.)
 Importation of uncertainty estimates from one surrogate to another
(Viana and Haftka)

9. Domain Segmentation based on Uncertainty in the Surrogate (DSUS)
9
• Based on the current level of
knowledge regarding the problem,
the designer may know what
levels of errors are acceptable for
particular design purposes.
Wind Farm Power Generation Model
Error Decision
< 2% error Desirable
2-10% error Acceptable
> 10% error Unacceptable
• The designer can estimate the
confidence of a new design, based on
the region into which the design
point is classified.
• These regions can correspond to
“good”, “acceptable”, and
“unacceptable” levels of accuracy.

10. Development of the DSUS
10
DSUS Key features:
I.Segregates the design domain into
regions based on the level of errors
(or level of fidelity).
II.Classifies any new point/design,
for which the actual functional
response is not known, into an error
class, and quantify the uncertainty in
its predicted function response.
III.Is readily applicable to a majority
of interpolative surrogate models.
The term "prediction uncertainty" denotes the
distribution of errors of the surrogate.

11. Cross-Validation
 Two popular strategies: (i) leave-one-out; and (ii) q-fold.
 In order to obtain the error at each training point, the leave-one-
out strategy is adopted in the DSUS framework.
 The Relative Accuracy Error (RAE) is used to classify the
training points into classes.
11
Actual
function value
Estimated value
by surrogate

12. Classifying the Training Points into Error Classes
• According to the RAE values, we classify the training points into error
classes, and define the lower and upper bounds of each class.
12
Class Design variable RAE
1 0.573 0.018
2 0.277 0.691
1 0.044 0.045
1 0.371 0.018
2 0.910 0.345
2 0.767 0.116
1 0.720 0.043
1 0.865 0.060
1 0.508 0.073
2 0.637 0.316
2 0.977 1.078
1 0.240 0.013
1 0.184 0.019
1 0.107 0.004
1 0.453 0.049
Class 1
RAE < 10%
Class 2
RAE > 10%

13. Pattern Classification
 A wide variety of pattern classification methods are available:
 Linear discriminant analysis (LDA);
 Principal components analysis (PCA);
 Kernel estimation and K-nearest-neighbor algorithms;
 Perceptrons;
 Neural Network; and
 Support Vector Machine (SVM) 1,2.
a competitive approach for multiclass
13
classification problem
1. Basudhar and Missoum; 2. Sakalkar and Hajela

14. Support Vector Machine (SVM)
 Four kernels are popularly used:
 Linear:
 Polynomial:
 Radial basis function:
 Sigmoid:
14
We have used an efficient SVM package, LIBSVM (A Library
for Support Vector Machines), developed by Chang and Lin.
One-against-one
classification

15. Classifying New Designs
15

16. Case Studies
 The DSUS framework is illustrated using two different surrogates.
 The Kriging; and
 The Adaptive Hybrid Functions (AHF).
 We apply the DSUS framework to a series of standard problems.
 1-variable function;
 2-variable Dixon & Price function;
 The Response Surface-Based Wind Farm Cost (RS-WFC) model; and
 The commonality representation in product family design.
16

17. Kriging
17
 The kriging approximation function consists of two parts: (i) a
global trend function, and (ii) a functional departure from the trend
function.
 In this paper, we use the
DACE (Design and Analysis
of Computer Experiments),
developed by Dr. Nielsen.

18. Adaptive Hybrid Functions (AHF)
 Determination of a trust region:
numerical bounds of the estimated
parameter (output) as functions of
the independent parameters (input).
 Definition of a local measure of
accuracy (using kernel functions) of
the estimated function value, and
representation of the corresponding
distribution parameters as functions
of the input vector.
 Weighted summation of different
surrogate models based on the local
measure of accuracy.
18

19. Onshore Wind Farm Cost Model
19
 Response Surface-BasedWind Farm Cost (RS-WFC) model
 The inputs for the surrogate model are
 The number, and
 The rated power of wind turbines.
 The output of the surrogate model is
 Total annual cost of a wind farm
 Why DSUS?
Surrogate:
 DSUS can quantify the errors and uncertainties in the predicted wind farm cost.
 The farm designer/analyst can estimate the uncertainty in the payback period and
the associated risk of the wind farm project.
Cost  f (N,P0)

20. Commonality Representation in Product Family Design
• This commonality index is essentially based on the ratio of “the number
of unique parts” to “the total number of parts” in the product family.
• For a n-variable scalable product family, the commonality matrix is
replaced by a tractable set of n integer variables.
• Each of these integer commonality variables belong to the feasible set of
integers: Z = [0, 1, . . . , 2N(N−1)/2−1].
• We develop a surrogate model to represent the commonality index as a
function of the integer variables.
Universal Motor No. of variables Integers: Z
2 products 7 [0, 1]
3 products 7 [0, 1, 2, 4, 7]
1 2 7 Surrogate: CI  f (x , x , , x )
* Chowdhury et al., 2011 20

21. SVM Kernels and Predefined Classes Bounds
21
Numerical setup for test problems
The uncertainty scale in each class

22. Representation of Prediction Uncertainty
• Gaussian distribution is adopted to represent the uncertainty in the
prediction accuracy of the surrogate.
• For any new point (design) candidate, the DSUS framework can classify
that point into one of these error classes.
22
Uncertainty scale of each class with AHF surrogate

23. Representation of Prediction Uncertainty
23
• Gaussian distribution is adopted to represent the uncertainty in the
prediction accuracy of the surrogate.
• For any new point (design) candidate, the DSUS framework can classify
that point into one of these error classes.
Uncertainty scale of each class with Kriging surrogate

24. Representation of Prediction Uncertainty
24
1-variable function
Dixon & Price function

25. Representation of Prediction Uncertainty
25
Wind farm cost model
Commonality matrix (2 products)

26. Representation of Prediction Uncertainty
26
Commonality matrix (3 products)
• It is observed that the prediction uncertainties vary significantly from problem to
problem.
• The standard deviation of class 3 for all test problems is generally larger than that
of classes 1 and 2.
 There is significant scope to improve the surrogate accuracies in class 3 regions.
 This design-space information pushes the paradigm in advanced surrogate
modeling and surrogate-based optimization.

27. Uncertainty Prediction Results
27
Classification accuracy of each problem
Problem AHF Kriging
1-variable function 90% (18/20) 95% (19/20)
Dixon & Price function 75% (15/20) 100% (20/20)
Wind farm cost 100% (20/20) 95% (19/20)
Commonality (2 products) 94% (33/35) 100% (35/35)
Commonality (3 products) 95% (20/21) 86% (18/21)

28. Classifying New Designs
28

29. Uncertainty Prediction Results
29
Classification accuracy of each problem
Problem AHF Kriging
1-variable function 90% (18/20) 95% (19/20)
Dixon & Price function 75% (15/20) 100% (20/20)
Wind farm cost 100% (20/20) 95% (19/20)
Commonality (2 products) 94% (33/35) 100% (35/35)
Commonality (3 products) 95% (20/21) 86% (18/21)
• The classification accuracy of the DSUS prediction is more than 90% for
most test problems.
Uncertainty characterization for new designs (wind farm cost)
New design No. of turbines Rated Power Class Uncertainty (μ, σ)
1 40 1.25 MW 1 0.0018, 0.0011
2 7 1 MW 2 0.0066, 0.0013
3 44 1.5 MW 3 0.0216, 0.0102

30. Conclusion
• We developed a method to characterize the uncertainty attributable to
surrogate models.
• In this framework, the whole design domain can be divided into
physically meaningful classes.
• The DSUS is applicable to a majority of surrogate models.
• The results showed that the DSUS framework can successfully
characterize and quantify the uncertainty (predictive errors) in surrogates.
• Future research should investigate:
• the training points balance between surrogate modeling and pattern classification;
• the physical implications of error class definitions; and
• the selection of surrogate and pattern classification tools for the DSUS framework.
30

31. Acknowledgement
• I would like to acknowledge my research adviser
Prof. Achille Messac, for his immense help and
support in this research.
• I would also like to thank my friend and colleague
Souma Chowdhury for his valuable contributions to
this paper.
• Support from the NSF Awards is also
acknowledged.
31

32. Selected References
1. Keane, A. J. and Nair, P. B., Computational Approaches for Aerospace Design: The Pursuit of Excellence, John Wiley and
Sons, 2005.
2. Basudhar, A. and Missoum, S., “Adaptive Explicit Decision Functions for Probabilistic Design and Optimization Using Support
Vector Machines,” Computers and Structures, Vol. 86, No. 19-20, 2008, pp. 1904–1917.
3. Forrester, A. and Keane, A., “Recent Advances in Surrogate-based Optimization,” Progress in Aerospace Sciences, Vol. 45, No.
1-3, 2009, pp. 50–79.
4. Wang, G. and Shan, S., “Review of Metamodeling Techniques in Support of Engineering Design Optimization,” Journal of
Mechanical Design, Vol. 129, No. 4, 2007, pp. 370–380.
5. Simpson, T., Toropov, V., Balabanov, V., , and Viana, F., “Design and Analysis of Computer Experiments in Multidisciplinary
Design Optimization: A Review of How Far We Have Come or Not,” 12th AIAA/ISSMO Multidisciplinary Analysis and
Optimization Conference, Victoria, Canada, September 10-12 2008.
6. Zhang, J., Chowdhury, S., and Messac, A., “An Adaptive Hybrid Surrogate Model,” Structural and Multidisciplinary
Optimization, 2012, doi: 10.1007/s00158-012-0764-x.
7. Forrester, A., Sobester, A., and Keane, A., Engineering Design via Surrogate Modelling: A Practical Guide, Wiley, 2008.
8. Apley, D. W., Liu, J., and Chen, W., “Understanding the Effects of Model Uncertainty in Robust Design With Computer
Experiments,” ASME Journal of Mechanical Design, Vol. 128, No. 4, 2006, pp. 945(14 pages).
9. Neufeld, D. and an J. Chung, K. B., “Aircraft Wing Box Optimization Considering Uncertainty in Surrogate Models,”
Structural and Multidisciplinary Optimization, Vol. 42, No. 5, 2010, pp. 745–753.
10. Viana, F. A. C. and Haftka, R. T., “Importing Uncertainty Estimates from One Surrogate to Another,” 50th
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Palm Springs, California, May 4-6
2009.
32

33. 33

34. RBHF Framework
 Step A  Step A.1: Determination of the Base Model
34
 Determination of a trust region:
numerical bounds of the estimated
parameter (output) as functions of
the input vector over the feasible
input space.
 Characterization of the local
reliability (using probability
distribution functions) of the
estimated function value.

35.  Wherever the density of training points is high, it is expected that the interpolative function
(in that region) has lower errors.
35
 Step A.2: Formulation of Crowding Distance-Based Trust Region (CD-TR)
 Crowding distance is used to evaluate the
density of points
 A parameter ρ is defined to represent the local
density of input data, as given by
 The adaptive distance:
RBHF Framework
It is important to note that the CD-TR estimation is particularly useful for recorded or measured
data based (commercial or experimental data) surrogate modeling. In the case of problems, where
the user has control over sampling (simulation-based), the initial sample data is expected to be
relatively evenly distributed; significant variation in crowding distance might not be observed.

36. 36
RBHF Framework
 Step A.3: Estimation of Probability Distributions
 We develop a Reliability Measure of Surrogate
Modeling, to represent the reliability in the
estimated function value.
 The probability density function (pdf) represents
the distribution of the estimated outputs of
component surrogates.
 The corresponding typical pdf coefficients are
represented as functions of the input vector,
thereby characterizing the reliability of the
estimated function over the entire input domain.
 Gaussian pdf is adopted here
a = 1

37. RBHF Framework
37
 We assume that the reliability of the estimated function value (pdf) is a maximum of
one at the actual output value y(xi); and, a minimum of 0.1 at the trust region
boundaries. The probability distribution is represented as
 σ1 and σ2 are controlled by the full width at one tenth maximum (Δz10), given by
and
where

38. Step B: Component Surrogates Development
 In this paper, the RBHF integrates:
 Kriging method
 Radial Basis Functions (RBF)
 Extended Radial Basis Functions (E-RBF)
38

39. Radial Basis Functions
39
 Radial Basis Functions
 The RBFs are expressed in terms of the Euclidean distance,
 (r)  r2 c2
where 0 c  is a prescribed parameter.
 
 
f x  x x
1
( )
np
i
i
i

i r  x  x
 One of the most effective forms is the multiquadric function:
 The final approximation function is a linear combination of these basis
functions across all data points.

40. Extended Radial Basis Function (E-RBF)
40
 Extended Radial Basis Functions (E-RBF) is a combination of Radial
Basis Functions (RBFs) and Non-Radial Basis Functions (N-RBFs).
 Non-Radial Basis Functions
 N-RBFs are functions of individual coordinates of generic points x
relative to a given data point xi, in each dimension separately
 It is composed of three distinct components
E-RBF
 The E-RBF approach incorporates both the RBFs and the N-RBFs
 Methods: (i) linear programming, or (ii) pseudo inverse.

41. Numerical Settings
41
 Sampling Strategies: Latin Hypercube Sampling
 Selection of Parameters
Method Parameter value
E-RBF λ = 4.75; c = 0.9; t = 2
RBF c = 0.9
Kriging θl = 0.1; θu = 20
 Performance Criteria
• Root Mean Squared Error (RMSE)
• Maximum Absolute Error (MAE)

42. Numerical Settings
42
Function No. of
variables
No. of
training points
No. of
test points
Test function 1 1 5 100
Test function 2 2 36 441
Goldstein & Price 2 36 1681
Branin-Hoo 2 36 961
Hartmann-3 3 49 216
Hartmann-6 6 150 729

43. 43
 Test Function 1: 1-Variable Function
 Test Function 2: 2-Variable Function
 Test Function 3: Goldstein & Price Function
 Test Function 4: Branin-Hoo Function
 Test Function 5 and 6: Hartmann Function

44. AHF: Determining Local Weights
44
 The AHF is a weighted summation of function values estimated by the component
surrogates:
 The weights are expressed in terms of the estimated measure of accuracy, expressed as
where, Pi(x) is the measure of accuracy of the ith surrogate for point x.