SlideShare a Scribd company logo
Quantifying Regional Error in Surrogates by Modeling its 
Relationship with Sample Density 
Ali Mehmani, Souma Chowdhury , Jie Zhang, Weiyang Tong, 
and Achille Messac 
Syracuse University, Department of Mechanical and Aerospace Engineering 
54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and 
Materials Conference 
April 8-11, 2013, Boston, Massachusetts
Surrogate model 
• Surrogate models are commonly used for providing a tractable and 
inexpensive approximation of the actual system behavior in many 
routine engineering analysis and design activities: 
2
3 
Broad Research Question 
Lower Fidelity How Low? 
Structural Blade Design 
ANSYS, Inc. 
Surrogate Model 
Expensive Inexpensive
4 
Broad Research Question 
How to quantify the level of the surrogate accuracy ? 
 further improvement of the surrogate, 
 domain exploration, 
 assessing the reliability of the optimal design, 
 quantifying the uncertainty associated with the surrogate, 
 construction of a weighted surrogate model, and 
 …
Research Objective 
 Develop a reliable method to quantify the surrogate error, 
5 
 This method should have the following characteristics: 
 model independent 
 no additional system evaluations 
 local/global error measurement 
 quantify the error of the actual surrogate
Regional Error Estimation of Surrogate 
6
Regional Error Estimation of Surrogate 
7 
(REES)
Presentation Outline 
• Review of surrogate model error measurement methods 
8 
• Relation of surrogate accuracy with sample density 
• Regional Error Estimation of Surrogate 
• Numerical examples: benchmark and an engineering 
design problems
Presentation Outline 
• Review of surrogate model error measurement methods 
9 
• Relation of surrogate accuracy with sample density 
• Regional Error Estimation of Surrogate 
• Numerical examples: benchmark and an engineering 
design problems
Surrogate Model Error Measurement Methods 
 Error quantification methods can be classified, 
 based on their computational expense, into methods that require 
additional data, and methods that use existing data. 
 based on the region of interest, into: 
- Global error measure 
(e.g., split sample, cross-validation, Akaike’s information criterion, and 
bootstrapping). 
- Local or point-wise error measure 
(e.g., the mean squared errors for Kriging and the linear reference model 
(LRM).) 
10
Surrogate Model Error Measurement Methods 
 Error metrics, 
11 
• The mean squared error (MSE) (or root mean square error (RMSE) ) 
• The maximum absolute error (MAE) 
• The relative absolute error (RAE) 
actual values on ith test point 
predicted values on ith test point
Surrogate Model Error Measurement Methods 
 Error metrics, 
• The prediction sum of square (PRESS) is based on the leave-one-out 
12 
cross-validation error 
• The root mean square of PRESS (PRESSRMS) based on the k-fold cross-validation. 
• The relative absolute error of cross-validation (RAECV) based on leave-one- 
out approach.
Presentation Outline 
• Review of surrogate model error measurement methods 
13 
• Relation of surrogate accuracy with sample density 
• Regional Error Estimation of Surrogate 
• Numerical examples: benchmark and an engineering 
design problems
Methodology: Concept 
14 
Model accuracy ∝ Available resources 
In general, this concept can be applied for different methodologies 
- Surrogate modeling, 
- Finite Element Analysis, and 
- ...
Methodology: Concept 
15 
 Finite Element Analysis (numerical methods) 
The finer mesh, the stresses are more precise due to the larger 
number of elements 
coarse mesh 
(4 solid brick element) 
medium mesh 
(32 solid brick element) 
fine mesh 
(256 solid brick element) 
Estimate total shear force and flexural moment at vertical 
sections using Finite Element Analysis.
Methodology: Concept 
 Surrogate (mathematical model) 
Surrogate accuracy generally improves with increasing training points. 
3 training points 
9 training points 
7 training points 
The location of additional points has 
strong impact on surrogate accuracy. 
This impact is highly problem and model 
dependent. 
16
Presentation Outline 
• Review of surrogate model error measurement methods 
17 
• Relation of surrogate accuracy with sample density 
• Regional Error Estimation of Surrogate 
• Numerical examples: benchmark and an engineering 
design problems
Methodology: REES 
 The REES method formulates the variation of error as a 
18 
function of training points using intermediate surrogates. 
 This formulation is used to predict the level of error in a 
final surrogate.
Methodology: REES 
Step 1 : Generation of sample data 
The entire set of sample points is represented by 푿 . 
Step 2 : Identification of sample points inside/outside region of interest 
{푿풊풏} 
{푿풐풖풕} 
푿 = 푿풊풏 + 푿풐풖풕 
푿풊풏 : Inside-region data set 
푿풐풖풕 : Outside-region data set 
user-defined region of interest 
19
Methodology: REES 
Step 3 : Estimation of the variation of the error with sample density 
Outside-Training region Point 
point 
Inside-region point 
user-defined region 
of interest 
First Iteration : 
Test Point 
Second iteration : 
Third iteration : 
Final Surrogate : 
Training Point
Methodology: REES 
Step 3 : Estimation of the variation of the error with sample density 
 A position of sample points which are selected as training 
points, at each iteration, is critical to the surrogate accuracy. 
 The proposed error measure should be minimally sensitive to 
the location of the test points at each iteration. 
21 
 Intermediate surrogates are 
iteratively constructed (at each 
iteration) over a sample set 
comprising all samples outside the 
region of interest and heuristic 
subsets of samples inside the region 
of interest.
Methodology: REES 
Step 3 : Estimation of the variation of the error with sample density 
 The intermediate subset for each 
 The number of iterations (푁푖푡) is defined 
combination at specific iteration is defined 
by 
{휷풌} ⊂ 푿풊풏 
#{휷풌} = 풏풕, 풏풕−ퟏ < 풏풕 
풌 = 1,2, … , 퐾푡 
- dimension of a problem, 
- number of inside sample points, and 
- preference of the user 
 The intermediate training points and test 
points for each combination at each 
iteration is defined by 
 The number of sample combinations 
(푲풕 ) is defined, 
푿푻푹 = 푿풐풖풕 + 휷풌 
푿푻푬 = 푿 − 푿푻푹 
 The intermediate surrogates 
푓푘 , 풌 = ퟏ, ퟐ, . . , 푲풕 
are constructed for all combinations using the 
intermediate training points ( 푿푻푹 ), and are 
tested over the intermediate test points ( 푿푻푬 ).
Methodology: REES 
Step 3 : Estimation of the variation of the error with sample density 
 The median and the maximum errors are 
estimated for each combination 
풎풕: the number of test points in tth iteration 
풆: the RAE value estimated on intermediate test points 
23
Methodology: REES 
Step 3 : Estimation of the variation of the error with sample density 
 The median and the maximum errors are 
estimated for each combination 
The median is a useful measures of central 
tendency which is less vulnerable to outliers. 
24 
Median error 
Overall Fidelity Information 
Maximum error 
Minimum Fidelity Information
Methodology: REES 
Step 3 : Estimation of the variation of the error with sample density 
 Probabilistic models are developed using 
The mode of distribution is selected to 
represent the errors at each iteration. 
a lognormal distribution to represent 
median and maximum errors estimated 
over all 푲풕 combinations at each 
iterations. 
Mode of median error distribution 
Mode of maximum error distribution 
 These values are used to relate the 
variation of the surrogate error with 
number of training points (sample 
density).
The relation of the error with sample density 
 12-D Test Problem (Dixon & Price, n=12) 
Number of sample points # 푿 = ퟓퟓퟎ, Number of inside sample points # 푿풊풏 = # 푿 
Number of training points at each iteration,풏풕 = 5푡 + 50, 푡 = 1,2, … , 70 
Number of sample combination, 푲풕 = 500 
Number of Training Points 
MOmax 
# 푿푻푹 = ퟓퟓ 
# 푿푻푬 = ퟒퟒퟓ 
Number of Training Points 
Estimated mode of median errors Estimated mode of maximum errors 
MOmed 
First iteration 
Last iteration 
# 푿푻푹 = ퟒퟎퟎ 
# 푿푻푬 = ퟏퟎퟎ
The relation of the error with sample density 
 12-D Test Problem (Dixon & Price, n=12) 
Number of Training Points 
Meanmean 
Number of Training Points 
Estimated mode of median errors Estimated mean of mean errors 
MOmed 
REES Method Normalized k-fold CV
Methodology: REES 
Step 4 : Prediction of regional error in the final surrogate 
 The final surrogate model is constructed using the full set of training data. 
 Regression models are applied to relate 
- the statistical mode of the median error distribution(푴풐풎풆풅) 
- the statistical mode of the maximum error distributions(푴풐풎풂풙), and 
- the absolute maximum error (푨푩푺풎풂풙) 
at each iteration to the size of the inside-region training points (nt), 
 These regression models are called the variation of error with sample density 
(VESD). 
The regression models are used to predict the level of the 
error in the final surrogate within the region of interest. 
28
Methodology: REES 
Modeling the Variation of Regional Error with Training Point Density 
 In this study, three types of the regression functions are used to represent 
the variation of regional error with respect to the inside-region training points 
Exponential regression model 
Multiplicative regression model 
Linear regression model 
 The choice of these functions assume a smooth monotonic decrease of the 
regional error with the training point density within that region. 
 The root mean squared error metric is used to select the best-fit regression 
model 
29
Presentation Outline 
• Review of surrogate model error measurement methods 
30 
• Relation of surrogate accuracy with sample density 
• Regional Error Estimation of Surrogate 
• Numerical examples: benchmark and an engineering 
design problems
Numerical Examples 
 The effectiveness of the REES method is explored for applications with 
- Kriging, 
- Radial Basis Functions (RBF), 
- Extended Radial Basis Functions (E-RBF), and 
- Quadratic Response Surface (QRS). 
 To evaluate practical and numerical efficiencies of the REES method, 
three benchmark problems and an engineering design problem are tested. 
 The error evaluated using REES, and the relative absolute error given by 
leave-one-out cross-validation (푹푨푬풄풗) are compared with the actual 
error evaluated using relative absolute error on additional test 
points (푹푨푬풂풄풕풖풂풍). 
31
Median of RAEs 
Numerical Examples 
Results and Discussion 
VESD regression models within the region of interest of surrogate models 
constructed for the Branin-Hoo Function to predict, 
Distribution of 
median errors 
Mode of the median error 
distribution, 
Predicted mode of median error 
in the final surrogate, 
VESDmed 
Number of Inside-region Training Points 
32
Numerical Examples 
Results and Discussion 
VESD regression models 
within the region of interest of 
surrogate models constructed 
for the Branin-Hoo Function 
to predict, 
Type and coefficients of 
VESDmed 
Kriging RBF 
E-RBF QRS
Maximum of RAEs 
Numerical Examples 
Results and Discussion 
VESD regression models within the region of interest of surrogate models constructed for the 
Branin-Hoo Function to predict the mode of maximum ( ) and the absolute 
maximum ( ) error. 
Distribution of 
maximum errors 
Mode of the maximum 
error distribution, 
Absolute maximum error 
Predicted absolute 
maximum error in 
the final surrogate 
Predicted mode of 
maximum error in 
the final surrogate 
34 Number of Inside-region Training Points
Numerical Examples 
Results and Discussion 
VESD regression models within the region of interest of surrogate models constructed for the 
Branin-Hoo Function to predict the mode of maximum ( ) and the absolute 
maximum ( ) error. 
Type and coefficients of VESDmax 
Type and coefficients of VESDABS 
Kriging RBF 
E-RBF QRS
Numerical Examples 
Wind Farm Power Generation 
Surrogates are developed using Kriging, RBF, E-RBF, and QRS to 
represent the power generation of an array-like wind farm. 
36
Numerical Examples 
Results and Discussion 
VESD regression models in different surrogates for the wind farm power 
generation problem 
It. 1 It. 2 It. 3 It. 4 Predicted Error 
37
Numerical Examples 
38 
Results and Discussion 
The closer to one, the better the corresponding error measure. 
predicted mode of median errors 
median of RAEs evaluated on test 
points 
median of relative absolute 
errors of cross-validation
Concluding Remarks 
 We developed a new method to quantify surrogate error based on the 
hypothesis that: 
“The accuracy of the approximation model is related to the amount 
of available resources” 
 This relationship can be reliably quantified when the error measures is 
less sensitive to sample locations or a type of application. 
 The REES method addresses this issue. 
 The preliminary results on benchmark and wind farm power generation 
problems indicate that in majority of cases the REES method is more 
accurate than other measures. 
39 
It is not possible using any existing methods
Future Works 
 The scope for improvement the method 
 The implementation of the proposed error measurement in 
surrogate developments. 
40
Acknowledgement 
41 
 I would like to acknowledge my research adviser 
Prof. Achille Messac, and my co-adviser Prof. 
Souma Chowdhury for their immense help and 
support in this research. 
 Support from the NSF Awards is also acknowledged.
42 
Thank you 
Questions 
and 
Comments
Median of RAEs 
Numerical Examples 
Results and Discussion 
VESD regression models within the region of interest of surrogate models 
constructed for the Branin-Hoo Function to predict, 
Distribution of 
median errors 
k-fold CV 
Mode of the median error 
distribution, 
Predicted mode of median error 
in the final surrogate, 
VESDmed 
Number of Inside-region Training Points 
43 
Mean mean 
Number of Inside-region Training Points

More Related Content

What's hot

Orthogonal array
Orthogonal arrayOrthogonal array
Orthogonal array
ATUL RANJAN
 
COSMOS-ASME-IDETC-2014
COSMOS-ASME-IDETC-2014COSMOS-ASME-IDETC-2014
COSMOS-ASME-IDETC-2014
OptiModel
 
Cross-validation aggregation for forecasting
Cross-validation aggregation for forecastingCross-validation aggregation for forecasting
Cross-validation aggregation for forecasting
Devon Barrow
 
Lecture7 cross validation
Lecture7 cross validationLecture7 cross validation
Lecture7 cross validationStéphane Canu
 
Classification using L1-Penalized Logistic Regression
Classification using L1-Penalized Logistic RegressionClassification using L1-Penalized Logistic Regression
Classification using L1-Penalized Logistic Regression
Setia Pramana
 
Orthogonal array approach a case study
Orthogonal array approach   a case studyOrthogonal array approach   a case study
Orthogonal array approach a case study
Karthikeyan Rajendran
 
Performance Evaluation for Classifiers tutorial
Performance Evaluation for Classifiers tutorialPerformance Evaluation for Classifiers tutorial
Performance Evaluation for Classifiers tutorialBilkent University
 
Taguchi design of experiments nov 24 2013
Taguchi design of experiments nov 24 2013Taguchi design of experiments nov 24 2013
Taguchi design of experiments nov 24 2013
Charlton Inao
 
Performance analysis of logic
Performance analysis of logicPerformance analysis of logic
Performance analysis of logic
ijcsa
 
Sampling techniques 2
Sampling techniques 2Sampling techniques 2
Sampling techniques 2
ssuser0903d3
 
Orthogonal array testing
Orthogonal array testingOrthogonal array testing
Orthogonal array testingPrince Bhanwra
 
IDA 2015: Efficient model selection for regularized classification by exploit...
IDA 2015: Efficient model selection for regularized classification by exploit...IDA 2015: Efficient model selection for regularized classification by exploit...
IDA 2015: Efficient model selection for regularized classification by exploit...
George Balikas
 
Sampling Theory Part 3
Sampling Theory Part 3Sampling Theory Part 3
Sampling Theory Part 3
FellowBuddy.com
 
Essentials of monte carlo simulation
Essentials of monte carlo simulationEssentials of monte carlo simulation
Essentials of monte carlo simulationSpringer
 
Quality of Analytical Procedures
Quality of Analytical ProceduresQuality of Analytical Procedures
Quality of Analytical Procedures
Brendon Naicker
 
Unit 6 input modeling
Unit 6 input modeling Unit 6 input modeling
Unit 6 input modeling
raksharao
 
Sampling Theory Part 2
Sampling Theory Part 2Sampling Theory Part 2
Sampling Theory Part 2
FellowBuddy.com
 

What's hot (19)

Orthogonal array
Orthogonal arrayOrthogonal array
Orthogonal array
 
COSMOS-ASME-IDETC-2014
COSMOS-ASME-IDETC-2014COSMOS-ASME-IDETC-2014
COSMOS-ASME-IDETC-2014
 
Cross-validation aggregation for forecasting
Cross-validation aggregation for forecastingCross-validation aggregation for forecasting
Cross-validation aggregation for forecasting
 
Lecture7 cross validation
Lecture7 cross validationLecture7 cross validation
Lecture7 cross validation
 
Classification using L1-Penalized Logistic Regression
Classification using L1-Penalized Logistic RegressionClassification using L1-Penalized Logistic Regression
Classification using L1-Penalized Logistic Regression
 
Orthogonal array approach a case study
Orthogonal array approach   a case studyOrthogonal array approach   a case study
Orthogonal array approach a case study
 
Performance Evaluation for Classifiers tutorial
Performance Evaluation for Classifiers tutorialPerformance Evaluation for Classifiers tutorial
Performance Evaluation for Classifiers tutorial
 
Taguchi design of experiments nov 24 2013
Taguchi design of experiments nov 24 2013Taguchi design of experiments nov 24 2013
Taguchi design of experiments nov 24 2013
 
Performance analysis of logic
Performance analysis of logicPerformance analysis of logic
Performance analysis of logic
 
Ad test
Ad testAd test
Ad test
 
Sampling techniques 2
Sampling techniques 2Sampling techniques 2
Sampling techniques 2
 
Orthogonal array testing
Orthogonal array testingOrthogonal array testing
Orthogonal array testing
 
IDA 2015: Efficient model selection for regularized classification by exploit...
IDA 2015: Efficient model selection for regularized classification by exploit...IDA 2015: Efficient model selection for regularized classification by exploit...
IDA 2015: Efficient model selection for regularized classification by exploit...
 
Sampling Theory Part 3
Sampling Theory Part 3Sampling Theory Part 3
Sampling Theory Part 3
 
Essentials of monte carlo simulation
Essentials of monte carlo simulationEssentials of monte carlo simulation
Essentials of monte carlo simulation
 
Quality of Analytical Procedures
Quality of Analytical ProceduresQuality of Analytical Procedures
Quality of Analytical Procedures
 
Unit 6 input modeling
Unit 6 input modeling Unit 6 input modeling
Unit 6 input modeling
 
Sampling Theory Part 2
Sampling Theory Part 2Sampling Theory Part 2
Sampling Theory Part 2
 
Sampling theory
Sampling theorySampling theory
Sampling theory
 

Similar to PEMF2_SDM_2012_Ali

AIAA-SDM-PEMF-2013
AIAA-SDM-PEMF-2013AIAA-SDM-PEMF-2013
AIAA-SDM-PEMF-2013
OptiModel
 
AIAA-MAO-RegionalError-2012
AIAA-MAO-RegionalError-2012AIAA-MAO-RegionalError-2012
AIAA-MAO-RegionalError-2012
OptiModel
 
AIAA-MAO-DSUS-2012
AIAA-MAO-DSUS-2012AIAA-MAO-DSUS-2012
AIAA-MAO-DSUS-2012
OptiModel
 
AIAA-SDM-SequentialSampling-2012
AIAA-SDM-SequentialSampling-2012AIAA-SDM-SequentialSampling-2012
AIAA-SDM-SequentialSampling-2012
OptiModel
 
WCSMO-ModelSelection-2013
WCSMO-ModelSelection-2013WCSMO-ModelSelection-2013
WCSMO-ModelSelection-2013
OptiModel
 
computer application in pharmaceutical research
computer application in pharmaceutical researchcomputer application in pharmaceutical research
computer application in pharmaceutical research
SUJITHA MARY
 
dimension reduction.ppt
dimension reduction.pptdimension reduction.ppt
dimension reduction.ppt
Deadpool120050
 
2. visualization in data mining
2. visualization in data mining2. visualization in data mining
2. visualization in data mining
Azad public school
 
APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...
APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...
APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...
Shibaprasad Bhattacharya
 
Slides sem on pls-complete
Slides sem on pls-completeSlides sem on pls-complete
Slides sem on pls-complete
Dr Hemant Sharma
 
NEURAL Network Design Training
NEURAL Network Design  TrainingNEURAL Network Design  Training
NEURAL Network Design TrainingESCOM
 
SEM
SEMSEM
Probability density estimation using Product of Conditional Experts
Probability density estimation using Product of Conditional ExpertsProbability density estimation using Product of Conditional Experts
Probability density estimation using Product of Conditional ExpertsChirag Gupta
 
Adaptive response surface by kriging using pilot points for structural reliab...
Adaptive response surface by kriging using pilot points for structural reliab...Adaptive response surface by kriging using pilot points for structural reliab...
Adaptive response surface by kriging using pilot points for structural reliab...
IOSR Journals
 
Parameter Optimisation for Automated Feature Point Detection
Parameter Optimisation for Automated Feature Point DetectionParameter Optimisation for Automated Feature Point Detection
Parameter Optimisation for Automated Feature Point DetectionDario Panada
 
Machine learning with scikitlearn
Machine learning with scikitlearnMachine learning with scikitlearn
Machine learning with scikitlearn
Pratap Dangeti
 
IEOR 265 Final Paper_Minchao Lin
IEOR 265 Final Paper_Minchao LinIEOR 265 Final Paper_Minchao Lin
IEOR 265 Final Paper_Minchao LinMinchao Lin
 
3 SAMPLING LATEST.pptx
3 SAMPLING LATEST.pptx3 SAMPLING LATEST.pptx
3 SAMPLING LATEST.pptx
ssuser504dda
 

Similar to PEMF2_SDM_2012_Ali (20)

AIAA-SDM-PEMF-2013
AIAA-SDM-PEMF-2013AIAA-SDM-PEMF-2013
AIAA-SDM-PEMF-2013
 
AIAA-MAO-RegionalError-2012
AIAA-MAO-RegionalError-2012AIAA-MAO-RegionalError-2012
AIAA-MAO-RegionalError-2012
 
AIAA-MAO-DSUS-2012
AIAA-MAO-DSUS-2012AIAA-MAO-DSUS-2012
AIAA-MAO-DSUS-2012
 
AIAA-SDM-SequentialSampling-2012
AIAA-SDM-SequentialSampling-2012AIAA-SDM-SequentialSampling-2012
AIAA-SDM-SequentialSampling-2012
 
WCSMO-ModelSelection-2013
WCSMO-ModelSelection-2013WCSMO-ModelSelection-2013
WCSMO-ModelSelection-2013
 
computer application in pharmaceutical research
computer application in pharmaceutical researchcomputer application in pharmaceutical research
computer application in pharmaceutical research
 
dimension reduction.ppt
dimension reduction.pptdimension reduction.ppt
dimension reduction.ppt
 
2. visualization in data mining
2. visualization in data mining2. visualization in data mining
2. visualization in data mining
 
APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...
APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...
APPLICATION OF STATISTICAL LEARNING TECHNIQUES AS PREDICTIVE TOOLS FOR MACHIN...
 
Slides sem on pls-complete
Slides sem on pls-completeSlides sem on pls-complete
Slides sem on pls-complete
 
NEURAL Network Design Training
NEURAL Network Design  TrainingNEURAL Network Design  Training
NEURAL Network Design Training
 
SEM
SEMSEM
SEM
 
Probability density estimation using Product of Conditional Experts
Probability density estimation using Product of Conditional ExpertsProbability density estimation using Product of Conditional Experts
Probability density estimation using Product of Conditional Experts
 
MyPoster1
MyPoster1MyPoster1
MyPoster1
 
Adaptive response surface by kriging using pilot points for structural reliab...
Adaptive response surface by kriging using pilot points for structural reliab...Adaptive response surface by kriging using pilot points for structural reliab...
Adaptive response surface by kriging using pilot points for structural reliab...
 
Parameter Optimisation for Automated Feature Point Detection
Parameter Optimisation for Automated Feature Point DetectionParameter Optimisation for Automated Feature Point Detection
Parameter Optimisation for Automated Feature Point Detection
 
Machine learning with scikitlearn
Machine learning with scikitlearnMachine learning with scikitlearn
Machine learning with scikitlearn
 
IEOR 265 Final Paper_Minchao Lin
IEOR 265 Final Paper_Minchao LinIEOR 265 Final Paper_Minchao Lin
IEOR 265 Final Paper_Minchao Lin
 
3 SAMPLING LATEST.pptx
3 SAMPLING LATEST.pptx3 SAMPLING LATEST.pptx
3 SAMPLING LATEST.pptx
 
OTTO-Report
OTTO-ReportOTTO-Report
OTTO-Report
 

More from MDO_Lab

MOWF_SchiTech_2015_Weiyang
MOWF_SchiTech_2015_WeiyangMOWF_SchiTech_2015_Weiyang
MOWF_SchiTech_2015_Weiyang
MDO_Lab
 
SA_SciTech_2014_Weiyang
SA_SciTech_2014_WeiyangSA_SciTech_2014_Weiyang
SA_SciTech_2014_Weiyang
MDO_Lab
 
MOMDPSO_IDETC_2014_Weiyang
MOMDPSO_IDETC_2014_WeiyangMOMDPSO_IDETC_2014_Weiyang
MOMDPSO_IDETC_2014_Weiyang
MDO_Lab
 
MOWF_WCSMO_2013_Weiyang
MOWF_WCSMO_2013_WeiyangMOWF_WCSMO_2013_Weiyang
MOWF_WCSMO_2013_Weiyang
MDO_Lab
 
WFSA_IDETC_2013_Weiyang
WFSA_IDETC_2013_WeiyangWFSA_IDETC_2013_Weiyang
WFSA_IDETC_2013_Weiyang
MDO_Lab
 
WM_MAO_2012_Weiyang
WM_MAO_2012_WeiyangWM_MAO_2012_Weiyang
WM_MAO_2012_Weiyang
MDO_Lab
 
CP3_SDM_2010_Souma
CP3_SDM_2010_SoumaCP3_SDM_2010_Souma
CP3_SDM_2010_Souma
MDO_Lab
 
ATI_SDM_2010_Jun
ATI_SDM_2010_JunATI_SDM_2010_Jun
ATI_SDM_2010_Jun
MDO_Lab
 
PF_MAO_2010_Souam
PF_MAO_2010_SouamPF_MAO_2010_Souam
PF_MAO_2010_Souam
MDO_Lab
 
WFO_MAO_2010_Souma
WFO_MAO_2010_SoumaWFO_MAO_2010_Souma
WFO_MAO_2010_Souma
MDO_Lab
 
ATE_MAO_2010_Jun
ATE_MAO_2010_JunATE_MAO_2010_Jun
ATE_MAO_2010_Jun
MDO_Lab
 
COST_MAO_2010_Jie
COST_MAO_2010_JieCOST_MAO_2010_Jie
COST_MAO_2010_Jie
MDO_Lab
 
WFO_IDETC_2011_Souma
WFO_IDETC_2011_SoumaWFO_IDETC_2011_Souma
WFO_IDETC_2011_Souma
MDO_Lab
 
COSTMODEL_IDETC_2010_Jie
COSTMODEL_IDETC_2010_JieCOSTMODEL_IDETC_2010_Jie
COSTMODEL_IDETC_2010_Jie
MDO_Lab
 
WFO_TIERF_2011_Messac
WFO_TIERF_2011_MessacWFO_TIERF_2011_Messac
WFO_TIERF_2011_Messac
MDO_Lab
 
WFO_SDM_2011_Souma
WFO_SDM_2011_SoumaWFO_SDM_2011_Souma
WFO_SDM_2011_Souma
MDO_Lab
 
RBHF_SDM_2011_Jie
RBHF_SDM_2011_JieRBHF_SDM_2011_Jie
RBHF_SDM_2011_Jie
MDO_Lab
 
WFO_IDETC_2011_Souma
WFO_IDETC_2011_SoumaWFO_IDETC_2011_Souma
WFO_IDETC_2011_Souma
MDO_Lab
 
AHF_IDETC_2011_Jie
AHF_IDETC_2011_JieAHF_IDETC_2011_Jie
AHF_IDETC_2011_Jie
MDO_Lab
 
WPPE_ES_2011_Jie
WPPE_ES_2011_JieWPPE_ES_2011_Jie
WPPE_ES_2011_Jie
MDO_Lab
 

More from MDO_Lab (20)

MOWF_SchiTech_2015_Weiyang
MOWF_SchiTech_2015_WeiyangMOWF_SchiTech_2015_Weiyang
MOWF_SchiTech_2015_Weiyang
 
SA_SciTech_2014_Weiyang
SA_SciTech_2014_WeiyangSA_SciTech_2014_Weiyang
SA_SciTech_2014_Weiyang
 
MOMDPSO_IDETC_2014_Weiyang
MOMDPSO_IDETC_2014_WeiyangMOMDPSO_IDETC_2014_Weiyang
MOMDPSO_IDETC_2014_Weiyang
 
MOWF_WCSMO_2013_Weiyang
MOWF_WCSMO_2013_WeiyangMOWF_WCSMO_2013_Weiyang
MOWF_WCSMO_2013_Weiyang
 
WFSA_IDETC_2013_Weiyang
WFSA_IDETC_2013_WeiyangWFSA_IDETC_2013_Weiyang
WFSA_IDETC_2013_Weiyang
 
WM_MAO_2012_Weiyang
WM_MAO_2012_WeiyangWM_MAO_2012_Weiyang
WM_MAO_2012_Weiyang
 
CP3_SDM_2010_Souma
CP3_SDM_2010_SoumaCP3_SDM_2010_Souma
CP3_SDM_2010_Souma
 
ATI_SDM_2010_Jun
ATI_SDM_2010_JunATI_SDM_2010_Jun
ATI_SDM_2010_Jun
 
PF_MAO_2010_Souam
PF_MAO_2010_SouamPF_MAO_2010_Souam
PF_MAO_2010_Souam
 
WFO_MAO_2010_Souma
WFO_MAO_2010_SoumaWFO_MAO_2010_Souma
WFO_MAO_2010_Souma
 
ATE_MAO_2010_Jun
ATE_MAO_2010_JunATE_MAO_2010_Jun
ATE_MAO_2010_Jun
 
COST_MAO_2010_Jie
COST_MAO_2010_JieCOST_MAO_2010_Jie
COST_MAO_2010_Jie
 
WFO_IDETC_2011_Souma
WFO_IDETC_2011_SoumaWFO_IDETC_2011_Souma
WFO_IDETC_2011_Souma
 
COSTMODEL_IDETC_2010_Jie
COSTMODEL_IDETC_2010_JieCOSTMODEL_IDETC_2010_Jie
COSTMODEL_IDETC_2010_Jie
 
WFO_TIERF_2011_Messac
WFO_TIERF_2011_MessacWFO_TIERF_2011_Messac
WFO_TIERF_2011_Messac
 
WFO_SDM_2011_Souma
WFO_SDM_2011_SoumaWFO_SDM_2011_Souma
WFO_SDM_2011_Souma
 
RBHF_SDM_2011_Jie
RBHF_SDM_2011_JieRBHF_SDM_2011_Jie
RBHF_SDM_2011_Jie
 
WFO_IDETC_2011_Souma
WFO_IDETC_2011_SoumaWFO_IDETC_2011_Souma
WFO_IDETC_2011_Souma
 
AHF_IDETC_2011_Jie
AHF_IDETC_2011_JieAHF_IDETC_2011_Jie
AHF_IDETC_2011_Jie
 
WPPE_ES_2011_Jie
WPPE_ES_2011_JieWPPE_ES_2011_Jie
WPPE_ES_2011_Jie
 

Recently uploaded

The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
Jisc
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
Celine George
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
Atul Kumar Singh
 
Sectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdfSectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdf
Vivekanand Anglo Vedic Academy
 
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdfESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
Fundacja Rozwoju Społeczeństwa Przedsiębiorczego
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
GeoBlogs
 
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
AzmatAli747758
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
MIRIAMSALINAS13
 
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptxMARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
bennyroshan06
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
Steve Thomason
 
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
MysoreMuleSoftMeetup
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
TechSoup
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
Special education needs
 
Chapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptxChapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptx
Mohd Adib Abd Muin, Senior Lecturer at Universiti Utara Malaysia
 
The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
Delapenabediema
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
GeoBlogs
 
Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)
rosedainty
 
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptxStudents, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
EduSkills OECD
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
Jisc
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
EverAndrsGuerraGuerr
 

Recently uploaded (20)

The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
 
Sectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdfSectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdf
 
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdfESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
 
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
 
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptxMARUTI SUZUKI- A Successful Joint Venture in India.pptx
MARUTI SUZUKI- A Successful Joint Venture in India.pptx
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
 
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
 
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup   New Member Orientation and Q&A (May 2024).pdfWelcome to TechSoup   New Member Orientation and Q&A (May 2024).pdf
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdf
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
 
Chapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptxChapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptx
 
The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
 
Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)
 
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptxStudents, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
 

PEMF2_SDM_2012_Ali

  • 1. Quantifying Regional Error in Surrogates by Modeling its Relationship with Sample Density Ali Mehmani, Souma Chowdhury , Jie Zhang, Weiyang Tong, and Achille Messac Syracuse University, Department of Mechanical and Aerospace Engineering 54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference April 8-11, 2013, Boston, Massachusetts
  • 2. Surrogate model • Surrogate models are commonly used for providing a tractable and inexpensive approximation of the actual system behavior in many routine engineering analysis and design activities: 2
  • 3. 3 Broad Research Question Lower Fidelity How Low? Structural Blade Design ANSYS, Inc. Surrogate Model Expensive Inexpensive
  • 4. 4 Broad Research Question How to quantify the level of the surrogate accuracy ?  further improvement of the surrogate,  domain exploration,  assessing the reliability of the optimal design,  quantifying the uncertainty associated with the surrogate,  construction of a weighted surrogate model, and  …
  • 5. Research Objective  Develop a reliable method to quantify the surrogate error, 5  This method should have the following characteristics:  model independent  no additional system evaluations  local/global error measurement  quantify the error of the actual surrogate
  • 6. Regional Error Estimation of Surrogate 6
  • 7. Regional Error Estimation of Surrogate 7 (REES)
  • 8. Presentation Outline • Review of surrogate model error measurement methods 8 • Relation of surrogate accuracy with sample density • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  • 9. Presentation Outline • Review of surrogate model error measurement methods 9 • Relation of surrogate accuracy with sample density • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  • 10. Surrogate Model Error Measurement Methods  Error quantification methods can be classified,  based on their computational expense, into methods that require additional data, and methods that use existing data.  based on the region of interest, into: - Global error measure (e.g., split sample, cross-validation, Akaike’s information criterion, and bootstrapping). - Local or point-wise error measure (e.g., the mean squared errors for Kriging and the linear reference model (LRM).) 10
  • 11. Surrogate Model Error Measurement Methods  Error metrics, 11 • The mean squared error (MSE) (or root mean square error (RMSE) ) • The maximum absolute error (MAE) • The relative absolute error (RAE) actual values on ith test point predicted values on ith test point
  • 12. Surrogate Model Error Measurement Methods  Error metrics, • The prediction sum of square (PRESS) is based on the leave-one-out 12 cross-validation error • The root mean square of PRESS (PRESSRMS) based on the k-fold cross-validation. • The relative absolute error of cross-validation (RAECV) based on leave-one- out approach.
  • 13. Presentation Outline • Review of surrogate model error measurement methods 13 • Relation of surrogate accuracy with sample density • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  • 14. Methodology: Concept 14 Model accuracy ∝ Available resources In general, this concept can be applied for different methodologies - Surrogate modeling, - Finite Element Analysis, and - ...
  • 15. Methodology: Concept 15  Finite Element Analysis (numerical methods) The finer mesh, the stresses are more precise due to the larger number of elements coarse mesh (4 solid brick element) medium mesh (32 solid brick element) fine mesh (256 solid brick element) Estimate total shear force and flexural moment at vertical sections using Finite Element Analysis.
  • 16. Methodology: Concept  Surrogate (mathematical model) Surrogate accuracy generally improves with increasing training points. 3 training points 9 training points 7 training points The location of additional points has strong impact on surrogate accuracy. This impact is highly problem and model dependent. 16
  • 17. Presentation Outline • Review of surrogate model error measurement methods 17 • Relation of surrogate accuracy with sample density • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  • 18. Methodology: REES  The REES method formulates the variation of error as a 18 function of training points using intermediate surrogates.  This formulation is used to predict the level of error in a final surrogate.
  • 19. Methodology: REES Step 1 : Generation of sample data The entire set of sample points is represented by 푿 . Step 2 : Identification of sample points inside/outside region of interest {푿풊풏} {푿풐풖풕} 푿 = 푿풊풏 + 푿풐풖풕 푿풊풏 : Inside-region data set 푿풐풖풕 : Outside-region data set user-defined region of interest 19
  • 20. Methodology: REES Step 3 : Estimation of the variation of the error with sample density Outside-Training region Point point Inside-region point user-defined region of interest First Iteration : Test Point Second iteration : Third iteration : Final Surrogate : Training Point
  • 21. Methodology: REES Step 3 : Estimation of the variation of the error with sample density  A position of sample points which are selected as training points, at each iteration, is critical to the surrogate accuracy.  The proposed error measure should be minimally sensitive to the location of the test points at each iteration. 21  Intermediate surrogates are iteratively constructed (at each iteration) over a sample set comprising all samples outside the region of interest and heuristic subsets of samples inside the region of interest.
  • 22. Methodology: REES Step 3 : Estimation of the variation of the error with sample density  The intermediate subset for each  The number of iterations (푁푖푡) is defined combination at specific iteration is defined by {휷풌} ⊂ 푿풊풏 #{휷풌} = 풏풕, 풏풕−ퟏ < 풏풕 풌 = 1,2, … , 퐾푡 - dimension of a problem, - number of inside sample points, and - preference of the user  The intermediate training points and test points for each combination at each iteration is defined by  The number of sample combinations (푲풕 ) is defined, 푿푻푹 = 푿풐풖풕 + 휷풌 푿푻푬 = 푿 − 푿푻푹  The intermediate surrogates 푓푘 , 풌 = ퟏ, ퟐ, . . , 푲풕 are constructed for all combinations using the intermediate training points ( 푿푻푹 ), and are tested over the intermediate test points ( 푿푻푬 ).
  • 23. Methodology: REES Step 3 : Estimation of the variation of the error with sample density  The median and the maximum errors are estimated for each combination 풎풕: the number of test points in tth iteration 풆: the RAE value estimated on intermediate test points 23
  • 24. Methodology: REES Step 3 : Estimation of the variation of the error with sample density  The median and the maximum errors are estimated for each combination The median is a useful measures of central tendency which is less vulnerable to outliers. 24 Median error Overall Fidelity Information Maximum error Minimum Fidelity Information
  • 25. Methodology: REES Step 3 : Estimation of the variation of the error with sample density  Probabilistic models are developed using The mode of distribution is selected to represent the errors at each iteration. a lognormal distribution to represent median and maximum errors estimated over all 푲풕 combinations at each iterations. Mode of median error distribution Mode of maximum error distribution  These values are used to relate the variation of the surrogate error with number of training points (sample density).
  • 26. The relation of the error with sample density  12-D Test Problem (Dixon & Price, n=12) Number of sample points # 푿 = ퟓퟓퟎ, Number of inside sample points # 푿풊풏 = # 푿 Number of training points at each iteration,풏풕 = 5푡 + 50, 푡 = 1,2, … , 70 Number of sample combination, 푲풕 = 500 Number of Training Points MOmax # 푿푻푹 = ퟓퟓ # 푿푻푬 = ퟒퟒퟓ Number of Training Points Estimated mode of median errors Estimated mode of maximum errors MOmed First iteration Last iteration # 푿푻푹 = ퟒퟎퟎ # 푿푻푬 = ퟏퟎퟎ
  • 27. The relation of the error with sample density  12-D Test Problem (Dixon & Price, n=12) Number of Training Points Meanmean Number of Training Points Estimated mode of median errors Estimated mean of mean errors MOmed REES Method Normalized k-fold CV
  • 28. Methodology: REES Step 4 : Prediction of regional error in the final surrogate  The final surrogate model is constructed using the full set of training data.  Regression models are applied to relate - the statistical mode of the median error distribution(푴풐풎풆풅) - the statistical mode of the maximum error distributions(푴풐풎풂풙), and - the absolute maximum error (푨푩푺풎풂풙) at each iteration to the size of the inside-region training points (nt),  These regression models are called the variation of error with sample density (VESD). The regression models are used to predict the level of the error in the final surrogate within the region of interest. 28
  • 29. Methodology: REES Modeling the Variation of Regional Error with Training Point Density  In this study, three types of the regression functions are used to represent the variation of regional error with respect to the inside-region training points Exponential regression model Multiplicative regression model Linear regression model  The choice of these functions assume a smooth monotonic decrease of the regional error with the training point density within that region.  The root mean squared error metric is used to select the best-fit regression model 29
  • 30. Presentation Outline • Review of surrogate model error measurement methods 30 • Relation of surrogate accuracy with sample density • Regional Error Estimation of Surrogate • Numerical examples: benchmark and an engineering design problems
  • 31. Numerical Examples  The effectiveness of the REES method is explored for applications with - Kriging, - Radial Basis Functions (RBF), - Extended Radial Basis Functions (E-RBF), and - Quadratic Response Surface (QRS).  To evaluate practical and numerical efficiencies of the REES method, three benchmark problems and an engineering design problem are tested.  The error evaluated using REES, and the relative absolute error given by leave-one-out cross-validation (푹푨푬풄풗) are compared with the actual error evaluated using relative absolute error on additional test points (푹푨푬풂풄풕풖풂풍). 31
  • 32. Median of RAEs Numerical Examples Results and Discussion VESD regression models within the region of interest of surrogate models constructed for the Branin-Hoo Function to predict, Distribution of median errors Mode of the median error distribution, Predicted mode of median error in the final surrogate, VESDmed Number of Inside-region Training Points 32
  • 33. Numerical Examples Results and Discussion VESD regression models within the region of interest of surrogate models constructed for the Branin-Hoo Function to predict, Type and coefficients of VESDmed Kriging RBF E-RBF QRS
  • 34. Maximum of RAEs Numerical Examples Results and Discussion VESD regression models within the region of interest of surrogate models constructed for the Branin-Hoo Function to predict the mode of maximum ( ) and the absolute maximum ( ) error. Distribution of maximum errors Mode of the maximum error distribution, Absolute maximum error Predicted absolute maximum error in the final surrogate Predicted mode of maximum error in the final surrogate 34 Number of Inside-region Training Points
  • 35. Numerical Examples Results and Discussion VESD regression models within the region of interest of surrogate models constructed for the Branin-Hoo Function to predict the mode of maximum ( ) and the absolute maximum ( ) error. Type and coefficients of VESDmax Type and coefficients of VESDABS Kriging RBF E-RBF QRS
  • 36. Numerical Examples Wind Farm Power Generation Surrogates are developed using Kriging, RBF, E-RBF, and QRS to represent the power generation of an array-like wind farm. 36
  • 37. Numerical Examples Results and Discussion VESD regression models in different surrogates for the wind farm power generation problem It. 1 It. 2 It. 3 It. 4 Predicted Error 37
  • 38. Numerical Examples 38 Results and Discussion The closer to one, the better the corresponding error measure. predicted mode of median errors median of RAEs evaluated on test points median of relative absolute errors of cross-validation
  • 39. Concluding Remarks  We developed a new method to quantify surrogate error based on the hypothesis that: “The accuracy of the approximation model is related to the amount of available resources”  This relationship can be reliably quantified when the error measures is less sensitive to sample locations or a type of application.  The REES method addresses this issue.  The preliminary results on benchmark and wind farm power generation problems indicate that in majority of cases the REES method is more accurate than other measures. 39 It is not possible using any existing methods
  • 40. Future Works  The scope for improvement the method  The implementation of the proposed error measurement in surrogate developments. 40
  • 41. Acknowledgement 41  I would like to acknowledge my research adviser Prof. Achille Messac, and my co-adviser Prof. Souma Chowdhury for their immense help and support in this research.  Support from the NSF Awards is also acknowledged.
  • 42. 42 Thank you Questions and Comments
  • 43. Median of RAEs Numerical Examples Results and Discussion VESD regression models within the region of interest of surrogate models constructed for the Branin-Hoo Function to predict, Distribution of median errors k-fold CV Mode of the median error distribution, Predicted mode of median error in the final surrogate, VESDmed Number of Inside-region Training Points 43 Mean mean Number of Inside-region Training Points