The document discusses various types of validation metrics that can be used to measure the difference between results obtained from a simulation and experimental measurement. It describes several common validation metrics, including classical hypothesis testing, Bayes factor, frequentist's metric, area metric, and various error/correlation metrics such as vector norms, average residual, coefficient of correlation, Sprague and Geers metric, and normalized integral square error. It also provides examples of how validation and correlation metrics can be applied to compare experimental and simulation data.

1. Seoul National University
Seoul National University System Health & Risk Management
2017/2/25 ‐ 1 ‐
Correlation metric
Seoul National University System Health & Risk Management
Jungho Park*
*hihijung@snu.ac.kr

2. Seoul National University ‐ 2 ‐
Validation metric
Validation metric : a mathematical operator that measures the difference between a
system response quantity (SRQ) obtained from a simulation result and one obtained
from experimental measurement.(verification and validation in scientific computing)
Figure reference : Verification, validation, and predictive capability in computational engineering and physics, Oberkampf et al. ,Applied mechanics(2004)

3. Seoul National University ‐ 3 ‐
Validation metric 의 종류
1. Classical hypothesis testing
‐ 평균 및 분산에 대한 가설을 세우고, 얻어진 실험 결과로부터 가설 검정 실시
‐ 장점 : 모델의 적합도 여부를 결정 가능
‐ 단점 : 실험의 개수가 적을 때는 이용 불가능
Liu, Yu, et al. "Toward a better understanding of model validation metrics."Transactions of the ASME‐R‐Journal of Mechanical Design 133.7
(2011): 071005.

4. Seoul National University ‐ 4 ‐
Validation metric 의 종류
2. Bayes factor
‐ Bayesian hypothesis testing 에서 유래
‐ Null, alternative 가설의 posterior distribution 의 비에 의해 결정
Liu, Yu, et al. "Toward a better understanding of model validation metrics."Transactions of the ASME‐R‐Journal of Mechanical Design 133.7
(2011): 071005.
B=bayes factor

5. Seoul National University ‐ 5 ‐
Validation metric 의 종류
3. Frequentist’s metric
‐ Hypothesis 로부터 모델의 적합도를 ‘yes ‘ or ‘no’를 결정하기보다는 실험과 시
뮬레이션 값의 차이를 정량화
Liu, Yu, et al. "Toward a better understanding of model validation metrics."Transactions of the ASME‐R‐Journal of Mechanical Design 133.7
(2011): 071005.
tan
e estimated predictionerror
s estimated s dard devidation
N numberof physicalobservation



Estimated error in the predictive model with a confidence
level of 100(1‐ α)% that the true error is in the interval =
e

6. Seoul National University ‐ 6 ‐
Validation metric 의 종류
4. Area metric
‐ Mean, variance 같은 moment 가 아닌 시험, 시뮬레이션 분포의 전체적 모양을
비교
‐ 시험, 시뮬레이션 개수가 적을 때 사용 가능
‐ U‐pooling method 와 함께 자주 쓰임
Liu, Yu, et al. "Toward a better understanding of model validation metrics."Transactions of the ASME‐R‐Journal of Mechanical Design 133.7
(2011): 071005.

7. Seoul National University ‐ 7 ‐
Error metric(or correlation metric )의 종류
1. Vector norms
2. Average residual and Its Standard Deviation
3. Coefficient of correlation and cross relation
Limitations: Not able to distinguish error due to phase from error due to magnitude
Limitations: Positive and negative differences at various point may cancel out
2
1
1
( )
1
N
i
N
R R
S
N




 ( )i iRi a b 
Limitations: Sensitive to phase difference
Not able to distinguish error due to phase from error due to magnitude
1 1 1
2 2 2 2
1 1 1 1
( )
( )
( ) ( ) ( ) ( )
N n N n N n
i i n i i n
i i i
N n N n N n N n
i i i n i n
i i i i
N n a b a b
n
N n a a N n b b

  
 
  
   
 
   
 

   
  
   
*Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics, H.Sarin, M.Kokkolaras, G.Hulbert, P.Papalambro, S.Barbat, R.‐J.Yang,
Journal of Dynamic Systems, Measurement, and Control(2010)

8. Seoul National University ‐ 8 ‐
Error metric(or correlation metric )의 종류
4. Sprague and Geers metric
5. Russel’s error measure
1
&G
1
cos ( ),AB
S
AA BB
P

  


& 1,
2 2
& & &S G S G S GC M P 
2 2
1 1 1
, , ,
N N N
i i i i
i i i
AA BB AB
a b a b
N N N
    
  
  
Characteristics: Phase error portion considered
Limitations: lumped the entire information into , ,
Magnitude :
Phase :
Total :
10
( )log (1 )AA BB
R AA BB
AA BB
M sign
 
 
 

  
Characteristics: Phase error portion considered
Limitations: lumped the entire information into , ,
No magnitude error
*Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics, H.Sarin, M.Kokkolaras, G.Hulbert, P.Papalambro, S.Barbat, R.‐J.Yang,
Journal of Dynamic Systems, Measurement, and Control(2010)

9. Seoul National University ‐ 9 ‐
Error metric(or correlation metric )의 종류
6. Normalized Integral Square Error(NISE)
7. Dynamic Time Warping
2 ∗ 2
∗
2 ∗
1 ∗
1
2
Phase : Magnitude: Shape :
Total :
Characteristics: Shape error portion considered
Limitations: Magnitude portion can be negative. (which mean magnitude portion can decrease overall error)
Characteristics: Algorithm for measuring discrepancy between time history
*Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics, H.Sarin, M.Kokkolaras, G.Hulbert, P.Papalambro, S.Barbat, R.‐J.Yang,
Journal of Dynamic Systems, Measurement, and Control(2010)

10. Seoul National University ‐ 10 ‐
Error metric(or correlation metric )의 종류
8. Weighted Integrated Factor (WIFac)
1
max , ⋅ 1
max 0, ⋅
max ,
max ,
0 1
1
∑
∑

11. Seoul National University ‐ 11 ‐
Correlation metric and validation metric
Validation metric : a mathematical operator that measures the difference between a
system response quantity (SRQ) obtained from a simulation result and one obtained
from experimental measurement.(verification and validation in scientific computing)
0 5 10 15 20 25 30
0
50
100
150
200
250
300
350
Simulation
Time(ms)
ResultantAcc(g)0 0.2 0.4 0.6 0.8 1
0
10
20
30
40
50
60
70
80
90
WIFac
Density
Exp :
Sim :
Time(s)
Acc(g)
0 0.005 0.01 0.015 0.02 0.025 0.03
0
20
40
60
80
100
120
140
160
180
Mean
of exp.
:
0 5 10 15 20 25 30
0
50
100
150
200
250
300
350
Simulation
Time(ms)
ResultantAcc(g)
0 5 10 15 20 25 30
0
50
100
150
200
250
300
350
Simulation
Time(ms)
ResultantAcc(g)
0 5 10 15 20 25 30
0
50
100
150
200
250
300
350
Simulation
Time(ms)
ResultantAcc(g)
+3σ +1.5σ
‐3σ ‐1.5σ
WIFac
Derivation of WIFac for Simulation
(0.5275, 0.5133,0.5293,0.5183)
0 5 10 15 20 25 30
0
50
100
150
200
250
300
350
Experiment
Time(ms)
ResultantAcc(g)
HIC 324
HIC 565
HIC 347
HIC 290
Derivation of WIFac for Experiment
(0.7738, 0.6186, 0.7648, 0.7308)
WIFac is not Validation
metric, but Area metric is
0 0.5 1
0
0.5
1
Funi
Fu
CDF
Um = 0.2641