Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software

Introduction
Mathematical Background & Concepts
Implementation of a MC Simulation
Post-processing and Analysis of a Simulation
Discussion

Review of Methodology and Rationale of
Monte Carlo Simulation
Application to Metrology with Open Source Software

Vishal Ramnath
vramnath@nmisa.org

Mechanical Metrology Group
National Metrology Institute of South Africa

November 6, 2008

Vishal Ramnath vramnath@nmisa.org NMISA-08-0121: Review of Methodology & Rationale of MC Simulation

Introduction
Discussion

Overview of Presentation
Introduction
Review of GUM Methodology
Review of Monte Carlo Methodology
Developing a Mathematical Model
Assigning Uncertainties and PDF’s to the Model
Illustrative Mass Unc Example
Analysing and Understanding the Data
Reporting Results in GUM Terms
Discussion


Introduction
Discussion

Introduction

This is an introductory presentation to convey the basic ideas behind
the mechanics of the Monte Carlo technique as applied to metrology
measurement uncertainty problems.
The rationale for the need to understand and implement Monte Carlo
(MC) techniques in the context of metrology is that with the advance
of science and technology more accurate measurements are for
various reasons increasingly necessary in many economies and MC
simulations present the most accurate and readily available numerical
technology to solve such challenges taking into account certain
limitations in existing approaches such as the well known ISO Guide
to Uncertainty in Measurement (GUM).


Introduction
Discussion

GUM Review 1 - Essential Information Needed
For an input quantity xi in the GUM framework three quantities are needed
viz.
◮ the expectation of xi which is just the estimate of this input
◮ the standard deviation of xi which is the standard deviation of this input
σ(xi )
◮ the corresponding degrees of freedom νi associated with xi
If there are dependencies with other input quantities xj , j = i then covariances
are also required:
in the case of two inputs xi and xj the covariance u(xi , xj ) and correlation
coefﬁcient r (xi , xj ) are related by

(1) u(xi , xj ) = r (xi , xj )u(xi )u(xj ), −1 r (xi , xj ) 1


Introduction
Discussion

GUM Review 1 cont. - Essential Information Needed

q
1
(2) u(xi , xj ) = ¯ ¯
(xi,k − xi )(xj,k − xj )
q(q − 1)
k =1

◮ If r (xi , xj ) = 0 then there is no correlation and if r (xi , xj ) ≈ 1 then
there is strong correlation
◮ Most uncertainty calculations assume r (xi , xj ) ≈ 0 for simplicity
i.e. no correlation between input quantities but if necessary
correlation can be explicitly incorporated into calculations
◮ In the case of correlation between more than two variables e.g.
xi , xj , xk with i = j = k then a covariance matrix and not a scalar
correlation coefﬁcient is required


Introduction
Discussion

GUM Review 1 cont. - Essential Information Needed

The GUM approach is the propagation of uncertainties associated
with input quantities in a measurement model to provide estimates of
the model output quantity (univariate) or quantities (multivariate) It
should be noted that:
◮ Within the framework of the GUM a mathematical model of
the measurand is a prerequisite in order to implement an
uncertainty calculation


Introduction
Discussion

GUM Review 2 - Standard Calculation Technique

(3a) y = f (x1 , . . . , xn ) ⇐ math model
∂f
(3b) ci = ⇐ sens coeff
∂xi
n 2
∂f
(3c) u 2 (y ) = u 2 (xi ) ⇐ std unc
∂xi
i=1
n
4
u (y ) ci4 u 4 (xi )
(3d) = ⇐ calc eff deg freedom
νeff νi
i=1
νeff +1
−
t
Γ[ νeff2+1 ] u2 2

k ⇔ √ 1+ du = p
−t πνeff Γ[ ν2 ]eff
νeff
⇑
(3e) coverage factor


Introduction
Discussion

GUM Review 2 - Brief Comment on Sensitivity coeff’s

Various possibilities will arise in practise with real inputs x ∈ Rn :
◮ univariate, explicit, real model or multivariate, explicit, real model

◮ univariate, implicit, real model or multivariate, implicit, real model
In the case of an implicit model i.e. where an explicit functional
relationship between the input and output(s) is not known then
additional matrix algebraic manipulations are necessary and such
manipulations require the solution of linear systems of equations
In addition as per the above but with complex models i.e. with x ∈ Cn
require analogous sensitivity conterparts where now partial
derivatives for both the real and imaginary components of an input
are necessary


Introduction
Discussion

GUM Review 3 - Assumptions & Limitations of the
GUM
There are three chief requirements that limit the applicability of the
GUM:
◮ the non-linearity for the measurand as modelled by a function
f (x ) must be insigniﬁcant [GUM Clause 5.1.2]
◮ the Central Limit Theorem must be assumed to apply for the
model of the measurand i.e. the PDF for the output must be
Gaussian (alternately in terms of a t-distribution) [GUM Clauses
G.2.1 and G.6.6] and
◮ the necessary conditions for a Welch-Satterthwaite formula to
calculate the effective degrees of freedom must apply [GUM
Clause G.4.2]


Introduction
Discussion

GUM Review 4 - When & how will the GUM not work

As per the three requirements for the GUM to adequately apply it will then by
implication not function adequately when:
◮ non-linearities in the model are significant - when the model can not
accurately be represented by a first order Taylor series expansion then
the probability distribution of the measurand can similarly not be
accurately represented in terms of the convolution integral of the
distributions of the input quantities;
◮ the conditions for the validity of the Central Limit Theorem as applicable
to the measurement model are not sufficiently strong - theoretically the
CLT predicts a Gaussian distribution for the measurand only in the limit
as the number of input quantities increases i.e. it is not necessarily a
true or accurate representation of the measurand PDF for a small finite
number of input parameters


Introduction
Discussion

GUM Review 4 cont. - When & how will the GUM not
work
As per the three requirements for the GUM to adequately apply it will then by
implication not function adequately when:
◮ the conditions for the validity of the Welch-Satterthwaite formula are not
present i.e. in the case for a univariate, real output y where the input
quantities x are not mutually independent - the GUM does not state how
νeff is to be calculated when the input quantities are correlated† i.e. even
though correlation coefﬁcients (alternately covariance matrix) may be
modelled / calculated from experimental data there is no methodology to
estimate νeff and hence a corresponding coverage factor k unless one
assumes u(xi , xj ) ≈ 0∀i = j
†Correlation coefﬁcients r (xi , xj ) are used for calculating the combined
standard uncertainty uc , cf. U = k(p, νeff ) · uc


Introduction
Discussion

Review of GUM Methodology - Background to why MC
is being utilized

With the three requirements for the GUM to adequately apply and
with limitations and lack of applicability that arises when these
conditions are not met for many practical measurement models of
real measurement systems and standards we then see that:
◮ Due to the sometimes restrictive conditions on the limitations and
applicability of the GUM that many NMI’s and possibly even
industrial metrology laboratories are starting to investigate and
implement Monte Carlo simulations for their own laboratory
standards and in inter-comparisons for e.g. CMC justiﬁcations


Introduction
Discussion

Review of MC Methodology

In a measurement uncertainty analysis one is concerned with
propagating uncertainties from inputs to outputs and the GUM
propogates uncertainties from a ﬁrst order approximation from a
model of a measurement system with the assumption that the
measurand has a Gaussian distribution whilst a MC method directly
propogates PDF information without any prior assumptions.
A MC method can be accurately described as a statistical sampling
technique.


Introduction
Discussion

Outline of MC Process Used for a Univariate Model

1. select M Monte Carlo trials
2. generate M vectors by sampling from the PDF’s for the set of N
input quantities
3. for each vector evaluate the model to give the corresponding
value of the output quantity
4. calculate the estimate of the output quantity i.e. the measurand
and its associated standard uncertainty
5. use the simulation data to build a discrete representation of the
distribution function
6. use the distribution function to calculate the coverage interval for
the measurand


Introduction
Discussion


For continous random variables recall: f (x ) is a PDF for a random
∞
variable x if (i) f (x ) 0∀x ∈ R, (ii) −∞ f (x )dx = 1, (iii)
b
P(a < X < b) = a f (x )dx
The corresponding cumulative distribution function is
x
F (x ) = −∞ f (t)dt
For MC work we will use the following nomenclature: Let the PDF for
input Xi be gi (ξi ), the PDF for the measurand Y be g(η), and
η
G(η) = −∞ g(z)dz denote the distribution function corresponding to
g(η)


Introduction
Discussion


There are additional mathematical deﬁnitions and terminology that
are necessary to more fully understand how a Monte Carlo simulation
works in practice but for our purposes we will not delve too deeply into
the ﬁner details in this presentation and rather concentrate on some
of the more practical considerations that are needed if one wishes to
undertake and implement a MC measurement uncertainty analysis


Introduction
Mathematical Background & Concepts Developing a Mathematical Model
Implementation of a MC Simulation Assigning Uncertainties and PDF’s to the Model
Post-processing and Analysis of a Simulation Illustrative Mass Unc Example
Discussion


A few preliminary points should be noted:
◮ a good random number generator is essential for reliable work -
the MS Excel RNG is not satisfactory and will introduce problems
◮ the software code used should allow definition of a model and
the parameters defining the PDF’s for the input quantities
◮ symmetry in the output PDF is not assumed
◮ no derivatives are required
◮ there is an avoidance of the concept of effective degrees of
freedom
◮ sensitivity coefficients are not calculated or needed: possible to
modify post-processing to calc a sensitivity coeff


Introduction
Discussion


When developing a mathematical model for a MC simulation it should
be noted that there is no distinction between Type A and Type B
uncertainty contributors and that the measurand is simply defined in
terms of a function e.g.

(4) y = f (x1 , x2 , . . . , xn )

where the inputs x1 , . . . , xn directly model and describe the influence
if an input is changed - it is this variation/change in the input
parameters that is propogated through the model and hence
influences the output expressible as an uncertainty.


Introduction
Discussion


A MC simulation is therefore different from the GUM in the sense that
one has to have a full and complete understanding of the entire
measurement system under investigation and one can not simply
assign an input uncertainty and a unity sensitivity coefﬁcient without
adequate justiﬁcation


Introduction
Discussion

Assigning Uncertainties and PDF’s to the Model

In a MC simulation the PDF’s of the input quantities g1 (ξ1 ), . . . , gN (ξN )
are required and the following options are possible:
◮ if the input xi is a quantity that has been measured/calibrated
then it will have a measurement/calibration certiﬁcate that was
done with the GUM so the quoted value is the mean µi and its
standard uncertainty is obtained by dividing the expanded
uncertainty by the applicable coverage factor - this is enough
information to infer its PDF since the GUM result is always
expressed in terms of a Gaussian PDF which is completely
deﬁned in terms of µ and σ
◮ similarly as above for rectangular (particularly if estimated from
e.g. literature), triangular, U shaped distributions etc.


Introduction
Discussion

Assigning Uncertainties and PDF’s to the Model cont.

◮ a statistical analysis based on relevant theory may indicate an
inputs PDF e.g. dimensional cosine terms and one then just has
to estimate some parameters to fully deﬁne the PDF
◮ there may be discrete numerical data for an input parameter
which means that input parameter’s PDF can be built up with its
frequency data (like a histogram)


Introduction
Discussion

Mass Meas Unc Example to Contrast the GUM vs. MC
Example (Math Model Explanation & Details)
Consider a mass calibration to
illustrate the principles of a MC
simulation and some of the
differences with the standard GUM
approach with a schematic
illustration of the measurement
system


Introduction
Discussion

Consider a mass calibration to δmR

simulation and some of the mW mR

pivot
system


Introduction
Discussion



pivot
system

The principle of a mass measurement with an equal arm force balance is of a
balance of moments generated which by applying a force balance with
Archimedes’ principle for buoyancy effects we then get


Introduction
Discussion



pivot
system

The principle of a mass measurement with an equal arm force balance is of a
balance of moments generated which by applying a force balance with
Archimedes’ principle for buoyancy effects we then get
mW mR + δmR
(5) mW g − ρair g = (mR + δmR )g − ρair g
ρW ρR


Introduction
Discussion

Developing the Meas Model

Example (Mass example cont.)
Rearranging


Introduction
Discussion

Developing the Meas Model

Example (Mass example cont.)
Rearranging

ρair ρair
mW 1− = (mR + δmR ) 1 −
ρW ρR

Symbol Description
mW mass of weight piece W
mR mass of reference weight piece R
δmR small test mass to add onto mR to achieve force balance
ρi mass density with i respectively that of the weight W
air medium air or that of the reference’s density R


Introduction
Discussion

Developing the Meas Model cont.
Example (Expressing model with lab speciﬁc stds)
Mass is an invariant quantity as any physical parameter in metrology but the
values quoted will differ depending on the lab system in use e.g. mass
is“heavier” in air than in water which is why astronauts train under water to
simulate weightlessness:


Introduction
Discussion

Developing the Meas Model cont.
Example (Expressing model with lab speciﬁc stds)
Mass is an invariant quantity as any physical parameter in metrology but the
values quoted will differ depending on the lab system in use e.g. mass
is“heavier” in air than in water which is why astronauts train under water to
simulate weightlessness:
In mass metrology laboratories use the concept of “conventional mass”

Deﬁnition (Conventional Mass)
The conventional mass mW ,c of a weight W is the apparent mass of a
hypothetical weight of density ρW 0 = 8000 kg.m−3 that balances W in air
when the air density is ρair 0 = 1.2 kg.m−3 i.e.
mW (1 − ρair 0 /ρW ) = mW ,c (1 − ρair 0 /ρW 0 )

Fact (Usage of Conventional Mass)
Conventional mass is simply a measurement tool used to incorporate the
invariance of inertial mass i.e. “compare apples with apples”


Introduction
Discussion

Final Measurement Model

We utilize the Open Source computer algebra system Maxima to
simplify our calculations due to the substitions that the conventional
mass introduces.
The reason for why one may prefer to use a CAS is in the cases when
fairly complicated expressions and hand calculations carry the risk of
lengthy time and error generation.


Introduction
Discussion

Review of calculas theory - cf. GUM assumptions
Recall that from multi-variable calculas that Taylor series expansions
for the case of a single variable

(6)
f ′ (a) f ′′ (a) f (n−1) (a)
f (x ) = f (a)+ (x −a)+ (x −a)2 +· · ·+ (x −a)n−1 +Rn (x )
1! 2! (n − 1)!
can be generalized to the case for multiple variables. An example for
two variables would be

∂f ∂f
f (x , y ) = f (a, b) + (a, b)(x − a) + (a, b)(y − b)
∂x ∂y
1 ∂ 2f ∂2f
+ (a, b)(x − a)2 + 2 (a, b)(x − a)(y − b)
2! ∂x 2 ∂x ∂y
∂ 2f
+ (a, b)(y − b)2 + · · ·
∂y 2

Introduction
Discussion

Approx. of functions - linearized in 1D

Let f (x ) = exp(x 2 ) and expand about a = 1
f = exp(x 2 )
9

8
y-axis

7
2
6 function ex
5 1st e + 2e(x − 1)
4 2nd e + 2e(x − 1) + 3e(x − 1)2
3

2

1
-1.5 -1 -0.5 0 0.5 1 1.5
x-axis


Introduction
Discussion

Let f (x ) = exp(x 2 + y 2 ) and expand about a = [1, 1]T

Linear approx: L(f ) = 2e2 (y − 1) + 2e2 (x − 1) + e2

9
8.5
9 8
8.5
7.5
8
z1 7
7.5
7 6.5
6.5 6
6 5.5
5.5
1.05
1.04
1.03
1.02
0.95 0.96 1.01
1
0.97 0.98 0.99 y1
0.99 1 0.98
1.01 1.02 0.97
x1 1.03 1.04 0.96
0.95
1.05

3d plot

1.05 0

1.04
-0.5
1.03

1.02
-1
1.01
y1

1 -1.5

0.99
-2
0.98

0.97
-2.5
0.96

0.95 -3
0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05
x1

% error


Introduction
Discussion

Quadratic approx:
Linear approx: L(f ) = 2e2 (y − 1) + 2e2 (x − 1) + e2 f = L(f ) + 1 [6e2 (y − 1)2 + 8e2 (x − 1)(y − 1) + 6e2 (x − 1)2 ]
2

9
8.5 9.5
9 8 9
8.5 9.5
7.5 8.5
8 9
z1 7 8
7.5 8.5
7 6.5 z1 7.5
8
6.5 6 7.5 7
6 5.5 7 6.5
5.5 6.5 6
1.05 6
1.04
1.03 1.05
1.02 1.04
0.95 0.96 1.01 1.03
1 1.02
0.97 0.98 0.99 y1 1.01
0.99 1 0.98 0.95 0.96 1
1.01 1.02 0.97 0.97 0.98 0.99 y1
x1 1.03 1.04 0.96 0.99 1 0.98
0.95
1.05 1.01 1.02 0.97
x1 1.03 1.04 0.96
0.95
1.05
3d plot
3d plot

1.05 0
1.05 0.3
1.04
-0.5 1.04 0.25
1.03
1.03 0.2
1.02
-1 0.15
1.02
1.01
0.1
1.01
y1

1 -1.5
0.05

y1
1
0.99 0
-2 0.99
0.98 -0.05
0.98
0.97 -0.1
-2.5
0.97 -0.15
0.96
0.96 -0.2
0.95 -3
0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 -0.25
x1 0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05
x1
% error
% error


Introduction
Discussion

Quadratic approx:
Linear approx: L(f ) = 2e2 (y − 1) + 2e2 (x − 1) + e2 f = L(f ) + 1 [6e2 (y − 1)2 + 8e2 (x − 1)(y − 1) + 6e2 (x − 1)2 ]
2

9
8.5 9.5
9 8 9
8.5 9.5
7.5 8.5
8 9
z1 7 8
7.5 8.5
7 6.5 z1 7.5
8
6.5 6 7.5 7
6 5.5 7 6.5
5.5 6.5 6
1.05 6
1.04
1.03 1.05
1.02 1.04
0.95 0.96 1.01 1.03
1 1.02
0.97 0.98 0.99 y1 1.01
0.99 1 0.98 0.95 0.96 1
1.01 1.02 0.97 0.97 0.98 0.99 y1
x1 1.03 1.04 0.96 0.99 1 0.98
0.95
1.05 1.01 1.02 0.97
x1 1.03 1.04 0.96
0.95
1.05
3d plot
3d plot

1.05 0
1.05 0.3
1.04
-0.5 1.04 0.25
1.03
1.03 0.2
1.02
-1 0.15
1.02
1.01
0.1
1.01
y1

1 -1.5
0.05

y1
1
0.99 0
-2 0.99
0.98 -0.05
0.98
0.97 -0.1
-2.5
0.97 -0.15
0.96
0.96 -0.2
0.95 -3
0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05 0.95 -0.25
x1 0.95 0.96 0.97 0.98 0.99 1 1.01 1.02 1.03 1.04 1.05
x1
% error
% error
The error for a linear approximation of the very non-linear model function i.e. using the GUM method is ∼ 3% and for a quadratic

approximation ∼ 0.3%


Introduction
Discussion

Taylor series for f (x1 , . . . , xd )

For n variables we then have the Taylor series T (x1 , . . . , xd ) for
f (x1 , . . . , xd ) as

∞ ∞
∂ n1 ∂ nd f (a1 , . . . , ad )
T (x1 , . . . , xd ) = ··· n1 · · · n
n1 =0 nd =0
∂x1 ∂xd d n1 ! · · · nd !
(7) ×(x1 − a1 )n1 · · · (xd − ad )nd

D α f (a)
(8) T (x1 , . . . , xd ) = (x − a)α
α!
i∈N0


Introduction
Discussion

Special cases for Taylor series expansions

For a model f (x) of the measurement system given with inputs
x = [x1 , . . . , xn ]T (x is a column vector with dimensions n × 1) and
with nominal value a which is the state that the measurement ystem
is in then making use of the general Taylor series expansion for
α
f (a)
multiple variables T (x) = |α| 0 D α! (x − a)α we note that:
◮ First order approximation:

∂f ∂f
(9) f (x ) ≈ f (a1 , . . . , an ) + (x1 − a1 ) + · · · + (xn − an )
∂x1 a ∂xn a


Introduction
Discussion

Special cases for Taylor series expansions cont.
In most cases it is seldom beneﬁcial to construct a multiple variable
Taylor series expansion for 3rd or higher order
◮ Second order approximation:

 
x1 − a1
∂f ∂f .
f (x ) ≈ f (a1 , . . . , an ) + ···
 .
.

∂xa a ∂xn a
 
xn − an
 ∂2f ∂2 f ∂2 f

∂x1 2 ∂x1 ∂x2 · · · ∂x1 ∂xn
 ∂2f ∂2 f ∂2 f
 
1  ∂x ∂x

∂x2 2 · · · ∂x2 ∂xn  x1 − a1
(10) +  2. 1 ···

. .

2!  . . .. .

 . . . .

 xn − an
∂2f ∂2 f ∂2 f
∂xn ∂x1 ∂xn ∂x2 ··· ∂x 2
n

Comment: the n × n square matrix above is the Hessian matrix
for f and all the entries are evaluated at a

Introduction
Discussion

Final Measurement Model - Calc details
Recall that we have a model for the mass to be measured

ρair ρair
(11) mW 1− = (mR + δmR ) 1 −
ρW ρR
and we wish to write the model in terms of the ‘conventional’ mass by
subsituting the formulae
−1
ρair 0 ρair 0
(12a) mW = mW ,c 1 − 1−
ρW 0 ρW
−1
ρair 0 ρair 0
(12b) mR = mR,c 1 − 1−
ρW 0 ρR
−1
ρair 0 ρair 0
(12c) δmR = δmR,c 1 − 1−
ρW 0 ρR
Once the equations are substited we then want to solve for mW ,c
which is the mass of the weight that we wish to calibrate

Introduction
Discussion

Measurement Model - Formulating and solving in
Maxima
In Maxima we use the following computer code:
LHS: mW*(1 - rhoair/rhoW);
RHS: (mR + deltamR)*(1 - rhoair/rhoR);
LHS1: ev(LHS, mW = mWc*(1 - rhoair0/rhoW0)/(1 - rhoair0/rhoW));
RHS1: ev(RHS, mR = mRc*(1 - rhoair0/rhoW0)/(1 - rhoair0/rhoR),
deltamR =
deltamRc*(1 - rhoair0/rhoW0)/(1 - rhoair0/rhoR));
soln: solve(LHS1 = RHS1, mWc);
mWc: rhs(soln[1]);
whence
1
mW ,c = ×
(ρR − ρair 0 )ρW − ρair ρR + ρair ρair 0
[((mR,c + δmR,c )ρR − ρair mR,c − ρair δmR,c )ρW
+(−ρair 0 mR,c − ρair 0 δmR,c )ρR
(13) +ρair ρair 0 mR,c + ρair ρair 0 δmR,c ]

Introduction
Discussion

Measurement Model - GUM Formulation
The measurement model consists of ﬁve input parameters which we list below:
symbol description PDF comments
mR,c reference mass Gaussian from meas cert
δmR,c balance mass Gaussian from meas cert
ρair density of air rectangular estimated from CIPM formula
ρW density of weight rectangular estimate that is equally likely
ρR density of reference rectangular from literature of physical
properties
Comment on parameters that are not included:
◮ the density ρW 0 does not explicitly appear in the model equation as it
cancels out when the model equation is algebraically solved for mW ,c
which would not be obvious in a spreadsheet
◮ the air density ρair 0 is not included in the model as a variable but as a
constant since this is known exactly


Introduction
Discussion

Practical Implementation of MC Model Inputs
Definition (Model constants & parameters)
In a measurement mathematical model working in SI units one should
distinguish between how to incorporate constants and parameters. A
parameter is a variable that one is uncertain of and which has a statistical
uncertainty (however small) and PDF whilst a constant is exactly known.1

Fact (Theories which use exact constants i.e. zero unc)
An example would be the speed of light which was historically measured
using various techniques with associated experimental uncertainties and with
Einstein’s Special Theory of Relativity fixed and then later defined as
def
c0 = 299792458 m.s−1 where σ(c0 ) = 0

Fact (Theories which use approx constants i.e. finite unc)
An example would be the Avogadro number NA = 6.02214179 × 1023 mol−1
which as a constant of nature is fixed but which is currently experimentally
known to an accuracy of σ(NA ) = 0.00000030 × 1023 mol−1
1For details see the CODATA website for physical and chemical reference values at
http://physics.nist.gov/cuu/Constants/international.html


Introduction
Discussion

Math Model - GUM Calcs 1
We will consider both ﬁrst and second order calc’s using the GUM for
comparison with a Monte Carlo simulation.
◮ 1st order: apply the GUM as usual with sums of products of
gradients etc.
◮ 2nd order: must build up f with additional terms using Hessian
matrix etc.

Fact (Practical observation of GUM calc’s)
The GUM makes use of the assumption that there is a linearized model of the
system to propogate the uncertainties and to be strictly consistent one should
apply a linearized model when calculating the standard uncertainty in order
not to mix of terms from different assumptions and approximations, however
we note that in practice most metrologists would most likely take the original
expression to evaluate the model and not its linearization - this is only valid if
the model is approximately linear in a neighbourhood of a where a is the
state space that the system is in.


Introduction
Discussion

Math Model - GUM Calcs 2
Since there are ﬁve input variables in our mathematical model the
distinction and implications of 1st and 2nd order approximations are
not immediately obvious to appreciate. Noting the variables

mR,c , δmR,c , ρair , ρW , ρR

we can reasonably conclude that the two variables that are most
likely to be uncertain and vary are
◮ δmR,c because this must be adequately controlled to achieve a
balance and equlibrium on the force beam and the equilibrium
can be a bit subjective if there isn’t an exact balance and the
beam is moving very slowly
◮ ρair the actual air density which will depend and vary with the
laboratory’s ambient temperature and pressure


Introduction
Discussion

Math Model - GUM Calcs 2 cont

Setting the nominal conditions for argument as

a = (mR,c , δmR,c , ρair , ρW , ρR )
= (0.099 kg, 0.001 kg,
1.17 kg.m−3 , 7800 kg.m−3 , 8000 kg.m−3 )

for mW ,c ≈ 0.100 kg we can then see in a limited sense the
implications of the GUM requirement for a linearized model.


Introduction
Discussion

Mass Unc Math Model - Approx

Linear approx of f error [ppm] Quadratic approx of f error [ppm]
O(4 × 10−3 ) O(5 × 10−8 )

ρair / kg.m−3
ρair / kg.m−3

1.35 0.004 1.35 5e-008

4e-008
0.003
1.3 1.3 3e-008
0.002
2e-008
0.001
1.25 1.25 1e-008

0 0

1.2 1.2 -1e-008
-0.001
-2e-008
-0.002
1.15 1.15 -3e-008
-0.003
-4e-008

1.1 -0.004 1.1 -5e-008
0 0.0005 0.001 0.0015 0.002 0.0025 0 0.0005 0.001 0.0015 0.002 0.0025
δmR,c / kg δmR,c / kg


Introduction
Discussion

Uncertainty results using the GUM 1
Assuming for argument that all the inputs for
mW ,c = f (mR,c , δmR,c , ρair , ρW , ρR ) have uncertainties of 0.1% and in addition
are not correlated we then have that the uncertainty estimate for the mass
being weighed mW ,c reported in standard uncertainty is:
◮ 1st order:

(14) u(mW ,c ) = 1.0000000740094588 × 10−6 kg
◮ 2nd order:
N 2
∂f
u 2 (f ) = u 2 (xi )
∂xi
i=1
N N 2
1 ∂2f ∂f ∂ 3 f
+ + u 2 (xi )u 2 (xj )
2 ∂xi ∂xj ∂xi ∂xi ∂xj2
i=1 j=1

= L+H
(15) u(f ) = 1.0001135041200297 × 10−6 kg
◮ difference in uncertainty is underestimated by approx. 113 ppm

Introduction
Discussion

Uncertainty results using the GUM 2

Full uncertainty can be misleading if linearization not accurate if f is
very non-linear – in the above it is not too signiﬁcant since f is not too
non-linear
It should be noted that H is formed out of double sum over the
number of variables N and that such a calculation can only
realistically be performed in a computer algebra system due to the
excessive number of partial derivatives that must be computed e.g.
with N = 5 then 100 partial derivatives must be calculated.
The computer code to perform this computation within a CAS e.g.
Maxima is relatively straightforward to implement but it should be
noted that the full expression can become algebraically large and
unwieldy – the non-linear terms correctly evaluate to zero when the
model is indeed linear.


Introduction
Discussion

Including non-linear terms using the GUM
Example (A nonlinear functional)
2 2
Work out the uncertainty for f (x1 , x2 ) = exp[x1 + x2 ] assuming
u(x1 ) = u(x2 ) = 0.1% at the point a = [x1 = 1, x2 = 1]T and compare
the linear and nonlinear answers using the GUM.
The linearization of f is 2e2 (x2 − 1) + 2e2 (x1 − 1) + e2 and now the
nonlinear term H where u 2 (f ) = L + H is
2 2
2 2 2 2 2 x2 +x1 x 2 +x 2 2 x 2 +x 2 x 2 +x 2
H = u1 u2 8 x1 x2 e + 2 x1 e 2 1 8 x1 x2 e 2 1 + 4 x1 e 2 1

2
 
2 2 2 2
 2 x +x1 + 2 ex2 +x1
4 x1 e 2 
2  x 2 +x 2 2 2
3 x +x1 x 2 +x 2
 
+u1  2 x1 e 2 1 8 x1 e 2 + 12 x1 e 2 1 +



 2 


2 2
2 2 2 2 2 x2 +x1 x 2 +x 2 2 x 2 +x 2 x 2 +x 2
+u2 u1 8 x1 x2 e + 2 x2 e 2 1 8 x1 x2 e 2 1 + 4 x2 e 2 1

2
 
2 2 2 2
 2 x +x1 + 2 ex2 +x1
4 x2 e 2 
2  x 2 +x1
2 2 2
3 x +x1 x 2 +x 2
 
+u2  2 x2 e 2 8 x2 e 2 + 12 x2 e 2 1 +



 2 



Introduction
Discussion

Non-linear calc - how signiﬁcant are the GUM approx?

2 2
For f = ex1 +x2 the non-linear contribution H is non-zero as indicated
above, and the difference between the linear and non-linear
uncertainty estimates is
◮ u(flinear ) = 0.020899406696487
◮ u(fnon−linear ) = 0.02089964181349
◮ the difference in uncertainty is therefore underestimated by
approx. 11 ppm

Fact (GUM linear model underestimates uncertainty)
It is seen that a linearized model may underestimate the actual
uncertainty by 10 – 100 ppm


Introduction
Discussion

Comparison of PDF’s for Math Model’s Calc Unc
When calculating uncertainties there are three PDF’s that one must
consider when interpreting the measurand’s uncertainty:
◮ the actual PDF for the measurand as computed in terms of a
convolution integral (Markov formula2 )
∞ ∞ ∞
(16) g(η) = ··· g(ξ)δ(y − f (ξ))dξN dξN−1 · · · dξ1
−∞ −∞ −∞

◮ a Gaussian3 like i.e. a t-distribution with νeff degrees-of-freedom
via. the Welch-Satterwaithe formula for the calculation of the
measurand y ’s PDF as per the GUM approach
◮ a discrete PDF in a Monte Carlo simulation that is built up with
sampled data from the input PDF’s g1 (ξ1 ), . . . , gN (ξN ) that will
converge to the measurand’s actual PDF (as calculated with a
covolution integral) as the number of MC events M → ∞
2 adequate mathematical statistics working knowledge is necessary to fully

understand the conditions/derivation of the Markov formula wrt. GUM
3 A Gaussian PDF i.e. N(µ = 0; σ 2 ) is entirely deﬁned in terms of the variance σ 2

whilst a t-distribution needs ν to deﬁne its shape

Introduction
Discussion

Comment on Application of Markov formula
◮ The GUM is based on the application of the Markov formula to
linearized models and all of the results and formulae in the GUM
can be derived (with certain assumptions) via. application of the
Markov formula
◮ Practical examples:
1. Higher order terms are necessary in the GUM for non-linear
models where the GUM will not work yielding incorrect results e.g.
Y = X 2 where u(y) = 2x · u(x)∀x if just linear terms of the form
u 2 (f ) = N [∂xi f · u(xi )]2 are used
i=1
2. The Markov formula will yield the correct result with
4
u(y) = u(x) 4x 2 + 5 u 2 (x) which is true even for x = 0
◮ In general a direct evaluation is only analytically possible for
certain simple cases whilst symbolic evaluation is only feasible
with a low order of variables requiring transformations and
evaluation/calculation of Jacobians with a numerical approach
preferred

Introduction
Discussion

Comment on Application of Markov formula cont.

◮ The joint PDF gX1 ,X2 ,... (ξ1 , ξ2 , . . .) built up in terms of matrix
multiplications requires the use of a Dirac delta function δ as
deﬁned in terms of a sum with derivative terms and in addition
manipulation of the inputs ξi wrt. the output η
◮ Such calculations in the GUM require the application of further
matrix algebra and will not be considered in this presentation
◮ The direct application of the Markov formula is in practice
awkward and difﬁcult to implement particularly in the case of
non-linear models and the use of a Monte Carlo approach is
entirely consistent with the Markov formula and is in fact a more
practical calculation method that does not rely on any of the
assumptions inherent as in the GUM


Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software

Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software

Similar to Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software (20)

Review of Methodology and Rationale of Monte Carlo Simulation - Application to Metrology with Open Source Software