phd thesis presentation

Measurement data analysis in quality
management systems.
Application to fuel test methods.
PhD Thesis
of
Dimitrios G. Theodorou
October 2015
NATIONAL TECHNICAL UNIVERSITY OF ATHENS
School οf Chemical Engineering
Department οf Synthesis and
Development οf Industrial Processes

PhD Thesis – D. Theodorou
Presentation Outline
Introduction and motivation
Statistical and numerical methods overview
Measurement uncertainty arising from sampling
Measurement uncertainty estimation of an analytical
procedure
Estimation of the standard uncertainty of a calibration
curve
The use of measurement uncertainty and precision data
in conformity assessment
Conclusions

Introduction and motivation
Fuels produced and placed on market should comply with
strict requirements introduced by relevant legislation
 Directive 98/70/EC
 Directive 2003/17/EC
Several laboratory test methods are used for the
evaluation and assessment of fuel properties
The social and economic impact of the laboratory getting
a wrong result and the customer consequently
reaching a false conclusion can be enormous.
The laboratory should provide a high quality service to
its customers

Quality = Fitness for purpose (i.e. intended use)
The quality of a result and its fitness for purpose is
directly related to the estimation of measurement
uncertainty
Measurement uncertainty – A key requirement for
accreditation according to international standards:
 ISO/IEC 17025, for testing and calibration laboratories
 ISO 15189, for medical laboratories
 ISO/IEC 17043, for proficiency testing providers
 ISO Guide 34, for reference material producers

Statistical and numerical methods overview
Measurement Uncertainty (MU) = non-negative
parameter characterizing the dispersion of the quantity
values being attributed to a measurand, based on the
information used.
International Vocabulary of Metrology (VIM)

Master document
on MU estimation
Guide to the
Expression of
Uncertainty in
Measurement,
GUM
All MU estimation
methodologies should give
results consistent with
GUM

Modelling approach – GUM uncertainty framework

Modelling approach – GUM uncertainty framework
),...,,( 21 NXXXfY 
 i
ii xucyu )()( 222
),()()(2 jij
ji
iji xxrxuxucc
Nxxxii
i
X
f
x
f
c
,...,, 21






)()(
),(
),(
ji
ji
ji
xuxu
xxu
xxr 
)(ykuU 

Modelling approach – Kragten approximation
)(
)())((
i
iii
i xu
xfxuxf
x
f 



)())(()()()( iiii
i
iii xfxuxfxu
x
f
xucyu 




Modelling approach – Monte Carlo Method

Empirical approaches
Data produced by single laboratory validation approach
and analyzed by
 ANOVA
 Robust ANOVA
 Range statistics
Data obtained by proficiency testing schemes
 Standard uncertainty estimated as pooled reproducibility limit
Bayesian uncertainty analysis
Type A uncertainty evaluated through a Bayesian
approach

Measurement uncertainty and precision data are used in
conformity assessment

SAMPLING
MEASUREMENT
PROCEDURE
REPORTING
RESULTS AND
ASSESSING
CONFORMITY
Chapter 3
Estimation of sampling
uncertainty
Chapter 4
Estimation of the
uncertainty of a typical
measurement procedure
Chapter 5
Estimation of the
uncertainty of a
measurement procedure
involving the construction
of a calibration curve
Chapter 6
Use of measurement
uncertainty in conformity
assessment

Measurement uncertainty arising from sampling
Sampling uncertainty is defined as the part of the total
measurement uncertainty attributable to sampling
Empirical approach
Statistical model
analysissamplingtrue   Xx
2
analysis
2
sampling
2
tmeasuremen  
2
analysis
2
sampling
2
tmeasuremen sss 
sU 2

Experimental protocol and experimental design
 Balanced nested
experimental design
 Duplicate diesel samples
were taken from 104
petroleum retail stations
 The sampling protocol used was consistent with the standard
method ASTM D 4057
 The duplicated samples were analyzed in duplicate under
repeatability conditions for sulful mass content
determination. (ANTEK 9000S sulfur analyzer - ASTM D 5453
/ISO 20846)
Sampling
target
Sample B
Sample A
Analysis A2
Analysis A1
Analysis B2
Analysis B1

Data analysis methods
1. Classical Analysis of Variance (ANOVA)
Variations associated with different sources (analysis and
sampling) can be isolated and estimated

Data analysis methods (continued)
2. Robust Analysis of Variance
Particularly appropriate for providing estimated of variances, in
cases where the validity of classical ANOVA is doubtful
 It is insensitive to distributional assumptions (such as normality)
 It can tolerate a certain amount of unusual observations (outliers)
 It uses robust estimates of the mean and standard deviation calculated by an
iterative process (Huber’s method). Extreme values that exceed a certain
distance from the sample mean are downweighted or brought in.

Data analysis methods (continued)
3. Range Statistics
The variance of sampling is calculated indirectly as the difference
of the variances of measurement and analysis.
2
analysis2
tmeasuremen
2
sampling
2 








s
ss
128.1
analysis
analysis
D
s 128.1
tmeasuremen
tmeasuremen
D
s 
Sampling
target
Sample B
Sample A
Analysis A2
Analysis A1
Analysis B2
Analysis B1

Results
•Robust ANOVA leads to
statistically significantly
different results (F-test)
compared to the other two
methodologies.
•Robust ANOVA, which is
not influenced by less than
10% outliers, is considered
as the method providing
the most reliable estimates

Discussion
 Different results is an indication
that the assumptions of classical
ANOVA and range statistics are not
justified
 Classical ANOVA and range
statistics are strongly affected by
the presence of outlying values (9
out of 104 datasets - 8.7 %).

Discussion (continued)
 The measurement uncertainty of manual sampling of fuels is
dominated by the analytical variance (accounts for the 71 % of
the measurement uncertainty)
 This leaves “room” for an effective reduction e.g. by making
more measurements and calculating their average, instead of
making a single measurement.
 Then the standard deviation of the mean gets smaller as the
number of data increases leading to smaller random error
uncertainty contributions.
2
analysiss -20 % Expanded uncertainty
of measurement

Measurement uncertainty estimation of an
analytical procedure
 Test method studied
Gross Heat of Combustion (GHC) (or Higher Calorific Value)
determination of a diesel fuel using a bomb calorimeter and
following the standard method ASTM D240
 Measurement principle
Heat of combustion is determined in this test
method by burning a weighed sample in
an oxygen bomb calorimeter under
controlled conditions. The heat of
combustion is computed from temperature
observations before, during and after
combustion, with proper allowance for
thermochemical and heat transfer
corrections.

 Equipment used – Parr Instruments
 Parr 6200 calorimeter
 Parr 1108 oxygen bomb
 Parr 6510 water handing system
 Reference material
 Benzoic acid traceable to NIST SRM 39j

Measurement system modeling
repg
g
eeeWt
Q 

 321
rep
t
eegQ
W '
'
''' 21



gSe  582

Uncertainty estimation methods
 GUM uncertainty framework
 Assumed normal distribution
 Assumed t-Student distribution (use of effective degrees of freedom)
 Monte Carlo Method (MCM)
 Fixed number of trials
 Adaptive MCM
 GUM with Bayesian statistics
 Empirical method using interlaboratory study
(Proficiency Testing Scheme) data

GUM uncertainty framework vs Monte Carlo Method (MCM)
Formulation
- Definition of the output quantity (measurand)
- Determination of the input quantities (sources of
uncertainty)
- Development of a model relating the output quantity
with the input quantities
- Assignment of PDFs to the input quantities on the
basis of available knowledge
Propagation
- Propagation of the PDFs of the input quantities through
the model to obtain the PDF for the output quantity
Summarizing
- Use of the PDF of the output quantity to obtain the
expectation (measurement result) of the output quantity
standard uncertainty associated with expectation
- Use of the PDF of the output quantity to obtain a
coverage interval containing the output quantity with a
specified probability
PDF: Probability Density Function
No difference

Formulation
uncertainty)
Propagation
Summarizing
x1, u(x1)
Y = f (X1,X2,X3) y, u(y)x2, u(x2)
x3, u(x3)
Assumed PDF for Y
GUM
MCM
PDF for X1
PDF for X2
PDF for X3
Y = f (X1,X2,X3)
PDF for Y
Propagation of uncertainties
Propagation of distributions

Formulation
uncertainty)
Propagation
Summarizing
GUM
MCM
U = k u(y)

GUM uncertainty framework – Uncertainty budget

GUM uncertainty framework – Uncertainty contributions

GUM uncertainty framework – Expanded uncertainty /
Coverage interval estimation
Assumed probability distribution
Standard uncertainty u(Qg) =141.7 J g-1 (33.84 cal g-1)
Normal (Gaussian)
distribution
t- distribution
Expanded uncertainty U(Qg)=k . u(Qg)
k=1.96
k=2.08
22 effective degrees
of freedom
U(Qg)=277.7 J g-1 (66.3 cal g-1)
U(Qg)=294.6 J g-1 (70.4 cal g-1)
Welch–Satterthwaite formula

Monte Carlo Method (MCM)– Development
The algorithm of MCM was developed in MATLAB®
1
2
…
n
Number of trials
o Fixed (106)
o Increasing number of
trials until the results have
stabilized in a statistical
sense (Adaptive MCM)

Monte Carlo Method (MCM)– MATLAB code flow diagram
Establishment of the process parameters and the
default values of the control variables
Creation of M size row vectors for each of the input
variables
Evaluation of the model. M size file vector
Calculation of the average, standard deviation and
symmetrical interval of the M value sequence
Shortest interval?
Calculation of the shortest coverage interval of the M
value sequence
Value matrices and parameter vectors are formed
First sequence?
Add row to value matrices and parameter vectors
Calculation of the standard deviation of the
parameters
Calculation of the total standard deviation
Calculation of the numerical tolerance related to the
standard deviation
Stabilization?
Calculation of the average and the symmetrical
coverage interval of all the values
Shortest interval?
Calculation of the shortest coverage interval of all the
values
Show results
YES: Interval=1
YES: h=1
YES: Interval=1
YES: comp=1
NO
NO
NO
Lines 3-7*
Lines 8-49*
Lines 50-53*
Lines 54-58*
Lines 59-61*
Lines 62-67*
Lines 68-70*
Lines 71-73*
Lines 74-76*
Lines 77-81*
Lines 82-83*
Lines 84-97*
Lines 98-103*
Lines 104-108*
Lines 109-110*
Lines 111-117*
Lines 118-132*

Monte Carlo Method (MCM)– MATLAB code Flow diagram

Monte Carlo Method (MCM)–– Expanded uncertainty /
Coverage interval estimation
1 using a PC equipped with Intel® Core™ i3 M330, 2.13GHz, 4GB RAM

MCM results vs GUM results (95% coverage intervals)
12% underestimation
7% underestimation

GUM with Bayesian treatment of Type A uncertainties
)(
3
1
)( iiBayes xs
m
m
xu



Standard uncertainty, u(Qg) = 160.7 Jg-1 (38 cal g-1)
95% coverage interval
[44.88 – 45.51] MJ kg-1
or [10719 – 10870] cal g-1
Comparable
to MCM results !!!

Uncertainty evaluated from proficiency testing data
  
zl
sl
s z
i
i
z
i
i
Ri
pooled
R







1
1
2)(
)1(
• The PTS provider is accredited according to
ISO/IEC 17043
• Most of the participants used the standard
method ASTM D240 for the measurement
li :number of participating laboratories in round i,
z: number of rounds
95% expanded uncertainty 0.30 MJ kg-1 (71 cal g-1)
pooled
Rg sQU 96.1)( 
-6,3 % compared to MCM

Estimation of the standard uncertainty of a
calibration curve
Calibration often
comprises an important
uncertainty component
of the uncertainty of the
whole analytical
procedure
The slope and the intercept of a linear calibration model
are only estimates based on a finite number of
measurements
Therefore their values are associated with uncertainties

calibration curve
2 stage - measurement model
xbbY 10 
1
00
pred
b
by
x


calibration data
(pairs of xi yi)
0y
correlated

calibration curve
The standard uncertainty of a calibration curve used for
the determination of sulfur mass concentration in fuels
has been estimated using 4 methodologies:
 GUM uncertainty framework
 Kragten numerical method
 Monte Carlo method (MCM)
 Approximate equation calculating the standard error of
prediction
 
 


 n
i
i xxb
yy
Nnb
SE
xs
1
22
1
2
0
1
regression
pred
11
)(
xbbY 10 
1
00
pred
b
by
x



calibration curve
Results
Mean value (ng μL-1)
Standard uncertainty
(ng μL-1)
GUM (correlation included) 8.000 0.175
Kragten method (correlation included) 8.000 0.172
MCM (correlation included) 8.003 0.175
Standard error of prediction equation
(including response uncertainty) 8.000 0.175
Standard error of prediction equation
(no response uncertainty included) 8.000 0.137
GUM (no correlation included) 8.000 0.283
Kragten method (no correlation
included) 8.000 0.279
MCM (no correlation included) 8.005 0.284
 
 


 n
i
i xxb
yy
Nnb
SE
xs
1
22
1
2
0
1
regression
pred
11
)(
 2
02
2
predpred )()(')( yucxsxu 
 
 


 n
i
i xxb
yy
nb
SE
xs
1
22
1
2
0
1
regression
pred
1
)('
Overestimation of uncertainty by 62%

calibration curve
Bivariate (or joint) Gaussian distribution N(E,V)
characterized by the expectation and the covariance (or
uncertainty) matrices, E and V







1b
b
E o







)(),(
),()(
1
2
10
100
2
bubbu
bbubu
V
A coverage region can be determined
Two types:
•rectangle centered coverage region (separately
determined coverage intervals for b1 and b0).
•ellipse centered coverage region
specifies a region in
2-dimensional space
that contains E with
probability p
Treating calibration curve as a bivariate measurement model

calibration curve
(η – E)T V-1 (η – E) = kp
2
  2
11
00
1
1
2
10
100
2
1100
)(),(
),()(
pk
bn
bn
bubbu
bbubu
bnbn 















Rectangular and elliptical coverage regions (p=0.95)
p=0.95

Measurement uncertainty and precision data in
Certain approaches should be used to support reliable
decisions in conformity assessment of fuels (EN 228, EN 590)
It is necessary to take into account the
dispersion of the values that can be
attributed to the measurand.

 The comparison of the result with the specified requirements should be based
on predefined decision rules, which are of key importance when the result is
close to the tolerance limit
 Use of guard bands to determine acceptance or rejection zones taking into
account measurement variability
Guarded acceptance
Guarded rejection
(Relaxed acceptance)
No rule

Guard bands minimize the probability of incorrect decisions (risks)
Guarded acceptance decision rules for upper
and lower specification limits (TU, TL) and
maximum probability of false
acceptance (Type II error) when guard
bands of width w are used
Guarded rejection (relaxed acceptance)
decision rules for upper and lower
specification limits (TU, TL) and maximum
probability of false rejection (Type I
error) when guard bands of width w are
used

Approaches for defining guard bands
 The between laboratory precision data approach (ISO 4259 approach)
 The intermediate precision (or uncertainty estimate) approach
ISO 4259:2006
Petroleum products -- Determination and application of precision
data in relation to methods of test
Widely used in fuel market for resolution of disputes

The between laboratory precision data approach (ISO 4259 approach)
Sulphur content test method limits
Product Automotive diesel
Specification 10 mg/kg
Dispute Sulphur content off specification
Test method ISO 20846
Reproducibility value R = 0.112*x + 1.12
x = sample result
For x=10, R = 2.24
Upper limit of guard band = TU+0.59*R
=10+0.59*2.24
=11.3 mg/kg
The product can be considered as FAILING the specification when a
single test results falls above 11.3 mg/kg (for 95% confidence level)

The between laboratory precision data approach (ISO 4259 approach)







k
rRR
1
122
1
For multiple (k) test results
The expressions have been applied for all the parameters related to
automotive fuel quality referred in EN 590:2009 and EN 228:2008 and
the acceptance limits were calculated using the precision data of the
relevant test methods

The between laboratory precision data approach (ISO 4259)
unleaded petrol
(gasoline) EN228
automotive diesel
EN 590

The intermediate precision (or uncertainty estimate) approach
Sulphur content test method limits
Product Automotive diesel
Specification 10 mg/kg
Dispute Sulphur content off specification
Test method ISO 20846
Standard uncertainty value u = 0.31
Upper limit of guard band = TU+1.64*u
=10+1.64*0.31
=10.5 mg/kg
The product can be considered as FAILING the specification when a
single test results falls above 10.5 mg/kg (for 95% confidence level)

The intermediate precision (or uncertainty estimate) approach
2
analysis2
sampling 






k
u
uu
For multiple (k) test results

Automotive diesel fuel samples were taken from 769 petroleum retail
stations and their sulfur mass content was determined in order to assess
their compliance with the EU regulatory limit of 10 mg kg-1

The effect of different approaches for defining guard bands, different levels
of confidence or different number of replicate measurements is
investigated
ISO 4259 approach

Intermediate precision approach

 General remarks
 The larger the width of the guard band used, the larger the proportion of
samples that will be judged incorrectly
 The differences in the calculations of the two approaches reflect a possible
number of samples judged incorrectly when using the ISO 4259 approach
(uncertainty estimates represent more precisely the dispersion of the values of
the measurand)
 Minimizing the guard band width by reducing the measurement uncertainty
(more replicates, more accurate measurement method) leads to fewer cases
of false acceptance or false rejection decisions, reducing as well the
costs associated with these decisions.
 At the same time the cost of analysis becomes higher, there is a need that
these two costs are balanced against each other in order to find an optimum
(target) measurement uncertainty

Conclusions
SAMPLING
MEASUREMENT
PROCEDURE
REPORTING
RESULTS AND
ASSESSING
CONFORMITY
Statistical and numerical methods have been developed
and/or applied concerning the estimation and use of the
measurement uncertainty in all parts of the measurement
process.

Conclusions
SAMPLING
MEASUREMENT
PROCEDURE
REPORTING
RESULTS AND
ASSESSING
CONFORMITY
Manual sampling from petroleum retail stations for sulphur determination
 Three alternative empirical statistical approaches
 Expanded uncertainty of sampling estimated in the range of 0.34 – 0.40
mg kg-1.
 The estimation of robust ANOVA (0.40 mg kg-1) is considered more
reliable, because of the presence of outliers

Conclusions
SAMPLING
MEASUREMENT
PROCEDURE
REPORTING
RESULTS AND
ASSESSING
CONFORMITY
Gross Heat of Combustion of diesel fuel
 Two alternative modelling approaches (GUM and Monte Carlo Method)
 GUM approaches (Gaussian or t-distribution) underestimate
measurement uncertainty
 Overall Monte Carlo Method is a more reliable tool (subject to fewer
assumptions) (0.32 MJ kg-1)
 Bayesian treatment of Type A uncertainties “corrects” GUM estimates

Conclusions
SAMPLING
MEASUREMENT
PROCEDURE
REPORTING
RESULTS AND
ASSESSING
CONFORMITY
Calibration curve construction (Sulphur content determination)
 Four alternative approaches (GUM, Monte Carlo Method, Kragten,
standard error equation)
 All approaches agree well (std uncertainty 0.172 – 0.175 ng μL-1)
 Importance of correlation – correct use of standard error equation

Conclusions
SAMPLING
MEASUREMENT
PROCEDURE
REPORTING
RESULTS AND
ASSESSING
CONFORMITY
Conformity assessment of automotive fuel products (EN 590, EN 228)
 Acceptance limits for guarded acceptance and guarded rejection for 95%
and 99% confidence levels
 Significant differences in the resulting number of non-conforming results
when using different approaches for defining guard bands, different
levels of confidence or different number of replicate measurements

Applications
The program codes developed in MATLAB in order to
apply the Monte Carlo method (adaptive and fixed trials)
may be used in any type of measurement
Decision Support System for the Evaluation of Conformity
of Fuel Products (key features defined)

Future Work
Bayesian uncertainty analysis – Development of numerical
techniques
Conformity assessment – Take into account the variability
of both the measuring system and the process (production
system)
Sampling - Development of methods for the estimation
and the inclusion of sampling bias

List of publications - Conferences
PUBLICATIONS
1. D. Theodorou, Y. Zannikou, G. Anastopoulos, F. Zannikos. Coverage interval estimation of the
measurement of Gross Heat of Combustion of fuel by bomb calorimetry: Comparison of ISO GUM and
adaptive Monte Carlo method. Thermochimica Acta (2011) 526: 122– 129
2. D. Theodorou, Y. Zannikou, F. Zannikos. Estimation of the standard uncertainty of a calibration curve:
application to sulfur mass concentration determination in fuels. Accreditation and Quality
Assurance (2012) 17: 275–281
3. D. Theodorou, N. Liapis, F. Zannikos. Estimation of measurement uncertainty arising from manual
sampling of fuels. Talanta (2013) 105: 360-365
4. D. Theodorou, F. Zannikos. The use of measurement uncertainty and precision data in conformity
assessment of automotive fuel products. Measurement (2014) 50: 141-151
5. D. Theodorou, Y. Zannikou, F. Zannikos. Components of measurement uncertainty from a measurement
model with two stages involving two output quantities. Chemometrics and Intelligent
Laboratory Systems (2015) 146: 305–312
CONFERENCES
 5th National Congress on Metrology "Metrologia 2014”
 EuroAnalysis 2013 - XVII European Conference on Analytical Chemistry
 4th National Congress on Metrology "Metrologia 2012”

Acknowledgments
Advisory Committee
 F. Zannikos –Professor – School of Chemical Engineering, NTUA
(Supervisor)
 K. Tzia - Professor – School of Chemical Engineering, NTUA
 D. Karonis – Associate Professor - School of Chemical Engineering,
NTUA
Fuels and Lubricants Technology Laboratory
 Staff
 Partners
 Graduate / Post graduate students

ΤΗΑΝΚ YOU FOR YOUR ATTENTION !
ΕΥΧΑΡΙΣΤΩ ΓΙΑ ΤΗΝ ΠΡΟΣΟΧΗ ΣΑΣ !

phd thesis presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to phd thesis presentation

Similar to phd thesis presentation (20)

phd thesis presentation