DOE - KVL
(1.1)
Design Of Experiments
(031025)
a 3-day reminder
Frans van den Berg
The Royal Veterinary and Agricultural University (KVL), Denmark
Dept. of Dairy and Food Science, Food Technology
Chemometrics group
fb@kvl.dk
www.models.kvl.dk
Per M. Bruun Brockhoff
The Royal Veterinary and Agricultural University (KVL), Denmark
Dept. of Mathematics and Physics,
Statistics group
pmb@kvl.dk
www.matfys.kvl.dk

DOE - KVL
(1.2)
Design Of Experiments
a 3-day reminder
Day 1 1. 13:00-13:45h - Introduction; what was experimental design again?
2. 14:00-14:45h - Design in one formula: y = X.b
3. 15:00-15:45h - Statistical inference and ANOVA
Day 2 4. 09:00-12:00h - Hands-on DOE: computer exercise I *)
5. 13:00-13:45h - Data inspection and plotting (+ some PCA)
6. 14:00-14:45h - Miscellaneous subjects
7. 15:00-15:45h - Example: case study I
Day 3 8. 09:00-12:00h - Hands-on DOE: computer exercise II *)
9. 13:00-13:45h - Blocking and split-plot
10. 14:00-14:45h - Example: case study I
11. 15:00-15:45h - Introduction to mixed linear models
Participants can choose to take part in the theory (3 afternoons) sessions or theory + computer exercise
*)
(2 mornings) sessions.

DOE - KVL
(1.3)
Design Of Experiments
Accompanying literature
• D.E. Coleman and D.C. Montgomery \"A Systematic Approach to Planning for a Designed
Industrial Experiment\" Technometrics 35/1(1993)1-12
Some ideas on the setup and execution of experimental (industrial) designs
• J. Guervenou el.al. “Experimental design methodology and data analysis techniques applied
to optimise an organic synthesis” Chemometrics and Intelligent Lab. Sys. 63(2002)81-89
A typical experimental design and optimization example
• P. BruunBrockhoff “Sensory profile average data: combining mixed model ANOVA with
measurement error methodology” Food Quality and Preference 12(2001)413-426
An advanced study in experimental design and statistical inference in food research

DOE - KVL
(1.4)
Design Of Experiments
Definitions
Experiment:
“A test or series of tests in which purposeful changes are made to the
input variables of a process or system so that we may observe and
identify the reasons for changes that may be observed in the output
response.” a)
Design:
“The art or process of deciding how something will look, work, etc.” b)
Motivation:
“The statistician’s aim in designing surveys and experiments is to meet a
desired degree of reliability at the lowest possible cost under the existing
budgetary, administrative, and physical limitations within which the work
must be conducted. In other words, the aim is efficiency - the most
information (smallest error) for the money.” c)
Definition, like many of the ideas in course, taken from Douglas C. Montgomery “Design and
a)
Analysis of Experiments” Wiley (2001, 5th)
b) Definition taken from Oxford Advanced Learner’s Dictionary (2000, 6th)
c) William E. Deming “Some theory of sampling” Dover (1950 reprint); quoting R. A. Fisher

DOE - KVL
(1.5)
Leading example
Making apple juice
Controllable factors Output
(experimental domain) (sensor score/response)
pH Sugar Sensors
Uncontrollable factors
(‘nuisance factors’)
…?
Material Production Technician Periphery

DOE - KVL
(1.6)
Variables
Rational, ordinal or nominal
• Interval or ration scales; e.g. pH or sugar concentration
• measurement or quantitative variables
• continuous or discrete (e.g. counting)
• most often encountered and easiest case
• Ordinal scale; e.g. “very poor”, “poor”, “average”, “good”, “excellent”
• called ranked variables
• distinct graduation, but scale-distance defined
• Nominal scale; e.g. “green”, “red”, “yellow” or “accept”, “reject”
• qualitative of categorical variables or attributes
• require some special “tricks” in statistical inference

DOE - KVL
(1.7)
Design Of Experiments
Objective
Controllable factors x … x
(Input) Output y
Process
Uncontrollable factors z … z
a) Does x influence y, and if so, how?
b) Which inputs x are the most influential on the output y
c) How to set x’s so that y is (almost) always near target or optimal
d) How to set x’s to minimize variability in y
e) How to set x’s so that influence of z’s on y is minimized
Strategy of experimentation is the most important job of the experimenter

DOE - KVL
(1.8)
Design Of Experiments
Different approaches
• Best guess (often works well due to good insight on the problem!)
• One factor at a time
(“pseudo scientific”)
score
score
low high less more
pH Sugar
• Factorial design
low pH
high
(22 to reveal interactions)
score
pH
high pH
low
less more less more
Sugar Sugar

DOE - KVL
(1.9)
Example
2 factor factorial design
• Five sensors score product (apple juice) for each design point
average is product score
(don’t like) 1 10 (like)
√
• Design is replicated twice: 22 x 2 = 4 x 2 = 8 experiments
(6,5) (7,7)
high
• Result
pH
low
(6,6) (9,8)
less more
Sugar

DOE - KVL
(1.10)
Example
2 factor factorial design
• Main effects
high
7+7+9+8 - 6+5+6+6
= 2.0
Sugar and pH 4 4
pH
low
high
less more
Sugar
pH
low
6+5+7+7 - 6+6+9+8 less more
= -1.0
4 4 Sugar
• Interaction
effect
high
7+7+6+6 - 6+5+9+8
= -0.5
Sugar.pH 4 4
pH
low
less more
Sugar

DOE - KVL
(1.11)
Example
2 factor factorial design
Important in interpretation are magnitude and direction of the effects:
• sweet juice has a clear preference
• a low pH leads to a higher score
• the interaction Sugar-pH is weak
-0.5
high
pH
low
-1.0
less more
Sugar
2.0

DOE - KVL
(1.12)
Design Of Experiments
Why DOE; different approaches revisited
• Best guess: good starting values, but “areas unvisited” remain unknown!
• One factor at a time:
inefficient use of the data
score
score
(“2 small factor designs”)
low high less more
pH Sugar
• Factorial design:
high
maximum use of the data,
since all observations are used
pH
for all the main and interaction
low
effects! And, the trend in the
less more
surface gives an indication for
Sugar
“areas unvisited”.

DOE - KVL
(1.13)
Example
3 factor factorial design
• Main effect Apple/Material 23 = 8 experiments (still!)
(6) (7)
(6) (9)
high
(5) (7)
pH
(6) less more (8)
low
Sugar
6+6+9+7 - 6+5+8+7
= 0.5
4 4

DOE - KVL
(1.14)
Design Of Experiments
Why DOE; relative efficiency
• Factorial design
high
Relative efficiency
4 observations
(6/4) = 1.5
pH
all effects estimated
as average over two
low
3.5
less more
Sugar
3
2.5
2
high
1.5
2 3 4 5 6
• One factor at a time
pH
2x
factors
Needs 6 observations
low
to get the same
4 8 16 32 64
less more
information
observations
Sugar

DOE - KVL
(1.15)
Example
4 factor fractional factorial design
• Main effect production ½ x 24 = 8 experiments (still!)
Full information on the main effects, partial information on the interactions
Production
Hand press Kitchen blender
(6) (7)
(9) (6)
green
Material
high
(5)
(7)
red
pH
low
(6) less more less more (8)
Sugar Sugar

DOE - KVL
(1.16)
Confounding
4 factor fractional factorial design
Of course you loose something by reducing the number of experiments…
• Main effects and interaction effects will be confounded
• Confounding means: we can not separate some effects/interactions
For the example:
• We have 4 factors (Sugar, pH, Material and Production)
• There are 4 blocks (2x Material plus 2x Production)
• In this case: block effects and threefold interactions are confounded
• E.g. Material (apple) effect and Sugar-pH-Material effect are confounded
• When reducing a full design, usually the assumption is made that high-order
interactions are unimportant
• When reducing the design you have to carefully select the ‘things’ confounding

DOE - KVL
(1.17)
Example
Quality of the response
(7)
10
(9)
(6)
(8)
(5)
(6)
1
high
more
pH
Sugar
low
less

DOE - KVL
(1.18)
Example
Quality of the response
• Five sensors ‘score’ a product for each design point
average is product score
repeated measurements
average = (6)
(don’t like) 1 10 (like)
average = (6)
• Even if not explicitly used in the statistical analysis of a design, it is of utmost
importance to have an impression of uncertainty in the response!!!
• Laboratory info, analysis replicates, a priory knowledge, literature values…

DOE - KVL
(1.19)
Example
Quality of the response
Signal-to-noise ratio
for repeated measurements
10
1
high
more
pH
Sugar
low
less
Subtle difference in definition of error in statistics: ‘error’ as in ‘wandering’ (e.g.
knight errant) rather than ‘incorrect’. Observations come from a population,
based on common part (e.g. average) and unique part (e.g. error).

DOE - KVL
(1.20)
Example
Quality of the response
Center point replicates are a good indication of the reproducibility of design points,
plus they can give a (cheap) indication of curvature in the response.
(3x ; 5x)
Total:
8 + 3 = 11
8 + 5 = 13

DOE - KVL
(1.21)
Errors
Random versus systematic
repeatability
↓xbar
Day 1
reproducibility
Day 2
↑xbar ↑µx
bias
variance is repeatability & reproducibility is a function of sample size n
bias is not!
Error: difference between true and observed value
x(i) - µx
e(i) =
e(i) = (x(i) - xbar) + (xbar - µx)
True value µx (ISO): “The value which
- random systematic
characterizes a quantity perfectly defined in
the conditions which exist at the moment when
- imprecision bias
that quantity is observed (or the subject of a
- precision accuracy
determination). It is an ideal value which could
be arrived at only if all causes of measurement
error were eliminated and the population was
repeatability reproducibility
infinite”.

DOE - KVL
(1.22)
The three basics
Replication, randomization and blocking
Signal-to-noise ratio
for design replicates
10
(less; high)
v.
(more; low)
1
high
more
pH
Sugar
low
less

DOE - KVL
(1.23)
The three basics
Replication, randomization and blocking
Signal-to-noise ratio
for design replicates
10
!
?
1
high
more
pH
Sugar
low
less
Design point has experimental error = statistical error = a random variable

DOE - KVL
(1.24)
The three basics
Replication, randomization and blocking
Underlying statistical methods require that the observations (or errors) are
independent distributed random variables. Randomization of e.g. starting
material and run-order of the design points (usually) makes this assumption valid.
“Reduce experimental error by training”
Less More Less More Center
Low Low High High points
time
experiment
Less More Less More Center
Low Low High High points

DOE - KVL
(1.25)
The three basics
Replication, randomization and blocking
All sorts of effects can influence a series of observation:
• learning by experience: reducing the uncertainty
• wear-and-tear in equipment: increasing the uncertainty
• a change in lab-assistants: a jump in uncertainty
Randomization “reshuffles” the observations, eliminating a potential confounding
between design- and run-order. It “averages out” the effect of uncontrollable
extraneous factors.*)
“Shake 10 minutes”
10
time
10
experiment
*)
Randomization is also the justification/motivation behind the so-called F-test, used excessively later in this course.

DOE - KVL
(1.26)
The three basics
Replication, randomization and blocking
So-called Blocking is capable to eliminate undesired/nuisance factors,
by asking a different question
2
1(1)
e.g. (1)
(0)
- =
0
high
more
pH
Sugar
low
(0) less
“Improve signal”, but at a price…

DOE - KVL
(1.27)
The three basics
Replication, randomization and blocking
Blocking can also be a ‘necessary evil’, destroying the desired randomization.
E.g. assume we don’t have enough green apples to run the full experiment twice.
(6) (7)
(6) (9)
high
(5) (7)
pH
(6) less more (8)
low
Sugar

DOE - KVL
(1.28)
The three basics
Replication, randomization and blocking
There is a link between the three basics!
E.g. we want to perform 30 experiments (replicates), but we can only do 10 runs
from one batch of raw material (a typical nuisance factor).
Block effect
block 1 block 2 block 3
Uncertainty
randomized

DOE - KVL
(1.29)
Some basic notions
Sample and population
pH 1
4.90
5.06
pH1
5.05
n1 = 10x 5.17
5.06
Samples: 10 pH-measurements 4.94
taken from flask 1 5.04
4.90
Population: all the possible pH-
5.00
values to be found in flask 1
5.00
We assume continuous
pH 1
distribution in population (not
always the case; e.g. pH in
European rivers)
4.7 4.8 4.9 5 5.1 5.2 5.3
pH

DOE - KVL
(1.30)
Some basic notions
Expectation and population parameters mean & variance
Expected value sample statistic for n observation
∑ x (i )
µ x = E ( x) → x = i =1:n
Mean Locality
n
(x(i ) − x )2
∑
( )
σ = E (x − µ x ) 2
→ sx = i =1:n
2 2
Spread
Variance x
n −1
Eg: Normal distribution N(µx,σx)
µ σ
68%
Notice: µ and σ are
95% Observations
population constants
100%

DOE - KVL
(1.31)
Some basic notions
(x(i ) − x )2
∑ x (i ) ∑
E (x ) = µ
x= sx = sx = sx
i =1:n i =1:n
2 2
n −1
n
Mean Variance Standard deviation
sx s
sr (% ) = 100 sr
SE x = sr =
x
n
Standard error Relative SD RSD in %
of the mean (coefficient of variation)
Pr (µ − t n −1SE x < x < µ + t n −1SE x ) = 95%
Pr ( x − t n −1SE x < µ < x + t n −1SE x ) = 95%
95% confidence interval/level; n large or sx given tn-1 = z0 = 1.96

DOE - KVL
(1.32)
Some basic notions
Critical t-values
(a) α is users choice
(b) Increasing for α
(c) Decreasing for n
(d) n large (or σ known)
(a) (b)
(c)
(d)

DOE - KVL
(1.33)
Some basic notions
Sample statistics (= descriptors in numbers)
pH 1
SEpH1 = 0.084/√10 = 0.026 t9 = 2.26
4.90
5.06
5.01 – 2.26x0.026 < µpH1 < 5.01 + 2.26x0.026
5.05
4.95 < µpH1 < 5.07
5.17
5.06
4.94 Assumption: Normal distribution N(xbar,sx)
5.04
xbar sx
4.90
5.00
5.00
pH 1
Sum 50.1
Mean 5.01
4.7 4.8 4.9 5 5.1 5.2 5.3
Variance 0.0070
pH
S.D. 0.084

DOE - KVL
(1.34)
Some basic notions
Pooled standard deviation
pH 1 pH 2
4.90 5.10
5.06 5.07
pH1 pH2
5.05 5.21
n1 = 10x n2 = 10x 5.17 4.91
5.06 5.14
Two ‘treatments’ 4.94 5.19
e.g. making pH-buffers with 5.04 5.17
two different stock solutions
4.90 5.16
5.00 5.10
5.00 5.17
pH 1
pH 2
4.7 4.8 4.9 5 5.1 5.2 5.3
pH

DOE - KVL
(1.35)
Some basic notions
Comparing two samples
pH 1 pH 2
4.90 5.10
5.06 5.07
pH 1
5.05 5.21
pH 2
5.17 4.91
5.06 5.14
4.7 4.8 4.9 5 5.1 5.2 5.3
4.94 5.19
pH
5.04 5.17
4.95 < µpH1 < 5.07
4.90 5.16
5.06 < µpH2 < 5.18
5.00 5.10
Assuming the variance in flask 1 and 2 is the same:
5.00 5.17
Sum 50.1 51.2
(n1-1).s21 + (n2-1).s22 0.0630 + 0.0666
s2 = = = 0.0072
Mean 5.01 5.12 pooled
(n1-1) + (n2-1) 9+9
Variance 0.0070 0.0074
9 degrees-of-freedom (df) in estimating
S.D. 0.084 0.086
each of the standard deviations

DOE - KVL
(1.36)
Design Of Experiments
In distinct steps
1. Recognition and definition of the problem
2. Choice of factors, levels and ranges pH Sugar
3. Selection of the response variable
4. Choice of experimental design
Sensors
5. Performing the experiment
6. Statistical analysis of the data
Design
7. Conclusions and recommendations
experiment
Statistical
analysis

DOE - KVL
(1.37)
Design Of Experiments
An iterative process; e.g. optimization
Factorial screening experiment
for initial optimization
pH
Unknown response surface
with score contour lines
Central composite
Sugar design for detail
pH
optimization
“(11)”
(9) (10)
(8)
Sensors
Sugar