This webinar looks at answering this question, not by going deeply into the various designed experiment types, but from a process improvement perspective. Progressing from a definition of a designed experiment, to Why and when do I need a designed experiment?, What’s the concept? (and why can’t I do a “one-factor-at-a-time” series of experiments? , to Will this tool solve REAL WORLD problems?

1. Why Do a Designed
Experiment
Jim Breneman
©2011 ASQ & Presentation Jim Breneman
Presented live on Sep 08th, 2011
http://reliabilitycalendar.org/The_Reli
ability_Calendar/Webinars_‐
_English/Webinars_‐_English.html

2. ASQ Reliability Division
English Webinar Series
One of the monthly webinars
on topics of interest to
reliability engineers.
To view recorded webinar (available to ASQ Reliability
Division members only) visit asq.org/reliability
To sign up for the free and available to anyone live
webinars visit reliabilitycalendar.org and select English
Webinars to find links to register for upcoming events
http://reliabilitycalendar.org/The_Reli
ability_Calendar/Webinars_‐
_English/Webinars_‐_English.html

3. Why do a Designed
Experiment?
- What is it?
- Why do I need it?
- How do I do it?
- Can it solve my problem?
9/8/2011 Jim Breneman1

4. What is Design of Experiments?
• A designed experiment is a test or series of tests in which purposeful
changes are made to the input variables of a process or system so
that we may observe and identify the reasons for changes in the
output response…
Doug Montgomery
9/8/2011 2

5. Why do I need a Designed experiment?
General Model of a Process or System
Process
Controllable factors
w1 w2 wp
...
NOISE
Inputs Transformation Vehicle Output
y
x1 x2… xp
NOISE We often want to
... Minimize
Maximize
z1 z2 zq or
Reduce variability
Uncontrollable factors in the Output(y)
9/8/2011 3

6. Where does a Designed
Experiment fit in?
Let’s look at Deming’s PDCA process
and then the DMAIC process
Control Define
Act Plan
Check Do
Improve Measure & Analyze
Major DOE use
9/8/2011 4

7. Step 1: What’s the objective of Your Experiment?
• Comparative objective:
– Primary goal is make a conclusion about one a-priori important factor.
– i.e. is this factor “significant” (and possibly what level maximizes or minimizes the
response)
• Process Improvement objective (Sequential experimentation):
– Step 1: Screening…the primary purpose of the 1st experiment is to select or screen out
the few important main effects from the many less important ones.
– Step 2: Followed up with experiment(s) to define the important 2-factor interactions,
(and any 3-factor interactions that may be important based on experience).
• Response Surface objective:
– The experiment is designed to allow us to estimate interaction (and even quadratic
effects), and to optimize the response or responses.
– Each factor is usually at 3 levels.
9/8/2011 5

8. Step 2 What level of Evidence will I accept?
I. Controlled Trials with complete randomization. DOE
II. Empirical Evidence.
a) Controlled Trials without complete randomization.
b) Case directed studies. Carefully observing cases as they occur.
c) Multiple Time Sequence Studies (Looking back through data files for
patterns and drastic changes)
“Scientific Studies have shown….”
III. Delphi (Agreement between a group of knowledgeable “experts”)
“8 out of 10 doctors recommend …..”
IV. Personal Antidote (“In my experience”)
V. Personal bullying (“I think we should do it this way”)
9/8/2011 6

9. Step 3: The DOE Roadmap
D
Design
Brainstorm Experiment
Run Experiment Sequential
M & Collect Data experimentation
Analyze data/
A Interpret results
Choose variables
& levels
I Run confirmation
test
C Incorporate into design
or process
9/8/2011 7

10. The Strategy of Experimentation
1. Screening experiments to find
the mountain range
2. Factorial/Fractional factorial Region of Interest
experiments to get close to
the peak.
3. Response Surface Modeling
to “climb” it.
Region of Operability
9/8/2011 8

11. Review of Basics
Language
• Factor: An independent variable. This is what we control and
change in an experiment. A factor is often generically
referred to as xi.
Examples: Reaction Temperature, Bake Time, Fuel flow, Stress
• Factor Setting or Level: A particular value for a factor.
For example, the factor Bake Temperature might have a setting of 275° F. Bake
Temperature is the factor, 275° F is one of the levels.
• Experimental Run: A particular combination of factor
settings.
For example, one run in an experimental design (for say a composite piece) might call for a
Bake Temperature of 275° F, a Bake Time of 30 minutes, and a Bake pressure of 5Atm.
9/8/2011 9

12. Review of Basics
Language
• Experimental Design: The complete set of runs that we plan
to do. It is sometimes called the Design Matrix. Experimental
design in general is often referred to as DOE or DOX (Design of Experiments).
• Response: A dependent variable. The level of the response is
measured rather than controlled like a factor. It is referred to as a
response because we think its level will change in response to changes in the factor settings.
One of the goals of experimental design is to relate changes in the factor levels to measured
changes in the response values.
Examples of Typical Response Variables: Tensile Strength, Elongation, Thrust
• Factor Effect: The way in which changes in the level of a
factor translate to changes in the response level.
9/8/2011 10

13. Review of Basics
Language
• Interaction: When the effect of a factor depends on the level
of another factor, the two factors are said to interact:
Life=f(Stress, Temp)
Temp 1
Life
(hrs)
Temp 2
Stress
9/8/2011 11

14. General Observations of DOEs
In General
1. Several factors (or main effects) will be significant
2. Some two-factor interactions will be significant
3. Very few (if any) three-factor and higher order interactions will be significant
Concentrate on main effects and 2-factor interactions in your experiments.
However, if a three-factor interaction is perceived to exist, then include it in the
experiment!
9/8/2011 12

15. Why Do Statistically Designed Experiments?
• Statistically designed experiments can detect and describe
factor-factor interactions.
Experiments that vary only one factor at a time and trial-and-error
experiments cannot.
• Statistically designed experiments offer more precise
estimates of factor effects for the same number of runs
compared to a one-factor-at-a-time (ladder) study.
This is because DOE’s use “hidden replication” and the
power of averaging to see through noise.
Let me illustrate this with an engineering example.
9/8/2011 13

16. DOE vs One-Factor-at-a-Time(OFAT)
Cooling Metal
Temp Air Temp Temp
Deg F Deg F Deg F
An engineer performed an experiment on a new piece
600 2500 1900
of equipment . As a function of three factors: 700 2500 1900
• Cooling Temp(°F) 800 2500 1900
• Air Temperature (°F) 900 2500 1900
• Metal temperature (°F) 1000 2500 1900
800 2300 1900
The objective was to maximize the response (y variable);
800 2400 1900
in this case, part life.
800 2500 1900
The engineer performed the experiment as a one-factor-
800 2600 1900
at-a-time for three factors in 15 runs 800 2700 1900
800 2500 1700
800 2500 1800
800 2500 1900
800 2500 2000
800 2500 2100
9/8/2011 14

17. DOE vs One-Factor-at-a-Time
Cooling Temp(°F)
1000
Illustrating this, we see that we can estimate that
effect, both linear & quadratic; however, we MetalTemp (°F)
900
cannot estimate interactions.
2300 2400 800
2500 2600 2700
Air Temp (°F)
700
600
9/8/2011 15

18. DOE beats OFAT – Round 1
The Box Behnken designed experiment shown
here and in the accompanying figure could have
been performed instead. Both the One-Factor-at-
a-Time and the designed experiments have 15
runs( if three center points are used in the Box
Behnken design to make the design rotatable
and to provide an estimate of natural variability).
And, the Box-Behnken estimates interactions
and their importance!
Cooling Temp(°F)
9/8/2011 16

19. OFAT vs DOE – Round 2
DOE’s “Hidden replication” beats OFAT
Cooling Temp(°F)
1000
4 points
MetalTemp (°F)
900
1 point
Cooling Temp(°F)
2300 2400 800
2500 2600 2700
Air Temp (°F)
700
600
1 point
9/8/2011 18

20. DOE Review
Planning Carefully
• DOE provides a useful framework for applied
experimentation. However, there’s no magic
involved and one of its advantages is that it
forces some rigorous thinking before an
experiment is started
9/8/2011 19

21. DOE Review
Planning Carefully
1. What do we want to have accomplished when the experiment is
finished?
2. What responses are in my job objectives?
3. How well can we measure these responses?
4. What factors are likely to cause these responses to vary?
5. How many factors can I reasonably investigate in a single
experiment?
6. Over what range should they be varied?
9/8/2011 20

22. Review
Planning Carefully
How will I manage the noise.
1. Consider making the noise factor into an experimental factor
for study.
2. Hold the noise factor as constant as possible during the
experiment.
3. Randomize the experiment.
4. Measure the noise factor levels for future analysis
(covariates).
5. Ignore it.
9/8/2011 21

23. Review
Noise Management
• Noise makes the effects of controllable factors more
difficult to see.
• If it happens that noise variables interact with controllable
variables, the conclusions we draw from an experiment will
only be valid at the noise variable settings experienced
during the experiment.
• As a result, the experiment may not repeat at a later time.
i.e there may be an indication that noise variables are
present with larger effects than the controllable variables!
9/8/2011 22

24. Types of Designed Experiments
– One-way ANOVA
• Randomized Complete Block Design (RCBD)
• Latin Square (& Graeco-Latin Square) Designs
• Balanced Incomplete Block Designs
– Screening Designs
– Factorial, Fractional Factorial Designs
– Response Surface Designs
– Custom Designs
– Nested Designs
– Split-Plot Designs
– Mixture experiments
9/8/2011 23

25. Can DOE solve my problem?
Responses:
1. Performance Model Assumptions
2. $
3. Safety The Model: Y=b1X1 + b2X2 + b12X1X2 + Noise
1. Each factor can be varied independently of the others.
2. All points in the experimental region are feasible:
There’s a difference between “infeasible” experimental runs and those
that are merely expected to give “bad” results.
3. Factors are continuous (so a center point makes some sense)
4. The experimental noise is consistent across the design space.
5. All factors can be run across the other factors.
6. Runs can be done in a random order.
7. There is no curvature in the model.
8. There are no CONSTRAINTS on Resources.
9/8/2011 24

26. A DOE Path Forward
Assumptions If No, then What
Each factor can be varied independently of
1 the others. Mixture or D-Optimal
All points in the experimental region are
2 feasible: D-Optimal
Factors are continuous (so a center point
3 makes some sense) Categorical Factorials
The experimental noise is consistent across
4 the design space. Blocking
5 All factors can be run across other factors Nested Designs
6 Runs can be done in a random order. Split Plots
RSM (Central Composite
7 There is no curvature in the model. Models)
8 There are no CONSTRAINTS on Resources. Fractional Factorials
9/8/2011 25

27. 0. Examples Of Experimental Designs
Full Factorial Experiment 1/2 FFE
A1 A2 A1 A2
B1 B2 B1 B2 B1 B2 B1 B2
C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2
D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2
G1 G1
F1 F1
G2 G2
E1 E1
G1 G1
F2 F2
G2 G2
G1 G1
F1 F1
G2 G2
= E2
G1 G1
F2 F2
G2 G2
1/4 FFE 1/8 FFE
A1 A2 A1 A2
B1 B2 B1 B2 B1 B2 B1 B2
C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2
D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2
G1 G1
F1 F1
G2 G2
E1 E1
G1 G1
F2 F2
G2 G2
G1 G1
F1 F1
G2 G2
E2 E2
G1 G1
F2 F2
G2 G2
9/8/2011 26

28. But, you could start by doing 8 experiments with these 7
variables to find which are MOST important
Taguchi L8 Orthogonal Array or Plackett-Burman
8-run screening design
Variables
Trial A B C D E F G
No. 1 2 3 4 5 6 7
1 1 1 1 1 1 1 1
2 1 1 1 2 2 2 2
3 1 2 2 1 1 2 2
4 1 2 2 2 2 1 1
5 2 1 2 1 2 1 2
6 2 1 2 2 1 2 1
7 2 2 1 1 2 2 1
8 2 2 1 2 1 1 2
9/8/2011 27

29. Before we proceed, one more “truism:”
After completing an experiment and analyzing
the significant factors, remember to recognize
the difference between practical and
statistical significance.
Practical Importance
Yes No
Move Stop
Yes (document Lessons
Forward learned)
Why not think
Statistical
about this
Significance at the beginning?
No Take more
Stop
data?
9/8/2011 28

30. Summary
• What is it?
A designed experiment is a test or series of tests in which purposeful changes are
made to the input variables of a process or system so that we may observe and
identify the reasons for changes in the output response…
• Why do I need it?
Efficiency with hidden replication and interaction detection.
• How do I do it?
Multi-factor sequential experimentation
• Can it solve my problem?
Yes, very often!
9/8/2011 29

31. RAMS Course Info
• 8-hour “Introduction to DOE” course,
[Jan. 26 1-5 pm + Jan. 27 8-Noon] following
RAMS conference
• Registration available soon at: www.rams.org
Jan. 23-26 in Reno, NV
• Course Topics:
*Background: Historical Perspective *Fractional Factorial Designs
*Completely Randomized Designs *Response Surface Designs
*Factorial Designs *In-Class Exercises
9/8/2011 30

32. Thaaaats
All
Folks!!
9/8/2011
Questions?? 31