This document provides an overview of basic design of experiments (DOE). It discusses how DOE is a more effective approach to experimentation than traditional trial and error or one-factor-at-a-time methods. The document reviews full and fractional factorial experimental designs and provides an example exercise involving optimization of a paper helicopter design through experimental testing. The overall goal is to introduce practitioners to DOE methodology and its benefits for process and product improvement.

1. UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
National Guard
Black Belt Training
Module 47
Basic Design of
Experiments (DOE)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO

2. UNCLASSIFIED / FOUO
CPI Roadmap – Improve
8-STEP PROCESS
6. See
1.Validate 2. Identify 3. Set 4. Determine 5. Develop 7. Confirm 8. Standardize
Counter-
the Performance Improvement Root Counter- Results Successful
Measures
Problem Gaps Targets Cause Measures & Process Processes
Through
Define Measure Analyze Improve Control
TOOLS
ACTIVITIES
•Brainstorming
• Develop Potential Solutions
•Replenishment Pull/Kanban
• Develop Evaluation Criteria
•Stocking Strategy
• Select Best Solutions
•Process Flow Improvement
• Develop Future State Process Map(s)
•Process Balancing
• Develop Pilot Plan
•Standard Work
• Pilot Solution
•Quick Change Over
• Develop Full Scale Action/
•Design of Experiments (DOE)
Implementation Plan
•Solution Selection Matrix
• Complete Improve Gate
•‘To-Be’ Process Mapping
•Poka-Yoke
•6S Visual Mgt
•RIE
Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive. UNCLASSIFIED / FOUO 2

3. UNCLASSIFIED / FOUO
Learning Objectives
 Learn benefits of DOE methodology
 Discuss differences between DOE and trial and
error (one-factor-at-a-time) approaches to
experimentation
 Learn basic DOE terminology
 Distinguish between the concepts of full
and fractional factorial designs
 Use Minitab to run and analyze a DOE
 Use results of DOE to drive statistically
significant improvements
Basic Design of Experiments UNCLASSIFIED / FOUO 3

4. UNCLASSIFIED / FOUO
Helicopter Simulation
Phase One
UNCLASSIFIED / FOUO 4

5. UNCLASSIFIED / FOUO
Exercise: Helicopter Simulation
 Customers at CHI (Cellulose Helicopters Inc.) have
been complaining about the limited flight time of CHI
helicopters
 Management wants to increase flight time to improve
customer satisfaction
 You are put in charge of this improvement project
UNCLASSIFIED / FOUO 5

6. UNCLASSIFIED / FOUO
Exercise: Constraints
 Project Mission: Find the combination of factors that
maximize flight time
 Project Constraints:
 Budget for testing = $1.5 M
 Cost to build one prototype = $100,000
 Cost per flight test = $10,000
 Prototype once tested can not be altered
 See allowable flight test factors and parameters on the
next page
UNCLASSIFIED / FOUO 6

7. UNCLASSIFIED / FOUO
Exercise: Test Factors and Parameters
 Paper Type Regular Card stock
 Paper Clip No Yes
 Taped Body No 3 in of tape
 Taped Wing Joint No Yes
 Body Width 1.42 in 2.00 in
 Body Length 3.00 in 4.75 in
 Wing Length 3.00 in 4.75 in
UNCLASSIFIED / FOUO 7

8. UNCLASSIFIED / FOUO
Exercise: Roles & Responsibilities
 Lead Engineer – Leads the team and makes final
decision on which prototypes to build and test
 Test Engineer – Leads the team in conducting the test
and has final say on how test are conducted
 Assembly Engineer – Leads the team in building
prototypes and has final say on building issues
 Finance Engineer – Leads the team in tracking
expenses and keeping the team on budget
 Recorder – Leads the team in recording data from the
trials
UNCLASSIFIED / FOUO 8

9. UNCLASSIFIED / FOUO
Exercise: Phase One Deliverables
 Prepare a Phase One Report showing:
 Recommendation for optimal design
 Predicted flight time at optimal setting
 How much money was spent
 Description of experimental strategy used
 Description of analysis techniques used
 Recommendations for future tests
UNCLASSIFIED / FOUO 9

10. UNCLASSIFIED / FOUO
Introduction to DOE
UNCLASSIFIED / FOUO 10

11. UNCLASSIFIED / FOUO
How Do We Learn?
1. Significant Event 2. Somebody Sees It
3. Research How can we learn
more efficiently?
Basic Design of Experiments UNCLASSIFIED / FOUO 11

12. UNCLASSIFIED / FOUO
How Do We Learn? (Cont.)
 Products and processes are continually providing data
that could lead to their improvement - so what has
been missing? There are several possibilities:
 We are not collecting and analyzing the data provided
 We are not proactive in data collection
 We are unable to translate the data into information
 A significant event has not occurred
“In order to learn, two things must occur simultaneously:
something must happen (informative event) and someone
must see it happen (perceptive observer).” – George Box
Basic Design of Experiments UNCLASSIFIED / FOUO 12

13. UNCLASSIFIED / FOUO
How Do We Improve?
 By creating significant events and observing them, we
can obtain knowledge faster
 That is basically what occurs in a designed
experiment
 Let‟s look at an example of these two things occurring
(significant event and perceptive observer)
simultaneously
Basic Design of Experiments UNCLASSIFIED / FOUO 13

14. UNCLASSIFIED / FOUO
Champagne Example
 Wine – The fermented juice of
fresh grapes used as a beverage.
Wine has been in existence since
the beginning of recorded history
 Champagne – A clear, sparkling
liquid made by way of the second
fermentation of wine. First
discovered by a French monk in
the late 1600s
Basic Design of Experiments UNCLASSIFIED / FOUO 14

15. UNCLASSIFIED / FOUO
Need Improved Observation
 Need to make sure that naturally occurring informative
events are brought to the attention of the perceptive
observer!
 Improved observation increases the probability of observing
naturally occurring informative events so appropriate action
can be taken.
SPC tools and techniques improve observation, but we
must wait for an event to happen in order to observe it
Basic Design of Experiments UNCLASSIFIED / FOUO 15

16. UNCLASSIFIED / FOUO
Passive Observation Is Not Enough
 We need to induce occurrence of informative events.
 An experiment is set-up so that an informative event
will occur!
Designed Experimentation – The manipulation of
controllable factors (independent variables) at different levels
to see their effect on some response (dependent variable)
 By manipulating inputs to see how the output
changes, we can understand and model Y (a
dependent variable) as a function of X (an
independent variable).
Basic Design of Experiments UNCLASSIFIED / FOUO 16

17. UNCLASSIFIED / FOUO
What Is Experimental Design?
Inputs (Factors) Outputs (Responses)
People
Responses related to
Material
producing a product
Equipment Experimental Process:
A controlled blending Responses related to
Policies
of inputs which completing a task
Procedures generates corresponding
measurable outputs. Responses related to
Methods
performing a service
Environment
From Understanding Industrial Designed Experiments, Schmidt & Launsby
Basic Design of Experiments UNCLASSIFIED / FOUO 17

18. UNCLASSIFIED / FOUO
Example of a Recruiting Process
Inputs Outputs
(Factors) (Responses)
Job Description
Marketing
Candidate Pool
Economic Environment Process:
Recruiting
Type of Job
Location of Job Hire Quickly
Job Application Process Hire Best Candidate
EEO Requirements Hire at Competitive Pay
DOE was originally used for manufacturing quality applications - it has now
expanded to many other areas where performance characteristics are of interest
Basic Design of Experiments UNCLASSIFIED / FOUO 18

19. UNCLASSIFIED / FOUO
Methods of Experimentation
 Experimentation has been used for a long time.
Some experiments have been good, some not so
good
 Our early experiments can be grouped into the
following general categories:
1. Trial and Error
2. One-Factor-at-a-Time (OFAT)
3. Full Factorial
4. Fractional Factorial
5. Others
Basic Design of Experiments UNCLASSIFIED / FOUO 19

20. UNCLASSIFIED / FOUO
Trial and Error - Increase Gas Mileage
 Problem: Gas mileage for car is 20 mpg. Would like to get > 30
mpg.
 Factors:
 Change brand of gas
 Change octane rating
 Drive slower
 Tune-up car
 Wash and wax car
 New tires
 Change tire pressure
 Remove hood ornament and external radio antenna
Basic Design of Experiments UNCLASSIFIED / FOUO 20

21. UNCLASSIFIED / FOUO
One-Factor-at-a-Time (OFAT)- Gas Mileage
 Problem: Gas mileage for vehicle is 20 mpg. Would like to get > 30 mpg
MPG Results
Speed Octane Tire Pressure Miles per Gallon
55 85 30 25
65 85 30 23
55 91 30 27
55 85 35 27
 How many more runs would you need to figure out the best configuration of
variables?
 How can you explain the above results?
 If there were more variables, how long would it take to get a good solution?
 What if there‟s a specific combination of two or more variables that leads to the
best mileage (the optimum)?
Basic Design of Experiments UNCLASSIFIED / FOUO 21

22. UNCLASSIFIED / FOUO
Results
Miles per Gallon as a Function of Speed and Tire Pressure
75
17
70
 How would we find
65
Speed (mph)
26 this optimum with
60
OFAT testing?
55 18 23 26 26 20
50
26  How would we
45
know that we‟d
18 found it?
40
35 35
30 Optimum MPG
26 28 30 32 34 36 38
Tire Pressure (lbs.)
Basic Design of Experiments UNCLASSIFIED / FOUO 22

23. UNCLASSIFIED / FOUO
One-Factor-at-a-Time (OFAT)
 While OFAT is simple, it is inefficient in determining
optimal results:
 Unnecessary experiments may be run
 Time to find causal factors is significant
 Don‟t know the effects of changing one factor while
other factors are also changing (no model)
 Inability to detect or learn about how factors work
together to drive the response
Is there a better way?
Basic Design of Experiments UNCLASSIFIED / FOUO 23

24. UNCLASSIFIED / FOUO
Cake Example - Interactions
 An Interaction occurs when the
effect of one factor, X1, on the
response, Y, depends on the setting
(level) of another factor, X2:
Y = f(x)
 For example, when baking a cake, the
temperature that you set the oven at
is dependent on the time that the cake
will be in the oven.
Basic Design of Experiments UNCLASSIFIED / FOUO 24

25. UNCLASSIFIED / FOUO
Cake Example - Interactions
 Where would you set Time to get a good cake?
 How would you experiment on this process
to learn about this interaction?
Temp = 100 degrees Temp = 500 degrees
Time = 45 minutes Time = 20 minutes
Time
Temp = 100 degrees Temp = 500 degrees
Time = 20 minutes Time = 45 minutes
100 Temp 500
Basic Design of Experiments UNCLASSIFIED / FOUO 25

26. UNCLASSIFIED / FOUO
Cake Example - Interactions
 Duncan Hines used designed
experiments in the 50‟s on their cake
mixes.
 Their goal was a robust design for
the most consistent product.
Basic Design of Experiments UNCLASSIFIED / FOUO 26

27. UNCLASSIFIED / FOUO
Why Use DOE?
“Often we have used a trial and error
approach to testing, or just changed one
variable at a time. Why is a statistically
designed experiment better?”
 The structured methodology provides a directed approach to avoid time wasted
with “hunt and peck” - don‟t need 30 years of experience to design the tests
 The designed experiment gives a mathematical model relating the variables and
responses - no more experiments where you can‟t draw conclusions
 The model is easily optimized, so you know when you‟re done
 The statistical significance of the results is known, so there is much greater
confidence in the results
 Can determine how multiple input variables interact to affect results
Basic Design of Experiments UNCLASSIFIED / FOUO 27

28. UNCLASSIFIED / FOUO
Full Factorial DOE
 Full Factorial examines every possible
combination of factors at the levels tested. The full
factorial design is an experimental strategy that
allows us to answer most questions completely.
 Full factorial enables us to:
 Determine the Main Effects that the factors being
manipulated have on the response variable(s)
 Determine the effects of factor interactions on the
response variables
 Estimate levels at which to set factors for best
results
Basic Design of Experiments UNCLASSIFIED / FOUO 28

29. UNCLASSIFIED / FOUO
Full Factorial
Minimum number of tests for a full factorial experiment: Xk
X = # of levels, k = # of factors
Factors
Level 2 3 4
2 4 8 16 # of Tests
3 9 27 81
Adding another level significantly increases the number of tests!
 Full Factorial Advantages
 Information about all effects
 Information about all interactions
What can we do when
 Quantify Y=f(x)
resources are limited?
 Limitations
 Amount of resources needed
 Amount of time needed
Basic Design of Experiments UNCLASSIFIED / FOUO 29

30. UNCLASSIFIED / FOUO
Full Factorial Notation
 2 level designs are the most common because
they provide a lot of information, but require the
fewest tests.
 The general notation for a full factorial design of 2
levels is:
2k = # Runs
 2 is the number of levels for each factor
(Range = High and Low)
 k is the number of factors to be investigated
 This is the minimum number of test runs required
for a full factorial
Basic Design of Experiments UNCLASSIFIED / FOUO 30

31. UNCLASSIFIED / FOUO
Full Factorial Experiment
Problem: Gas Mileage is 20 mpg
Speed Octane Tire Pressure Miles per Gallon
55 85 30 25
65 85 30 23
55 91 30 27
65 91 30 23
55 85 35 27
65 85 35 24
55 91 35 32
65 91 35 25
OFAT Runs
Do we think 32 is best?
 What conclusion do you make now?
 How many runs? MPG = f(Speed, Octane, Tire Pressure)
 How many runs at each level?
Basic Design of Experiments UNCLASSIFIED / FOUO 31

32. UNCLASSIFIED / FOUO
Fractional Factorial
 Looks at only a fraction of all the possible combinations contained in
a full factorial.
 If many factors are being investigated, information can be obtained
with smaller investment.
 Resources necessary to complete a fractional factorial are
manageable.
 Limitations - give up some interactions
 Benefits
 Economy
 Speed
 Fewer runs
Basic Design of Experiments UNCLASSIFIED / FOUO 32

33. UNCLASSIFIED / FOUO
Fractional Factorial Notation
 The general notation to designate a fractional factorial
design is:
k p
2R = # Runs
 2 is the number of levels for each factor
 k is the number of factors to be investigated
 2-p is the size of the fraction
(p = 1  1/2 fraction, p = 2  1/4 fraction, etc.)
 2k-p is the number of runs
 R is the resolution
Basic Design of Experiments UNCLASSIFIED / FOUO 33

34. UNCLASSIFIED / FOUO
Fractional Factorial Notation – Resolution
When we go to a fractional factorial design, we
are not able to estimate all of the interactions
The amount that we are able to estimate is
indicated by the resolution of an experiment
The higher the resolution, the more interactions
we can measure
 Example: The designation below means fifteen factors will be
investigated in 16 runs. This design is a resolution III:
1511
2 III Note: A deeper discussion of design resolution is beyond the scope of the lesson. The
content, above, is intended to only provide a brief explanation of the design resolution term.
Basic Design of Experiments UNCLASSIFIED / FOUO 34

35. UNCLASSIFIED / FOUO
Gas Mileage Example
 Problem: Gas mileage for vehicle is 20 mpg
Speed Octane Tire Pressure Mileage
(A) (B) (C) (Y)
55 85 35 27
65 85 30 23
55 91 30 27
65 91 35 25
 Compare with previous full factorial:
 How many runs?
 How much information?
Basic Design of Experiments UNCLASSIFIED / FOUO 35

36. UNCLASSIFIED / FOUO
DOE Will Help Us Identify Factors
 Factors which shift the average
A1 A2
 Longer line = greater effect
Main Effects Plot (data means) for Mileage
55 65 85 91 30 35
27.8
26.8
In the gas mileage
Mileage
25.8 example, Speed,
Octane, and Tire
24.8 Pressure all look to
have an effect on
23.8
average mileage.
Speed Octane Tire Pressure
Basic Design of Experiments UNCLASSIFIED / FOUO 36

37. UNCLASSIFIED / FOUO
DOE Will Help Us Identify Factors
 Factors which affect variation B2
B1
 Flat line = no effect
Main Effects Plot for Standard Deviation
55 65 85 91 30 35
3.0
2.5
Only Tire Pressure
Standard Dev
is affecting standard
2.0
1.5 deviation
1.0
Speed Octane Tire Pressure
Basic Design of Experiments UNCLASSIFIED / FOUO 37

38. UNCLASSIFIED / FOUO
DOE Will Help Us Identify Factors
 Factors which shift the average and C2
affect variation C1
Main Effects Plot (data means) for Mileage Main Effects Plot for Standard Dev
55 65 85 91 30 35 55 65 85 91 30 35
27.8 3.0
26.8 2.5
Standard Dev
Mileage
25.8 2.0
24.8 1.5
23.8 1.0
Speed Octane Tire Pressure Speed Octane Tire Pressure
Only Tire Pressure affects both the
average mileage and also the variability
Basic Design of Experiments UNCLASSIFIED / FOUO 38

39. UNCLASSIFIED / FOUO
DOE Will Help Us Identify Factors
 Factors which have no effect D1 = D2
Main Effects Plot (data means) for Mileage Main Effects Plot for Standard Dev
-1 1 -1 1 -1 1 -1 1
28
3.0
27
2.5
Standard Dev
Mileage
26
2.0
25
1.5
24
1.0
Driver Radio Driver Radio
An expanded study investigated the effect of driver and radio on
mileage. These factors show no effect. This is also valuable information,
because these factors can be set at their most economical (least cost) or
most convenient levels.
Basic Design of Experiments UNCLASSIFIED / FOUO 39

40. UNCLASSIFIED / FOUO
Benefits of DOE
 Determine input settings which optimize results and
minimize costs
 Quick screening for significant effects
 Obtain a mathematical model relating inputs and results
 Reduction in the number of tests required
 Verification of the statistical significance of results
 Identification of low-impact areas allows for increased
flexibility/tolerances
 Standardized methodology provides a directed approach
Basic Design of Experiments UNCLASSIFIED / FOUO 40

41. UNCLASSIFIED / FOUO
When Could I Use Design of Experiments?
 Identification of critical factors to improve
performance
 Identification of unimportant factors to reduce costs
 Reduction in cycle time
 Reduction of scrap/rework
 Scientific method for setting tolerances
 Whenever you see repetitive testing
Basic Design of Experiments UNCLASSIFIED / FOUO 41

42. UNCLASSIFIED / FOUO
DOE Review
 What does DOE offer us that trial and error
experimentation and OFAT do not?
 What are the differences between full and fractional
factorial DOE‟s?
 What is the minimum number of runs required for a
2-level, 3-factor full factorial experiment?
Basic Design of Experiments UNCLASSIFIED / FOUO 42

43. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 A contract airline is interested in reducing overall late
take-off time in order to improve Soldier satisfaction
 Previous Black Belt work has identified 4 key process
input variables (KPIVs) that affect late time:
 Dollars spent on training
 Number of jets
 Number of employees
 % Overbooked
Basic Design of Experiments UNCLASSIFIED / FOUO 43

44. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Using the Airline DOE Data.mpj file and Minitab, the
instructor will walk the class through the following
activities:
 A DOE to identify which factors affect “minutes late” in
terms of both the mean and standard deviation
 Use the DOE results to determine new process settings
 Hypothesis test to prove statistical significance of change.
Basic Design of Experiments UNCLASSIFIED / FOUO 44

45. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Current settings for these factors are as follows:
 Dollars spent on training 200
 Number of jets 52
 Number of employees 850
 % Overbooked 15
 The target is zero minutes late, with a specification
of +/- 10 minutes.
Basic Design of Experiments UNCLASSIFIED / FOUO 45

46. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Our goal in this experiment is to reduce late take-
off times - we will measure late time in minutes
 Here are the factors and their levels that we are
going to investigate:
Factors Levels
 Dollars spent on training 100 300
 Number of Jets 50 55
 Number of Employees 800 900
 % Overbooked 0 25
Basic Design of Experiments UNCLASSIFIED / FOUO 46

47. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 First, we need to set up the test matrix
 Select Stat>DOE>Factorial>Create Factorial
Design
Basic Design of Experiments UNCLASSIFIED / FOUO 47

48. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 We left the default at: 2-level factorial design. That means we will
test each factor at 2 different levels
 We also selected 4 factors, since there are 4 variables that we want to
test in this experiment. Select Display Available Designs to display
possible experiments we can run…
Basic Design of Experiments UNCLASSIFIED / FOUO 48

49. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 For 4 factors, we can either do an 8 run half-fraction or a
16 run full factorial. We will go with the 16 run full
factorial experiment.
 Click OK to return to the previous dialog box.
Basic Design of Experiments UNCLASSIFIED / FOUO 49

50. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Click the Designs button and highlight the 16-run Full
Factorial design.
 Leave the other settings at their defaults, click on OK.
Basic Design of Experiments UNCLASSIFIED / FOUO 50

51. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Next, click the Factors button
 Enter the names of the Factors – Change -1 and 1 to actual
levels per chart below
Factors
Levels
Dollars spent on training 100
300
Number of Jets 50 55
Number of Employees 800
900
Basic Design of Experiments UNCLASSIFIED / FOUO 51

52. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Click the Options button and uncheck Randomize runs.
 We do want to randomize our tests when we actually run an
experiment. However, for this in-class demo, it will be easiest if
everyone‟s screen is the same.
Basic Design of Experiments UNCLASSIFIED / FOUO 52

53. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Here are the tests that we need to run. Example, row 1 indicates that we
first need to collect data at the low level for all four factors.
(Tip: First check that you have the same test matrix. If you don‟t, it‟s
likely that you did not uncheck “Randomize Runs.”)
Basic Design of Experiments UNCLASSIFIED / FOUO 53

54. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Next, we collect the data.
 To allow us to measure variation, we need to run 3 repetitions at each
set of settings.
 Copy the data from the DOE data worksheet and paste into the design as
shown below.
Min Late 1 = C9
Min Late 2 = C10
Min Late 3 = C11
Basic Design of Experiments UNCLASSIFIED / FOUO 54

55. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Run #1, with settings of $100 spent on training, 50 jets,
800 employees, and 0% overbooked, was 46.35 minutes
late on the first repetition, 61.92 minutes late on the
second repetition, and 75.18 minutes late on the third
repetition.
$100.00 50 Jets 800 Employees 0% Overbooked
Note: This row of coded variables
were all at their Low (or –1) settings
Basic Design of Experiments UNCLASSIFIED / FOUO 55

56. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Since we ran our DOE with 3 repetitions, and we want to analyze
the variation in our DOE results, we need to prepare the worksheet
by having Minitab calculate means and standard deviations.
 First, we need to name some blank columns. Name a blank column
StdDev Min Late, a second blank column Count Min Late, and a
third blank column Mean Min Late.
Basic Design of Experiments UNCLASSIFIED / FOUO 56

57. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Now, we will have Minitab calculate the means and standard
deviations.
 Select Stat>DOE>Factorial>Pre-Process Responses for
Analyze Variability
Basic Design of Experiments UNCLASSIFIED / FOUO 57

58. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Click on Compute for repeat responses across rows, then click in
the cell under Repeat responses across rows of: and select Min
late 1, Min late 2, and Min late 3 from the columns pane.
 Click in the box Store
standard deviations in and
select the column you named
StdDev Min Late
 Click in the box Store
number of repeats in and
select the column you named
Count Min Late
 Click in the box Store Means
(optional) in and select the
column you named Mean Min
Late
 Click OK
Basic Design of Experiments UNCLASSIFIED / FOUO 58

59. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Minitab calculates means and standard deviations for each combination of
factors. (Remember: there were 24, or 16, combinations.)
 Minitab also determines the counts. (Remember: there were 3 data points at
each combination, since we ran 3 repetitions at each setting of the DOE.)
Looking at this data Practically, there appears to be some
significance to the factors, but nothing definitive…yet.
Basic Design of Experiments UNCLASSIFIED / FOUO 59

60. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Before we view the statistics, we always start with the graphs.
 Select Stat>DOE>Factorial>Factorial Plots.
Basic Design of Experiments UNCLASSIFIED / FOUO 60

61. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Select Main Effects Plot.
 Choose Setup and click
on OK to go to next dialog
box
Basic Design of Experiments UNCLASSIFIED / FOUO 61

62. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Select StdDev Min Late and Mean Min Late for Responses,
and move all four factors from Available to the Selected box
to have them included in the analysis. Click on OK.
>
Basic Design of Experiments UNCLASSIFIED / FOUO 62

63. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 The Main Effects Plot shows that the number of Employees is the
only driver for Mean Min Late
Main Effects Plot for Mean Min Late
Data Means
Training Dollars Jets
60
45
30
15
0
Mean
-1 1 -1 1
Employees %Overbooked
60
45
30
15
0
-1 1 -1 1
Looking at this data Graphically, it appears that Employees might
be a significant factor influencing the Mean of Time Late
Basic Design of Experiments UNCLASSIFIED / FOUO 63

64. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 The Main Effects Plot shows that the number of Jets AND
Employees are driving the StdDev Min Late
Main Effects Plot for StdDev Min Late
Data Means
Training Dollars Jets
8
6
4
2
Mean
-1 1 -1 1
Employees % Overbooked
8
6
4
2
-1 1 -1 1
Looking at this data Graphically, it appears that Jets AND Employees
might be a significant factor influencing the StdDev of Time Late
Basic Design of Experiments UNCLASSIFIED / FOUO 64

65. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Now we will run the analysis.
 Select Stat>DOE>Factorial>Analyze Factorial Design
Basic Design of Experiments UNCLASSIFIED / FOUO 65

66. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Select Mean Min Late and StdDev Min Late as the Response.
 Choose Terms to get to next dialog box
Basic Design of Experiments UNCLASSIFIED / FOUO 66

67. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Include Terms in the model up through second order (2).
This will include the main effects and two-way interactions.
Click on OK to go back to previous dialog box.
Basic Design of Experiments UNCLASSIFIED / FOUO 67

68. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Click on Graphs and select the Pareto Chart. Click OK
in both dialog boxes.
Basic Design of Experiments UNCLASSIFIED / FOUO 68

69. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 This chart confirms what we saw earlier in the Main Effects Plot – the
number of Employees has a significant impact on Mean Min Late.
We also see that the interaction term BD* is significant.
This is the „Critical F-statistic‟ used to determine significance.
Pareto Chart of the Standardized Effects
(response is Mean Min Late, Alpha = 0.05)
2.57
F actor N ame
C A Training D ollars
B Jets
BD C E mploy ees
D % O v erbooked
D
AD
BC
Term
B
* - BD is the interaction
A between the factors Jets
AB and %Overbooked. Pareto Chart for
AC Mean Min Late
CD
0 10 20 30 40 50 60 70 80 90
Standardized Effect
Basic Design of Experiments UNCLASSIFIED / FOUO 69

70. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 This chart confirms only part of what we saw earlier in the Main Effects
Plot – the number of Jets has a significant impact on StdDev Min Late
but Employees does not.
Pareto Chart of the Standardized Effects
(response is StdDev Min Late, Alpha = 0.05)
2.571
F actor N ame
B A Training Dollars
B Jets
AD C E mploy ees
D % O v erbooked
C
AC
BC
Term
AB
BD Pareto Chart for
CD
StdDev Min Late
D
A
0 1 2 3 4
Standardized Effect
Basic Design of Experiments UNCLASSIFIED / FOUO 70

71. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 In the Session window, we see that Employees and the interaction
Jets*%Overbooked are the only statistically significant factors for Mean
Min Late. All other main effects and 2-way interactions have a p-value >
0.05.
Factorial Fit: Mean Min Late versus Training Dollars, Jets, ...
Estimated Effects and Coefficients for Mean Min Late (coded units)
This data shows
Term Effect Coef SE Coef T P
Constant 30.42 0.3191 95.34 0.000 Analytically that
Training Dollars
Jets
-0.71
-0.77
-0.35
-0.39
0.3191
0.3191
-1.11
-1.21
0.319
0.279 Employees and
Employees -53.89 -26.94 0.3191 -84.44 0.000 the Jets*%
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Training Dollars*Jets 0.68 0.34 0.3191 1.06 0.337 Overbooked
Training Dollars*Employees
Training Dollars*%Overbooked
0.12
1.13
0.06
0.56
0.3191
0.3191
0.18
1.77
0.861
0.138 interaction are
Jets*Employees -0.81 -0.41 0.3191 -1.27 0.259 statistically
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020
Employees*%Overbooked -0.10 -0.05 0.3191 -0.16 0.881 significant.
S = 1.27632 PRESS = 83.4049
R-Sq = 99.93% R-Sq(pred) = 99.28% R-Sq(adj) = 99.79%
Basic Design of Experiments UNCLASSIFIED / FOUO 71

72. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 In the Session window, we see that Jets is the only statistically
significant factor for Stdev Min Late. The negative sign for Effect
indicates that standard deviation decreases as Jets increases. All other
main effects and 2-way interactions have a p-value > 0.05.
Factorial Fit: StdDev Min Late versus Training Dollars, Jets, ...
Estimated Effects and Coefficients for StdDev Min Late (coded units)
Term Effect Coef SE Coef T P
Constant 4.575 0.7241 6.32 0.001
Training Dollars -0.105 -0.052 0.7241 -0.07 0.945
Jets -5.930 -2.965 0.7241 -4.09 0.009
Employees -2.477 -1.239 0.7241 -1.71 0.148
% Overbooked 0.146 0.073 0.7241 0.10 0.923
Training Dollars*Jets -0.987 -0.493 0.7241 -0.68 0.526
Training Dollars*Employees 2.032 1.016 0.7241 1.40 0.219
Training Dollars*% Overbooked 2.786 1.393 0.7241 1.92 0.112
Jets*Employees 1.655 0.828 0.7241 1.14 0.305
Jets*% Overbooked -0.935 -0.468 0.7241 -0.65 0.547
Employees*% Overbooked 0.932 0.466 0.7241 0.64 0.548
S = 2.89641 PRESS = 429.527
R-Sq = 84.84% R-Sq(pred) = 0.00% R-Sq(adj) = 54.52%
Basic Design of Experiments UNCLASSIFIED / FOUO 72

73. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Summarizing what we have found in the initial analysis:
 Employees and the interaction between Jets and %Overbooked
had a significant impact on Mean Min Late.
 As seen from the Effects, increasing Jets or %Overbooked
decreases Mean Min Late.
 In addition, when the product of Jets*%Overbooked is positive,
Mean Min Late will increase. If the product is negative, Mean Min
Late will decrease.
Term Effect Coef SE Coef T P
Employees -53.89 -26.94 0.3191 -84.44 0.000
Jets -0.77 -0.39 0.3191 -1.21 0.279
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020
 Jets had a significant impact on Stdev Min Late. As seen from its
Effect, increasing Jets decreases Stdev.
Term Effect Coef SE Coef T P
Jets -5.930 -2.965 0.7241 -4.09 0.009
Basic Design of Experiments UNCLASSIFIED / FOUO 73

74. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 The next step in the DOE analysis is to eliminate
the insignificant terms. This is called “reducing the
model.”
 Every study is different; in this particular case, let‟s
take the following approach:
 Reduce the StdDev model to identify the needed
setting for „Jets‟ since it:
 is the only significant factor influencing StdDev
 plays only a small role in driving the Mean (Coef = -0.39)
 will determine where to set the factor % Overbooked in
the interaction term.
Basic Design of Experiments UNCLASSIFIED / FOUO 74

75. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Select Stat>DOE>Factorial>Analyze Factorial Design
Basic Design of Experiments UNCLASSIFIED / FOUO 75

76. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
1. Click on Terms
2. Remove all Selected Terms: except B:Jets
3. Select OK and OK
2
1 3
Basic Design of Experiments UNCLASSIFIED / FOUO 76

77. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 The mathematical model is taken from the „Coef‟ column of the
Session Window: StdDev Min Late* = 4.575 – (2.965 x Jets)
Factorial Fit: StdDev Min Late versus Jets
Estimated Effects and Coefficients for StdDev Min Late (coded units)
Term Effect Coef SE Coef T P
Constant 4.575 0.7792 5.87 0.000
Jets -5.930 -2.965 0.7792 -3.81 0.002
S = 3.11695 PRESS = 177.653
R-Sq = 50.84% R-Sq(pred) = 35.79% R-Sq(adj) = 47.33%
 Conclusion: To reduce StdDev Min Late, we should set the
factor „Jets‟ to the +1 level (55).
Basic Design of Experiments UNCLASSIFIED / FOUO 77

78. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Now, let‟s reduce the model for the response, Mean.
 Select Stat>DOE>Factorial>Analyze Factorial Design
Basic Design of Experiments UNCLASSIFIED / FOUO 78

79. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
1. Click on Terms
2. Remove all Selected Terms: except B:Jets, C:
Employees, D: % Overbooked and the interaction BD.
3. Select OK and OK
2
1 3
Basic Design of Experiments UNCLASSIFIED / FOUO 79

80. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 The mathematical model is taken from the „Coef‟ column of the
Session Window:
Mean Min Late = 30.42 – (0.39 x Jets) – (26.94 x Employees)
–(0.64 x %Overbooked) + (1.07 x Jets x %Overbooked)
Factorial Fit: Mean Min Late versus Jets, Employees, %Overbooked
Estimated Effects and Coefficients for Mean Min Late (coded units)
Term Effect Coef SE Coef T P
Constant 30.42 0.3354 90.71 0.000
Jets -0.77 -0.39 0.3354 -1.15 0.273
Employees -53.89 -26.94 0.3354 -80.34 0.000
%Overbooked -1.28 -0.64 0.3354 -1.91 0.082
Jets*%Overbooked 2.14 1.07 0.3354 3.19 0.009
S = 1.34148 PRESS = 41.8811
R-Sq = 99.83% R-Sq(pred) = 99.64% R-Sq(adj) = 99.77%
Basic Design of Experiments UNCLASSIFIED / FOUO 80

81. UNCLASSIFIED / FOUO
Minitab: Airline DOE Example
 Of the four factors that were investigated, two (Employees and
Jets), plus the Jets*%Overbooked interaction, were significant.
 Jets – to reduce variation, we need to increase Jets to 55.
 Employees - to reduce the average late time from 30 minutes,
we need to increase Employees from 850 to 900.
 Training Budget - had no effect, and can be reduced to $100k
as a budget savings.
 % Overbooked - had marginal effect on time late and on
variation – should be reduced to 0% to improve customer
satisfaction.
 What would you do if this were your organization?
Basic Design of Experiments UNCLASSIFIED / FOUO 81

82. UNCLASSIFIED / FOUO
Is the Change Significant?
1. Conduct a hypothesis test.
2. Open the Capability Data worksheet within the Airline
DOE Data.mpj file.
3. Since we have the baseline sample and the improved
sample, select Stat>Basic Statistics>2-Sample t…
Basic Design of Experiments UNCLASSIFIED / FOUO 82

83. UNCLASSIFIED / FOUO
Is the Change Significant? (continued)
4. Select „Samples in different columns‟
5. Select „Baseline Data‟ for First: and „New Data‟ for Second:
6. Select „Graphs…‟
4
5
6
Basic Design of Experiments UNCLASSIFIED / FOUO 83

84. UNCLASSIFIED / FOUO
Is the Change Significant? (continued)
7. Select Boxplots of data
8. Click on OK
9. Interpret boxplot. Does there appear to be a graphical
difference?
7 50
Boxplot of Baseline Data, New Data
40
30
Data
20
8
9
10
0
Baseline Data New Data
Basic Design of Experiments UNCLASSIFIED / FOUO 84

85. UNCLASSIFIED / FOUO
Is the Change Significant? (continued)
10. Review Minitabs Session Window output.
11. Can we state, with 95% confidence, that there is a
statistical difference between our Baseline Data and the
New data? (i.e. Does the Confidence Interval contain „0‟
or is the P-value less than 0.05?)
10
11
Basic Design of Experiments UNCLASSIFIED / FOUO 85

86. UNCLASSIFIED / FOUO
Helicopter Simulation
Phase Two
UNCLASSIFIED / FOUO 86

87. UNCLASSIFIED / FOUO
Exercise: Helicopter Simulation
 Customers at CHI (Cellulose Helicopters Inc.) have
been complaining about the limited flight time of CHI
helicopters
 Management wants to increase flight time to improve
customer satisfaction
 You are put in charge of this improvement project
UNCLASSIFIED / FOUO 87

88. UNCLASSIFIED / FOUO
Exercise: Constraints
 Project Mission: Find the combination of factors that
maximize flight time
 Project Constraints:
 Budget for testing = $1.5 M
 Cost to build one prototype = $100,000
 Cost per flight test = $10,000
 Prototype once tested can not be altered
 See allowable flight test factors and parameters on the
next page
UNCLASSIFIED / FOUO 88

89. UNCLASSIFIED / FOUO
Exercise: Test Factors and Parameters
 Paper Type Regular Card stock
 Paper Clip No Yes
 Taped Body No 3 in of tape
 Taped Wing Joint No Yes
 Body Width 1.42 in 2.00 in
 Body Length 3.00 in 4.75 in
 Wing Length 3.00 in 4.75 in
UNCLASSIFIED / FOUO 89

90. UNCLASSIFIED / FOUO
Exercise: Roles & Responsibilities
 Lead Engineer – Leads the team and makes final
decision on which prototypes to build and test
 Test Engineer – Leads the team in conducting the test
and has final say on how test are conducted
 Assembly Engineer – Leads the team in building
prototypes and has final say on building issues
 Finance Engineer – Leads the team in tracking
expenses and keeping the team on budget
 Recorder – Leads the team in recording data from the
trials
UNCLASSIFIED / FOUO 90

91. UNCLASSIFIED / FOUO
Exercise: Phase Two Deliverables
 Prepare a Phase Two Report showing:
 Recommendation for optimal design
 Predicted flight time at optimal setting
 How much money was spent
 Description of experimental strategy used
 Description of analysis techniques used
 Recommendations for future tests
 Comparison of Phase One and Phase Two approaches
Which Team Has The Best Design?
UNCLASSIFIED / FOUO 91

92. UNCLASSIFIED / FOUO
Takeaways
 Types of experiments – Trial and Error, OFAT, DOE
 Introductory DOE terminology
 Benefits of full factorial vs. fractional designs
 How to use Minitab to design, run, and analyze a DOE
 Use DOE results to drive statistical improvements
Basic Design of Experiments UNCLASSIFIED / FOUO 92

93. UNCLASSIFIED / FOUO
What other comments or questions
do you have?
UNCLASSIFIED / FOUO 93

94. UNCLASSIFIED / FOUO
References
 Schmidt & Launsby, Understanding Industrial
Designed Experiments
Basic Design of Experiments UNCLASSIFIED / FOUO 94