Design Of Experiments (DOE) - Outline
1.   Introduction
2.   Preparation
3.   Components of Experimental Design
4.   Purpo...
an uncontrollable which causes variability under normal conditions, but we
           can control it during the experiment...
factors, and what are the levels of those factors, to achieve the exact taste and
            consistency of Mom's chocola...
machines, multiple shifts, raw materials, humidity, etc., can be built into the
   experiment so that their variation does...
fertilizer by washing it away. Factors may generate non-linear effects that are
          not additive, but these can only...
Scheduling the experimental runs is necessary to ensure
independence of observations. You can randomize your
runs using pe...
were used to calculate the variance of the means, minus one degree of freedom for the error
 term. The denominator of the ...
interval, and the boxes denote the 95% confidence interval. You can see that the 95%
confidence level boxes for Route A an...
number of runs by evaluating only a subset of all possible combinations of the factors. These
 designs are very cost effec...
adds this cost to society (consumers) of poor quality to the production cost of the product to
arrive at the total...
Upcoming SlideShare
Loading in...5

DOE - Design Of Experiments - Outline


Published on

DOE - Design Of Experiments - Outline

Published in: Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Transcript of "DOE - Design Of Experiments - Outline"

  1. 1. Design Of Experiments (DOE) - Outline 1. Introduction 2. Preparation 3. Components of Experimental Design 4. Purpose of Experimentation 5. Design Guidelines 6. Design Process 7. One Factor Experiments 8. Multi-factor Experiments (Downloadable Spreadsheets) 9. Taguchi Methods Introduction The term experiment is defined as the systematic procedure carried out under controlled conditions in order to discover an unknown effect, to test or establish a hypothesis, or to illustrate a known effect . When analyzing a process, experiments are often used to 1 evaluate which process inputs have a significant impact on the process output, and what the target level of those inputs should be to achieve a desired result (output). Experiments can be designed in many different ways to collect this information. Design of Experiments (DOE) is also referred to as Designed Experiments or Experimental Design - all of the terms have the same meaning. Experimental design can be used at the point of greatest leverage to reduce design costs by speeding up the design process, reducing late engineering design changes, and reducing product material and labor complexity. Designed Experiments are also powerful tools to achieve manufacturing cost savings by minimizing process variation and reducing rework, scrap, and the need for inspection. This Toolbox module includes a general overview of Experimental Design, instructions and templates for conducting simple experiments, and links and other resources to assist you in conducting more complex experiments. A glossary of terms is also available at any time through the Help function, and we recommend that you read through it to familiarize yourself with any unfamiliar terms. Preparation If you do not have a general knowledge of statistics, review the Histogram, Statistical Process Control, and Regression and Correlation Analysis modules of the Toolbox prior to working with this module. Although you can use the DOE spreadsheet templates to analyze several experimental designs using Excel or Lotus, you will need statistical analysis software to go beyond these basic experiment design templates. Several free packages can be downloaded through the Statistical Software module of the Toolbox. In addition, the book DOE Simplified, by Anderson and Whitcomb, comes with a sample of excellent DOE software that will work for 180 days after installation. Components of Experimental Design Consider the following diagram of a cake-baking process (Figure 1). There are three aspects of the process that are analyzed by a designed experiment: Factors, or inputs to the process. Factors can be classified as either controllable or uncontrollable variables. In this case, the controllable factors are the ingredients for the cake and the oven that the cake is baked in. The controllable variables will be referred to throughout the material as factors. Note that the ingredients list was shortened for this example - there could be many other ingredients that have a significant bearing on the end result (oil, water, flavoring, etc). Likewise, there could be other types of factors, such as the mixing method or tools, the sequence of mixing, or even the people involved. People are generally considered a Noise Factor (see the glossary) -
  2. 2. an uncontrollable which causes variability under normal conditions, but we can control it during the experiment using blocking and randomization. Potential factors can be categorized using the Fishbone Chart (Cause & Effect Diagram) available from the Toolbox. Levels, or settings of each factor. Examples include the oven temperature setting and the amounts of sugar, flour, and eggs. Response, or output of the experiment. In the case of cake baking, the taste, consistency, and appearance of the cake are measurable outcomes potentially influenced by the factors and their respective levels. Experimenters often desire to avoid optimizing the process for one response at the expense of another. For this reason, important outcomes are measured and analyzed to determine the factors and their settings that will provide the best overall outcome for the critical-to-quality characteristics - both measurable variables and assessable attributes. Figure 1 Purpose Designed experiments have many potential uses in improving processes and products, including: Comparing Alternatives. In the case of our cake-baking example, we might want to compare the results from two different types of flour. If it turned out that the flour from different vendors was not significant, we could select the lowest-cost vendor. If flour were significant, then we would select the best flour. The experiment(s) should allow us to make an informed decision that evaluates both quality and cost. Identifying the Significant Inputs (Factors) Affecting an Output (Response) - separating the vital few from the trivial many. We might ask a question: quot;What are the significant factors beyond flour, eggs, sugar and baking?quot; Achieving an Optimal Process Output (Response). quot;What are the necessary
  3. 3. factors, and what are the levels of those factors, to achieve the exact taste and consistency of Mom's chocolate cake? Reducing Variability. quot;Can the recipe be changed so it is more likely to always come out the same?quot; Minimizing, Maximizing, or Targeting an Output (Response). quot;How can the cake be made as moist as possible without disintegrating?quot; Improving process or product quot;Robustnessquot; - fitness for use under varying conditions. quot;Can the factors and their levels (recipe) be modified so the cake will come out nearly the same no matter what type of oven is used?quot; Balancing Tradeoffs when there are multiple Critical to Quality Characteristics (CTQC's) that require optimization. quot;How do you produce the best tasting cake with the simplest recipe (least number of ingredients) and shortest baking time?quot; Experiment Design Guidelines The Design of an experiment addresses the questions outlined above by stipulating the following: The factors to be tested. The levels of those factors. The structure and layout of experimental runs, or conditions. A well-designed experiment is as simple as possible - obtaining the required information in a cost effective and reproducible manner. Reminder: Like Statistical Process Control, reliable experiment results are predicated upon two conditions: a capable measurement system, and a stable process. If the measurement system contributes excessive error, the experiment results will be muddied. You can use the Measurement Systems Analysis module from the Toolbox to evaluate the measurement system before you conduct your experiment. Likewise, you can use the Statistical Process Control module to help you evaluate the statistical stability of the process being evaluated. Variation impacting the response must be limited to common cause random error - not special cause variation from specific events. Experimentation can contribute to stabilization of an unstable process by identifying the variables that are the true process controls. When designing an experiment, pay particular heed to four potential traps that can create experimental difficulties: 1. In addition to measurement error (explained above), other sources of error, or unexplained variation, can obscure the results. Note that the term quot;errorquot; is not a synonym with quot;mistakesquot;. Error refers to all unexplained variation that is either within an experiment run or between experiment runs and associated with level settings changing. Properly designed experiments can identify and quantify the sources of error. 2. Uncontrollable factors that induce variation under normal operating conditions are referred to as quot;Noise Factorsquot;. These factors, such as multiple
  4. 4. machines, multiple shifts, raw materials, humidity, etc., can be built into the experiment so that their variation doesn't get lumped into the unexplained, or experiment error. A key strength of Designed Experiments is the ability to determine factors and settings that minimize the effects of the uncontrollable factors. 3. Correlation can often be confused with causation. Two factors that vary together may be highly correlated without one causing the other - they may both be caused by a third factor. Consider the example of a porcelain enameling operation that makes bathtubs. The manager notices that there are intermittent problems with quot;orange peelquot; - an unacceptable roughness in the enamel surface. The manager also notices that the orange peel is worse on days with a low production rate. A plot of orange peel vs. production volume below (Figure 2) illustrates the correlation: Figure 2 If the data are analyzed without knowledge of the operation, a false conclusion could be reached that low production rates cause orange peel. In fact, both low production rates and orange peel are caused by excessive absenteeism - when regular spray booth operators are replaced by employees with less skill. This example highlights the importance of factoring in operational knowledge when designing an experiment. Brainstorming exercises and Fishbone Cause & Effect Diagrams are both excellent techniques available through the Toolbox to capture this operational knowledge during the design phase of the experiment. The key is to involve the people who live with the process on a daily basis. 4. The combined effects or interactions between factors demand careful thought prior to conducting the experiment. For example, consider an experiment to grow plants with two inputs: water and fertilizer. Increased amounts of water are found to increase growth, but there is a point where additional water leads to root-rot and has a detrimental impact. Likewise, additional fertilizer has a beneficial impact up to the point that too much fertilizer burns the roots. Compounding this complexity of the main effects, there are also interactive effects - too much water can negate the benefits of
  5. 5. fertilizer by washing it away. Factors may generate non-linear effects that are not additive, but these can only be studied with more complex experiments that involve more than 2 level settings. Two levels is defined as linear (two points define a line), three levels are defined as quadratic (three points define a curve), four levels are defined as cubic, and so on. Experiment Design Process The flow chart below (Figure 3) illustrates the experiment design process: Figure 3 Test of Means - One Factor Experiment One of the most common types of experiments is the comparison of two process methods, or two methods of treatment. There are several ways to analyze such an experiment depending upon the information available from the population as well as the sample. One of the most straight-forward methods to evaluate a new process method is to plot the results on an SPC chart that also includes historical data from the baseline process, with established control limits. Then apply the standard rules to evaluate out-of-control conditions to see if the process has been shifted. You may need to collect several sub-groups worth of data in order to make a determination, although a single sub-group could fall outside of the existing control limits. You can link to the Statistical Process Control charts module of the Toolbox for help. An alternative to the control chart approach is to use the F-test (F-ratio) to compare the means of alternate treatments. This is done automatically by the ANOVA function of statistical software, but we will illustrate the calculation using the following example: A commuter wanted to find a quicker route home from work. There were two alternatives to bypass traffic bottlenecks. The commuter timed the trip home over a month and a half, recording ten data points for each alternative. MoreSteam Reminder: Take care to make sure your experimental runs are randomized - run in random order. Randomization is necessary to avoid the impact of lurking variables. Consider the example of measuring the time to drive home: if a major highway project is started at the end of the sample period increases commute time, then the highway project could bias the results if a given treatment (route) is sampled during that time period.
  6. 6. Scheduling the experimental runs is necessary to ensure independence of observations. You can randomize your runs using pennies - write the reference number for each run on a penny with a pencil, then draw the pennies from a container and record the order. The data are shown below along with the mean for each route (treatment), and the variance for each route: As shown on the table above, both new routes home (B&C) appear to be quicker than the existing route A. To determine whether the difference in treatment means is due to random chance or a statistically significant different process, an F-test is performed. The F-ratio provides a statistic that can be compared to a probability distribution table for a given confidence level to determine whether the treatment means are significantly different. There are an infinite number of F-distributions based upon confidence levels, degrees of freedom for factors, and degrees of freedom for the unexplained variation. Generally we use two levels of confidence: 95% and 99%. Following is the F-ratio calculation for the quot;drive homequot; example above: Note: n = sample size. The F-ratio of 3.61 is compared to a value from the F-distribution table for the chosen confidence level. These charts are available at the back of most statistics books. There is a separate chart for different confidence levels (higher confidence = lower probabilities of significance). You will also need to know the degrees of freedom for the numerator and the denominator of the F-ratio in order to select an F-value from the statistical table. In this case, there are two degrees of freedom in the numerator because three numbers
  7. 7. were used to calculate the variance of the means, minus one degree of freedom for the error term. The denominator of the F-ratio, called the pooled variance within each treatment, has 27 degrees of freedom (30 individual data points less 3 degrees of freedom for calculating the sample averages). In total, the degrees of freedom add up to the total number of data points in the analysis (2 numerator + 1 error + 27 denominator = 30). The higher the degrees of freedom, the more powerful the analysis. Thankfully, most statistical analysis software will calculate the degrees of freedom automatically! In this case, the factor studied was the route taken home, and it had three level settings. Three settings minus 1 yields the degrees of freedom for the factor, or 2. The total degrees of freedom for a data set is always one less than the total number of data points, or in this case 30 driving route times less 1 is 29 total degrees of freedom. The unexplained degrees of freedom is the total degrees of freedom less the degrees of freedom associated with the factors, or in this case 29 - 2 = 27. Returning to the F- ratio of 3.61 for the quot;drive homequot; experiment, we now choose a level of confidence and look up an F-value for 2 (numerator) & 27 (denominator) degrees of freedom. F-Ratio Table Using a 95% confidence level, the F-value is 3.35. Since the F-ratio of 3.61 exceeds 3.35, we can conclude that there is less than a 5% probability that the difference in means is due to random chance. MoreSteam Note: At a confidence level of 95% or higher, any F-ratio of 1.5 or lower will not meet the test of significance for any number of degrees of freedom in the denominator less than 100,000. The F-test analysis that we have described is the basis for model evaluation of both single factor and multi-factor experiments. This analysis is commonly output as an ANOVA table by statistical analysis software, as illustrated by the table below: The most important output of the table is the F-ratio (3.61). The F-ratio is equivalent to the Mean Square (variation) between the groups (treatments, or routes home in our example) of 19.9 divided by the Mean Square error within the groups (variation within the given route samples) of 5.51. Again, the Model F-ratio of 3.61 implies the model is significant. There is only a 4.08% probability that a Model F-ratio this large could occur due to noise (random chance). The following graph (Figure 4) shows quot;Least Significant Differencequot; intervals (a form of confidence interval) for each of the treatment means - the bars denote the 99% confidence
  8. 8. interval, and the boxes denote the 95% confidence interval. You can see that the 95% confidence level boxes for Route A and Route B do not overlap - illustrating the significant difference that we tested for using the F-ratio. Conversely, the 99% confidence level bars do overlap slightly, so we cannot state with 99% confidence that the mean transit time is significantly different (remember from the ANOVA that there is a 4.1% chance that the sample difference is due to random error). Figure 4 Other statistical approaches to the comparison of two treatments are available through the online statistics handbook - Chapter 7: Statistics Handbook Multi-Factor Experiments Multi-factor experiments are designed to evaluate multiple factors set at multiple levels. One approach is called a Full Factorial experiment, in which each factor is tested at each level in every possible combination with the other factors and their levels. Full factorial experiments that study all paired interactions can be economic and practical if there are few factors and only 2 or 3 levels per factor. The advantage is that all paired interactions can be studied. However, the number of runs goes up exponentially as additional factors are added. Experiments with many factors can quickly become unwieldy and costly to execute, as shown by the chart below: To study higher numbers of factors and interactions, Fractional Factorial designs can be used to reduce the
  9. 9. number of runs by evaluating only a subset of all possible combinations of the factors. These designs are very cost effective, but the study of interactions between factors is limited, so the experimental layout must be decided before the experiment can be run (during the experiment design phase). MoreSteam Reminder: When selecting the factor levels for an experiment, it is critical to capture the natural variation of the process. Levels that are close to the process mean may hide the significance of factor over its likely range of values. For factors that are measured on a variable scale, try to select levels at plus/minus three standard deviations from the mean value. offers two downloadable MS Excel spreadsheet templates to facilitate the design and analysis of experiments using two common designs: Full Factorial experiment with 4 factors at 2 levels, 16 total runs (24). Fractional Factorial Yates Algorithm experiment with 4 factors at 2 levels, 2 interactions, and 8 total runs (1/2 the runs of a Full Factorial design for 4 factors at 2 levels). This is a lower resolution experiment best suited to situations where there is prior knowledge of the likely interactions. Each spreadsheet includes instructions and a case study example: You can also download a 30-day free trial of excellent D.O.E. software from Stat-Ease at the following website: The downloadable tutorials provide a straightforward description of the design and analysis of experiments. Sections 2-4 of the tutorial are particularly useful to understand one-factor analysis, two-level factorials (full and fractional), and general factorials. Advanced Topic - Taguchi Methods Dr. Genichi Taguchi is a Japanese statistician and Deming prize winner who pioneered techniques to improve quality through Robust Design of products and production processes. Dr. Taguchi developed fractional factorial experimental designs that use a very limited number of experimental runs. The specifics of Taguchi experimental design are beyond the scope of this tutorial, however, it is useful to understand Taguchi's Loss Function, which is the foundation of his quality improvement philosophy. Traditional thinking is that any part or product within specification is equally fit for use. In that case, loss (cost) from poor quality occurs only outside the specification (Figure 5). However, Taguchi makes the point that a part marginally within the specification is really little better than a part marginally outside the specification. As such, Taguchi describes a continuous Loss Function that increases as a part deviates from the target, or nominal value (Figure 6). The Loss Function stipulates that society's loss due to poorly performing products is proportional to the square of the deviation of the performance characteristic from its target value.
  10. 10. Taguchi adds this cost to society (consumers) of poor quality to the production cost of the product to arrive at the total loss (cost). Taguchi uses designed experiments to produce product and process designs that are more robust - less sensitive to part/process variation. References Websters Ninth New Collegiate Dictionary Additional Online Resources University will be offering an online course in Design of Experiments - to be available soon. will also be launching its D.O.E. Doc SM service for online D.O.E. consulting in the near future. Software for D.O.E. can be downloaded for free in a 30-day trial version from Stat-Ease at . An excellent online Statistics Handbook is available that covers Design of Experiments and many other topics. See Section 5 - quot;Improvequot; for a complete tutorial on Design of Experiments. Check the Article Archive for related online articles. Books Mark J. Anderson and Patrick J. Whitcomb, DOE Simplified (Productivity, Inc. 2000). ISBN 1-56327-225-3. Recommended - This book is easy to understand and comes with copy of excellent D.O.E. software good for 180 days. George E. P. Box, William G. Hunter and J. Stuart Hunter, Statistics for Experimenters - An Introduction to Design, Data Analysis, and Model Building (John Wiley and Sons, Inc. 1978). ISBN 0-471-09315-7 Douglas C. Montgomery, Design and Analysis of Experiments (John Wiley & Sons, Inc., 1984) ISBN 0-471-86812-4. Genichi Taguchi, Introduction to Quality Engineering - Designing Quality Into Products and Processes (Asian Productivity Organization, 1986). ISBN 92-833-1084-5