1.
ChE/MatE 166 Advanced Thin Films San Jose State University LAB 4: Experiment with a Single Factor: Analysis of Variance (ANOVA)Learning Objectives1. Write clear objectives and statement of problem for an experiment.2. Identify controllable and uncontrollable factors in an experimental set-up.3. Choose a factor for a single factor ANOVA based on expected outcome and given time and equipment constraints.4. Determine the appropriate levels to be researched for a factor based on expected outcome, equipment control, metrology precision, and time constraints.5. Design an experiment using proper replication, randomization, and control of variables.6. Calculate the variation between levels using a sum of squares method.7. Calculate the variation using an F-test.8. Plot data of all the levels to show variation between levels.9. Organize technical information into a clear and concise formal laboratory report.EquipmentEach team will be using ANOVA techniques to determine the effect of a single variableon the process. The group tasks are: Team 1: Oxidation Team 2: Wet or Dry Etching of Oxide Team 3: Wet Etching of Al Team 4: Deposition of AlThe equipment and metrology tools are the same as those discussed in Labs 2 and 3. Seethose lab handouts for more details.Design of a Single-Factor ExperimentIn this lab module, you will be designing an experiment using a simple, single factorapproach. One way ANOVA (analysis of variance) is a method used to compare two ormore data sets to determine if the means are the same or different. This will allow youto compare the data collected when you vary your factor to determine if the factor has astatistically significant impact.Note: The design steps and golf analogy used below are taken from D.C. Montgomery,Design and Analysis of Experiments, John Wiley & Sons, (1997).1. Statement of ProblemThe first step in designing an experiment is to clearly state what the problem is or whatyou are trying to investigate. The statement of your problem is your overall goal forperforming the experiment. Some typical goals for experimenting include improvingprocess yield, reducing variability or obtaining closer precision to targeted goals,reducing time or cost, or improving the product. Lab 1 Handout - 1
2.
ChE/MatE 166 Advanced Thin Films San Jose State UniversityExample: My golf score varies wildly from game to game. I want to determine whatfactors affect my golf score and quantify the influence each factor has on the overallscore.2. Determination of VariablesIn order to design a controlled experiment, all the factors that can influence the overalloutcome must be assessed. Variables can be classed into two groups: controllablevariables (ones you can change) and uncontrollable variables (ones that you can notalter).Example: I developed a list of variables that influence my golf score (or may influencemy golf score) and classified them as ones I can control and ones I cant: weather: uncontrollable riding a cart or walking: controllable type of clubs used: controllable type of ball used: controllable difficulty of the course: uncontrollable number of players assigned to play with me: uncontrollable ability level of other players: uncontrollable type of shoes: controllable type of beverage: controllable time of day: controllable amount of time I warm-up at the driving range: controllable3. Choosing a Factor to InvestigateYou need to choose variables and controls that are within the limits of your experimentalframework (can give valuable data with the time, supplies, and cost you have allotted).There are several different kinds of experiments you can design. A common experimentis one factor at a time where the engineer varies one factor and holds all the restconstant. (However, the problem with this is that it does not investigate the fact thatfactors may be inter-related. In future labs, we will design full factorial experiments toinvestigate the influence of multiple factors at once.)When choosing a factor to investigate, you need to consider a number of things. The firstis that you want to choose a factor that previous experience or engineering knowledgetells you is significant. You need to determine how the influence of the factor will bequantified (what metrology and statistics you will use) to ensure that the influence ismeasurable.One-way ANOVA (analysis of variance) will investigate the influence of the factor byperforming the experiment at different levels (such high, medium, and low) andstatistically comparing the data. The value of the levels needs to be controllable. Thelevels must be chosen to have a significant, quantifiable, and measurably different effecton the end result. You need to choose a range of levels that is broad enough toinvestigate the full impact of the variable. Lab 1 Handout - 2
3.
ChE/MatE 166 Advanced Thin Films San Jose State UniversityExample: I believe that the amount of time I warm up at the driving range will have asignificant impact on my game. My hypothesis is that if I do not warm up at all or onlyfor a brief time (less than 15 minutes), I will be stiff and my score will be poor.However, if I warm up too much (over 40 minutes), I will be tired and my game scorewill also suffer. I need to choose levels of warming up to test this hypothesis that aresignificantly different enough. The levels I will test are warming up for 0, 10, 30, and 50minutes. The levels are different enough to be controllable (I can accurately time 10minutes versus 30 minutes. Investigating the influence of 10 minutes versus 12 minuteswould not be controllable. This is because when a warm up starts and ends is vagueenough that a 2-minute difference is not significant.) Note that in this experiment, I amchoosing a broad range of levels to investigate a maximum- minimum type of influence:not warming up enough and warming up too much.4. Control of Other VariablesIn this experiment, you are only trying to investigate the influence of one factor. Theother controllable variables should be set to values that you have a tight control over andare within normal range for a typical process.Example: I want to wear my normal shoes and use my normal ball and clubs. Ideally,these would all be done at the same time of day and under the same weather conditions.5. Randomization & ReplicationOnce you run your experiments, you are going to make conclusions on the influence thefactors had. (In the golf example, I would compare golf scores to see if how the factorscaused the scores to vary.) To do this, you must also factor out the fact that the datacomes from two (or more) different experiments. In other words, you want to be able tosay the variation is due to those variables not just the fact that there is statistical variationfrom run to run. You also need to guarantee that some other, uncontrollable variablewasn’t influencing your results. To do this, your experiments need to contain bothreplication and randomization.Replication involves repeating the same run more than once such as playing a golf gameusing the same warm up regiment twice. You could compare the scores of the replicatesto determine the run-to-run variation. The more replicates you have, the more statisticalconfidence you will have in your data. However, there is a limit (based on time, money,and supplies) of how many runs you can do.Example: I have determined that my resources (time and money) allow me to play eightrounds of golf. Therefore, I can have 2 replicates at every data point.It is also important to randomize the order in which you run the experiments. This willaverage out the influence of any other factors that may contributing that you cant control.Example: Some factors that may influence my golf game but I cant control are theweather and my assigned golf partners. Also, my score may improve just because I amplaying more games. To average out these factors, I perform the eight games withvarying levels in a random order. Lab 1 Handout - 3
4.
ChE/MatE 166 Advanced Thin Films San Jose State University6. Monitoring the Experiment & Quantifying ResultsFor the golf game example, it is very easy to quantify the results. I total up my golf scorefor the entire game. When I am collecting my data, I want to be careful to note as manyother factors as possible (weather, partners, physical condition) in case I need to refer tothat information when discussing my data.7. Analyzing ResultsAnalysis of variance (ANOVA) is a statistical technique that can be used to quantify thesignificant differences between levels. An analysis of variance is done by comparing thevariance within a level against the variance across the whole population. The sum ofsquares is used (this is the sum of comparing each replicates value with the averagevalue). The square term is used to factor out whether the sample is above or below themean. That is, if the square wasn’t taken variations below the mean may cancelvariations above the mean to make it seem as if there is no variation.ANOVAFigure 1 shows the plot of three samples of data taken from three populations. It cannotbe determined whether the means are statistically the same or not by plotting the data.ANOVA needs to be done to ascertain whether the means are same or not. Figure 2details the steps needed to determine if the means of the three samples are statisticallydifferent. Table 1 gives the factors calculated in ANOVA. Table 2 shows the raw datalabeled as it is used in an ANOVA calculation.Is there any difference in mean? +---------+---------+---------+---------+ 9.0 10.0 = Sample from Level (treatment) 1 x1 x2 x3 = Sample from Level (treatment) 2 = Sample from Level (treatment) 3Figure 1: Data from three samples where, x1 = Mean of sample 1; x 2 = Mean of sample ;x 3 = Mean of sample 3. Lab 1 Handout - 4
5.
ChE/MatE 166 Advanced Thin Films San Jose State UniversityPopulation 1 Population 2 Population 3 … Sample 1 Sample 2 Sample 3(x1 , x 2 , x3 ,...xn ) (x1 , x 2 , x3 ,...xn ) (x1 , x 2 , x3 ,...xn ) _ Obtain each mean and standard deviation (x, s) Perform F Test Confidence intervals Interpreting resultFigure 2: Steps needed to determine if the means of the three samples are statisticallydifferent.Table 1: Summary of ANOVA calculation. Hypothesis H 0 : µ1 = µ 2 = µ 3 = µ 4 = .... = µ n H 1 : At least one µ is different from the others Where, µ i = the average of population i F test MS Treatments F0 = (See Tables 2 and 3 for the detailed calculation.) MS E Where, MS Treatments = the sum squares due to differences in the treatment means MS E = the overall variation due to random error Confidence MS E MS E Interval on y i • − tα 2 , N − a ≤ µ i ≤ y i • + tα 2 , N − a i th treatment n n Where, y i • = the average in treatment i N = total observations , a = number of treatments (levels) n = number of observations in a treatment Reject H 0 F0 > Fα ,a −1, N − a Where d.f = (a − 1, N − a )Table 2: Raw data of ANOVA.Treatment Data Total Averages (level) 1 y11 y12 … y1n y1• y1• 2 y 21 y 22 … y 2n y 2• y 2• M M M … M M M a y a1 ya2 … y an y a• y a• y •• y •• Lab 1 Handout - 5
6.
ChE/MatE 166 Advanced Thin Films San Jose State UniversityANOVA requires mathematical manipulation of all the values in Table 2. Table 3 detailsthe calculations needed in ANOVA.Table 3: The analysis of variance table for single- factor.Source of Sum of Square Degree Mean F0Variation of Square Freedom Within a 2 a −1 MS Treatments = MS Treatmentstreatment SS Treatments = n∑ ( y i• − y •• ) F0 = i =1 SS Treatments MS E a −1Error SS E = SS T − SS Treatments N −a SS E(within MS E =treatments) N −a Total 2 N −1 SS T = ∑∑ ( y ij − y •• ) a n i =1 j =1The sum of squares for levels is the sum squares due to differences in the treatmentmeans a 2SS Treatments = n∑ ( y i• − y •• ) i =1 nWhere, y i• is the sum of all the values in Level (treatment) i: y i• = ∑ y ij j =1 y i• y i• is the average of all the values in Level (treatment) i: y i• = n y •• y •• is the average of all the samples (all levels) : y •• = Nn is the number of replicates in that levela is the number of levels: a ⋅ n = NThe sum of squares for the total population is given as: 2SS T = ∑∑ ( y ij − y •• ) a n i =1 j =1where SStotal is the sum of square for the entire sample population N is the total number of samplesThe sum of squares errors is the overall variation due to random errorSS E = SS T − SS TreatmentsAn F-test is used to compare the sum of squares of the levels with the sum of squares ofthe random errors. Lab 1 Handout - 6
7.
ChE/MatE 166 Advanced Thin Films San Jose State University MS TreatmentsF0 = MS EA criteria is used that says that if the calculated Fo is less than a critical value than thelevels are not statistically different. A confidence level needs to be set, typically anα=0.05 confidence level is chosen. This signifies that the criterion is (1- α) or 0.95accurate. If Fo is greater than Fcritical = Fα ,a −1, N − a (Rejection Region), the levels aresignificantly different.The Fcritical value can be found using a chart that is specific to the confidence levelchosen. (See Lab 2 Handout, Table 3 for Fcritical for α=0.05.) To use the table you needthe degrees of freedom of MSlevel and MSerror that are a-1 and N-a respectively.ExampleAn experiment was run to investigate the influence of DC bias voltage on the amountsilicon dioxide etched from a wafer in a plasma etch process. Three different levels of DCbias were being studied and four replicates were run in random order, resulting in the datain Table 4.Table 4: Amount of silicon dioxide etched as a function of DC bias voltage in a plasmaetcher. DC Bias (Volts) Amount Etched (in Angstroms) Total Average 1 2 3 4 s398 283.5 236 231.5 228 979 244.75 485 329 330 336 384.5 1379.5 344.875 571 474 477.5 470 474.5 1896 474 4254.5 354.54HypothesisH 0 : µ1 = µ 2 = µ 3H 1 : At least one µ is different from the othersSum of Squares for Levels a 2SS Treatments = n∑ ( y i• − y •• ) i =1 [ = 4 (244.75 − 354.54) + (344.87 − 354.54) + (474 − 354.54) 2 2 2 ] = 105672Degree of freedom = a − 1 = 3 – 1= 2Sum of Squares for the Total Population 2SS Total = SS T = ∑∑ ( y ij − y •• ) a n i =1 j =1 [= (238.5 − 354.54 ) + (236 − 354.54) + (231.5 − 354.54) + L + (474.5 − 354.54) 2 2 2 2 ] Lab 1 Handout - 7
8.
ChE/MatE 166 Advanced Thin Films San Jose State University= 109857Degree of freedom= N − 1 = 12 – 1 = 11Sum of Squares ErrorsSS Error = SS Total − SS Treatments = 109857-105672 = 4185Degree of Freedom = N − a = 12 - 3 = 9F-Statistic SS Treatments 105672MS Treatments = = = 52836 a −1 2 SS E 4185MS E = = = 465 N −a 9 MS Treatments 52836F0 = = = 113.63 MS E 465Where, Fcritial = Fα ,a −1, N − a F0.05, 2,9 = 4.26 (determined from Lab 2, Table 3).The summary of the ANOVA calculation for the example is given in Table 5.Table 5: Summary of the ANOVA calculation for the plasma etcher example. Source of Sum of Degree of Mean F0 Variation Square Freedom SquareTreatments 105672 2 52836 113.63(Levels)Error 4185 9 465Total 109857 11Because the F0 = 113.63 > 4.26 , we reject the null hypothesis ( H 0 ). We can concludethat the mean of the oxide thickness etched at the different DC bias voltages are different.Confidence IntervalWith 95 % confidence, the true mean of each treatment is within:tα 2, N − a t 0.025,9 = 2.262 MS E MS Ey i • − tα 2 , N − a ≤ µ i ≤ y i • + tα 2 , N − a n n 465 465µ1 = 244.75 − 2.262 ≤ µ1 ≤ 244.75 + 2.262 4 4 = 220.36 Angstroms ≤ µ1 ≤ 269.14 Angstroms Lab 1 Handout - 8
9.
ChE/MatE 166 Advanced Thin Films San Jose State University 465 465µ 2 = 344.87 − 2.262 ≤ µ 2 ≤ 344.87 + 2.262 4 4 = 320.48 Angstroms ≤ µ 2 ≤ 369.26 Angstroms 465 465µ 3 = 474 − 2.262 ≤ µ 3 ≤ 474 + 2.262 4 4 =449.61 Angstroms ≤ µ 3 ≤ 498.39 AngstromsResults Using MinitabFrom Minitab results (Table 6), we can conclude that we reject the null hypothesisbecause the P-value is smaller than 0.05 and F0 = 113.63 > 5.71 . In other words, weaccept the alternative hypothesis that means the mean of the oxide thickness etched aredifferent, and the level of DC bias voltage affects the mean of the oxide etched.Table 6: Results of ANOVA analysis of plasma etcher example using Minitab.Analysis of VarianceSource DF SS MS F PFactor 2 105672 52836 113.63 0.000Error 9 4185 465Total 11 109857 Individual 95% CIs For Mean Based on Pooled StDevLevel N Mean StDev ---+---------+---------+---------+---398 4 244.75 26.04 (--*--)485 4 344.87 26.60 (--*--)571 4 474.00 3.08 (--*--) ---+---------+---------+---------+-- 240 320 400 480Experimental PlanComplete the one-way ANOVA worksheet to develop a detailed designed experiment toinvestigate the influence of one variable on your assigned process.The designed experiment needs to be carried out and analyzed in two weeks so projectmanagement is a very important part of the design process. It may be necessary to dividethe workload up between team members.Laboratory ReportYour report should include the following:-Experiment objective-A brief discussion about the theory of each process-One-way ANOVA including controlled and uncontrolled variables, level chosen and why,procedures,-A detailed statistical analysis Average value for each level ANOVA analysis showing at table 3-Plot of your data showing variation within level and trend between levels Lab 1 Handout - 9
10.
ChE/MatE 166 Advanced Thin Films San Jose State University-Discussion of influence of your variable on the process Is it statistically significant? Is it what your hypothesis predicted? Why or why not?-Discussion of possible future experiments to further improve the process-Summary/ conclusionAcknowledgmentsThe lab was created by Prof. G. Young and Prof. S. Gleixner of the Chemical andMaterials Engineering department and Irene Susanti Wibowo as part of her M.S. projectin Industrial Systems and Engineering.Lab EvaluationThis portion of the lab will not be graded. I would greatly appreciate it if you did take thetime to answer the following questions in regards to your overall satisfaction with the lab.Answers to these questions will be collected anonymously at the same time you turn inyour lab report. Please take the time to turn in the attached ANONYMOUS survey.Even if you’ve got no comments please turn in the blank form so I know that you at leastread the survey questions. Lab 1 Handout - 10
11.
ChE/MatE 166 Advanced Thin Films San Jose State UniversityANONYMOUS LAB SURVEY: NO NAME PLEASE1.What was the most useful learning experience that this lab provided you?2.Do you feel the one-way ANOVA instructions provided you with new information that will behelpful in designing future experiments?3. Do you feel the step by step process of designing an experiment provided you with newinformation that will be helpful in designing future experiments?4.What would you suggest to improve how the lab requires you to explore the concepts ofdesigning a single factor experiment? Lab 1 Handout - 11
Be the first to comment