Published on

Published in: Technology
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Experimental and Quasi-Experimental Research EDUU 600 Notes from McMillan and Schumacher
  2. 2. Characteristics of Experimental Research <ul><li>Theory-driven research hypothesis </li></ul><ul><li>Statistical equivalence of subjects in intervention and control and/or comparison groups (achieved through random assignment) </li></ul><ul><li>Researcher-controlled interventions independently and uniformly applied to all subjects </li></ul><ul><li>Measurement of each dependent variable </li></ul><ul><li>Use of inferential statistics </li></ul><ul><li>Rigorous control of conditions and extraneous variables </li></ul>McMillan and Schumacher, p. 254
  3. 3. Planning Experimental Research <ul><li>Define research problem </li></ul><ul><li>Select subjects from a defined population </li></ul><ul><li>Assign subjects to different groups </li></ul><ul><ul><li>Experimental or treatment group </li></ul></ul><ul><ul><li>Control or comparison group </li></ul></ul><ul><li>Determine the nature of the value, forms, or conditions each group receives. </li></ul><ul><ul><li>Could be two or more levels with varying degrees of condition </li></ul></ul><ul><ul><li>Determine the treatment conditions </li></ul></ul><ul><ul><li>Maximize internal validity </li></ul></ul>McMillan and Schumacher, p. 257
  4. 4. Validity <ul><li>Statistical Conclusion Validity </li></ul><ul><ul><li>Is the conclusion valid in determining a relationship or difference between groups? </li></ul></ul><ul><ul><li>Is the hypothesis supported or not supported by the results or findings? </li></ul></ul><ul><li>Construct Validity </li></ul><ul><ul><li>How well measured variables and interventions represent the theoretical constructs that have been hypothesized. </li></ul></ul>McMillan and Schumacher, p. 260
  5. 5. Experimental Validity <ul><li>Internal Validity </li></ul><ul><ul><li>Low statistical power </li></ul></ul><ul><ul><li>Violated Assumptions of statistical tests </li></ul></ul><ul><ul><li>“ Fishing” and the error rate problem </li></ul></ul><ul><ul><li>Unreliability of measures </li></ul></ul><ul><ul><li>Restriction of range </li></ul></ul><ul><ul><li>Unreliability of treatment implementation </li></ul></ul><ul><ul><li>Extraneous variance in the experimental study </li></ul></ul>McMillan and Schumacher, pp. 258-259
  6. 6. Internal Validity Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/intval.htm
  7. 7. External Validity <ul><li>The extent to which the results of an experiment can be generalized to people and environmental conditions outside the context of the experiment. </li></ul><ul><li>Population </li></ul><ul><ul><li>Selection of Subjects </li></ul></ul><ul><ul><li>Characteristics of Subjects </li></ul></ul><ul><ul><li>Subject/treatment Interaction </li></ul></ul><ul><li>Ecological </li></ul><ul><ul><li>Description of Variables </li></ul></ul><ul><ul><li>Multiple-treatment interference </li></ul></ul><ul><ul><li>Setting/treatment interaction </li></ul></ul><ul><ul><li>Pretest-posttest sensitization </li></ul></ul><ul><ul><li>Novelty or disruption effect </li></ul></ul>McMillan and Schumacher, p. 261
  8. 8. External Validity Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/external.htm
  9. 9. Proximal Similarity Model for External Validity Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/external.htm
  10. 10. Pre-Experimental Design Notation <ul><li>R – Random Assignment </li></ul><ul><li>O – Observation, a measure that records observations of a pretest or posttest </li></ul><ul><li>X – Treatment conditions (subscripts 1 through n indicate different treatments) </li></ul><ul><li>A,B,C,D,E,F – Groups of subjects, or for single-subject designs, baseline or treatment condition </li></ul>McMillan and Schumacher
  11. 11. Introduction to Design Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/desintro.htm Go to Website
  12. 12. Single–Group Pretest-Posttest Design <ul><li>Group Treatment Posttest </li></ul><ul><ul><li>A X O </li></ul></ul><ul><ul><li>B O </li></ul></ul><ul><li>Group Pretest Treatment Posttest </li></ul><ul><ul><li>A O X O </li></ul></ul>Nonequivalent Groups Posttest-Only Control Group McMillan and Schumacher
  13. 13. Nonequivalent Groups Pretest-Posttest Control Group <ul><li>Useful for educational research since it is often difficult to randomly assign subjects </li></ul><ul><li>Researcher uses intact, already established groups of subjects </li></ul><ul><li>Subjects given a pretest </li></ul><ul><li>Treatment is administered to one group </li></ul><ul><li>Subjects given a posttest </li></ul><ul><li>Group Pretest Treatment Posttest </li></ul><ul><ul><li>A O X O </li></ul></ul><ul><ul><li>B O O </li></ul></ul>McMillan and Schumacher
  14. 14. Quasi-Experimental Design Pretest-Posttest O 1 X O 2 O 1 O 2 Pre-test Post-test Intervention
  15. 15. Quasi-experimental Design Interrupted Time Series O 1     O 2     O 3     O 4     X     O 5     O 6     O 7     O 8 Pre-Intervention Post-Intervention Intervention
  16. 16. Interrupted Time Series Designs A time series design capitalizes on many observations over time to detect and rule out threats to internal validity. O O O O O O O O O O O X O O O O O O O O O O
  17. 17. Control-Group Interrupted Time-Series Group Preobservations Postobservations A O O O O O O O O O X O O O O O O O O B O O O O O O O O O O O O O O O O O
  18. 18. Inferential Statistics <ul><li>Statistics derived from a small group and then generalized to a population </li></ul><ul><li>Procedures that are used to indicate the probability associated with saying something about a population based on data from a sample.” (McMillan & Schumacher) </li></ul><ul><li>Two Common Statistical Tests </li></ul><ul><ul><li>The t-test </li></ul></ul><ul><ul><li>ANOVA – Analysis of Variance </li></ul></ul>
  19. 19. The Statistical Hypothesis <ul><li>If X , then Y </li></ul><ul><li>and </li></ul><ul><li>If not X , then not Y </li></ul><ul><li>If the program is given, then the outcome occurs </li></ul><ul><li>and </li></ul><ul><li>If the program is not given, then the outcome does not occur </li></ul>Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/desexper.htm
  20. 20. Null Hypothesis <ul><li>The working hypothesis that there is NO difference between groups </li></ul><ul><li>A formal statistical statement of NO relationship between two or more variables (McMillan & Schumacher) </li></ul><ul><li>State your question in the form of a null hypothesis and an alternative hypothesis </li></ul>
  21. 21. Statistical Hypothesis <ul><li>Null Hypothesis </li></ul><ul><ul><li>States that a population parameter is equal to some specific value. </li></ul></ul><ul><ul><li>Symbol for the null hypothesis - H sub zero </li></ul></ul><ul><ul><li>Thought of as the hypothesis of no difference. (For example: no difference between the experimental group and the control group) </li></ul></ul><ul><li>Alternative Hypothesis </li></ul><ul><ul><li>States that a population parameter is equal to some value other than that stated by the null hypothesis. </li></ul></ul><ul><ul><li>States the direction we would wish our experiment to turn out and thus is really a statement of the research question in the form of a statistical hypothesis. </li></ul></ul><ul><ul><li>Symbol for the alternative hypothesis or </li></ul></ul>Wasson - http://www.mnstate.edu/wasson/ed602lesson9.htm
  22. 22. Null and Alternative Hypotheses <ul><li>The first step is to specify the null hypothesis and an alternative hypothesis . For experiments testing differences between means, the null hypothesis is that the difference between means is some specified value. Usually the null hypothesis is that the difference is zero. For this example, the null and alternative hypotheses are: Ho: µ1 - µ2 = 0 H1: µ1 - µ2 ≠ 0 </li></ul>HyperStat Online - http://davidmlane.com/hyperstat/B58842.html
  23. 23. The t-test <ul><li>An inferential statistical procedure for determining the probability level of rejecting the null hypothesis that two means are the same (McMillan & Schumacher) </li></ul><ul><li>Compares the means of two groups </li></ul><ul><li>Considers the error in estimating the population mean from a sample mean </li></ul>
  24. 24. What do you need to determine? <ul><li>Probability - How likely is your result accurate? </li></ul><ul><li>Sampling Error - Do samples accurately reflect the population? </li></ul><ul><li>Standard Error - Standard deviation of the population means </li></ul><ul><li>Standard Deviation - Indicates average variability of scores </li></ul>
  25. 25. Probabilistic Equivalence Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/expequi.htm
  26. 26. The t-test Formula Research Knowledge Base - http://www.socialresearchmethods.net/kb/stat_t.htm
  27. 27. T-test Options <ul><li>Statistics in Plain English - http://www.statisticallysignificantconsulting.com/Statistics101.htm </li></ul><ul><ul><li>“ classic example for explaining statistical hypothesis testing and statistical inference .” - http://www.statisticallysignificantconsulting.com/Ttest.htm </li></ul></ul><ul><ul><li>Most commonly used statistical test </li></ul></ul><ul><ul><li>Three varieties of t-tests: </li></ul></ul><ul><ul><ul><li>1) the two-sample t-test (the student’s t-test or independent samples t-test ) </li></ul></ul></ul><ul><ul><ul><ul><li>“ The most common of these three is the two-sample t-test. The two-sample t-test is used to compare the means of two independent samples. There are two key concepts here, there is a measurement that you will take the average (or mean) of, and there are two separate groups.  As in all statistical hypothesis testing procedures, two hypotheses are stated, only one of which can be true, and one or the other must be true. The “null hypothesis”, is what we presume to be true and the “alternative hypothesis”, is what we will accept as true, if the facts are strong enough. The statistical hypothesis testing procedure (the t-test in this example) produces a p-value, and if this p-value is less than 0.05, then by convention, this is considered very strong evidence, and we will reject the null hypothesis and assume the alternative hypothesis must by true.” </li></ul></ul></ul></ul><ul><ul><ul><li>2) the paired samples t-test ( dependent t-test ) </li></ul></ul></ul><ul><ul><ul><li>3) the one-sample t-test </li></ul></ul></ul>Independent t-test when n is equal Dependent t-test
  28. 28. Example in Excel Using Data Analysis Tool Excel Websites for t-tests Wake Forest - http://www.wfu.edu/~massd2/T_test.htm Dr. Wasson - http://www.mnstate.edu/wasson/ed602excelss11.htm Control Group Test Group 65 85 59 87 60 88 61 92 58 88 59 88 60 85 61 89 62 88 63 85 Mean 60.8 87.5 t-Test: Two-Sample Assuming Equal Variances       Variable 1 Variable 2 Mean 60.8 87.5 Variance 4.4 4.722222 Observations 10 10 Pooled Variance 4.561111   Hypothesized Mean Difference 0   df 18   t Stat -27.9551   P(T<=t) one-tail 1.39E-16   t Critical one-tail 1.734064   P(T<=t) two-tail 2.79E-16   t Critical two-tail 2.100922  
  29. 29. t-test Resources <ul><li>About t-tests (QMSS - Columbia University) - http://www.columbia.edu/ccnmtl/projects/qmss/t_about.html </li></ul><ul><li>Wikipedia - http://en.wikipedia.org/wiki/Student's_t-test </li></ul><ul><li>t-test Calculator from GraphPad - http://www.graphpad.com/quickcalcs/ttest1.cfm </li></ul><ul><li>SISA - http://home.clara.net/sisa/t-thlp.htm </li></ul><ul><li>Research Methods Knowledge Database - http://www.socialresearchmethods.net/kb/statsimp.htm </li></ul><ul><li>Dr. Wasson’s Internet Research Course </li></ul><ul><ul><li>One sample t-test - http://www.mnstate.edu/wasson/ed602tsinglex.htm </li></ul></ul><ul><ul><li>The independent t-test - http://www.mnstate.edu/wasson/ed602lesson11.htm </li></ul></ul><ul><ul><li>The dependent t-test - http://www.mnstate.edu/wasson/ed602lesson12.htm </li></ul></ul><ul><li>HyperStat - http://davidmlane.com/hyperstat/B58842.html </li></ul><ul><li>Student’s t-test - http://helios.bto.ed.ac.uk/bto/statistics/tress4a.html#Student's%20t-test </li></ul>
  30. 30. Analysis of Variance (ANOVA) <ul><li>Allows the comparison of the means of more than two groups </li></ul><ul><li>Has multiple forms </li></ul><ul><li>About ANOVA from Columbia University (QMSS) - http://www.columbia.edu/ccnmtl/projects/qmss/anova_about.html </li></ul><ul><li>See factorial designs - http://www.socialresearchmethods.net/kb/expfact.htm </li></ul>
  31. 31. Multiple Regression <ul><li>Multiple linear regression – a statistical procedure for using several variables to predict an outcome </li></ul><ul><li>About Multiple Regression (Columbia University QMSS) - http://www.columbia.edu/ccnmtl/projects/qmss/multreg_about.html </li></ul><ul><li>Interactive Website - http://people.hofstra.edu/faculty/Stefan_Waner/Realworld/multlinreg.html </li></ul><ul><li>Multiple Regression in Excel - http://www.jeremymiles.co.uk/regressionbook/extras/appendix2/excel/ </li></ul><ul><li>StatSoft - http://www.statsoft.com/textbook/stmulreg.html </li></ul>
  32. 32. Statistics Websites <ul><li>American Statistics Association - http://www.amstat.org/ </li></ul><ul><li>Journal on Statistics Education - http://www.amstat.org/publications/jse/ </li></ul><ul><ul><li>Inferential Statistics: Understanding Expert Knowledge and its Implications for Statistics Education - http://www.amstat.org/publications/jse/v12n2/alacaci.html </li></ul></ul><ul><ul><li>A Visualization Tool for One- and Two-Way Analysis of Variance - http://www.amstat.org/publications/jse/v13n1/sturm-beiss.html </li></ul></ul><ul><ul><li>Visualizing Multiple Regression - http://www.amstat.org/publications/jse/v9n1/ip.html </li></ul></ul><ul><ul><li>An Investigation of the Median-Median Method of Linear Regression - http://www.amstat.org/publications/jse/v14n2/morrell.html </li></ul></ul><ul><li>NCES </li></ul><ul><ul><li>Statistical Standards Program - http://nces.ed.gov/StatProg/index.asp </li></ul></ul><ul><ul><li>NCES Digest of Statistics Education - http://nces.ed.gov/programs/digest/d05/ </li></ul></ul><ul><ul><li>Statistical Standards - http://nces.ed.gov/statprog/2002/std5_1.asp </li></ul></ul>