The document discusses experimental and quasi-experimental research methods. It defines key characteristics of experimental research such as random assignment, control and intervention groups, and pre- and post-testing. Issues of internal and external validity are examined. Common statistical analyses for experimental designs are introduced, including t-tests, ANOVA, and multiple regression. Examples of experimental designs like single-group, non-equivalent groups, interrupted time series, and factorial designs are also summarized.

1. Experimental and Quasi-Experimental Research EDUU 600 Notes from McMillan and Schumacher

2. Characteristics of Experimental Research Theory-driven research hypothesis Statistical equivalence of subjects in intervention and control and/or comparison groups (achieved through random assignment) Researcher-controlled interventions independently and uniformly applied to all subjects Measurement of each dependent variable Use of inferential statistics Rigorous control of conditions and extraneous variables McMillan and Schumacher, p. 254

3. Planning Experimental Research Define research problem Select subjects from a defined population Assign subjects to different groups Experimental or treatment group Control or comparison group Determine the nature of the value, forms, or conditions each group receives. Could be two or more levels with varying degrees of condition Determine the treatment conditions Maximize internal validity McMillan and Schumacher, p. 257

4. Validity Statistical Conclusion Validity Is the conclusion valid in determining a relationship or difference between groups? Is the hypothesis supported or not supported by the results or findings? Construct Validity How well measured variables and interventions represent the theoretical constructs that have been hypothesized. McMillan and Schumacher, p. 260

5. Experimental Validity Internal Validity Low statistical power Violated Assumptions of statistical tests “ Fishing” and the error rate problem Unreliability of measures Restriction of range Unreliability of treatment implementation Extraneous variance in the experimental study McMillan and Schumacher, pp. 258-259

6. Internal Validity Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/intval.htm

7. External Validity The extent to which the results of an experiment can be generalized to people and environmental conditions outside the context of the experiment. Population Selection of Subjects Characteristics of Subjects Subject/treatment Interaction Ecological Description of Variables Multiple-treatment interference Setting/treatment interaction Pretest-posttest sensitization Novelty or disruption effect McMillan and Schumacher, p. 261

8. External Validity Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/external.htm

9. Proximal Similarity Model for External Validity Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/external.htm

10. Pre-Experimental Design Notation R – Random Assignment O – Observation, a measure that records observations of a pretest or posttest X – Treatment conditions (subscripts 1 through n indicate different treatments) A,B,C,D,E,F – Groups of subjects, or for single-subject designs, baseline or treatment condition McMillan and Schumacher

11. Introduction to Design Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/desintro.htm Go to Website

12. Single–Group Pretest-Posttest Design Group Treatment Posttest A X O B O Group Pretest Treatment Posttest A O X O Nonequivalent Groups Posttest-Only Control Group McMillan and Schumacher

13. Nonequivalent Groups Pretest-Posttest Control Group Useful for educational research since it is often difficult to randomly assign subjects Researcher uses intact, already established groups of subjects Subjects given a pretest Treatment is administered to one group Subjects given a posttest Group Pretest Treatment Posttest A O X O B O O McMillan and Schumacher

14. Quasi-Experimental Design Pretest-Posttest O 1 X O 2 O 1 O 2 Pre-test Post-test Intervention

15. Quasi-experimental Design Interrupted Time Series O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8 Pre-Intervention Post-Intervention Intervention

16. Interrupted Time Series Designs A time series design capitalizes on many observations over time to detect and rule out threats to internal validity. O O O O O O O O O O O X O O O O O O O O O O

17. Control-Group Interrupted Time-Series Group Preobservations Postobservations A O O O O O O O O O X O O O O O O O O B O O O O O O O O O O O O O O O O O

18. Inferential Statistics Statistics derived from a small group and then generalized to a population Procedures that are used to indicate the probability associated with saying something about a population based on data from a sample.” (McMillan & Schumacher) Two Common Statistical Tests The t-test ANOVA – Analysis of Variance

19. The Statistical Hypothesis If X , then Y and If not X , then not Y If the program is given, then the outcome occurs and If the program is not given, then the outcome does not occur Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/desexper.htm

20. Null Hypothesis The working hypothesis that there is NO difference between groups A formal statistical statement of NO relationship between two or more variables (McMillan & Schumacher) State your question in the form of a null hypothesis and an alternative hypothesis

21. Statistical Hypothesis Null Hypothesis States that a population parameter is equal to some specific value. Symbol for the null hypothesis - H sub zero Thought of as the hypothesis of no difference. (For example: no difference between the experimental group and the control group) Alternative Hypothesis States that a population parameter is equal to some value other than that stated by the null hypothesis. States the direction we would wish our experiment to turn out and thus is really a statement of the research question in the form of a statistical hypothesis. Symbol for the alternative hypothesis or Wasson - http://www.mnstate.edu/wasson/ed602lesson9.htm

22. Null and Alternative Hypotheses The first step is to specify the null hypothesis and an alternative hypothesis . For experiments testing differences between means, the null hypothesis is that the difference between means is some specified value. Usually the null hypothesis is that the difference is zero. For this example, the null and alternative hypotheses are: Ho: µ1 - µ2 = 0 H1: µ1 - µ2 ≠ 0 HyperStat Online - http://davidmlane.com/hyperstat/B58842.html

23. The t-test An inferential statistical procedure for determining the probability level of rejecting the null hypothesis that two means are the same (McMillan & Schumacher) Compares the means of two groups Considers the error in estimating the population mean from a sample mean

24. What do you need to determine? Probability - How likely is your result accurate? Sampling Error - Do samples accurately reflect the population? Standard Error - Standard deviation of the population means Standard Deviation - Indicates average variability of scores

25. Probabilistic Equivalence Research Methods Knowledge Base http://www.socialresearchmethods.net/kb/expequi.htm

26. The t-test Formula Research Knowledge Base - http://www.socialresearchmethods.net/kb/stat_t.htm

27. T-test Options Statistics in Plain English - http://www.statisticallysignificantconsulting.com/Statistics101.htm “ classic example for explaining statistical hypothesis testing and statistical inference .” - http://www.statisticallysignificantconsulting.com/Ttest.htm Most commonly used statistical test Three varieties of t-tests: 1) the two-sample t-test (the student’s t-test or independent samples t-test ) “ The most common of these three is the two-sample t-test. The two-sample t-test is used to compare the means of two independent samples. There are two key concepts here, there is a measurement that you will take the average (or mean) of, and there are two separate groups. As in all statistical hypothesis testing procedures, two hypotheses are stated, only one of which can be true, and one or the other must be true. The “null hypothesis”, is what we presume to be true and the “alternative hypothesis”, is what we will accept as true, if the facts are strong enough. The statistical hypothesis testing procedure (the t-test in this example) produces a p-value, and if this p-value is less than 0.05, then by convention, this is considered very strong evidence, and we will reject the null hypothesis and assume the alternative hypothesis must by true.” 2) the paired samples t-test ( dependent t-test ) 3) the one-sample t-test Independent t-test when n is equal Dependent t-test

28. Example in Excel Using Data Analysis Tool Excel Websites for t-tests Wake Forest - http://www.wfu.edu/~massd2/T_test.htm Dr. Wasson - http://www.mnstate.edu/wasson/ed602excelss11.htm Control Group Test Group 65 85 59 87 60 88 61 92 58 88 59 88 60 85 61 89 62 88 63 85 Mean 60.8 87.5 t-Test: Two-Sample Assuming Equal Variances Variable 1 Variable 2 Mean 60.8 87.5 Variance 4.4 4.722222 Observations 10 10 Pooled Variance 4.561111 Hypothesized Mean Difference 0 df 18 t Stat -27.9551 P(T<=t) one-tail 1.39E-16 t Critical one-tail 1.734064 P(T<=t) two-tail 2.79E-16 t Critical two-tail 2.100922

29. t-test Resources About t-tests (QMSS - Columbia University) - http://www.columbia.edu/ccnmtl/projects/qmss/t_about.html Wikipedia - http://en.wikipedia.org/wiki/Student's_t-test t-test Calculator from GraphPad - http://www.graphpad.com/quickcalcs/ttest1.cfm SISA - http://home.clara.net/sisa/t-thlp.htm Research Methods Knowledge Database - http://www.socialresearchmethods.net/kb/statsimp.htm Dr. Wasson’s Internet Research Course One sample t-test - http://www.mnstate.edu/wasson/ed602tsinglex.htm The independent t-test - http://www.mnstate.edu/wasson/ed602lesson11.htm The dependent t-test - http://www.mnstate.edu/wasson/ed602lesson12.htm HyperStat - http://davidmlane.com/hyperstat/B58842.html Student’s t-test - http://helios.bto.ed.ac.uk/bto/statistics/tress4a.html#Student's%20t-test

30. Analysis of Variance (ANOVA) Allows the comparison of the means of more than two groups Has multiple forms About ANOVA from Columbia University (QMSS) - http://www.columbia.edu/ccnmtl/projects/qmss/anova_about.html See factorial designs - http://www.socialresearchmethods.net/kb/expfact.htm

31. Multiple Regression Multiple linear regression – a statistical procedure for using several variables to predict an outcome About Multiple Regression (Columbia University QMSS) - http://www.columbia.edu/ccnmtl/projects/qmss/multreg_about.html Interactive Website - http://people.hofstra.edu/faculty/Stefan_Waner/Realworld/multlinreg.html Multiple Regression in Excel - http://www.jeremymiles.co.uk/regressionbook/extras/appendix2/excel/ StatSoft - http://www.statsoft.com/textbook/stmulreg.html

32. Statistics Websites American Statistics Association - http://www.amstat.org/ Journal on Statistics Education - http://www.amstat.org/publications/jse/ Inferential Statistics: Understanding Expert Knowledge and its Implications for Statistics Education - http://www.amstat.org/publications/jse/v12n2/alacaci.html A Visualization Tool for One- and Two-Way Analysis of Variance - http://www.amstat.org/publications/jse/v13n1/sturm-beiss.html Visualizing Multiple Regression - http://www.amstat.org/publications/jse/v9n1/ip.html An Investigation of the Median-Median Method of Linear Regression - http://www.amstat.org/publications/jse/v14n2/morrell.html NCES Statistical Standards Program - http://nces.ed.gov/StatProg/index.asp NCES Digest of Statistics Education - http://nces.ed.gov/programs/digest/d05/ Statistical Standards - http://nces.ed.gov/statprog/2002/std5_1.asp