Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quantitative methodology part two.compressed 2

144 views

Published on

Non-experimental designs are explained. Threats to validity are defined and exemplified. Tips to deal with these threats are provided.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Quantitative methodology part two.compressed 2

  1. 1. Exploiting Rapid Change in Technology Enhanced Learning … for Post Graduate Education Quantitative Methodology Part Two
  2. 2. Ethical Research
  3. 3. Our Focus for Today 1. Discussion of the exercises presented in part one 2. Pre-experimental quantitative research methods 3. Threats to validity
  4. 4. Exercises from Part One
  5. 5. Establish what research design is appropriate for this study… •Problem: A researcher has three different groups of employees (engineers, administrators, sales personal) which are not performing at the expected level. Data shows three different levels of current performance: high, average, and low. •Purpose: To determine the effects of different types of feedback on employees’ performance •Research Question: What is the effect of different types of feedback over the performance of employees? • Values of the independent variable: positive, negative, no feedback
  6. 6. Establish what research design is appropriate for this study… •Problem: Faculty morale is low at a university according to the results of a questionnaire applied. •Purpose: To establish the effect of sensitivity training on faculty morale •Research Question: What is the effect of sensitivity training workshops on faculty morale? • Values of the independent variable: treatment, no treatment
  7. 7. Establish what research design is appropriate for this study… •Problem: A new calculus program has been suggested to improve students performance •Purpose: To establish the effect of the suggested program on students’ calculus scores •Research Question: What is the effect of the suggested calculus program on students’ scores? • Values of the independent variable: treatment, no treatment
  8. 8. Now do the same with your own study… •Problem: • Research Question: • Research Design: •Research Method:
  9. 9. Sharing Your Experience 1. What was the most clear point? 2. What was the least clear point? 3. What would you like to say about the experience? 4. Any specific exercise you want to talk about?
  10. 10. Pre-Experimental Designs • Ex Post Facto Research • Survey Design
  11. 11. Ex Post Facto Research • Description: Variables are studied after the fact has occurred without interference from the researcher. Possible relationships and effects are searched. It is applied in contexts in which it is not possible or acceptable to manipulate the characteristics of human participants or in situations in which it is not practical or ethically acceptable to apply the full protocol of a true experimental design. They are widely used in social sciences. • Types: – Causal-Comparative Design – Correlational Design
  12. 12. Causal-Comparative Design • Description: Researchers try to establish the possible cause-effect relationship that already exists between variables in a nonexperimental setting. There are comparison groups rather than experimental and control groups. • Basic approaches: – Retrospective – Prospective
  13. 13. Correlational Design • Description: It determines whether or not two variables are correlated, which means that an increase or decrease in one variable corresponds to an increase or decrease in the other variable. It is very important to note that correlation does not imply causation, just an association. • Types : – Relational Design – Prediction design
  14. 14. Survey Design • Description: It intends to describe a situation using the answers to questions • Types of Designs: – Longitudinal Design • Trend Study • Cohort Study • Panel Study – Cross-Sectional Design • Types of Surveys: – Questionnaire – Interview
  15. 15. Difference-in-Difference Estimation The simplest set up is one where outcomes are observed for two groups for two time periods. One of the groups is exposed to a treatment in the second period, but not in the first period. The second group is not exposed to the treatment during either period. Then, the researcher estimates the “difference of differences” An estimator, not a design. Used to study policy questions by estimating treatment effects with nonexperimental data
  16. 16. Example A study about the effect of the higher minimum wage in fast-food restaurants in Pennsylvania (where the wage is constant) and New Jersey (where the wage has been increased). These are some of the results: State Before Increase After Increase Difference New Jersey (treatment) 20.44 (0.51) 21.03 (0.52) 0.59 (0.54) Pennsylvania (control) 23.33 (1.35) 21.07 (0.94) -2.16 (1.25) Difference -2.89 (1.44) -0.14 (1.07) 2.76 (1.36) Note: Standard errors in parentheses Taken from: Albouy (n.d)
  17. 17. Taken from: Wiersma, W. & Jurs, S.G. (2009) Threats to Internal Validity
  18. 18. History Description: Unanticipated events occurring while the experiment is in progress that affect the dependent variable
  19. 19. Maturation Description: Processes operating within the subjects as a function of time
  20. 20. Testing Description: The effect that taking one test has on the results of a subsequent test
  21. 21. Instrumentation Description: An effect due to inconsistent use of the measuring instruments, observers, or scorers that may affect the results
  22. 22. Statistical Regression Description: An effect caused by a tendency for subjects selected on the basis of extreme scores to regress toward an average performance on a subsequent test. It is also known as “regression to the mean”
  23. 23. Selection Description: An effect due to the groups of subjects not being randomly assigned to groups; a selection bias is operating such that the groups are not equivalent
  24. 24. Mortality Description: An effect due to subjects dropping out of the experiment. The subjects that stay may be more motivated or capable and that affects the results
  25. 25. Selection- Maturation Interaction Description: An effect of maturation not being consistent across the groups because of some selection factor may lead to confusing results and an erroneous interpretation of the effect of the treatment.
  26. 26. Taken from: Wiersma, W. & Jurs, S.G. (2009) Threats to External Validity
  27. 27. Interaction Effect of Testing Description: Pretesting interacts with the experimental treatment and causes some effect such that the results cannot be not generalized to an unpretested population
  28. 28. Interaction Effects of Selection Biases and the Experimental Treatment Description: An effect of some selection factor of intact groups interacting with the experimental treatment that would not be the case if the groups were formed randomly
  29. 29. Reactive Effects of Experimental Arrangements Description: An effect that is due to the artificial or novel experimental setting. It may also threaten internal validity
  30. 30. Multiple-Treatment Interference Description: When the same subjects receive two or more treatments, there may be a carryover effect between treatments and therefore the results cannot be generalized to single treatments
  31. 31. Taken from: Wiersma, W. & Jurs, S.G. (2009) Threats to Construct Validity
  32. 32. Inadequate Preoperational Explication of Constructs Description: Insufficient definition of the variables
  33. 33. Mono-Operation Bias Description: Only one form of the experimental variable is implemented
  34. 34. Mono-Method Bias Description: Only one form of the dependent variable is implemented
  35. 35. Hypothesis-Guessing within Experimental Conditions Description: Participants behave differently when they know they are part of an experiment Participants’ behavior can also threaten internal validity
  36. 36. Confounding Constructs and Levels of Constructs Description: Drawing conclusions about variables when some levels of the variable are absent
  37. 37. Taken from: Wiersma, W. & Jurs, S.G. (2009) Threats to Statistical Conclusion Validity
  38. 38. Low-Statistical Validity Description: Using a sample size that is too small to detect differences between groups
  39. 39. Violated Assumptions of Statistical Tests Description: Failing to meet the underlying assumptions
  40. 40. Fishing and the Error Rate Problem Description: Capitalizing on chance findings
  41. 41. Reliability of Measures Description: Using technically inadequate measures
  42. 42. Now you know 1. The pre-experimental designs 2. What difference-in-difference estimation is 3. Different types of threats to validity
  43. 43. Reference Albouy, D. (n.d). Program evaluation and the difference in difference estimator. Retrieved from http://eml.berkeley.edu/~webfac/saez/e131_s04/diff.pdf Wiersma, W. & Jurs, S.G. (2009). Research methods in education: An introduction (9th ed.). Boston, MA: Allyn and Bacon
  44. 44. What’s Up at MN & DN this summer? 1. VERY IMPORTANT – backwards map your summer work 2. 30 day writing challenge, 30 day work-life balance challenge, and 365s - keep you in touch with your work 3. Group work – RLC writing OR Lingerers
  45. 45. Avoid this one 1. VERY IMPORTANT – backwards map your summer work 2. 30 day writing challenge, 30 day work-life balance challenge, and 365s - keep you in touch with your work 3. Group work – RLC writing OR Lingerers

×