Psbe2 08 research methods 2011-2012 - week 2

307 views

Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Psbe2 08 research methods 2011-2012 - week 2

  1. 1. PSBE2-08Research Methods Week 2 Tassos Sarampalis 1
  2. 2. Null Hypothesis Significance Testing and Power 2
  3. 3. Null Hypothesis Significance Testing• Goal – determine whether mean differences among groups in an experiment are greater than differences expected simply because of chance (error variation)• First step – assume that the groups do not differ (H0) • = null hypothesis • assume the independent variable did not have an effect 3
  4. 4. Null Hypothesis Significance Testing• Next steps – Probability theory: estimate likelihood of observed outcome, while assuming null hypothesis is true. – “statistically significant” • outcome has small likelihood of occurring under H0 • reject H0 • conclude IV had an effect on DV – difference between means is larger than what would be expected if error variation alone caused the outcome 4
  5. 5. 5
  6. 6. probability 0 2 4 6 8 10 12 14 16 18 “heads” count 6
  7. 7. probability 0 2 4 6 8 10 12 14 16 18 “heads” count 7
  8. 8. Null Hypothesis Significance Testing• How small does the likelihood have to be to decide outcome isn’t due to chance? • scientific consensus: p < .05 • = alpha (α) or level of significance • What does a statistically significant outcome tell us? – outcome at p ≈ .05 has about a 50/50 chance of being repeated (at p < .05) in an exact replication – as probability of outcome decreases (e.g., p = .025, p = .01), likelihood of observing a statistically significant outcome (p < .05) in an exact replication increases – APA recommends reporting exact probability of outcome 8
  9. 9. 9
  10. 10. Null Hypothesis Significance Testing• What do we conclude when a finding is not statistically significant? – do not reject the null hypothesis of no difference – don’t accept the null hypothesis • don’t conclude that the IV didn’t produce an effect – cannot make a conclusion about the effect of an IV • some factor in experiment may have prevented us from observing an effect of the IV • most common factor: too few participants 10
  11. 11. NHST Criticisms• A difference between populations can almost always be found, given a large enough sample• A statistically significant finding may not be relevant in practice, whilst a true effect of practical significance may not appear statistically significant if the test lacks the power• Fairness of exclusion• Publication bias and the file-drawer problem 11
  12. 12. Experimental Sensitivity and Power• Sensitivity – likelihood an experiment will detect the effect of an IV when in fact, the IV has an effect • affected by experiment methods and procedures • sensitivity increases with good research design and methods – high degree of experimental control – little opportunity for biases 12
  13. 13. Experimental Sensitivity and Power• Power – likelihood that a statistical test will allow researchers to reject correctly H0 • low statistical power increases Type II errors • Power = 1 - β • three factors affect power of statistical tests – level of significance (alpha) – size of the effect of the IV – sample size (N) 13
  14. 14. Experimental Sensitivity and Power• Prospective Power Analysis • step 1: estimate effect size of IV – examine previous research involving the IV • step 2: refer to “Power Tables” – identify sample size needed to observe effect of IV • step 3: use adequate sample size – most studies in psychology are “underpowered” because of low sample size• Retrospective Power Analysis • Determine the power of a study based on the effect size, sample size, and significance level 14

×