Power, effect size, and Issues in NHST

1,774 views

Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Power, effect size, and Issues in NHST

  1. 1. Calculating Effect Size Power Analysis Issues in Null Hypothesis Significance Testing Carlo Magno, PhD. De La Salle University, Manila
  2. 2. A researcher wanted to look at the effect of behavior modification technique on the aggression of clients. A group of participants in the experimental group were given behavior modification technique and no treatment in the control. The aggression of the two groups were measured after. n 30 60 100 500 1000 df 28 58 98 498 998 X1 X2 4.6 4.6 4.6 4.6 4.6 4.1 4.1 4.1 4.1 4.1 SD1 SD2 2.3 2.3 2.3 2.3 2.3 2.3 2.3 2.3 2.3 2.3 t 0.60 0.84 1.09 2.43 3.44 p value 0.56 0.4 0.28 0.02* 0.00*
  3. 3. Criticism on NHST • 1. NHST does not provide the information which the researcher wants to obtain • 2. Logical problems derived from the probabilistic nature of NHST. • 3. NHST does not enable psychological theories to be tested. • 4. The fallacy of replication. • 5. NHST fails to provide useful information because H0 is always false. • 6. Problems associated with the dichotomous decision to reject/not reject the H0. • 7. NHST impedes the advance of knowledge.
  4. 4. Alternatives to NHST • Effect size • Confidence levels • Power analysis
  5. 5. Effect Size • Cohen (1988) defines the effect size as the extent to which the phenomenon is found within the population or, in the context of statistical significance testing, the degree to which the H0 is false. • Snyder and Lawson (1993) argue that the effect size indicates the extent to which the dependent variable can be controlled, predicted and explained by the independent variable(s).
  6. 6. Effect Size Measures • Effect size measures of Two In/dependent Groups – Cohen’s d – Hedges g – Glass Delta • Correlation Measure of Effect Size –r – χ2 ►Φ; t ► r; F ► r; d►r • Effect size for Analysis of Variance – Eta Squared – Omega Square Index of Strength – Intercalss correlation
  7. 7. M1 − M2 d= 2 s 1 + s 22 2 Cohen’s d Formula M1 − M2 t= 2 s 1 + s 22 n1 n2 t-test for independent Means Formula
  8. 8. Computation A research compared students who engaged in group and individual sports on their passion on the sport. Passion was measured using the Passion Scale by Vallerand with tow factors, harmonious and obsessive passion. The t-test for two independent samples was used to determine the significant difference between the students in the group and individual sports on the two factors of passion. The following statistical output was obtained:
  9. 9. Statistical Results M1 M2 t-value df p N1 N2 SD1 SD2 HP 5.51 5.68 -1.01 58 0.315 30 30 0.70 0.61 OP 4.91 5.36 -1.40 58 0.167 30 30 1.51 0.87 Compute for the effect size http://www.uccs.edu/~lbecker/ http://effect-size-generator.software.informer.com/download/
  10. 10. Cohen's Standard LARGE MEDIUM SMALL Effect Size Percentile Standing 2 1.9 1.8 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 97.7 97.1 96.4 95.5 94.5 93.3 91.9 90 88 86 84 82 79 76 73 69 66 62 58 54 50 Percent of Nonoverlap 81.10% 79.40% 77.40% 75.40% 73.10% 70.70% 68.10% 65.30% 62.20% 58.90% 55.40% 51.60% 47.40% 43.00% 38.20% 33.00% 27.40% 21.30% 14.70% 7.70% 0%
  11. 11. Cohen's Standard LARGE MEDIUM SMALL d 2 1.9 1.8 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 r 0.707 0.689 0.669 0.648 0.625 0.6 0.573 0.545 0.514 0.482 0.447 0.41 0.371 0.33 0.287 0.243 0.196 0.148 0.1 0.05 0 r2 0.5 0.474 0.448 0.419 0.39 0.36 0.329 0.297 0.265 0.232 0.2 0.168 0.138 0.109 0.083 0.059 0.038 0.022 0.01 0.002 0
  12. 12. Statistical Power Reject H0 No real effect Real Effect Type 1 error α (.01, .05) Ho not rejected Slim chance of concluding that the treatment is effective, despite the fact that it is Type 2 error β (small as possible) 1-β Statistical power
  13. 13. Statistical Power • β=.20 (the error of rejecting a true Ho is 4x more serious than the error of not rejecting a false Ho) • .80=acceptable power
  14. 14. Statistical Power • The probability of rejecting a false null hypothesis. • The likelihood that a study will detect an effect when there is an effect to be detected. • If statistical power is high, the probability of making a Type II error, (or concluding there is no effect when, in fact, there is one) goes down.
  15. 15. Statistical Power • The power of any test of statistical significance will be affected by four main parameters: – the effect size – the sample size (N) – the alpha significance criterion (α) – statistical power, or the chosen or implied beta (β)
  16. 16. Statistical Power Statistics Power small r .80 26 t .80 29 medium Large 63 393 85 781 http://danielsoper.com/statcalc3/default.aspx http://www.statisticalsolutions.net/pss_calc.php https://www.dssresearch.com/KnowledgeCenter/toolkitcalculators/statisticalpow ercalculators.aspx http://homepage.stat.uiowa.edu/~rlenth/Power/
  17. 17. Influence of Effect Size on Power High school n=65 College n=153
  18. 18. Influence of Effect Size on Power
  19. 19. Influence of Effect Size on Power N=82 Taiwanese in Taiwan N=98 Taiwanese in the Philippines
  20. 20. Influence of Effect Size on Power
  21. 21. • What inference can be gained between effect size and power with fixed sample size and alpha level?
  22. 22. Influence of Significance Level on Power • Study of De Frias, Dixon, and Strauss (2006) • N=418 • r=.14 (not significant) • α=.01 (power=.23) α=.05 Power=.45 α=.10 Power=.58 α=.15 Power=.65 α=.20 Power=.71
  23. 23. • What inference can be gained between level of significance and power with fixed sample size?
  24. 24. Influence of Sample size on Power Magno (2005) Monitoring and metacognition N=280 r=.14 Power=.65 Magno, Mamauag, & Parinas (2007) Independence and self-esteem N=373 r=.14 Power=.78 Chemers, Hu, & Garcia (2001) N=381 Challenge-threat and self-efficacy r=.15 Power=.83
  25. 25. Influence of Sample size on Power
  26. 26. • What inference can be gained between sample size and power with fixed effect size and significance level?

×