Effect Size


Published on

Presentation introducing a better hypothesis testing methodology: Effect Size.

Published in: Education, Technology, Sports
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Effect Size

  1. 1. Beyond p value: Effect Size April 4, 2008 Guy Lion
  2. 2. P value vs Effect Size <ul><li>By increasing sample size, you can show there is a statistically significant* difference between two Means. The Effect Size evaluates how material is that difference. </li></ul>P value = probability sample Means are the same . (1 – P) or C.L. = probability sample Means are different . Effect Size = how different sample Means are. *Statistically significant does not imply “significant.” [Webster: … of consequence.]
  3. 3. Effect Size in Plain English Large Effect Size is visible without looking at a large sample. <ul><li>Other Effect Size examples: </li></ul><ul><li>The Kaplan course raises SAT math scores by 60 points; </li></ul><ul><li>This WOW initiative increases # Solutions by 30 units. </li></ul><ul><li>When analyzing Effect Size we deal with standardized units… </li></ul>With sea lions gender has a large Effect Size. With pugs gender has a small Effect Size
  4. 4. Effect Size Measures Cohen’s d = (Mean Pilot – Mean Control )/Pooled Stand. Deviation Cohen’s d is similar to the unpaired t test t value. It relies on Standard Deviations instead of Standard Errors. Hedges’ g is a more accurate version of Cohen’s d Hedges’ Cohen’s d Adjustment for small sample
  5. 5. Pilots vs Controls In this example, there is a 1.9 Standard deviation difference between Pilots and Controls.
  6. 6. Effect Size Info
  7. 7. An Example
  8. 8. ES Confidence Interval The Effect Size standard deviation formula allows to build Confidence Intervals around Effect Size values.
  9. 9. 1 st Nonparametric test: Gamma Index Recalculated Gamma Index to make it the same sign as Cohen's d and Hedges' g.
  10. 10. 2 nd Nonparametric test: Cliff Delta Cliff Delta ranges between +1 when all values of one group are higher than the values of the other group and – 1 when reverse is true. Two overlapping distributions would have a Cliff Delta of 0.
  11. 11. Cliff Delta efficiently… [(1 x 0) – (1 x 8)]/10 = -0.8
  12. 12. Cliff Delta vs Cohen’s d If distributions are Normal, Cliff Delta = Percent of Nonoverlap within a Cohen's d framework.
  13. 13. Effect Size and Sample Size requirement The above formula results from the algebraic transformation of: t stat or Z value = Difference in Means/Group Standard Error.
  14. 14. Testing Sample Size
  15. 15. Conclusion <ul><li>P value does not tell us how different two samples are. </li></ul><ul><li>Effect Size and its Confidence Intervals give much information on how different two samples are. </li></ul><ul><li>Effect Size in units allows us to derive a required sample size to meet a p value  threshold. </li></ul>