Beyond p value: Effect Size April 4, 2008 Guy Lion
P value vs Effect Size <ul><li>By increasing sample size, you can show there is a statistically significant* difference between two Means. The Effect Size evaluates how material is that difference. </li></ul>P value = probability sample Means are the same . (1 – P) or C.L. = probability sample Means are different . Effect Size = how different sample Means are. *Statistically significant does not imply “significant.” [Webster: … of consequence.]
Effect Size in Plain English Large Effect Size is visible without looking at a large sample. <ul><li>Other Effect Size examples: </li></ul><ul><li>The Kaplan course raises SAT math scores by 60 points; </li></ul><ul><li>This WOW initiative increases # Solutions by 30 units. </li></ul><ul><li>When analyzing Effect Size we deal with standardized units… </li></ul>With sea lions gender has a large Effect Size. With pugs gender has a small Effect Size
Effect Size Measures Cohen’s d = (Mean Pilot – Mean Control )/Pooled Stand. Deviation Cohen’s d is similar to the unpaired t test t value. It relies on Standard Deviations instead of Standard Errors. Hedges’ g is a more accurate version of Cohen’s d Hedges’ Cohen’s d Adjustment for small sample
Pilots vs Controls In this example, there is a 1.9 Standard deviation difference between Pilots and Controls.
ES Confidence Interval The Effect Size standard deviation formula allows to build Confidence Intervals around Effect Size values.
1 st Nonparametric test: Gamma Index Recalculated Gamma Index to make it the same sign as Cohen's d and Hedges' g.
2 nd Nonparametric test: Cliff Delta Cliff Delta ranges between +1 when all values of one group are higher than the values of the other group and – 1 when reverse is true. Two overlapping distributions would have a Cliff Delta of 0.
Cliff Delta efficiently… [(1 x 0) – (1 x 8)]/10 = -0.8
Cliff Delta vs Cohen’s d If distributions are Normal, Cliff Delta = Percent of Nonoverlap within a Cohen's d framework.
Effect Size and Sample Size requirement The above formula results from the algebraic transformation of: t stat or Z value = Difference in Means/Group Standard Error.
Conclusion <ul><li>P value does not tell us how different two samples are. </li></ul><ul><li>Effect Size and its Confidence Intervals give much information on how different two samples are. </li></ul><ul><li>Effect Size in units allows us to derive a required sample size to meet a p value threshold. </li></ul>