Carma internet research module sample size considerations


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Carma internet research module sample size considerations

  1. 1. Sample Size Considerations<br />CARMA Internet Research Module<br />Jeff Stanton<br />
  2. 2. Key Considerations<br />Sample size versus response rate – planning for the number of usable data points you will actually obtain<br />Attrition – Repeated measures, panel designs, and diary studies all lose participants over time<br />Statistical power – ability to draw inferences from the sample obtained<br />Margin of error – to the extent that the resulting statistics must be projectable to the larger population<br />
  3. 3. May 15-17, 2008<br />Internet Data Collection Methods (Day 2-3)<br />Response Rate Reminder<br />70%<br />65%<br />60%<br />55%<br />50%<br />45%<br />40%<br />1975<br />1995<br />Academic Surveys<br />
  4. 4. Hope for the best / Plan for the worst<br />Try to achieve an 80% response rate<br />Hope to achieve a 50% response rate<br />Plan ahead for a 30% response rate<br />Means you need to sample 1000 people to obtain a sample of 300<br />
  5. 5. Bad Data<br />Unproctored, anonymous self report instruments generally have a higher percentage of:<br />Unusual outliers<br />Missing data<br />Carelessly entered data<br />Intentionally sabotaged data<br />Another aspect of dealing with nonresponse is to anticipate, prepare for, and deal with item level data losses<br />
  6. 6. Attrition<br />
  7. 7. The Best Articles on Statistical Power<br />Cohen, J. (1992). "A power primer." Psychological bulletin 112(1): 155-159.<br />Cohen, J. (1992). "Statistical power analysis." Current Directions in Psychological Science: 98-101.<br />Kraemer, H. and S. Thiemann (1987). How many subjects?: Statistical power analysis in research, Sage Publications, Inc.<br />
  8. 8. May 15-17, 2008<br />Internet Data Collection Methods (Day 2-8)<br />Sample Size “Guestimates”(With apologies to Jacob Cohen)<br />
  9. 9. May 15-17, 2008<br />Internet Data Collection Methods (Day 2-9)<br />Estimating Effect Size(Also with apologies to Jacob Cohen)<br />Mean differences, calibrated in standard deviations: Large = .8+, Medium = .5, Small = .2<br />Multiple regression, size of R-squared: Large =.35+, Medium = .15, Small = .02<br />Chi-square, calibrated in the difference between null and alternate population proportions: Large = . 50, Medium = .30, Small = .10 <br />
  10. 10. Margin of Error<br />Generally represents only sampling error: Other sources of error will often make the margin much larger<br />Assumes a large population, with no more than 5% drawn into the sample<br />Margin of error is half the width of a confidence interval<br />Straightforward calculation for a CI around a mean or a mean difference: generally about 1.96 standard errors<br />CI around a proportion/percentage is more complex:<br />Use 1.96 times this SE; works fine for even splits; can be a little funky for extreme proportions<br />
  11. 11. Margin of Error Calculators<br /><br />Trades off sample size and margin of error<br /><br />Explains terminology<br /><br />Various tools for assessing poll data<br /><br />Confidence intervals for correlations<br /><br />Java-based applet<br />
  12. 12. An Overall Sampling Plan<br />Estimate the expected effect size for the most important tests you plan to conduct<br />For inferential testing, use power estimation tools to plan sample size<br />For projectability, use margin of error tools to plan sample size<br />Take into account item level data loss due to bad data<br />Take attrition into account for longitudinal designs<br />Take overall response rate into account for all types of designs<br />Determine overall initial sample size based on all of the factors listed above<br />