How to estimate sample size for surveys and experiments

1. Choosing the correct
sample size
Roger Watson

2. Sample size
Question:
Why is it important?
Answer:
• We usually cannot access the whole population
• We usually do not need to access the whole population
• We need samples to be representative of the population in surveys
• We need samples to be large enough to show real effects where inferential
statistics are being used to analyse data

3. Sample size - surveys
How many people should the questionnaire be sent to?
Answer:
• Enough to make the results representative of the population
• No more than is necessary to achieve the above
• Sample size will be related to the size of the population

4. Minimum sample sizes for selected small populations
95% confidence
Population size with 5% margin of error
<200 Sample whole population
500 218
1,000 278
2,000 323
5,000 357
10,000 370
100,000 383
(Jackson & Furnham 1999)

5. Minimum sample sizes for selected small populations
95% confidence
Population size with 5% margin of error
<200 Sample whole population
500 218
1,000 278
2,000 323
5,000 357
10,000 370
100,000 383
The confidence level is the
amount of uncertainty you can
tolerate. Suppose that you have
20 yes-no questions in your
survey. With a confidence level
of 95%, you would expect that
for one of the questions (1 in
20), the percentage of people
who answer yes would be more
than the margin of error away
from the true answer. The true
answer is the percentage you
would get if you exhaustively
interviewed everyone.
Higher confidence level
requires a larger sample size.

6. Minimum sample sizes for selected small populations
95% confidence
Population size with 5% margin of error
<200 Sample whole population
500 218
1,000 278
2,000 323
5,000 357
10,000 370
100,000 383
The margin of error is
the amount of error that
you can tolerate. If 90%
of respondents
answer yes, while 10%
answer no, you may be
able to tolerate a larger
amount of error than if
the respondents are split
50-50 or 45-55.
Lower margin of error
requires a larger sample
size.

7. http://www.raosoft.com/samplesize.html

8. Populations and samples
Population: all the members of a particular group
Sample: a subset of a population
Representative sample: sample is like population
Biased sample: sample is unlike population
Sampling error: difference between the above

9. Biased samples
•Result from poor sampling
•May result from poor response rates
• For example, only middle class, educated and motivated people may
respond to a survey

10. Sampling methods
Probability sampling:
simple random
stratified random
Non-probability sampling:
opportunistic
systematic
purposive

11. Sample size
Also depends on the statistical tests you are going to use
Power analysis*
Related to effect size
* Cohen J (1992) A power primer Psychological Bulletin 112 155-159

12. Effect size
In statistics, an effect size is a quantitative measure of the strength of a
phenomenon. Examples of effect sizes are the correlation between two
variables, the regression coefficient, the mean difference, or even the risk
with which something happens, such as how many people survive after a heart
attack for every one person that does not survive. For each type of effect-size,
a larger absolute value always indicates a stronger effect. Effect sizes
complement statistical hypothesis testing, and play an important role
in statistical power analyses, sample size planning, and in meta-analyses.
(Wikipedia)
Cohen suggested that d=0.2 be considered a 'small' effect size, 0.5
represents a 'medium' effect size and 0.8 a 'large' effect size. This means
that if two groups' means don't differ by 0.2 standard deviations or more, the
difference is trivial, even if it is statistically significant. (Walker)

15. Total sample
(100)
Men Women
(50) (50)
Young Old Young Old
(25) (25) (25) (25)
Well Sick Well Sick Well Sick Well Sick
(12) (13) (10) (15) (13) (12) (11) (14)

16. email: r.watson@hull.ac.uk
@rwatson1955