Section 8 Ensure Valid Test and Survey Results Trough Proper Sample Size Estimation Rhonda Knehans Drake Associate Professor, New York University Data Analytics, Interpretation and Reporting Copyright © 2013 2 • Prior to conducting your surveys, marketing tests, or obtaining your estimates we need to determine how large of a sample is required. • The more accuracy required in our estimates, the more we will need to sample. • However, we should only sample enough to obtain the level of accuracy needed to help us make a decision. • Some situations will require more accuracy than others. • The key here is to determine how much accuracy you will need to make the decision and only sample the required number of names. • We will discuss 4 sample size formulas in this section. Introduction 3 We will learn the following formulas for Sample Size estimation when concerned with: 1. The error associated with a single sample mean 2. The error associated with a single sample proportion or response rate 3. The error associated with the difference between 2 sample means or averages 4. The error associated with the difference between 2 sample proportions or response rates Sample Size Estimation 4 1. Sample Size Formula When Concerned with the Accuracy of a Single Sample Mean. n = Z2S2 E2 Where: • S is an estimate of the standard deviation. If unsure use 25% of your estimated average. • E is the ± error you can tolerate • Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99% confidence level. Single Sample Mean I 5 Example Every year you conduct a survey to determine student satisfaction at NYU. The scale is 1 -10 (1 = extremely unhappy and 10 = extremely Happy) Last year the survey yielded an average and standard deviation of 7.5 and 1.2. Your goal for this year was to increase satisfaction by 1 full point to 8.5. If unsuccessful you will not receive your bonus of $20,000. You work hard at increasing satisfaction over the course of the year by holding town hall meetings with students, putting suggestion boxes in all dorms, upgrading housing conductions, enhancing the student union with free coffee and snacks, etc. Single Sample Mean II 6 Example (continue…) Based on what you are hearing students say, you believe you will just meet your goal of moving the needle one point to a new average of 8.5. But, you do not believe it will be much higher than this value. How many students should you survey to ensure a tight read here with a maximum error of ± .1 with 99% confidence. Single Sample Mean II 7 • The resulting sample size is: Single Sample Mean III n = (2.575)2(1.2)2 (0.1)2 = (6.63)*(1.44) 0.01 = 954 1. 2. 3. 8 • Let’s do the previous example again but using the Plan-alyzer. 1. 2. 3. Select th.

1. Section 8
Ensure Valid Test and Survey Results Trough
Proper Sample Size Estimation
Rhonda Knehans Drake
Associate Professor, New York University
Data Analytics, Interpretation and Reporting
Copyright © 2013
2
• Prior to conducting your surveys, marketing tests, or obtaining
your estimates we need to determine how large of a sample is
required.
• The more accuracy required in our estimates, the more we will
need to sample.
• However, we should only sample enough to obtain the level of
accuracy needed to help us make a decision.
• Some situations will require more accuracy than others.
• The key here is to determine how much accuracy you will
need to make the decision and only sample the required number
of names.

2. • We will discuss 4 sample size formulas in this section.
Introduction
3
We will learn the following formulas for Sample Size
estimation
when concerned with:
1. The error associated with a single sample mean
2. The error associated with a single sample proportion or
response rate
3. The error associated with the difference between 2 sample
means or averages
4. The error associated with the difference between 2 sample
proportions or response rates
Sample Size Estimation
4

3. 1. Sample Size Formula When Concerned with the Accuracy of
a
Single Sample Mean.
n = Z2S2
E2
Where:
• S is an estimate of the standard deviation.
If unsure use 25% of your estimated average.
• E is the ± error you can tolerate
• Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%
confidence level.
Single Sample Mean I
5
Example
Every year you conduct a survey to determine student
satisfaction at NYU.
The scale is 1 -10 (1 = extremely unhappy and 10 = extremely
Happy)

4. Last year the survey yielded an average and standard deviation
of 7.5 and 1.2.
Your goal for this year was to increase satisfaction by 1 full
point to 8.5. If
unsuccessful you will not receive your bonus of $20,000.
You work hard at increasing satisfaction over the course of the
year by holding
town hall meetings with students, putting suggestion boxes in
all dorms,
upgrading housing conductions, enhancing the student union
with free coffee
and snacks, etc.
Single Sample Mean II
6
Example (continue…)
Based on what you are hearing students say, you believe you
will just meet
your goal of moving the needle one point to a new average of
8.5. But, you do

5. not believe it will be much higher than this value.
How many students should you survey to ensure a tight read
here with a
maximum error of ± .1 with 99% confidence.
Single Sample Mean II
7
• The resulting sample size is:
Single Sample Mean III
n = (2.575)2(1.2)2
(0.1)2
= (6.63)*(1.44)
0.01
= 954
1.
2.

6. 3.
8
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for Averages”
Select “One Sample”
Single Sample Mean IV

7. 9
Input the required info.
Single Sample Mean V
10
See the answer.
Single Sample Mean VI
11
• Is a 99% confidence level appropriate here in your opinion?
What
would be a more appropriate level to use?
• Had you felt you had moved the needle by almost 1.5 points
instead of
only one point, could you have been able to sample less names
and
tolerate more error while not putting your bonus in jeopardy?
Explain.
Single Sample Mean VII

8. A 99% confidence interval is a bit extreme for a survey.
A 90% - 95% interval is more appropriate.
An error of + 0.5 is still tolerable.
n = (6.63)*(1.44)
(0.25)
n = 38.2
12
2. Sample Size Formula When Concerned with the Accuracy of
a
Single Sample Proportion or Response Rate
n = Z2(p)·(1-p)
E2
Where:
• P is estimate of population proportion. You will base this
figure on prior experience.
• E is the ± error you can tolerate
• Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%
confidence level.

9. Single Sample Proportion I
13
Example
You are about to test a new prospect list
You expect the response rate of this new list to be some where
around 1% based on your list brokers experience.
Your break-even (the lowest response rate you can tolerate) for
prospecting is .9%.
How many names should you sample so that should the response
rate come in at 1% you will be able to make a decision
regarding
using the entire list?
Single Sample Proportion II
14
• So

10. n = (1.96)2·(.01)(.99) = (3.8416)(.0099) = 38,032
(.01-,009)2 .000001
• Do so will ensure should the response rate of the test come in
at 1%
the resulting confidence interval will look like
.01 ± (1.96) ·√ (.01)(.99)/38,032 = .01 ± .000877
(.9%, 1.1%)
and we can make our decision with actually same worse case
our
response rate is at or above break even!
Single Sample Proportion III
n = (1.96)2(0.01)·(0.99)
(0.001)2
n = (3.8416)(0.0099)
(0.000001)
n = 38,032

11. 1.
2.
3.
15
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of

12. Calculators”
Select “Sample Size
Calculators for
Percentages”
Select “One Sample”
Single Sample Proportion IV
16
Input the required info.
Single Sample Proportion V
17
See the answer.
Single Sample Proportion VI
18
3. Sample Size Estimation When Concerned with Accurately
Measuring the Difference Between 2 Means or Averages

13. n1 = n2 = Z2(S12 + S22)
d2
Where:
•d is the minimum difference you wish to detect as
significant should it be observed.
•S1 and S2 are estimates of the standard deviation
associated with each sample. In most cases you will use
the same estimate for both samples and if unsure, you
will use 25% of your expected average.
•Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99%
confidence level
Difference Between Sample Means I
19

14. Example
You work for MasterCard and you wish to test an incentive for
new
card members to increase spend over their first 3 months as a
card holder.
Based on a break even analysis you will need spend to increase
by
$5 to cover the costs of your incentive (a few bonus sky miles
that
is costing you about 10 cents per card member) Currently, new
card members spend on average $325 over the first 3 months
with
a standard deviation of $25.
How many names should we sample to ensure that if we find the
test to yield a spending level of $330 we can read the results as
significant?
Difference Between Sample Means II
20
Difference Between Sample Means III

15. n1 = n2 = (1.96)2(252 + 252)
52
n1 = n2 = (3.8416)*(625 + 625)
25
n1 = n2 = 192
1.
2.
3.
21
• Let’s do the previous
example again but using
the Plan-alyzer.

16. 1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for
Averages”
Select “Test vs. Control”
Difference Between Sample Means IV
22
Input the required info.
Difference Between Sample Means V
23

17. See the answer.
Difference Between Sample Means VI
24
4. Sample Size Estimation When Concerned with Accurately
Measuring the Difference Between 2 Proportions or Response
Rates
n1 = n2 = (Z2) (p1)(1-p1) + (p2)(1-p2)
d2
Where:
•p1 is an estimate of one of the samples. You typically know
the
response rate of your control group.
•d is difference you wish to detect as significant.
•p2 is p1 + d
•Z is 1.645, 1.96 or 2.575 for a 90%, 95% or 99% confidence

18. level
Difference Between Sample Proportions I
25
Example
You are testing the addition of a premium to your control
package.
Based on a break even analysis, you determine that you need
two
additional order per thousand names mailed to break even with
the
control.
Your control response rate is typically 1.00%.
How many names should we sample to ensure that if we obtain
two additional orders for our test package we will be able to
detect
it as a significant increase.
Difference Between Sample Proportions II

19. 26
Difference Between Sample Proportions III
n1 = n2 = (1.962) (0.01)(0.99) + (0.012)(0.988)
(0.002)2
n1 = n2 = (3.8416) (0.0099) + (0.011856)
(0.000004)
n1 = n2 = 20,894
1.
2.
3.

20. 27
• Let’s do the previous
example again but using
the Plan-alyzer.
1.
2.
3.
Select the tab “Table of
Calculators”
Select “Sample Size
Calculators for
Percentages”
Select “Test vs. Control”
Difference Between Sample Proportions IV
28
Input the required info.

21. Difference Between Sample Proportions V
29
See the answer.
Difference Between Sample Proportions VI
30
There are two break-evens we typically calculate as marketers:
• The break-even response rate required for a new list or
product
test such that profit exactly offsets revenue – breakeven.
• The break-even for a new and more expensive format or
creative
test such that net profit generated equals that of the control
format.
Break-Even Analysis
31

22. The break-even response rate for a new list or product test is the
lowest
response rate you can tolerate and not lose any money. It is
easily
calculated. It is the response rate such that:
Revenue – Costs = $0, or
(MQ x RR x PPP) – (MQ x PC) = $0
Where: MQ = Mail Quantity
RR = Response Rate
PPP = Profit Prior Promotional Costs
PC = Promotional Costs
Break-Even Response I
32
By rearranging the formula and solving for the RR, we find that
the
break-even response rate is equal to:

23. RR = PC / PPP
Break-Even Response II
33
Consider the following example:
Assume you sell collector plates via direct mail. The average
profit per
order before promotional costs is $55.00. You are planning to
test a
new list on the market that will cost you $650.00 per 1,000
names
promoted.
What is the minimum response rate you must achieve on this list
test in
order to break-eve and not lose any money?
And, if you typically have never seen a response rate above
1.00%
historically (regardless of how good the list is) do you
recommend
testing the list?

24. Break-Even Response III
34
Break-even is calculated as:
Our decision to test this list is:
Break-Even Response IV
RR = PC / PPP
= 0.65/55
= 0.118 or 1.18%
No. we will most likely not see this level of response, so do not
test.
35
The increase in response that must be obtained on a new and

25. more
expensive format test in order to generate at least the same
profit as the
control format is the response rate such that:
Test Rev – Test Cost = Control Rev – Control Cost, or
(MQ*RRT*PPP) – (MQ*PCT) = (MQ*RRC*PPP) – (MQ*PCC)
Where: MQ = Mail Quantity
RRT = Test Response Rate
RRC = Control Response Rate
PPP = Profit Prior Promotional Costs
PCT = Test Promotional Costs
PCC = Control Promotional Costs
Increase in Response Required Break - Even
36
Consider the following example:

26. Your current control format is known to yield a 5% response
rate and
has a promotional cost of $1 per piece. The profit prior
promotional
costs per order is $30.
Your creative director has come up with a new format but it is
quite
expensive. This new format will cost you $1.75 per piece to
mail.
What is the increase in response required for this new format to
break-
even wit the control format?
Increase in Response Required Break - Even
37
Break-even for the new format test is calculated as:
Increase in Response Required Break - Even
(1,000*RRT*30) – (1,000*1.75) = (1,000*0.05*30) – (1,000*1)

27. Divide both sides by 1,000
(30*RRT) – (1.75) = (1.5) – (1)
30*RRT = 2.25
RRT = 0.075 or 7.50%
1.
3.
2.
4.
5.

28. 38
8.1 A researcher wants to determine a 95% confidence interval
for the mean
number of hours that high school students spend doing
homework per
week. She believes based on prior research that the average
study time
per week is about 20 hours with a standard deviation of 7 hours.
How
large a sample should the researcher select this year so that the
estimate
will be within 1.5 hours of the population mean?
Do by hand and using the Plan-alyzer.
8.2 A U.S. government agency wants to estimate at a 95%
confidence level
the mean speed for all cars traveling on Interstate Highway I-
95. From a
previous study last year, the agency knows that the average is
about 63
miles per hour with a standard deviation of 3.5 miles per hour.
What

29. sample size should the agency choose this year so that the
estimate will
be within 1.5 miles per hour of the population?
Do by hand and using the Plan-alyzer.
Section 8 Exercises I
39
8.3 Tony’s Pizza guarantees all pizza deliveries within 30
minutes of the
placement of orders. The Federal Trade Commission is
concerned with
Tony’s advertisements and feels, based on customer complaints,
that they
only meet their guarantee about 50% of the time. As such the
FTC has
requested that Tony conduct a study. What sample size should
the FTC
require of Tony’s to ensure the estimate obtained is within 2%
of the true
percentage with 99% confidence?
Do by hand and using the Plan-alyzer.

30. 8.4 A consumer agency wants to estimate the proportion of all
drivers who
wear seat belts while driving. Assume that a preliminary study
has shown
that 76% of drivers wear seat belts while driving. How large
should the
sample size be so that a 99% confidence interval for the
population
proportion has a maximum error of .03?
Do by hand and using the Plan-alyzer.
Section 8 Exercises II
40
8.5 The marketing director at ACME Direct is planning to test
the addition of a
4-color flyer to his current direct mail control format. The 4-
color flyer will
contain testimonials from famous celebrities praising the
product being
offered. The control format is expected to yield a 4.50%
response rate. In

31. order to cover the cost of the flyer (break-even) the test format
will need to
yield an additional 3 orders per thousand names promoted. To
ensure the
marketing director will be able to read the break-even response
rate with
statistical significance, how large should each test panel be?
Assume a
95% confidence level.
Do by hand and using the Plan-alyzer.
Section 8 Exercises III
41
8.6 Jet Music is a direct marketer of music packages covering
all genres. Their active
music buyer market is shrinking and fast. There is much
competition. Based on prior
mailings, one of Jet Music’s most popular CD packages, “Dance
Till You Drop” is
known to yield a net response rate of 3.63% at a $9.97 price
point. In an effort to

32. help increase response rates, the marketing manager has tested
this title at a $1
lower price. Order intake is just beginning to come in for this
test. After two weeks of
intake the net response rate is approaching 3.95% and climbing.
At least 6 more
weeks of intake is expected. It is looking good.
The marketing managers boss is curious if the test of a $1 price
decrease is at the
break-even response rate level yet. Calculate the minimum net
order rate required to
break-even with the $1 price decrease test so that the marketing
manager can
answer her boss.
Section 8 Exercises IV
42
8.7 You are the marketing manager at ACME Publishing. You
test promoted a new
cookbook concept to a very large compiled list file and received
a response rate of
2.54%. Based on an examination of age information, you notice

33. that for those over 50
years of age you received a response rate of 4.30% (an index of
169 to total or a 69%
gain over total).
Assume the following:
- Cook book profit prior promotion costs = $9.92
- Promotion costs including list rental costs = $0.4217 per
Should you promote those on this complied list file that are
over the age of 50 if your
goal of this promotion is to break even?
Section 8 Exercises IV