Just finished designing your survey and have no idea how many people you should send it to? We've got you covered! Our in-house Survey Scientist, Sarah Cho the Survey Pro, answers this question and more.
Thanks for setting the stage Sheila. Before we get started on answering your questions, I just want to go over a general overview of how many respondents you might need to take your survey from the universe of survey respondents. For instance, you don’t need everyone in the whole world to take your survey, you just need a well-selected sample of people to take your survey. That’s where sample size calculations and sampling come in.
Let’s walk through an example to show you how you might make a decision on how many people you need to take your survey.
Let me introduce you to Kathy. Kathy is a gold star teacher who organized an educational conference for her fellow teachers and wants to send out a survey to gather feedback on the conference. She’s got her survey programmed and ready to go, but is unsure how many respondents she needs for her survey to get actionable results. We’re going to help out Kathy by walking through the steps on determining your sample size.
Before Kathy or you, our viewer, can figure out how many respondents you need, there are five things you need to consider: Who is your target population? Your target population is every single person you are interested in surveying. For example, what is the target population of a politician who is running for president who is surveying folks to figure out how many people are going to vote for him? For that politician, his target population is everyone eligible to vote in the country. Another example – what about someone who is sending out a survey to their book club to figure out what book they are reading next? Well, their target population is everyone who is a member of the book club. What is the size of your target population? If you had unlimited resources how many people meet the criteria to take your survey?
First – who is your target population? For Kathy, she actually has two target populations. First are conference attendees – who she wants to gather feedback about the conference food, amenities, and the quality of the speakers from. Second are teachers who were invited to the conference, but did not attend. She wants to ask these people how she can improve the conference to entice them to come the following year.
Next step to think about is how large is the target population?
For Kathy, she invited 2500 teachers to the conference and 500 of them registered and attended. That means her target population consists of 500 conference attendees and 2000 non-attendees
The next thing Kathy has to figure out is how much error they are willing to tolerate in their survey results. In order to do so, they also have to think about the goals of their survey and how they are going to use the survey data once all the responses have been collected.
Kathy plans to use the data to make improvements to the conference agenda, with a special target on offering conference sessions that appeal most to people who didn’t register in hopes of increasing attendance. She also hopes to use the data to assess whether she needs to change some of the conference amenities like the location or catering. So judging by Kathy’s goals for her survey, she doesn’t need to pinpoint the exact percentage who liked the location or food offered, but needs to be somewhere in the ballpark.
NEED TO NUMBER CHECK
Now let’s put together a few of the things we’ve covered so far to get at how many people we need for these surveys. As a general rule of thumb, the more people you get to take your survey, the less error you will have around your numbers, but as a word of caution, just because you have tons of people taking your survey doesn’t mean it is error or bias-free. There are many ways that error and bias can enter a survey and having an adequate sample size is just one way in which you can reduce the amount of error or bias.
Let’s start again with Kathy. We’ve already figured out that her target population of conference attendees is 500. She doesn’t need exact results, so she has decided that she is willing to accept a plus or minus five percentage point error. What does that mean? Well, if Kathy finds that 60 percent of conference attendees say they were satisfied with the food at the conference, she can be fairly confident that the true number ranges somewhere between 55 percent and 65 percent. So what sample size does she need? Taking into consideration the amount of error and size of the target population, looks like Kathy needs 220 conference attendees responding to her survey.
What about her survey of teachers who were invited to the conference but didn’t attend? We know that the size of that population is 2000, and if she wants around the same amount of precision as her previous group, she needs 350 non-attendees to complete her survey.
Before we move on to the next step, I just want to go over two things. First, there are some underlying assumptions to this table that I won’t get into the nitty gritty details, but if you are curious, just chat in your question and we can cover it later Second, if you want to survey everyone in your target population, go for it! Then you’ll have zero SAMPLING error around your numbers. Kathy might be able to do this since the size of her target population is small and manageable, but for whose target population is 1 million people, it isn’t feasible. Note just because you survey everyone and don’t have any error doesn’t mean your survey is perfect. As I mentioned previously, there are many areas in which bias or error could enter into your survey, for example question wording and order, so always be aware of things like that as well.
Okay, now that we’ve figured out what sample size Kathy needs, we’re almost there. The next thing they have to think about is how many people do they need to SEND the survey out to. For example, Kathy can’t just send out her survey to 220 randomly selected conference attendees and expect all of them to respond. So Kathy and Fred need to do some research and think about what their response rates might be.
Kathy expects about 75 percent of conference attendees to respond to her survey because she publicized the survey throughout the conference and told them that it was important to provide feedback. She also thinks that people are engaged enough to provide feedback since they came to the conference. But for non-attendees she expects a lower response rate of 25 percent since she didn’t publicize it with this group and they might not be engaged in the topic since they decided that the conference wasn’t right for them.
It’s time to do a little bit of math. Don’t worry, I’m not going to make you do calculus, but you might want to grab a calculator. To figure out the number of survey invitations, you’ll need to divide the total sample size you need by your expected response rate.
For attendees of Kathy’s conference, we divide the desired sample size of 220 we found when we were looking at the table a few slides ago by Kathy’s expected response rate of 75 percent or .75. After doing the math, looks like Kathy will need to send out 300 invites to her survey. How should she select these 300 attendees? The best way is to do a RANDOM sample and pick 300 folks randomly, so she knows that she is getting a good mix of attendees, for example young teachers and old teachers, to take her survey.
For non-attendees, since the response rate is lower, she’ll need to send out 1400 invites in order to get her desired sample size of 350.
Now one of the last steps you need to consider is whether you are thinking about breaking the data down by subgroups, for example men versus women.
This is a little bit more advanced and depending on your needs it may come earlier in the process, but you’ll have to think forward to how you’ll break down the data in the analysis phase. If you do some advanced planning, you’ll be sure that you’ll have a large enough sample size to say something meaningful about those smaller groups.
Kathy for example, wants to break her data down for conference non-attendees by geographic area to see whether those who lived further away from the conference location were more likely to say that the location was the main reason for not attending.
Kathy for example wants to break her data down by teachers who are seasoned veterans versus those who are newbies because their opinions on the conference could differ.
In order for Kathy to figure out if her sample sizes of 220 and 330 are large enough after breaking her data down by subgroups for her two populations, she needs to know how many people in her target population fall into each group and do more math to see how many respondents she might get in each group. To make the math easy, let’s say that half of Kathy’s target population, conference attendees, are seasoned veterans, and the other half are newbies. That means she can expect 110 respondents to be veterans and 110 respondents to be newbies which is enough to make meaningful conclusions about. A few things to note: This situation assumes that her respondents are selected randomly. And because the sample size is smaller for the subgroups than the overall, the error for each of the subgroups will be larger than the plus or minus 5 percentage points Kathy originally selected.
But what if her two groups – vets versus newbies – aren’t divided in half but instead only 10% will be newbies and 90% vets? That means she can only expect 22 newbies in her sample, which really isn’t a large enough number to make solid conclusions from. If she is very interested in looking at these newbies, in this situation where only 10% of the target population is a newbie, even though her target sample size of 220 is enough to help guide her decisions of the group as a whole, it isn’t enough for newbies and she’ll need a larger sample size in order to get enough newbie respondents. More sample also means more invites, so don’t forget to redo that calculation as well.
And that wraps up the quick primer on how many respondents you might need. If you have any questions about that or questions that are specific to your project, chat them in to us!
How Many People Do I Need to Take My Survey?
SurveyMonkey Office Hours
Sarah Cho, Survey Research Scientist
JUNE 9, 2014
Today’s topic: How many people do I
need to take my survey?
Meet Kathy, a teacher and event organizer
• Organized a conference for teachers and wants
to gather feedback on the conference
do I need?
Questions to answer before you start surveying
• Who is your target population?
• What is the size of your target population?
• How much error are you willing to tolerate?
• What type of response rate do you expect?
• Are you thinking of breaking the data down by
Who is your target population?
• Conference attendees
• Invited teachers who did not attend
What is the size of your target population?
• 2500 teachers were invited to the conference
- Target population #1: 500 teachers registered and
- Target population #2: 2000 teachers who were invited
but did not attend (non-attendees)
How much error are you willing to tolerate?
• Attendees: Make improvements to conference
agenda and amenities
• Non-attendees: Improve program quality or
relocate conference to improve attendance rate
What type of response rate do you expect?
• Attendees: 75%
• Non-attendees: 25% How many
teachers will take
What type of response do you expect? Let’s do some math!
Total sample desired ÷ expected response rate = total invites needed
• Total sample desired:
- 220 Attendees
- 350 Non-attendees
• Total invites needed:
- Attendees: 200 ÷ 0.75 = ~300
- Non-attendees: 350 ÷ 0.25 = ~1400
Are you thinking about breaking down the data by subgroups?
• For non-attendees: geographic area
• For conference attendees: Seasoned veterans versus newbies
• 50 % newbies, 50% seasoned veterans
• Overall sample size: 220
• 110 newbies
• 110 seasoned veterans
• 10% newbies, 90% seasoned veterans
• Overall sample size: 220
• 22 newbies
• 198 seasoned veterans