This presentation is a guide to creating great surveys. It was originally prepared for CoFoundersLab as a two-part webinar series. This content was also used in various presentations including at the annual AMA (American Marketing Association) Conference. And it's an incredibly useful guide that seeks to help demystify great survey creation along with how to run and manage a survey/feedback project.

1. FROM QUESTIONS TO
CONFIDENCE
USING SURVEYS TO HELP DRIVE
GROWTH IN YOUR BUSINESS

2. ABOUT ME
BRENT CHUDOBA
▸ Currently COO @PicMonkey, a leading
provider of image editing and design
software that empowers everyday
creatives. Board member @Schoology
▸ 2009-2016: Chief Revenue Officer
@SurveyMonkey where I managed
functions including sales, business
intelligence, business development and
M&A
▸ 2007-2009: Associate @Spectrum Equity
Investors
▸ Pre-2007: Piper Jaffray, Andor Capital

3. CONTENTS
▸ Quick Wins: 3 top recommendations to help you today
▸ The Process: Overview of the survey process, start to finish
▸ Tools & Services: The vendors you should consider
▸ Step by Step: A guided walk through The Process

4. QUICK WINS

5. QUICK WINS
CREATE RELIABLE ANSWER OPTIONS
▸ Use word scales that you can assign numerical values to
▸ If I ask you how likely you are to apply techniques in this presentation? I would
use the following answer options:
▸ Extremely likely, very likely, moderately likely, slightly likely, not at all likely.
▸ The words that precede “likely” are very deliberate. They equate to 100, 75, 50,
25, 0 for the 5 options, respectively
▸ Any chance you get, use likert scales like this, and use these two resources:
▸ https://www.slideshare.net/bchudoba/surveymonkey-audience-survey-
writing-guide-76542708
▸ https://www.slideshare.net/bchudoba/sample-likertscales

6. QUICK WINS
START WITH THE HYPOTHESIS
▸ Avoid the temptation to start writing questions
▸ Draft your hypothesis first
▸ X% of respondents prefer Product A. The most important reasons are Y, Z
and Q
▸ Respondents aged X to Y and Z to Q had the highest likelihood to purchase
▸ Y% of respondents said a mobile experience was extremely or very
important
▸ You will be surprised at how easy it becomes to ask the right questions (and
with the right types of answer options) when you start by outlining how you will
express your findings

7. QUICK WINS
SAMPLING AND SAMPLE SIZE
▸ Sample size, confidence level, margin of error, statistical significance can be complex topics to understand
▸ If you want some quick and dirty advice:
▸ 400 respondents is the “silver standard”
▸ 1,000 respondents is the “gold standard”
▸ Margin of error is important to understand
▸ “The margin of error is a statistic expressing the amount of random sampling error in a survey's results.”
Wikipedia
▸ Example: You are asking whether people prefer product A or B.
▸ If you survey 400 people and 90% prefer A and 10% prefer B, the answer is pretty clear
▸ If you survey 400 people and 52% prefer A and 48% prefer B, the margin of error of a 400 person
sample is typically around 5%. So your results are inconclusive since you could conceivably reverse if
the experiment was performed again, or more respondents were sampled.
▸ This is a simple resource for calculating sample size and margin of error.

8. THE PROCESS

9. THE PROCESS
OVERVIEW
▸ Hypothesis
▸ Experiment Design
▸ Survey Design
▸ Test/Deploy
▸ Analyze
▸ Iterate
▸ Present
▸ Implement

10. TOOLS &
SERVICES

11. ▸ SurveyMonkey
▸ Google Forms
▸ Qualtrics
TOOLS & SERVICES
SURVEYS

12. TOOLS & SERVICES
COLLABORATION
▸ Create a very simple markup syntax
▸ Example: Use [brackets] to indicate instructions
▸ [Randomize], [Add Other], [Skip to PAGE: X]
▸ Google Docs is excellent for collaboration in survey design

13. TOOLS & SERVICES
SAMPLE: RESPONDENTS
▸ SurveyMonkey Audience
▸ Google Surveys
▸ ResearchNow

14. TOOLS & SERVICES
THIRD PARTY SERVICES
▸ “Solutions”
▸ Boutique Consultants
▸ Full-Service Research Firms

15. STEP BY STEP

16. HYPOTHESIS
STEP BY STEP

17. HYPOTHESIS
OVERVIEW
▸ Let’s get scientific (method)
▸ Start with a hypothesis and use your survey to test it
▸ You probably have a POV on how some of the data will
come out
▸ If you do not have a clue what respondents may say,
creating a “fake hypothesis” will help Experiment Design
▸ You can always iterate with small samples to get a POV if
you don’t have one

18. HYPOTHESIS
DRAFT A CONCLUSION FIRST

19. HYPOTHESIS
DRAFT A CONCLUSION FIRST (CONT’D)

20. HYPOTHESIS
DRAFT A CONCLUSION FIRST (CONT’D)

21. EXPERIMENT DESIGN
STEP BY STEP

22. EXPERIMENT DESIGN
KEY QUESTIONS
▸ Who are you surveying?
▸ How many respondents do you need?
▸ How are you surveying them?
▸ How frequently will you conduct this survey (e.g., once,
monthly, yearly)?
▸ Are there steps in your experiment?

23. EXPERIMENT DESIGN
WHO ARE YOU SURVEYING?
▸ Existing Customers
▸ Active/Passive/Random
▸ External Sample

24. EXPERIMENT DESIGN
HOW MANY RESPONDENTS?
▸ It depends on several things. But here is a simple guide:

25. EXPERIMENT DESIGN
HOW ARE YOU SURVEYING?

26. EXPERIMENT DESIGN
SURVEY FREQUENCY
▸ One time or infrequently
▸ Monthly/Quarterly/Annual: tracking surveys
▸ Constant/Ongoing

27. EXPERIMENT DESIGN
EXPERIMENT STEPS
Being thoughtful about how you may want to iterate, and
building enough time into your process will increase your
odds of success
▸ Is there a “pre-survey”?
▸ Test your questions or validate language
▸ Will there be a follow up?
▸ Inevitably you’ll want to dig deeper on some findings

28. SURVEY DESIGN
STEP BY STEP

29. SURVEY DESIGN
KEY TOPICS
▸ Keep it short
▸ Design questions & answer options for the respondent
▸ Create scales with words, not numbers
▸ Provide mutually exclusive, exhaustive answer options
▸ Put yourself in the respondent’s shoes

30. SURVEY DESIGN
KEEP IT SHORT
▸ Survey questions take ~12-15 seconds (on average) for a
respondent to process and answer
▸ Short multiple choice questions are quick, open ended
take longer
▸ Matrix/Grid questions should be thought of as one
question per answer required (e.g., 5 rows = 5 questions)
▸ Use the timing guidance to balance what you are asking of
your respondents. 25 questions = ~5 minutes

31. SURVEY DESIGN
DESIGN FOR THE RESPONDENT
▸ How many minutes per month do you use a tablet computer?
▸ How frequently do you use tablet computers (e.g., iPad, Android Tablet, Windows Surface)?
▸ Every day
▸ A few times a week
▸ A few times a month
▸ Once a month
▸ A few times a year
▸ Rarely
▸ I don’t use tablet computers

32. SURVEY DESIGN
CREATE SCALES WITH WORDS
▸ Rate your satisfaction with your stay at our hotel from 0-100.
▸ How satisfied were you with your stay at our hotel?
▸ Very satisfied
▸ Moderately satisfied
▸ Neither satisfied nor unsatisfied
▸ Moderately unsatisfied
▸ Very unsatisfied

33. SURVEY DESIGN
MUTUALLY EXCLUSIVE, EXHAUSTIVE ANSWER OPTIONS
▸ How many times per week to you work out?
▸ 0
▸ 1-2
▸ 2-4
▸ 4-7
▸ 7+
▸ In a typical week, how many times do you visit the gym?
▸ None
▸ 1-2 times a week
▸ 3-4 times a week
▸ 5-6 times a week
▸ 7 or more times a week

34. SURVEY DESIGN
EMPATHIZE WITH YOUR RESPONDENTS
▸ Matrix/Grid questions are a pain in the ass to respond to
▸ Employee surveys can get away with more than customer surveys (e.g.,
employee respondents may be required to take surveys)
▸ Assume respondents are on mobile device (stay simple, short, test your
survey on mobile)
▸ Be careful to use nomenclature that may not make sense to respondents
(e.g., avoid acronyms)
▸ Sometimes it helps to tell respondents what you are trying to accomplish
(e.g., we are looking to improve our product)
▸ Thank them for their time, and don’t waste it by not acting on their input

35. TEST & DEPLOY
STEP BY STEP

36. TEST & DEPLOY
THE PRE-SURVEY
▸ A small sample pre-survey can help in survey drafting
▸ Clarifying that your nomenclature makes sense can be
aided by getting some open ended responses
▸ Crowdsourcing or checking competitor sets is helpful too

37. TEST & DEPLOY
TEST LAUNCH
▸ Always launch to a small group first
▸ Make sure they are navigating through your survey as
expected to spot any bugs (e.g., broken logic)
▸ Check the open-ended responses in case you missed
something obvious in your closed-end responses
▸ Get a sense of your drop-off rates to validate sample size
▸ See if there are obvious questions you should add before
full launch

38. TEST & DEPLOY
DEPLOYMENT
▸ Several key mediums to deploy
▸ Email: Straightforward to launch, low response rates and some
sampling bias
▸ Can be random sampled or triggered based on goals
▸ In-product (embedded): More complex to launch (and change),
higher response rates (if short), can avoid some sampling bias
▸ Can surface in-product and open off property (not a product/UX
favorite)
▸ Social/Other: essentially distribution channels for links or embeds

39. ANALYZE
STEP BY STEP

40. ANALYZE
FIND TOOLS WITH GOOD ANALYTICS
▸ You will inevitably export to excel/sheets for some analysis
▸ The better your survey tool’s analytics capabilities, the
fewer times you’ll need to export
▸ Setup things like:
▸ Filters & Crosstabs
▸ Shared views
▸ Custom labeling or variables

41. ITERATE
STEP BY STEP

42. ITERATE
BE AGILE
▸ After you do some analysis, you will have unanswered
questions
▸ Build time into your process to tweak your survey or do
follow ups

43. PRESENT
STEP BY STEP

44. PRESENT
SHOW YOUR FINDINGS
▸ Once you’ve sliced and diced, pull out the relevant
conclusions and put them into a summary
▸ People will always love the raw data, but you need to
synthesize and use all your variables (e.g., demographic,
usage) to tell the story. Tell it with slides or prose
▸ Use verbatims (exact text from respondents) frequently to
give a taste of actual verbiage

45. IMPLEMENT
STEP BY STEP

46. IMPLEMENT
PUT RECOMMENDATIONS TO WORK
▸ The worst thing to do with survey data is to not analyze
and present on it
▸ The second worst thing is to not act on it (at least you
learned something in the first)
▸ You surveyed (created the project, went through the
process, asked hundreds/thousands of people for their
time). Act on it.