A Primer on Primary Market Research

4,443 views

Published on

This slide deck covers why primary market research (aka customer development, customer research or customer empathy) is important and necessary, outlines how to organize a successful research program, provides a sampling of common qualitative and quantitative primary market research techniques, and provides an FAQ section on common questions.

Published in: Business, Technology

A Primer on Primary Market Research

  1. 1. © 2015 ConceptSpring Elaine Chen 2015 A Primer on Primary Market Research
  2. 2. Why do research?
  3. 3. People are different. • If you are a potential customer / user: – Your understanding of the problem will be colored by your work on the solution – You will constantly have to fight confirmation bias because it is your idea – Risk of the “Focus group of one” – you may be an extreme outlier • If you are not a potential customer / user: – You cannot develop a good understanding of who the customer / user is via web research. – You can only understand the target customer / user by meeting them, talking to them, observing them in the situation in which the product will be used, and testing MVPs with them and measuring their reaction. 3
  4. 4. A few tenets
  5. 5. It’s not a sales pitch Stay in inquiry mode, not advocacy mode. • You are here to learn. You are not here to sell. • Don’t talk about the product at all during problem research. • Don’t ask leading questions. Let the subject lead.
  6. 6. Keep your eyes and ears open Be a great listener and observer. • Be 100% present when listening. Don’t multitask. • Listen to listen, not to formulate a response • In interviews and observation sessions, pay close attention to non- verbal cues like body language and facial expression. Much more meaningful than the words they use.
  7. 7. Be open minded Be open minded • Embrace what your subject is teaching you especially if it goes against what you believe • If you find yourself thinking “she’s wrong about that”, or working on a rebuttal on why your solution is still great, you have already lost.
  8. 8. Sausage Making
  9. 9. 7 Steps to a successful primary market research program 1. Write a research plan 2. Define recruitment criteria for interviewees 3. Develop a recruitment questionnaire 4. Develop supporting content (e.g. discussion guide, landing page, on-line survey, etc) 5. Recruit subjects 6. Run the research program 7. Digest results, next steps
  10. 10. Components of a research plan • Goals / objectives • Description of research technique • Recruitment questionnaire • Recruitment strategy • Paperwork / incentives (NDA? Photo/Video release form? Check?) • Equipment required (AV, laptop, etc) • Researchers name list and roles • Discussion guide if appropriate
  11. 11. 2 classes of research goals • Problem Research – To understand who the buyers and users are – To understand the problem statement – To understand the context in which the product will be used – To understand use cases • Solution Research – To understand the usability and utility of the product – To help prioritize the feature set – To understand pricing elasticity – To understand customer satisfaction
  12. 12. Example recruitment questionnaire (Segment: people with sleep issues) • What is your age, gender and profession? (terminate once age/gender quota has been filled for the matching segment) • What is your household income? (terminate if <$xxk) • Are you interested in learning more about your sleep? (terminate if not) • Are you currently encountering sleep problems? (terminate if not) • Are you currently under the care of a doctor for your sleep issues? (terminate if true) • How many nights in an average week are you encountering sleep issues? (select between 2-5) • Please describe the sleep issues you are currently encountering (check all that apply). (cannot fall asleep; multiple awakenings; cannot go back to sleep after awakening; snoring spouse; etc) • Do you currently share your bed? (partner, child, pet) If so: how many nights in an average week is your sleep affected by your bed partner(s)?
  13. 13. Some common techniques
  14. 14. Qualitative versus quantitative • Open ended: you are having a conversation • Anecdotal: you derive meaning from each conversation • Requires excellent listening and observation skills • Best done face to face (or at least by video Skype or by voice call) • Best used for exploratory research where you are trying to get to know a problem or a persona • Qualitative (<50 samples) • Quantitative (>1000 samples) • Close ended: you are administering a questionnaire • Statistical: you derive meaning from looking at aggregate results • Requires systematic analysis skills • Best done via on-line survey (or, in the olden days, via phone survey) • Best used for confirmation research where you are trying to verify / quantify what you think is important
  15. 15. Qualitative versus quantitative • Qualitative (<30 samples) • Contextural interview • Observation / shadowing • Immersion • Longitudinal diary study • Photo essay • Usability benchmark • Focus groups • Landing page tests (small sample size) • … etc • Quantitative (>1000 samples) • Surveys – On line surveys – Conjoint analysis – Pricing studies • Monadic • Multiple monadic • Van Westendorp • … etc – Customer satisfaction: NPS, P/M fit • Web testing – A/B split, Multivariate – Web analytics – … etc
  16. 16. Qualitative, then quantitative The qualitative pass helps you develop your quantitative pass and interpret the result with context and meaning
  17. 17. Example Qualitative Techniques
  18. 18. Contextural interview: Interview subject where product will be used
  19. 19. Structure the interview for consistency, ask open ended q’s, let the subject lead Set up gear Warm up chat “Tell us the story of sleep in your house” House tour Discuss sleep problems Probe expect- ations Show product idea if and only if there’s if time left over and they ask
  20. 20. Observation: Learning things people cannot articulate In this project, we were trying to understand people’s showering habits. We contracted the research to an ethnographic research company, who took shower footage of volunteer subjects in a nudist colony. We were able to measure things like % time people stand facing the showerhead versus the other way, etc. that no one can tell us with words.
  21. 21. Observation: Learning things people will not articulate In this project, we were trying to understand people’s infotainment needs and multitasking behavior in the car. In this example, the subject (a real estate agent) was talking on the phone and scheduling appointments in her book while driving at 60mph (With our coworkers in the back, who were shadowing her for 8h that day )
  22. 22. Immersion: become the user In this project, we were trying to understand the usability of competitive CAD programs. We had our in house CAD experts try out 3 or 4 applications. They spent 3 days on each application, designing the same set of parts and documenting their experience learning and mastering each application.
  23. 23. Longitudinal diary studies: learn things over days/weeks, not over 1-2h
  24. 24. Photo essays: use imagery to capture the context
  25. 25. Usability benchmark: Give users a task, watch them struggle and learn why
  26. 26. Focus groups: not a fan. Dynamics hard to manage, expensive to do right Image credit: msresearchinc.com
  27. 27. Landing page interest/purchase intent tests One way to test vaporware: • Put up a landing page with a call to action and see if anyone clicks • Call to action may lead to: • Email capture (“Notify me when Product X is ready”) • Credit card capture (“Your CC will not be charged until we ship you Product X) • Use a hosted drag-drop design interface with integrated analytics (e.g. unbounce.com or quickmvp.com) to minimize development time • Conversion metrics will validate/invalidate your hypotheses about a whole host of things
  28. 28. Example Quantitative Techniques
  29. 29. First things first: the product concept In an on line survey, you have 2 pages to make your point about your product’s value proposition and its relevance to your survey respondents. For all practical purposes, you are creating an ad. The effectiveness of your survey depends on this ad. Invest in the creative content to make your point. Get a marketer to help do it right.
  30. 30. On line surveys – gauging interest Construct your survey carefully for easy tabulation of results. Best practice is to use 5 point scales, except for specific methodologies that require 10 point scales (e.g. Net promoter score – see slide in customer satisfaction section) There are techniques to “code” open ended responses – read more here. http://www4.uwm.edu/cuir/resources/upload/Planning-Council-qualitative-analysis- handout.pdf
  31. 31. Conjoint analysis • Conjoint analysis measures the value the market places on each feature of your product and predicts the value of any combination of features. • How it works: – Ask questions that force respondents to make trade-offs among features – Determine the value they place on each feature based on the trade-offs they make – Simulate how the market reacts to various feature trade-offs you are considering • Read more at: http://www.pragmaticmarketing.com/resources/conjoint- analysis-101#sthash.dqQGvTpS.dpuf
  32. 32. Pricing research: Monadic – a.k.a. “Intent to Purchase at Price” • Show the product concept, then ask: • How likely would you to be purchase this product at this price? – Not at all likely – Moderately likely – Neutral – Moderately likely – Extremely likely • “Top 2 box” gives estimate of purchase intent
  33. 33. Pricing research: Multiple Monadic • Repeat the Monadic survey with the exact same product concept, but with a different price point • Tabulate top 2 box responses and see if you see a lift in purchase intent as price goes up or down • Note: the product concept in a survey is the key. It’s a 2 page ad where you get to explain your value proposition. Invest in the creative content and the marketing messaging - get help from a marketer to do it right.
  34. 34. Pricing research: Van Westendorp (the Kindle team did this and came up with $99 ) • Show the product concept, then ask these 4 questions with a free response: – At what price would you consider the product to be so expensive that you would not consider buying it? (Too expensive) – At what price would you consider the product to be priced so low that you would feel the quality couldn’t be very good? (Too cheap) – At what price would you consider the product starting to get expensive, so that it is not out of the question, but you would have to give some thought to buying it? (Expensive/High Side) – At what price would you consider the product to be a bargain—a great buy for the money? (Cheap/Good Value) • Chart cumulative frequencies. Intersection = ideal price. • Read more here. http://en.wikipedia.org/wiki/Van_Westendorp's_Price_Sensitivity_Me ter
  35. 35. Customer satisfaction measure: Net Promoter Score (NPS) • Show the product concept, then ask this question on a 10 point scale (1 = not at all likely, 10 = very likely): – “How likely is it that you would recommend [your product] to a friend [or family, or colleague etc]?” • Categorize responses: – Promoters = 9, 10 – your loyal enthusiasts – Passives = 7, 8 – satisfied but unenthusiastic – Detractors = 0-6 – unhappy, can damage brand by word of mouth • Calculate the Net Promoter Score as follows: (NPS) – NPS = % Promoters - %Detractors • Calibrating yourself – 2013 scores: Apple laptop (best computer company) – 76%; Kaiser (best health insurance company) – 36% - find your industry for comparables • Read more here: http://www.netpromoter.com/why-net- promoter/know/
  36. 36. Customer satisfaction measure: Product / Market Fit (P/M Fit) • Show the product concept, then ask this question on a 3 point scale (1 = not at all disappointed, 2 = somewhat disappointed, 3 = very disappointed) – “How would you feel if they could no longer use [the product]?” • If over 40% of responses = 3 (very disappointed) you have achieved Product Market Fit • Read more here: http://www.startup- marketing.com/using-survey-io/
  37. 37. A word of caution: know your statistics and experimental design • The mean of any metric is meaningless without the variance. Learn how to do analysis of variance (ANOVA) properly to interpret your results correctly. Start with the student-t test and build up from there • The best ANOVA effort with the best experimental design falls down with insufficient sample size. – 500 is the absolute minimum – 5000 is the best practice number – 1000 is a good compromise
  38. 38. A/B split testing Image credit: http://www.smashingmagazine.com
  39. 39. Multivariate testing Image credit: http://visualwebsiteoptimizer.com
  40. 40. Web analytics Image credits: emaginewebmarketing.com, http://www.openwebanalytics.com/
  41. 41. The possibilities are endless
  42. 42. Frequently Asked Questions
  43. 43. Is it a good idea to start an interview with an “on a scale of 1-10” question? • No. To get the most learning out of an interview, use open ended questions to get open ended responses. • Closed end questions requiring yes-no answers or multiple-choice answers will cut the conversation short and reduce the potential learnings from the interview. • As a general rule of thumb, face to face conversations should involve open-ended responses, so the subject can tell you what you never would have thought to ask. On-line surveys should invove close-ended responses so you have a good way to tabulate responses.
  44. 44. How do you run a good detailed interview? • Read Talking to Humans by Giff Constable – free PDF download (also available on Amazon) • Use words like this to run the interview – “Tell me about the last time…” – “Tell me the story of…” – “You said X. Tell me more…” – “Why?” – “Why not?” • Go in pairs so one person can focus 100% on the person while the other person take notes.
  45. 45. How do you build/use a discussion guide for a detailed interview? • Think about what you want to have learned at the end of the conversation • Back out the topics and questions to ask and write them down so they fit on one page. 3-5 topics will do. • Memorize it. Then don’t bring it. • At the interview, use open ended questions and active listening to let the subject lead while guiding them through any content they didn’t cover on their own.
  46. 46. What if the interview subject simply won’t talk? • First, check your technique. Is everybody not talking to you or is it just one or two people? • If it is just one or two people, perhaps they are shy or naturally reticent. Spend a little more time up front breaking the ice and building rapport. Give them a little more time to respond. Avoid filling in all their silences with words. Embrace pauses. • If it’s everybody you need to check your technique. Role play with a friend then get some feedback. • Sometimes if the subject is otherwise talkative but you can’t keep them talking about the subject matter it may mean they don’t have any pain points in that area. That is important data you should take into account.
  47. 47. How do you recruit subjects? • If B2C and you and your friends can be buyers/users: – Directly recruit friends of friends (so they don’t know you) – Post the request on your social networks and your friends’ social networks – Where applicable, put up signage with a landing page URL at the watering holes where your target users hang out (e.g. at a preschool if recruiting parents of young children) • If B2B or you/your friends aren’t buyers/users: – Reach out into your network to see if you know someone who knows someone – Use LinkedIn to find suspects (and how you are connected) – Tap into alumni networks of your affiliated universities • You can also use a research service who can recruit subjects for you (works better for B2C). This costs money. • Stopping people in the mall or putting up flyers in the local library is generally not a great way to recruit subjects.
  48. 48. How do you get a company to give you access to its people for a B2B project? • #1 requirement is to have contact with a high ranking person who can get you time with others in the organization. • First way is via old fashioned networking straight to the high ranking person • Second way is via networking with someone else in that organization who may be able to introduce you to a next person, then a next person. It may take 4-5 introductions to get to the right person. • In some cases you may have to violate good technique in discovery research and share what you are working on to get access. Do what you can with what access you have.
  49. 49. How do you gauge purchase intent in an interview? • What you should NOT do is ask them “Would you buy it?” especially not “Would you buy it for $xxx?” • What you can do is ask them what they have done to solve the problem you are talking about. For example, did they spend $$ in the past 12 months to buy some product or service that solves some aspect of this problem? That is a good indication that there is a real pain that they will pay money for. • You should NEVER do pricing research in person. Always do it via one of the standard survey vehicles, quantitatively. See PMR primer for details.
  50. 50. How many interviews do you need to build a good persona? • If you don’t know what a persona is: Read “Flat Stanley doesn’t live here” by Jennifer Doctor. • If you don’t have a good first guess: Cast a wide net and have many short (15 minute) conversations with many more people (30-50) to help you come up with a tighter target. • Once you have a tight target: Interview 20-30 for each unique persona • For B2B you will need to interview many people with the same role per company as you will be able to get into many fewer companies than 20-30. 51
  51. 51. What if you keep hearing the same thing over and over? • Congratulations! You have found your persona and each conversation is validating the knowledge you have built. • If you have more than 10 or 15 interviews, you can stop now and build your persona. • If you have only 2 or 3 down, don’t stop. This can be sampling error. 52
  52. 52. What if every conversation with an interview subject is a whole new conversation? • What this means is that your targeting isn’t tight enough. This often happens in the beginning of trying to identify the persona. • Step back and ask yourself how else you can sub- segment your target market to tighten it up. See if you can have lots of short conversations with lots more people to help you sub-segment. • You will probably end up having several tighter targets, each sharing more nuanced attributes with each other. Now you can recruit a few people per target and look for the sweet spot. 53
  53. 53. @chenelaine blog.conceptspring.com Thank you

×