1.
© 2015 ConceptSpring
Elaine Chen
2015
A Primer on
Primary Market Research

2.
Why do research?

3.
People are different.
• If you are a potential customer / user:
– Your understanding of the problem will be colored by your work
on the solution
– You will constantly have to fight confirmation bias because it is
your idea
– Risk of the “Focus group of one” – you may be an extreme
outlier
• If you are not a potential customer / user:
– You cannot develop a good understanding of who the customer /
user is via web research.
– You can only understand the target customer / user by meeting
them, talking to them, observing them in the situation in which
the product will be used, and testing MVPs with them and
measuring their reaction.
3

4.
A few tenets

5.
It’s not a sales pitch
Stay in inquiry mode,
not advocacy mode.
• You are here to learn.
You are not here to
sell.
• Don’t talk about the
product at all during
problem research.
• Don’t ask leading
questions. Let the
subject lead.

6.
Keep your eyes and ears open
Be a great listener and
observer.
• Be 100% present when
listening. Don’t multitask.
• Listen to listen, not to
formulate a response
• In interviews and
observation sessions, pay
close attention to non-
verbal cues like body
language and facial
expression. Much more
meaningful than the
words they use.

7.
Be open minded
Be open minded
• Embrace what your
subject is teaching
you especially if it
goes against what you
believe
• If you find yourself
thinking “she’s wrong
about that”, or
working on a rebuttal
on why your solution
is still great, you have
already lost.

8.
Sausage Making

9.
7 Steps to a successful primary market
research program
1. Write a research plan
2. Define recruitment criteria for interviewees
3. Develop a recruitment questionnaire
4. Develop supporting content (e.g. discussion guide,
landing page, on-line survey, etc)
5. Recruit subjects
6. Run the research program
7. Digest results, next steps

10.
Components of a research plan
• Goals / objectives
• Description of research technique
• Recruitment questionnaire
• Recruitment strategy
• Paperwork / incentives (NDA? Photo/Video
release form? Check?)
• Equipment required (AV, laptop, etc)
• Researchers name list and roles
• Discussion guide if appropriate

11.
2 classes of research goals
• Problem Research
– To understand who the buyers and users are
– To understand the problem statement
– To understand the context in which the product will be
used
– To understand use cases
• Solution Research
– To understand the usability and utility of the product
– To help prioritize the feature set
– To understand pricing elasticity
– To understand customer satisfaction

12.
Example recruitment questionnaire
(Segment: people with sleep issues)
• What is your age, gender and profession? (terminate once age/gender
quota has been filled for the matching segment)
• What is your household income? (terminate if <$xxk)
• Are you interested in learning more about your sleep? (terminate if not)
• Are you currently encountering sleep problems? (terminate if not)
• Are you currently under the care of a doctor for your sleep issues?
(terminate if true)
• How many nights in an average week are you encountering sleep issues?
(select between 2-5)
• Please describe the sleep issues you are currently encountering (check all
that apply). (cannot fall asleep; multiple awakenings; cannot go back to
sleep after awakening; snoring spouse; etc)
• Do you currently share your bed? (partner, child, pet) If so: how many
nights in an average week is your sleep affected by your bed partner(s)?

13.
Some common techniques

14.
Qualitative versus quantitative
• Open ended: you are having
a conversation
• Anecdotal: you derive
meaning from each
conversation
• Requires excellent listening
and observation skills
• Best done face to face (or at
least by video Skype or by
voice call)
• Best used for exploratory
research where you are
trying to get to know a
problem or a persona
• Qualitative (<50 samples) • Quantitative (>1000 samples)
• Close ended: you are
administering a questionnaire
• Statistical: you derive meaning
from looking at aggregate
results
• Requires systematic analysis
skills
• Best done via on-line survey
(or, in the olden days, via
phone survey)
• Best used for confirmation
research where you are trying
to verify / quantify what you
think is important

15.
Qualitative versus quantitative
• Qualitative (<30 samples)
• Contextural interview
• Observation / shadowing
• Immersion
• Longitudinal diary study
• Photo essay
• Usability benchmark
• Focus groups
• Landing page tests (small
sample size)
• … etc
• Quantitative (>1000 samples)
• Surveys
– On line surveys
– Conjoint analysis
– Pricing studies
• Monadic
• Multiple monadic
• Van Westendorp
• … etc
– Customer satisfaction: NPS,
P/M fit
• Web testing
– A/B split, Multivariate
– Web analytics
– … etc

16.
Qualitative, then quantitative
The qualitative pass helps you develop your quantitative pass and interpret the result with
context and meaning

17.
Example Qualitative
Techniques

18.
Contextural interview: Interview subject
where product will be used

19.
Structure the interview for consistency, ask
open ended q’s, let the subject lead
Set up
gear
Warm
up chat
“Tell us the story of sleep
in your house”
House
tour
Discuss
sleep
problems
Probe
expect-
ations
Show product
idea if and only
if there’s if time
left over and
they ask

20.
Observation: Learning things people
cannot articulate
In this project, we were trying to understand people’s showering habits. We contracted the
research to an ethnographic research company, who took shower footage of volunteer subjects
in a nudist colony.
We were able to measure things like % time people stand facing the showerhead versus the
other way, etc. that no one can tell us with words.

21.
Observation: Learning things people will not
articulate
In this project, we were trying to understand people’s infotainment needs and multitasking
behavior in the car. In this example, the subject (a real estate agent) was talking on the phone
and scheduling appointments in her book while driving at 60mph (With our coworkers in the
back, who were shadowing her for 8h that day )

22.
Immersion: become the user
In this project, we were trying to understand the usability of competitive CAD programs. We
had our in house CAD experts try out 3 or 4 applications. They spent 3 days on each application,
designing the same set of parts and documenting their experience learning and mastering each
application.

23.
Longitudinal diary studies: learn things
over days/weeks, not over 1-2h

24.
Photo essays: use imagery to capture the
context

25.
Usability benchmark: Give users a task,
watch them struggle and learn why

26.
Focus groups: not a fan. Dynamics hard
to manage, expensive to do right
Image credit: msresearchinc.com

27.
Landing page interest/purchase intent tests
One way to test vaporware:
• Put up a landing page with a call to action
and see if anyone clicks
• Call to action may lead to:
• Email capture (“Notify me when
Product X is ready”)
• Credit card capture (“Your CC will not
be charged until we ship you Product
X)
• Use a hosted drag-drop design interface
with integrated analytics (e.g.
unbounce.com or quickmvp.com) to
minimize development time
• Conversion metrics will validate/invalidate
your hypotheses about a whole host of
things

28.
Example Quantitative
Techniques

29.
First things first: the product concept
In an on line survey, you have 2 pages to make your point about your product’s value
proposition and its relevance to your survey respondents. For all practical purposes, you are
creating an ad. The effectiveness of your survey depends on this ad. Invest in the creative
content to make your point. Get a marketer to help do it right.

30.
On line surveys – gauging interest
Construct your survey carefully for easy tabulation of results.
Best practice is to use 5 point scales, except for specific methodologies that require 10 point
scales (e.g. Net promoter score – see slide in customer satisfaction section)
There are techniques to “code” open ended responses – read more here.
http://www4.uwm.edu/cuir/resources/upload/Planning-Council-qualitative-analysis-
handout.pdf

31.
Conjoint analysis
• Conjoint analysis measures the value the market places
on each feature of your product and predicts the value of
any combination of features.
• How it works:
– Ask questions that force respondents to make trade-offs among
features
– Determine the value they place on each feature based on the
trade-offs they make
– Simulate how the market reacts to various feature trade-offs you
are considering
• Read more at:
http://www.pragmaticmarketing.com/resources/conjoint-
analysis-101#sthash.dqQGvTpS.dpuf

32.
Pricing research: Monadic – a.k.a. “Intent
to Purchase at Price”
• Show the product concept, then ask:
• How likely would you to be purchase this product at
this price?
– Not at all likely
– Moderately likely
– Neutral
– Moderately likely
– Extremely likely
• “Top 2 box” gives estimate of purchase intent

33.
Pricing research: Multiple Monadic
• Repeat the Monadic survey with the exact same
product concept, but with a different price point
• Tabulate top 2 box responses and see if you see a lift
in purchase intent as price goes up or down
• Note: the product concept in a survey is the key. It’s a
2 page ad where you get to explain your value
proposition. Invest in the creative content and the
marketing messaging - get help from a marketer to do
it right.

34.
Pricing research: Van Westendorp (the
Kindle team did this and came up with $99 )
• Show the product concept, then ask these 4 questions with a free
response:
– At what price would you consider the product to be so expensive that
you would not consider buying it? (Too expensive)
– At what price would you consider the product to be priced so low that
you would feel the quality couldn’t be very good? (Too cheap)
– At what price would you consider the product starting to get expensive,
so that it is not out of the question, but you would have to give some
thought to buying it? (Expensive/High Side)
– At what price would you consider the product to be a bargain—a great
buy for the money? (Cheap/Good Value)
• Chart cumulative frequencies. Intersection = ideal price.
• Read more here.
http://en.wikipedia.org/wiki/Van_Westendorp's_Price_Sensitivity_Me
ter

35.
Customer satisfaction measure:
Net Promoter Score (NPS)
• Show the product concept, then ask this question on a 10 point
scale (1 = not at all likely, 10 = very likely):
– “How likely is it that you would recommend [your product] to a friend [or
family, or colleague etc]?”
• Categorize responses:
– Promoters = 9, 10 – your loyal enthusiasts
– Passives = 7, 8 – satisfied but unenthusiastic
– Detractors = 0-6 – unhappy, can damage brand by word of mouth
• Calculate the Net Promoter Score as follows: (NPS)
– NPS = % Promoters - %Detractors
• Calibrating yourself
– 2013 scores: Apple laptop (best computer company) – 76%; Kaiser
(best health insurance company) – 36% - find your industry for
comparables
• Read more here: http://www.netpromoter.com/why-net-
promoter/know/

36.
Customer satisfaction measure:
Product / Market Fit (P/M Fit)
• Show the product concept, then ask this question on a 3
point scale (1 = not at all disappointed, 2 = somewhat
disappointed, 3 = very disappointed)
– “How would you feel if they could no longer use [the product]?”
• If over 40% of responses = 3 (very disappointed) you
have achieved Product Market Fit
• Read more here: http://www.startup-
marketing.com/using-survey-io/

37.
A word of caution: know your statistics
and experimental design
• The mean of any metric is meaningless without the
variance. Learn how to do analysis of variance
(ANOVA) properly to interpret your results correctly.
Start with the student-t test and build up from there
• The best ANOVA effort with the best experimental
design falls down with insufficient sample size.
– 500 is the absolute minimum
– 5000 is the best practice number
– 1000 is a good compromise

38.
A/B split testing
Image credit: http://www.smashingmagazine.com

39.
Multivariate testing
Image credit: http://visualwebsiteoptimizer.com

40.
Web analytics
Image credits: emaginewebmarketing.com, http://www.openwebanalytics.com/

41.
The possibilities are endless

42.
Frequently Asked Questions

43.
Is it a good idea to start an interview with
an “on a scale of 1-10” question?
• No. To get the most learning out of an interview, use
open ended questions to get open ended responses.
• Closed end questions requiring yes-no answers or
multiple-choice answers will cut the conversation short
and reduce the potential learnings from the interview.
• As a general rule of thumb, face to face conversations
should involve open-ended responses, so the subject
can tell you what you never would have thought to
ask. On-line surveys should invove close-ended
responses so you have a good way to tabulate
responses.

44.
How do you run a good detailed interview?
• Read Talking to Humans by Giff Constable – free PDF
download (also available on Amazon)
• Use words like this to run the interview
– “Tell me about the last time…”
– “Tell me the story of…”
– “You said X. Tell me more…”
– “Why?”
– “Why not?”
• Go in pairs so one person can focus 100% on the
person while the other person take notes.

45.
How do you build/use a discussion guide for a
detailed interview?
• Think about what you want to have learned at the end
of the conversation
• Back out the topics and questions to ask and write
them down so they fit on one page. 3-5 topics will do.
• Memorize it. Then don’t bring it.
• At the interview, use open ended questions and active
listening to let the subject lead while guiding them
through any content they didn’t cover on their own.

46.
What if the interview subject simply won’t
talk?
• First, check your technique. Is everybody not talking to you
or is it just one or two people?
• If it is just one or two people, perhaps they are shy or
naturally reticent. Spend a little more time up front breaking
the ice and building rapport. Give them a little more time to
respond. Avoid filling in all their silences with words.
Embrace pauses.
• If it’s everybody you need to check your technique. Role
play with a friend then get some feedback.
• Sometimes if the subject is otherwise talkative but you can’t
keep them talking about the subject matter it may mean
they don’t have any pain points in that area. That is
important data you should take into account.

47.
How do you recruit subjects?
• If B2C and you and your friends can be buyers/users:
– Directly recruit friends of friends (so they don’t know you)
– Post the request on your social networks and your friends’
social networks
– Where applicable, put up signage with a landing page URL at
the watering holes where your target users hang out (e.g. at a
preschool if recruiting parents of young children)
• If B2B or you/your friends aren’t buyers/users:
– Reach out into your network to see if you know someone who
knows someone
– Use LinkedIn to find suspects (and how you are connected)
– Tap into alumni networks of your affiliated universities
• You can also use a research service who can recruit subjects for
you (works better for B2C). This costs money.
• Stopping people in the mall or putting up flyers in the local library
is generally not a great way to recruit subjects.

48.
How do you get a company to give you
access to its people for a B2B project?
• #1 requirement is to have contact with a high ranking
person who can get you time with others in the
organization.
• First way is via old fashioned networking straight to the high
ranking person
• Second way is via networking with someone else in that
organization who may be able to introduce you to a next
person, then a next person. It may take 4-5 introductions to
get to the right person.
• In some cases you may have to violate good technique in
discovery research and share what you are working on to
get access. Do what you can with what access you have.

49.
How do you gauge purchase intent in an
interview?
• What you should NOT do is ask them “Would you buy
it?” especially not “Would you buy it for $xxx?”
• What you can do is ask them what they have done to
solve the problem you are talking about. For example,
did they spend $$ in the past 12 months to buy some
product or service that solves some aspect of this
problem? That is a good indication that there is a real
pain that they will pay money for.
• You should NEVER do pricing research in person.
Always do it via one of the standard survey vehicles,
quantitatively. See PMR primer for details.

50.
How many interviews do you need to
build a good persona?
• If you don’t know what a persona is: Read “Flat
Stanley doesn’t live here” by Jennifer Doctor.
• If you don’t have a good first guess: Cast a wide net
and have many short (15 minute) conversations with
many more people (30-50) to help you come up with a
tighter target.
• Once you have a tight target: Interview 20-30 for each
unique persona
• For B2B you will need to interview many people with
the same role per company as you will be able to get
into many fewer companies than 20-30.
51

51.
What if you keep hearing the same thing
over and over?
• Congratulations! You have found your persona and
each conversation is validating the knowledge you
have built.
• If you have more than 10 or 15 interviews, you can
stop now and build your persona.
• If you have only 2 or 3 down, don’t stop. This can be
sampling error.
52

52.
What if every conversation with an interview
subject is a whole new conversation?
• What this means is that your targeting isn’t tight
enough. This often happens in the beginning of trying
to identify the persona.
• Step back and ask yourself how else you can sub-
segment your target market to tighten it up. See if you
can have lots of short conversations with lots more
people to help you sub-segment.
• You will probably end up having several tighter targets,
each sharing more nuanced attributes with each other.
Now you can recruit a few people per target and look
for the sweet spot.
53

53.
@chenelaine blog.conceptspring.com
Thank you