APF orlando diy survey workshop 071114 final

www.aperioinsights.com
Revealing Insights for Clear Decisions
July 11, 2014
APF Orlando

1 Revealing Insights for Clear Decisions
Welcome to DIY Survey Research

Mike Courtney
Researcher | Ethnographer | Futurist
Educated and trained in both traditional
research and long range scenario planning.
MMR degree in research, published
author, angel investor.
Data Analysis | Ethnography
Ideation | End User Requirements
New Product Validation | Roadmaps

Do YOU Research?
 Term Research is vague
 Research? Sure! I use Google almost every day.
– Many don’t know if they need it.
 We don’t really need research. What we need is sales and marketing help.
– Our ads aren’t getting the response we expected. We might need to look
for a new ad agency.
– Competition just stole our top 3 customers. We need a better sales
trainer or maybe a new sales contest.
– I already did the research. Both my mom AND dad loved the idea. My
golfing buddies also think its awesome. Now I just need a couple million $
in funding so I can hire someone to do the marketing and sell it.
– Sales haven’t yet taken off, I think I just need better SEO.

Why Do Research?
Three Simple Reasons
 To reduce risks
 To discover opportunities
 To improve business results
One Metric for Success — Answers that Lead to Actions
“After conducting thousands of marketing research studies and asking
hundreds of thousands of questions, the author has come to understand one
thing: There are no bad questions, only irrelevant ones. In other words, the
majority of questions asked are irrelevant. That is, they don't result in
answers that lead to actions.”
The Secret to Great Marketing Research: Ask the Right Questions by Robert J. Kaden

Research has Changed
COSTS
• Even straightforward projects
cost big $
• Simple projects are
inexpensive or free
TIMING
• Took months, hard to predict
completion date
• Often done in weeks if not
days or hours
TOOLS
• Tools were basic – skilled
experts needed
• Tools are powerful and
geared towards non-experts
OUTPUT
• Hard to change final output
once done
• Easier to adjust or change
final output
BACK THEN NOW

Today’s Research Tools are Easy and Powerful
But could they be dangerous?

Safe Surveys –What We Will Cover
 When to use Quant vs. Qual
 Research brief basics– capturing the objectives upfront (Team Exercise)
 Turning objectives into questions – subtle differences that add depth
 Common Scales and other answer options – how to choose, examples, etc
 Common mistakes and traps – rules of thumb (Featuring Dr. Bishop handout)
 Survey question practice - you can do it, we can help (Team Exercise)
– Show and tell – share your questions and get reactions/critiques
 I need how many? – The significance of sample size
 Finding Respondents - Survey Sample Options
 Survey and analysis tools - low cost options
 Final Q & A

Refresher — Qualitative vs. Quantitative
 Qualitative (Qual):
– Discover and understand the “why” or the rationale behind actions
– Explore the range of possible perceptions and beliefs
– Understand the language used and how it impacts understanding
– Helps develop ideas and hypothesis to test
 Quantitative (Quant):
– Measure the percentage who think a certain way
– Project findings onto larger population
– Gather feedback from a larger audience
– Benchmark and periodically re-measure

Research Brief Keeps Everyone on Target
 Background – Relevant background to understand context of the request including the scope of
project & information currently available:
 Existing Research – what data/knowledge do you already have? Primary, secondary, anecdotal
 Decision[s] to be made – What are the key decisions that will be made? What are the primary
reasons to consider conducting this research?:
 Specific objectives/Hypotheses – Are there any theories about what will be learned? What
information will affect the decision[s]? What do you need to prove/find out? Ideally, what will
you know when the research is completed?:
 Additional decision data required – If the research IS successful in gathering the required
information, what other data might ALSO be needed before making the decision[s] in question?:
 Target respondents – What criteria (demographic, product usage and other specific
requirements) must the respondents meet in order to qualify for this study? Please note any
hard quotas or splits:
 Justification – $ implications / ROI / reasons why project is important:
 Deadline – For final results:

Team Exercise: Use a Research Brief to Outline your Objectives
 Team Task
– Complete the research brief and outline the specific survey objectives
that will form the outline of a survey. Pick a survey topic or use below.
 Example Survey Topic
– As an APF member you understand the value a good futurist can bring to
an organization.
– Unfortunately there are still many people in the world who have no idea
what you do and how it might help them.
– Imagine you are working on a project to identify potential NEW clients to
approach in order to educate, work with, etc.
– Your team has been asked to create a survey to understand and measure
WHO may be interested in working with a futurist.

TEAMS DEVELOP AND PRESENT
OBJECTIVES
OBJECTIVE = SOMETHING YOU WANT
TO LEARN/CONFIRM/MEASURE
(TEN MINUTES? DON’T OVERTHINK IT)

12
Watch out for too many or too few
response options
“Capital punishment should be
reintroduced for “serious crimes”
1 = Agree 2 = Disagree
1 = Very, Very Strongly Agree 7 = Slightly Disagree
2 = Very Strongly Agree 8 = Disagree
3 = Strongly Agree 9 = Strongly Disagree
4 = Agree 10 = V. Strongly Disagree
5 = Slightly Agree 11 = V, V Strongly Disagree
6 = Neutral

13
Example: How could
this question be improved?
How old are you?
___ 18-20
___ 20-22
___ 22-30
___ 30 and over

14
Are you satisfied with your marriage
and your job?
__________________________
Example: How could

15
You didn’t think the food was very
good, did you?
_____ Yes _____ No
Example: How could

16
Environmental issues have become
increasingly important when
choosing respectable hotels. Are
environmental considerations an
important factor when deciding on
your choice of hotel accommodation?
____ Yes ____ No
Example: How could

17
What information sources did you
use to locate your restaurant for
today’s meal?
(please tick appropriate spaces)
____ yellow pages
____ Internet
____ word of mouth
Example: How could

Lenovo Survey Example

Lenovo Survey Example (Cont)

Lenovo Survey Example (Cont)
Remember – pick only
FIVE items to rate

SURVEY QUESTION CRASH COURSE:
TIPS, TRICKS AND RULES OF THUMB

Survey Templates Readily Available

Basic Questions Get You Started

Subtle Question Differences Add Depth (PS -don’t use this many)
Management and Fairness
My manager treats all his/her employees fairly.
My manager knows what is going on in my work group.
Favoritism is not an issue in raises or promotions.
My manager is always consistent when administering policies concerning employees.
I am always treated fairly by my manager.
Everybody is treated fairly in this organization.
My manager establishes plans and work objectives with me.
My manager gives me clear instructions.
My manager is available when I need advice.
My manager (immediate supervisor) trusts me.
My manager helps me to improve myself.
My manager takes prompt and fair corrective action on employees who fail to perform their work satisfactorily.
I feel free to talk openly and honestly to my manager.
My manager praises me when I do a good job.
My manager holds regular meetings with my work groups.
My manager is effective in making decisions.
TIP: Build and fine tune questions over time; share question lists with other DIY researchers.

Survey Tips
 Whose opinions matter? Need to ask a sample of respondents who you think might be worth
targeting in the future. Don’t ask mom if you doubt she would be a future customer etc.
 Understand without Influence - Trying to get true perceptions – not trying to influence, sway,
convince etc.
 Bias comes in many forms – question wording, concern for interviewer, those who know you,
those who take pity on you and want to help, fatigue, low category involvement etc.
 Behavior then perception – ask first about what they have done, they what they think about it
 KISS principal – Simple wording is often best.
 Define Timelines - Have you …how much do you…how often do you….ALL require some sort of
timeline.
 Define categories & avoid mental math– What % of your beverages are coca cola products?
Does % mean number of drinks per day? Are they asking about % as in how many ounces of
product or bottles? Cups? Fountain? Should I include bottled water sold by coca cola corp? etc.
 Chocolate Cake Factor - Most real world decisions involve many factors. Need to tell the rest
of the story – or enough to make sense. Don’t forget to ask about the major factors that would
influence behavior.

Basic tips for Choosing Scales
 Don’t recreate the wheel
 Choose question stems carefully
 Reverse word some of the items
 Use the same scales each time data is collected
 Avoid mixing too many scales in one survey
 Format scales for one direction – negative to positive etc.
 Balance Scales – Agreement, quality and satisfaction scales use an equal
number of options on either side (positive vs. negative) of a scale, each with
similar wording.
 Scales vs. Checklists – Sometimes it’s better to use a checklist instead of
having everyone mark the same scale item multiple times.
 https://www.case.edu/president/cir/wordfiles/tips_choosingscales.docx

27
I need how many?
The significance of sample size

Example Survey Flow
 Screeners – Questions that confirm the respondent meets some minimum
criteria
 Category Involvement – Questions that explore the level of activity in the
category
 Stated Needs & Pain Points – What do they like/not like about their current___?
 Concept Introduction – Imagine there is a new __ available. Here’s a description
of it...
 5 Key Questions – Appeal, unique/different, satisfies a need, believability, value
for money/Interested in learning more etc.
 Suggestions – How would you improve or change it?
 Demographics – Age, gender, income, occupation, education, etc.
 Bizographics – Company size, industry sectors, locations, etc.
 Psychographics – Personality, values, attitudes, interests, or lifestyles

Common Questionnaire Mistakes – Dr. Bishop’s List
Issue Definition Example
Ambiguous terms Terms that can mean more than one thing. Do you believe in freedom for all?
Anchoring
The response is biased by the last information
presented to the respondent, usually dealing
with numerical information.
The U.S. has over 39 million without health insurance. How many uninsured
does Germany have? versus
The U.S. has over 200 million with health insurance. How many insured
does Germany have?
Biased wording
Terms that indicate the attitude of the
questioner and influence the respondent to
answer the same way.
Do you agree that worthless people should be excluded from welfare
payments?
Complex questions
Items that are too difficult to answer because
the question is too complicated.
For each of the following courses, indicate whether the course should be
offered, when it should be offered, and by whom it should be offered. In the
last column, show whether the course should be offered by adjuncts or full-
time professors and whether the courses should include only undergraduates,
graduates, or both.
Double-barreled questions Asking two questions in one item.
Do you think the President should cut defense spending and reduce the
national debt?
Incomplete list
Response categories that do not contain all
possible responses.
Protestant, Baptist, Catholic, Jewish
Jargon Terms that are unique to a particular field. All health education curricula should contain courses on RVRBS.
Lack of a filter
Questions that apply to only a subgroup without
filtering.
What was your major in college?
Lack of knowledge
The respondent does not have the information
requested.
When was the last time a President visited Japan?
Mutually inclusive categories
Response categories that overlap and contain
the same material.
Protestant, Baptist, Catholic, Jewish
Negative questions Items that contain negations. Do you agree that the U.S. should not bomb the Serbs in Bosnia?
Spurious exactitude
Items that ask respondents to make distinctions
that are too fine.
Rank the following 20 items how important they are for the future.

Survey question practice - you can do it, we can help (Team
Exercise)
TEAM TASK
–SELECT 2 SPECIFIC SURVEY OBJECTIVES AND
CREATE 2-3 SURVEY QUESTIONS FOR EACH.
–USE OBJECTIVES FROM LIST PROVIDED OR
YOU MAY USE YOUR OWN OBJECTIVES.
–QUESTION EXAMPLES FROM EACH TEAM
WILL BE PRESENTED AND CRITIQUED.

Examples of objectives: (see handout)
 Screening Criteria: Understanding who should be surveyed/not surveyed. Will you target any particular industry, sector
or organization type? Your list of emails to survey is a big hot mess, should anyone be screened out?
 Targeting/Sampling: Which job roles or titles are most likely to be relevant to this study? Who would normally deal with
long term strategy and scenarios? Who would benefit and/or be a potential champion for the process?
 General Background - What background or previous experience will the respondents have? What about their
background would be useful to understand?
 Awareness of Futurists and Foresight. Do these organizations worry about the future? How much?
 History of Involvement - Experience or history with doing this type of process? Do these respondents have any
knowledge of, or experience working with futurists?
 Current State/Situation - Rate of change being experienced by org/industry now. Key issues they are dealing with that
create disruption/risk. Does org experience much change? Where are the major sources of change coming from? Do
different rates of change impact interest in working with futurists?
 Outsourcing attitude – how often do they hire an outside group for help of any kind? How they view Futurists vs
working with some other type of consultant/agency Do they often (ever?) use outside help/consultants?
 Time Horizons - What time horizons do these organizations typically deal with? What does short vs long term typically
mean to their organization or industry?
 Understanding/Common Lexicon - Do they understand the same terms, acronyms and concepts you do? Are you sure
you are speaking the same language?
 Internal Capabilities/Readiness - What parts of the process would they most like help with vs feel confident to do
themselves? Is their organization ready to take on a project that focuses on the future?
 Attitudes/Perceptions - Does their mindset or attitude towards your process matter? What would success look like and
how would they measure results of a project?
 Reaction to the Concept – Once the basics of the process is explained, what are their reactions to it? Do they
understand what is possible, services offered, value potential, etc.?
 Sources of learning/knowledge – How do they keep up to date on new methods? Where do they look to learn about
new topics or approaches to organizational issues?
 Alternatives - If they wanted to think about the future, where would they look for help? If they didn’t hire a futurist to
help think about the future, who WOULD they likely hire?

TEAMS DEVELOP AND PRESENT
SURVEY QUESTIONS
(15 MINUTES?)

DIY RESEARCH TOOLS

Rapid Rise of DIY Research Tools
Tools for Easier Data Capture and Analysis
 Quant survey and qualitative interviewing software
 Analysis tools for the visual display of data
 Tools that offer new hybrid methodologies
Features Anyone Can Use
 Survey tools with more analysis features—built in cross tabs
 Automate common tasks—graph output to PowerPoint and stat testing
 Templates, canned reports and standard graphics
 Tools marketed directly to non-research end users

Finding Respondents - Survey Sample Options
Competition among traditional sample sources continues to reduce price points. Some offer DIY “self-serve”
sample tools for easy access. Non-traditional sample sources such as Google and Mechanical Turk can appear
cheap but are not always nationally representative, or the best value depending on the survey length. Respondent
quality is often and unexpectedly uneven for both sources and data cleaning is critical to meaningful results.
FULL FEATURE, FULL PRICE
• GreenBook.org – List of sample/panel
companies such as:
• Survey Sampling
• Cint
• ResearchNow
• Wide-open, nationally representative
sample is currently running $3-6 per
complete. Lower incidence studies can
still easily top $10-14 per complete.
SCRAPPY ALTERNATIVE
• Mechanical Turk
• Google Sample

Quantitative Survey Tools
Many companies have, but don’t fully or frequently use their expensive software
licenses. If you really need the full featured versions consider partnering with or
consolidating your survey software activity among several companies.
• Qualtrics – Lots of features, popular
with academic crowd ($5-10K+/year)
• ConfirmIt – Industrial strength feature
set, popular with larger research shops
($10K+/year)
• SawTooth Software – The standard
for classic conjoint and MaxDiff
($3-10K 1x fee)
SCRAPPY ALTERNATIVE
• SurveyGizmo – Free version (8
question types & 350 responses) or
Basic version $19/month
(28+ question types & unlimited
responses)
• SawTooth Software – Make friends
(and a deal) with someone who owns
a license

Quantitative Analysis Tools
Many survey tools now have built-in analytics capabilities that can handle the basics.
If you really need the full featured versions consider partnering with or consolidating
your survey software activity among several companies.
• SPSS – One of the original standard
analytic tools. Lots of statistical
features, poor graphics. Now IBM
owned and 5-10x the old prices
($5-40K+)
• MarketSight – Super easy to use online
analytics tool. Can create PPT charts in
batches from crosstabs
($99/year academic license, $1,600
standard license, free 30 day trial)
SCRAPPY ALTERNATIVE
• R Project for Statistical Computing –
Popular open source stats package,
tricky to install with basic graphics
(FREE)
• Excel – Easy for basic crosstabs and
many analysis templates exist for
FREE. Most of us already have an old
copy laying around right? Or use
Google docs

Visualization Tools
Information overload means effective researchers need to have valuable data AND be
capable of using it to tell a clear and compelling story. Visualization tools are useful
for exploring data in the analytics phase as well as turning key findings into easily
understood and eye catching output.
• Tableau Software – Drag and drop
visual analytics tool. Capable of
handling various data types and
creating great visual outputs with a
little practice ($999/year hosted
edition, $1,999 pro version)
SCRAPPY ALTERNATIVE
• RAW –Newly announced online
visualization tool with 7 graphic styles
and ability to export finished result
(FREE)
• Tableau Public – “Lite” version of drag
and drop visual analytics tool, main
caveat is your output must be publicly
shared

Consider Methods Like MaxDiff
Requires either
SawTooth OR Survey
Analytics survey
software

When Should I Use All This?
 Trend is towards conducting smaller, more frequent studies that
provide a constant stream of feedback instead of a few large studies
 Common Research Touch Points Include:
– Idea or concept stage, before any development has begun
– MVP or proto stage, testing a working version of the core offer
– Development milestones as features are added/perfected
– Friendly user stage – full product but pre-full public release
– Post launch – periodic testing to track how expectations are
evolving as users move past novelty and into routine use phases
– Timing also dependent on pace of change and response times in
the industry sector

Keeping Score
 Don’t look at just one score
 Averages vs. Segmentation

Q & A

www.aperioinsights.com
Revealing Insights for Clear Decisions
For more information, contact:
Mike Courtney
m: 469-363-0109
e: mike@aperioinsights.com

Links! Survey basics and Tips
https://explorable.com/course/the-survey-guide
http://www.slideshare.net/jtneill/survey-design-workshop?qid=307afda6-63e0-45ce-
a197-3fda3491513f&v=default&b=&from_search=2
http://blog.askyourtargetmarket.com/2011/04/survey-design-tips-bad-questions-
made.html
http://www.marketingprofs.com/short-articles/2109/three-errors-that-lead-to-bad-
survey-questions
http://www.focusgrouptips.com/marketing-research-surveys.html
http://www.focusgrouptips.com/qualitative-research.html
http://www.b2binternational.com/publications/questionnaire-design/
http://www.qualtrics.com/blog/writing-survey-questions/
http://www.researchplan.com/blog/?p=51
http://www.surveysystem.com/sscalc.htm
http://www.slideshare.net/horatjitra/designing-questionnaire-and-survey-research

Links! Scales and Sample Frame basics
Scales
http://www.360degreefeedback.net/media/ResponseScales.pdf
http://www.uwex.edu/ces/4h/evaluation/documents/Wordingforratingscales.pdf
Sample frame PPT’s
http://www.qualtrics.com/blog/determining-sample-size/
http://www.slideshare.net/heenadhawan86/sampling-bigslides
http://www.slideshare.net/hafizahhajimia/research-method-sampling?qid=d8d874cd-
dae6-4b76-bcde-51ffe3a168f6&v=default&b=&from_search=6
http://www.slideshare.net/faizan-anjum/project-of-sampling-types?qid=d8d874cd-
dae6-4b76-bcde-51ffe3a168f6&v=default&b=&from_search=17
http://www.slideshare.net/nirbhay4shalu/sampling-14703106
Tool mentioned during APF session
http://www.sensemaker-suite.com/smsite/index.gsp

APF orlando diy survey workshop 071114 final

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to APF orlando diy survey workshop 071114 final

Similar to APF orlando diy survey workshop 071114 final (20)

More from Mike Courtney

More from Mike Courtney (12)

Recently uploaded

Recently uploaded (20)

APF orlando diy survey workshop 071114 final