SlideShare a Scribd company logo
Jobs That Scale
Holly Fake, Betsy Bland, and Michele Hansen
RECAP
JOBS TO BE DONE RECAP
JTBD Meet-Up
1: Overview
User
Interviews
STEP 1
Create Job
Statements
STEP 02
Do our
findings scale?
STEP 03
RESEARCH
QUESTIONS
Why & How  Qualitative
Methods
What  Quantitative
Methods
RESEARCH
QUESTIONS
Why do people buy our
service?
How do people shop for [ ].
TO WHAT EXTENT DO THESE “JOBS” SCALE?
SURVEYS
WHAT ARE SURVEYS GOOD FOR?
WHAT ARE
THEY GOOD
FOR?
SURVEYSOPINIONS BELIEFS
VALUES
ATTITUDES/
PERCEPTIONS
BEHAVIORS FACTUAL ITEMS
WHAT ARE
THEY GOOD
FOR?
SURVEYSPATTERNS FREQUENCY
EXPECTATIONSEASE OF USE
PRIORITIES SATISFACTION
BUT WAIT…
THERE’S MORE!
QUANTITATIVE
PEOPLE LIKE
THEM!
SURVEYS
SUPPORT
COLLABORATION!
SURVEYS
*MULTIPLE PEOPLE CAN WORK ON A SURVEY
WITHOUT AFFECTING THE VALIDITY OR
RELIABILITY OF THE RESULTS!
PROVIDE
BASELINES
OVER TIME
SURVEYS
SECURE
ANONYMITY &
CONFIDENTIALITY
SURVEYS
ALLOW
RESEARCHERS
TO ASK A LOT OF
QUESTIONS!
SURVEYS
CAUTION!
THE BAD…
SURVEYS AREN’T
SO GOOD AT…
*Unless designed otherwise
MEASUREMENT
OVER TIME
POINT IN TIME
*Unless designed otherwise
SURVEYS AREN’T
SO GOOD AT…
HONEST
RESPONSES
(aspirational)
SURVEYS AREN’T
SO GOOD AT…
HONEST
RESPONSES
(social bias)
SURVEYS AREN’T
SO GOOD AT…
CAPTURING THE
RANGE OF
HUMAN
EMOTIONS
SURVEYS AREN’T
SO GOOD AT…
ADDRESSING
SUBJECTIVITY
We will cover online surveys for this session.
POTENTIAL
ERRORS
Coverage Error
Non-
Response
Error
Sampling Error
Measurement Error
CRAFTING SURVEY
QUESTIONS
ARE THESE SURVEY QUESTIONS
GOOD OR BAAAAAAAAAAAD ?
To what extent do you agree or disagree that
there are enough
arts in your local
area?
[1- strongly disagree- 5 strongly agree]
in your local area?
[1- strongly disagree- 5 strongly agree]
Assumes prior
knowledge or
understanding
Define what you
mean by “local
area”
Please agree or disagree with the following statement:
Cars should be faster
and safer.
[1- strongly disagree- 5 strongly agree]
Double-Barreled
Question
Cars should be
faster and safer.
Split the question
into two
How short was
Napoleon?
Leading Bias
How short was
Napoleon?
Reword the
question
Describe Napoleon’s
height
How long have you
worked in this job?
-1 to 2 years
-2 to 5 years
-5 years or more
Inadequate
response
options
-1 to 2 years
-2 to 5 years
-5 years or more
Inadequate
response
options
-less than a year
-1 to 2 years
-2 to 5 years
-5 or more years
DoyouagreeordisagreethatKFC’sDoubleDown]
Do you agree or disagree that
KFC’s Double Down
Sandwich is not a
healthy option for
our growing
children? [Agree/ Disagree/ Don’t know]
DoyouagreeordisagreethatKFC’sDoubleDown]
Negative
question
wording & bias
Do you agree or disagree that
KFC’s Double Down
Sandwich is not a
healthy option for
our growing
children? [Agree/ Disagree/ Don’t know]
Reword the
question
WHAT IS WRONG WITH THIS QUESTION?
SOURCE: https://measuringu.com/left-side-bias/
Knowing that the population of the
U.S. is 316 million, what is the
population of Canada?
DoyouagreeordisagreethatKFC’sDoubleDown]
ANCHORING
Knowing that the
population of the
U.S. is 316 million,
what is the
population of
Canada?
DoyouagreeordisagreethatKFC’sDoubleDown]
REWORD THE
QUESTION
Knowing that the
population of the
U.S. is 316 million,
what is the
population of
Canada?
DoyouagreeordisagreethatKFC’sDoubleDown]
What is the
population of
Canada?
36.29 Million
IMPACT OF STRONG LANGUAGE
SOURCE: https://measuringu.com/extreme-items/
I think this is one of my all-time favorite websites.
I never want to use the website again.
IMPACT OF STRONG LANGUAGE
SOURCE: https://measuringu.com/extreme-items/
• User Experience Professional
Association Website (N=60)
• 95% confidence level
ASKING
UNNECESSARY
QUESTIONS
TOO MANY
OPEN-ENDED
QUESTIONS
OTHER POTENTIAL
SURVEY ERRORS
SURVEY FORMAT
WHICH ONE IS BIASED?
SOURCE: https://measuringu.com/left-side-bias/
WHICH ONE IS BIASED?
SOURCE: https://measuringu.com/left-side-bias/
FIVE OR SEVEN?
SOURCE: https://measuringu.com/scale-points/
Finstad, 2010
SEVEN! (95% Confidence)
SOURCE: (Nunnally 1978).
https://measuringu.com/scale-points/
2.5% of 858 responses
users provided
responses between the
points.,
0% of 840 responses
users provided
responses between the
points.
Finstad, 2010
WHAT’S
WRONG WITH
THESE
QUESTIONS?
What was your total family income
from all sources in 2013?
Do you support or oppose making
immigration into the United States
difficult?
What is your favorite sport to watch on
television?
(If employed) Does your employer
provide you with health insurance?
CASE STUDY
MARKET
PASS
User
Interviews
Job
Statements
Design
Parameters for
Market Pass
Evaluating
Impact
Keeping track of my stock ideas
Investing advice based on a potential market theme (e.g., Artificial Intelligence,
eSports, Personalized Medicine)
Compare my holdings to a recommended investment allocation
Figuring out the best time to buy a stock
SURVEY
• Market Pass Members
(N=253)
• 6.6% response rate
• Satisfaction &
Importance Survey
• Usage of site features
• Demographics
• Misc. Questions Based
on findings from
interviews/Operating
Hypotheses
Evaluating Member Satisfaction and Importance of Job
Statements
SIZE IMPACT
OF WHAT WE DID
Keep track of
my stock ideas
Keep track of
my stock ideas
Invest in a
market theme
SIZE IMPACT
OF WHAT WE DIDN’T
DO
Comparing my
holdings to a
recommended
investment
allocation
Understand
how much
money I am
getting from
my investment
each month
Figuring out
the best time
to buy a stock
MARKET PASS
THE GOOD
1. Keeping track of my
stock ideas
2. Investing advice based
on a potential market
theme (e.g., Artificial
Intelligence, eSports, etc.)
8. Finding stocks I believe
in
THE OPPORTUNITY
5. Understanding how
much money I am getting
from my investment each
month
6. Figuring out the best
time to buy a stock
9. Determining the best
time to sell a stock
10. Learning how to invest
NEXT STEPS
• Evaluate Scalability of Findings
• Determine Future Priorities
• Identify Surprises and Outliers
• Rinse/Repeat
• Frequency vs. Pain
• Focused Job Interviews
• More iterations for addressing those jobs
YOUR NEXT STEPS
What jobs do you think would scale? Which ones would you want
to survey?
How could you use this in your work?
Reflection Questions
What biases will you need to control for in your questions?
How can you minimize survey error?

More Related Content

What's hot

Getting started with Job to be Done research
Getting started with Job to be Done researchGetting started with Job to be Done research
Getting started with Job to be Done research
Firmhouse
 
Jobs To Be Done Framework
Jobs To Be Done FrameworkJobs To Be Done Framework
Jobs To Be Done Framework
Rupali Shah
 
The 10-Slide Pitch Deck Structure
The 10-Slide Pitch Deck StructureThe 10-Slide Pitch Deck Structure
The 10-Slide Pitch Deck Structure
Ethos3
 
Start Up Business Proposal Powerpoint Presentation Slides
Start Up Business Proposal Powerpoint Presentation SlidesStart Up Business Proposal Powerpoint Presentation Slides
Start Up Business Proposal Powerpoint Presentation Slides
SlideTeam
 
Company Summary of Business a Plan PowerPoint Presentation Slides
Company Summary of Business a Plan PowerPoint Presentation SlidesCompany Summary of Business a Plan PowerPoint Presentation Slides
Company Summary of Business a Plan PowerPoint Presentation Slides
SlideTeam
 
Top 20 Twitter Influencers In Singapore
Top 20 Twitter Influencers In SingaporeTop 20 Twitter Influencers In Singapore
Top 20 Twitter Influencers In Singapore
Happy Marketer
 
Go-to-market strategy, positioning and viral growth
Go-to-market strategy, positioning and viral growthGo-to-market strategy, positioning and viral growth
Go-to-market strategy, positioning and viral growth
Lisa Enckell
 
Sales Decks for Founders - Founding Sales - December 2015
Sales Decks for Founders - Founding Sales - December 2015 Sales Decks for Founders - Founding Sales - December 2015
Sales Decks for Founders - Founding Sales - December 2015
Peter Kazanjy
 
Minimum viable product
Minimum viable productMinimum viable product
Minimum viable product
Sofian Hadiwijaya
 
The History of Facebook Advertising
The History of Facebook AdvertisingThe History of Facebook Advertising
The History of Facebook Advertising
HubSpot
 
Workshop MVP
Workshop MVPWorkshop MVP
Workshop MVP
Avatech Accelerator
 
Aplplying Jobs To Be Done To UX Strategy
Aplplying Jobs To Be Done To UX StrategyAplplying Jobs To Be Done To UX Strategy
Aplplying Jobs To Be Done To UX Strategy
Jim Kalbach
 
Multicultural marketing strategy by michael
Multicultural marketing strategy by michaelMulticultural marketing strategy by michael
Multicultural marketing strategy by michael
Michael Hong
 
MVP: What it is and why we all (should) care
MVP: What it is and why we all (should) careMVP: What it is and why we all (should) care
MVP: What it is and why we all (should) careKate Rutter
 
Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides
Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides
Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides
SlideTeam
 
Startup weekend - your 5 minute pitch
Startup weekend - your 5 minute pitchStartup weekend - your 5 minute pitch
Startup weekend - your 5 minute pitch
nzangels
 
Measuring & Understanding Product/Market Fit Qualitatively
Measuring & Understanding Product/Market Fit QualitativelyMeasuring & Understanding Product/Market Fit Qualitatively
Measuring & Understanding Product/Market Fit Qualitatively
Hiten Shah
 
Start Up Business Process And Strategy PowerPoint Presentation Slides
Start Up Business Process And Strategy PowerPoint Presentation Slides Start Up Business Process And Strategy PowerPoint Presentation Slides
Start Up Business Process And Strategy PowerPoint Presentation Slides
SlideTeam
 
Revolutionizing JTBD Research: Evan Shore on AI
Revolutionizing JTBD Research: Evan Shore on AIRevolutionizing JTBD Research: Evan Shore on AI
Revolutionizing JTBD Research: Evan Shore on AI
Jim Kalbach
 
What Is Jobs-To-Be-Done?
What Is Jobs-To-Be-Done?What Is Jobs-To-Be-Done?
What Is Jobs-To-Be-Done?
Andrei Radulescu
 

What's hot (20)

Getting started with Job to be Done research
Getting started with Job to be Done researchGetting started with Job to be Done research
Getting started with Job to be Done research
 
Jobs To Be Done Framework
Jobs To Be Done FrameworkJobs To Be Done Framework
Jobs To Be Done Framework
 
The 10-Slide Pitch Deck Structure
The 10-Slide Pitch Deck StructureThe 10-Slide Pitch Deck Structure
The 10-Slide Pitch Deck Structure
 
Start Up Business Proposal Powerpoint Presentation Slides
Start Up Business Proposal Powerpoint Presentation SlidesStart Up Business Proposal Powerpoint Presentation Slides
Start Up Business Proposal Powerpoint Presentation Slides
 
Company Summary of Business a Plan PowerPoint Presentation Slides
Company Summary of Business a Plan PowerPoint Presentation SlidesCompany Summary of Business a Plan PowerPoint Presentation Slides
Company Summary of Business a Plan PowerPoint Presentation Slides
 
Top 20 Twitter Influencers In Singapore
Top 20 Twitter Influencers In SingaporeTop 20 Twitter Influencers In Singapore
Top 20 Twitter Influencers In Singapore
 
Go-to-market strategy, positioning and viral growth
Go-to-market strategy, positioning and viral growthGo-to-market strategy, positioning and viral growth
Go-to-market strategy, positioning and viral growth
 
Sales Decks for Founders - Founding Sales - December 2015
Sales Decks for Founders - Founding Sales - December 2015 Sales Decks for Founders - Founding Sales - December 2015
Sales Decks for Founders - Founding Sales - December 2015
 
Minimum viable product
Minimum viable productMinimum viable product
Minimum viable product
 
The History of Facebook Advertising
The History of Facebook AdvertisingThe History of Facebook Advertising
The History of Facebook Advertising
 
Workshop MVP
Workshop MVPWorkshop MVP
Workshop MVP
 
Aplplying Jobs To Be Done To UX Strategy
Aplplying Jobs To Be Done To UX StrategyAplplying Jobs To Be Done To UX Strategy
Aplplying Jobs To Be Done To UX Strategy
 
Multicultural marketing strategy by michael
Multicultural marketing strategy by michaelMulticultural marketing strategy by michael
Multicultural marketing strategy by michael
 
MVP: What it is and why we all (should) care
MVP: What it is and why we all (should) careMVP: What it is and why we all (should) care
MVP: What it is and why we all (should) care
 
Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides
Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides
Sales And Marketing Summary Annual Report Complete PowerPoint Deck With Slides
 
Startup weekend - your 5 minute pitch
Startup weekend - your 5 minute pitchStartup weekend - your 5 minute pitch
Startup weekend - your 5 minute pitch
 
Measuring & Understanding Product/Market Fit Qualitatively
Measuring & Understanding Product/Market Fit QualitativelyMeasuring & Understanding Product/Market Fit Qualitatively
Measuring & Understanding Product/Market Fit Qualitatively
 
Start Up Business Process And Strategy PowerPoint Presentation Slides
Start Up Business Process And Strategy PowerPoint Presentation Slides Start Up Business Process And Strategy PowerPoint Presentation Slides
Start Up Business Process And Strategy PowerPoint Presentation Slides
 
Revolutionizing JTBD Research: Evan Shore on AI
Revolutionizing JTBD Research: Evan Shore on AIRevolutionizing JTBD Research: Evan Shore on AI
Revolutionizing JTBD Research: Evan Shore on AI
 
What Is Jobs-To-Be-Done?
What Is Jobs-To-Be-Done?What Is Jobs-To-Be-Done?
What Is Jobs-To-Be-Done?
 

Similar to Jobs at Scale: Developing and Analyzing JTBD Surveys

Jobs That Scale
Jobs That ScaleJobs That Scale
Jobs That Scale
Helen Fake, Ph.D.
 
What do my customers really want
What do my customers really wantWhat do my customers really want
What do my customers really wantThe URL Dr.
 
How to Use Feedback and Online Surveys to Drive Action
How to Use Feedback and Online Surveys to Drive ActionHow to Use Feedback and Online Surveys to Drive Action
How to Use Feedback and Online Surveys to Drive Action
The URL Dr.
 
Human-Centred Design and Experimentation for Impact — SIMNA Breakfast Workshop
Human-Centred Design and Experimentation for Impact — SIMNA Breakfast WorkshopHuman-Centred Design and Experimentation for Impact — SIMNA Breakfast Workshop
Human-Centred Design and Experimentation for Impact — SIMNA Breakfast Workshop
Julia Birks
 
Dine America
Dine AmericaDine America
Dine America
Chris Stiehl
 
MBTI Huang (1)
MBTI Huang (1)MBTI Huang (1)
MBTI Huang (1)Shuo Huang
 
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Frithjof Petscheleit
 
Listening through customer insights
Listening through customer insightsListening through customer insights
Listening through customer insights
Mai Bằng
 
Voice of the Member Research
Voice of the Member ResearchVoice of the Member Research
Voice of the Member Research
guest392313
 
Marketing Research & Strategic Applications
Marketing Research & Strategic ApplicationsMarketing Research & Strategic Applications
Marketing Research & Strategic Applications
Matthew A. Gilbert, MBA
 
Listening through Customer Insights
Listening through Customer InsightsListening through Customer Insights
Listening through Customer Insights
Delvinia
 
Beware Biases: Path to better UX research
Beware Biases: Path to better UX researchBeware Biases: Path to better UX research
Beware Biases: Path to better UX research
HP
 
Measuring & Maintaining Employee Engagement
Measuring & Maintaining Employee EngagementMeasuring & Maintaining Employee Engagement
Measuring & Maintaining Employee EngagementPeople Lab
 
What You Need to Know About Surveys
What You Need to Know About SurveysWhat You Need to Know About Surveys
What You Need to Know About Surveys
Amanda Kaiser
 
General Customer Experience Perspective Survey for The Alley Luj.docx
General Customer Experience Perspective Survey for The Alley Luj.docxGeneral Customer Experience Perspective Survey for The Alley Luj.docx
General Customer Experience Perspective Survey for The Alley Luj.docx
shericehewat
 
Qualitative vs quantitative survey questions
Qualitative vs quantitative survey questionsQualitative vs quantitative survey questions
Qualitative vs quantitative survey questions
Pollfish
 
Spot opportunities for strategy
Spot opportunities for strategySpot opportunities for strategy
Spot opportunities for strategy
Anna Witteman
 
Online Survey Creation Guide
Online Survey Creation GuideOnline Survey Creation Guide
Online Survey Creation Guide
SimplyCast
 
Exaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilitiesExaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilities
Exaudio
 
Insights on Research Design
Insights on Research DesignInsights on Research Design
Insights on Research Design
UXPA Boston
 

Similar to Jobs at Scale: Developing and Analyzing JTBD Surveys (20)

Jobs That Scale
Jobs That ScaleJobs That Scale
Jobs That Scale
 
What do my customers really want
What do my customers really wantWhat do my customers really want
What do my customers really want
 
How to Use Feedback and Online Surveys to Drive Action
How to Use Feedback and Online Surveys to Drive ActionHow to Use Feedback and Online Surveys to Drive Action
How to Use Feedback and Online Surveys to Drive Action
 
Human-Centred Design and Experimentation for Impact — SIMNA Breakfast Workshop
Human-Centred Design and Experimentation for Impact — SIMNA Breakfast WorkshopHuman-Centred Design and Experimentation for Impact — SIMNA Breakfast Workshop
Human-Centred Design and Experimentation for Impact — SIMNA Breakfast Workshop
 
Dine America
Dine AmericaDine America
Dine America
 
MBTI Huang (1)
MBTI Huang (1)MBTI Huang (1)
MBTI Huang (1)
 
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
 
Listening through customer insights
Listening through customer insightsListening through customer insights
Listening through customer insights
 
Voice of the Member Research
Voice of the Member ResearchVoice of the Member Research
Voice of the Member Research
 
Marketing Research & Strategic Applications
Marketing Research & Strategic ApplicationsMarketing Research & Strategic Applications
Marketing Research & Strategic Applications
 
Listening through Customer Insights
Listening through Customer InsightsListening through Customer Insights
Listening through Customer Insights
 
Beware Biases: Path to better UX research
Beware Biases: Path to better UX researchBeware Biases: Path to better UX research
Beware Biases: Path to better UX research
 
Measuring & Maintaining Employee Engagement
Measuring & Maintaining Employee EngagementMeasuring & Maintaining Employee Engagement
Measuring & Maintaining Employee Engagement
 
What You Need to Know About Surveys
What You Need to Know About SurveysWhat You Need to Know About Surveys
What You Need to Know About Surveys
 
General Customer Experience Perspective Survey for The Alley Luj.docx
General Customer Experience Perspective Survey for The Alley Luj.docxGeneral Customer Experience Perspective Survey for The Alley Luj.docx
General Customer Experience Perspective Survey for The Alley Luj.docx
 
Qualitative vs quantitative survey questions
Qualitative vs quantitative survey questionsQualitative vs quantitative survey questions
Qualitative vs quantitative survey questions
 
Spot opportunities for strategy
Spot opportunities for strategySpot opportunities for strategy
Spot opportunities for strategy
 
Online Survey Creation Guide
Online Survey Creation GuideOnline Survey Creation Guide
Online Survey Creation Guide
 
Exaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilitiesExaudio, llc Background + MR capabilities
Exaudio, llc Background + MR capabilities
 
Insights on Research Design
Insights on Research DesignInsights on Research Design
Insights on Research Design
 

Recently uploaded

Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...
Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...
Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...
Mansi Shah
 
Book Formatting: Quality Control Checks for Designers
Book Formatting: Quality Control Checks for DesignersBook Formatting: Quality Control Checks for Designers
Book Formatting: Quality Control Checks for Designers
Confidence Ago
 
原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样
原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样
原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样
gpffo76j
 
RTUYUIJKLDSADAGHBDJNKSMAL,D
RTUYUIJKLDSADAGHBDJNKSMAL,DRTUYUIJKLDSADAGHBDJNKSMAL,D
RTUYUIJKLDSADAGHBDJNKSMAL,D
cy0krjxt
 
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理
jyz59f4j
 
White wonder, Work developed by Eva Tschopp
White wonder, Work developed by Eva TschoppWhite wonder, Work developed by Eva Tschopp
White wonder, Work developed by Eva Tschopp
Mansi Shah
 
Connect Conference 2022: Passive House - Economic and Environmental Solution...
Connect Conference 2022: Passive House -  Economic and Environmental Solution...Connect Conference 2022: Passive House -  Economic and Environmental Solution...
Connect Conference 2022: Passive House - Economic and Environmental Solution...
TE Studio
 
Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...
Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...
Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...
ameli25062005
 
SECURING BUILDING PERMIT CITY OF CALOOCAN.pdf
SECURING BUILDING PERMIT CITY OF CALOOCAN.pdfSECURING BUILDING PERMIT CITY OF CALOOCAN.pdf
SECURING BUILDING PERMIT CITY OF CALOOCAN.pdf
eloprejohn333
 
Moldes de letra 3D Alfabeto completo esp
Moldes de letra 3D Alfabeto completo espMoldes de letra 3D Alfabeto completo esp
Moldes de letra 3D Alfabeto completo esp
Hess9
 
一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理
一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理
一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理
h7j5io0
 
20 slides of research movie and artists .pdf
20 slides of research movie and artists .pdf20 slides of research movie and artists .pdf
20 slides of research movie and artists .pdf
ameli25062005
 
Game Concept Presentation for Ukrainian Mythology Based Game With Designs
Game Concept Presentation for Ukrainian Mythology Based Game With DesignsGame Concept Presentation for Ukrainian Mythology Based Game With Designs
Game Concept Presentation for Ukrainian Mythology Based Game With Designs
184804
 
Technoblade The Legacy of a Minecraft Legend.
Technoblade The Legacy of a Minecraft Legend.Technoblade The Legacy of a Minecraft Legend.
Technoblade The Legacy of a Minecraft Legend.
Techno Merch
 
vernacular architecture in response to climate.pdf
vernacular architecture in response to climate.pdfvernacular architecture in response to climate.pdf
vernacular architecture in response to climate.pdf
PrabhjeetSingh219035
 
Borys Sutkowski portfolio interior design
Borys Sutkowski portfolio interior designBorys Sutkowski portfolio interior design
Borys Sutkowski portfolio interior design
boryssutkowski
 
Design Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinkingDesign Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinking
cy0krjxt
 
Can AI do good? at 'offtheCanvas' India HCI prelude
Can AI do good? at 'offtheCanvas' India HCI preludeCan AI do good? at 'offtheCanvas' India HCI prelude
Can AI do good? at 'offtheCanvas' India HCI prelude
Alan Dix
 
Top Israeli Products and Brands - Plan it israel.pdf
Top Israeli Products and Brands - Plan it israel.pdfTop Israeli Products and Brands - Plan it israel.pdf
Top Israeli Products and Brands - Plan it israel.pdf
PlanitIsrael
 
Design Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinkingDesign Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinking
cy0krjxt
 

Recently uploaded (20)

Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...
Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...
Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...
 
Book Formatting: Quality Control Checks for Designers
Book Formatting: Quality Control Checks for DesignersBook Formatting: Quality Control Checks for Designers
Book Formatting: Quality Control Checks for Designers
 
原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样
原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样
原版定做(penn毕业证书)美国宾夕法尼亚大学毕业证文凭学历证书原版一模一样
 
RTUYUIJKLDSADAGHBDJNKSMAL,D
RTUYUIJKLDSADAGHBDJNKSMAL,DRTUYUIJKLDSADAGHBDJNKSMAL,D
RTUYUIJKLDSADAGHBDJNKSMAL,D
 
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证成绩单如何办理
 
White wonder, Work developed by Eva Tschopp
White wonder, Work developed by Eva TschoppWhite wonder, Work developed by Eva Tschopp
White wonder, Work developed by Eva Tschopp
 
Connect Conference 2022: Passive House - Economic and Environmental Solution...
Connect Conference 2022: Passive House -  Economic and Environmental Solution...Connect Conference 2022: Passive House -  Economic and Environmental Solution...
Connect Conference 2022: Passive House - Economic and Environmental Solution...
 
Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...
Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...
Коричневый и Кремовый Деликатный Органический Копирайтер Фрилансер Марке...
 
SECURING BUILDING PERMIT CITY OF CALOOCAN.pdf
SECURING BUILDING PERMIT CITY OF CALOOCAN.pdfSECURING BUILDING PERMIT CITY OF CALOOCAN.pdf
SECURING BUILDING PERMIT CITY OF CALOOCAN.pdf
 
Moldes de letra 3D Alfabeto completo esp
Moldes de letra 3D Alfabeto completo espMoldes de letra 3D Alfabeto completo esp
Moldes de letra 3D Alfabeto completo esp
 
一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理
一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理
一比一原版(Bolton毕业证书)博尔顿大学毕业证成绩单如何办理
 
20 slides of research movie and artists .pdf
20 slides of research movie and artists .pdf20 slides of research movie and artists .pdf
20 slides of research movie and artists .pdf
 
Game Concept Presentation for Ukrainian Mythology Based Game With Designs
Game Concept Presentation for Ukrainian Mythology Based Game With DesignsGame Concept Presentation for Ukrainian Mythology Based Game With Designs
Game Concept Presentation for Ukrainian Mythology Based Game With Designs
 
Technoblade The Legacy of a Minecraft Legend.
Technoblade The Legacy of a Minecraft Legend.Technoblade The Legacy of a Minecraft Legend.
Technoblade The Legacy of a Minecraft Legend.
 
vernacular architecture in response to climate.pdf
vernacular architecture in response to climate.pdfvernacular architecture in response to climate.pdf
vernacular architecture in response to climate.pdf
 
Borys Sutkowski portfolio interior design
Borys Sutkowski portfolio interior designBorys Sutkowski portfolio interior design
Borys Sutkowski portfolio interior design
 
Design Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinkingDesign Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinking
 
Can AI do good? at 'offtheCanvas' India HCI prelude
Can AI do good? at 'offtheCanvas' India HCI preludeCan AI do good? at 'offtheCanvas' India HCI prelude
Can AI do good? at 'offtheCanvas' India HCI prelude
 
Top Israeli Products and Brands - Plan it israel.pdf
Top Israeli Products and Brands - Plan it israel.pdfTop Israeli Products and Brands - Plan it israel.pdf
Top Israeli Products and Brands - Plan it israel.pdf
 
Design Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinkingDesign Thinking Design thinking Design thinking
Design Thinking Design thinking Design thinking
 

Jobs at Scale: Developing and Analyzing JTBD Surveys

Editor's Notes

  1. affordable, scalable, speedy
  2. Good at measuring attitudes, opinions, beliefs, values, behaviors, factual items, and perceptions quantitative people like them Multiple people can conduct a survey without affecting the validity or reliability of the results Provides baselines over time Anonymity/confidentiality for users You can ask a lot of questions OK at measuring behaviors patterns, frequency, ease and success of use user needs, expectations, perspectives, priorities and preferences user satisfaction with collections and services shifts in user attitudes and opinions relevance of collections and services to user needs trends (by repetition over time). HUFF POST – some wise guy had feminism students signing away their suffrage/Daily Show
  3. Multiple people can conduct a survey without affecting the validity or reliability of the results Provides baselines over time Anonymity/confidentiality for users You can ask a lot of questions OK at measuring behaviors patterns, frequency, ease and success of use user needs, expectations, perspectives, priorities and preferences user satisfaction with collections and services shifts in user attitudes and opinions relevance of collections and services to user needs trends (by repetition over time).
  4. Multiple people can conduct a survey without affecting the validity or reliability of the results
  5. Provides baselines over time Anonymity/confidentiality for users You can ask a lot of questions OK at measuring behaviors patterns, frequency, ease and success of use user needs, expectations, perspectives, priorities and preferences user satisfaction with collections and services shifts in user attitudes and opinions relevance of collections and services to user needs trends (by repetition over time).
  6. Anonymity/confidentiality for users You can ask a lot of questions OK at measuring behaviors patterns, frequency, ease and success of use user needs, expectations, perspectives, priorities and preferences user satisfaction with collections and services shifts in user attitudes and opinions relevance of collections and services to user needs trends (by repetition over time).
  7. Anonymity/confidentiality for users You can ask a lot of questions
  8. Disadvantages - what they aren’t so good at (5 minutes) Difficult to measure attitude changes over time (unless designed that way) Example: Conducting a survey right after a massive organizational restructure Aspirational Responses/Dishonesty May not feel encouraged to provide accurate, honest answers (social bias) May not feel comfortable providing answers that present themselves in an unfavorable manor Users may not have realistic perceptions of our behaviors Many argue that surveys are inadequate at capturing emotions, feelings and behaviors No ways to tell how truthful a respondent is being Differences in Understanding and Interpreting Questions Understanding Questions Subjectivity of Respondent Who in here is someone who never gives an establishment a 10? Maybe you give them a 9.4 or a 9.5, but never a ten? Subjectivity of the Researcher/Analyst Ways to mitigate Need to get a representative sample Ways to encourage users to participate
  9. Disadvantages - what they aren’t so good at (5 minutes) Difficult to measure attitude changes over time (unless designed that way) Example: Conducting a survey right after a massive organizational restructure Aspirational Responses/Dishonesty May not feel encouraged to provide accurate, honest answers (social bias) May not feel comfortable providing answers that present themselves in an unfavorable manor Users may not have realistic perceptions of our behaviors Many argue that surveys are inadequate at capturing emotions, feelings and behaviors No ways to tell how truthful a respondent is being Differences in Understanding and Interpreting Questions Understanding Questions Subjectivity of Respondent Who in here is someone who never gives an establishment a 10? Maybe you give them a 9.4 or a 9.5, but never a ten? Subjectivity of the Researcher/Analyst Ways to mitigate Need to get a representative sample Ways to encourage users to participate
  10. Disadvantages - what they aren’t so good at (5 minutes) Difficult to measure attitude changes over time (unless designed that way) Example: Conducting a survey right after a massive organizational restructure Aspirational Responses/Dishonesty May not feel encouraged to provide accurate, honest answers (social bias) May not feel comfortable providing answers that present themselves in an unfavorable manor Users may not have realistic perceptions of our behaviors Many argue that surveys are inadequate at capturing emotions, feelings and behaviors No ways to tell how truthful a respondent is being Differences in Understanding and Interpreting Questions Understanding Questions Subjectivity of Respondent Who in here is someone who never gives an establishment a 10? Maybe you give them a 9.4 or a 9.5, but never a ten? Subjectivity of the Researcher/Analyst Ways to mitigate Need to get a representative sample Ways to encourage users to participate
  11. Disadvantages - what they aren’t so good at (5 minutes) Difficult to measure attitude changes over time (unless designed that way) Example: Conducting a survey right after a massive organizational restructure Aspirational Responses/Dishonesty May not feel encouraged to provide accurate, honest answers (social bias) May not feel comfortable providing answers that present themselves in an unfavorable manor Users may not have realistic perceptions of our behaviors Many argue that surveys are inadequate at capturing emotions, feelings and behaviors No ways to tell how truthful a respondent is being Differences in Understanding and Interpreting Questions Understanding Questions Subjectivity of Respondent Who in here is someone who never gives an establishment a 10? Maybe you give them a 9.4 or a 9.5, but never a ten? Subjectivity of the Researcher/Analyst Ways to mitigate Need to get a representative sample Ways to encourage users to participate
  12. Disadvantages - what they aren’t so good at (5 minutes) Difficult to measure attitude changes over time (unless designed that way) Example: Conducting a survey right after a massive organizational restructure Aspirational Responses/Dishonesty May not feel encouraged to provide accurate, honest answers (social bias) May not feel comfortable providing answers that present themselves in an unfavorable manor Users may not have realistic perceptions of our behaviors Many argue that surveys are inadequate at capturing emotions, feelings and behaviors No ways to tell how truthful a respondent is being Differences in Understanding and Interpreting Questions Understanding Questions Subjectivity of Respondent Who in here is someone who never gives an establishment a 10? Maybe you give them a 9.4 or a 9.5, but never a ten? Subjectivity of the Researcher/Analyst Ways to mitigate Need to get a representative sample Ways to encourage users to participate
  13. Coverage error is when the list from which sample members are drawn does not accurately represent the population on the characteristic(s) one wants to estimate with the survey. Coverage error is the difference between the estimate produced when the list is inaccurate and what would have been produced with an accurate list. If you herd had way more brown goats then what was sampled, that would be coverage error. Coverage error is another important source of variability in survey statistics; it is the degree to which statistics are off due to the fact that the sample used does not properly represent the underlying population being measured.
  14. Response rate is only an indirect indicator of survey quality. Error occurs when the characteristics of those who chose to respond different from those who chose not to respond in a way that is relevant to the survey results. Favorable environment example – If those who have positive views of the environment are the only ones who responded, than the sample would be biased because of the non-response error. [take something questionable – like a big mac burger or something] HIV testing example with 84% response rate with remarkably low levels of HIV. Later learned that those who participated did not report behaviors that would be consistent with the contraction of HIV. At the Motley Fool we look for at least a 10-15% response rate for external surveys. Internal surveys typically hit 30-40% response rates ne early example of a finding was reported by Visser, Krosnick, Marquette and Curtin (1996) who showed that surveys with lower response rates (near 20%) yielded more accurate measurements than did surveys with higher response rates (near 60 or 70%).[3] In another study, Keeter et al. (2006) compared results of a 5-day survey employing the Pew Research Center’s usual methodology (with a 25% response rate) with results from a more rigorous survey conducted over a much longer field period and achieving a higher response rate of 50%. In 77 out of 84 comparisons, the two surveys yielded results that were statistically indistinguishable. Among the items that manifested significant differences across the two surveys, the differences in proportions of people giving a particular answer ranged from 4 percentage points to 8 percentage points.[4]
  15. How many of you are familiar with statistics? When you send out a survey you likely won’t want to send it out to your whole file. Sampling error is the difference between the population you survey and the actual population.
  16. Measurement error occurs when respondents are unwilling or unable to provide accurate answers. This can be due to poor question design, survey mode effects, or data collection mistakes. Good examples could be: Socially deviant or illegal behaviors Cannot answer because the words are not understood or phrases are confusing Question structure may encourage certain answers when another would not. For example, items that ask respondents to check all that apply tend to result in fewer selections as opposed to those who ask for explicit positive or negative answers Question order could impact measurement error Fun fact: scalar questions are likely to be answered differently in visual versus aural survey
  17. Assuming prior knowledge or understanding To what extent to you agree or disagree that there are enough arts and cultural activities in your local area? [1- strongly disagree- 5 strongly agree]   To what extent to you agree or disagree that there are enough arts and cultural activities in your local area?  By ‘local area’ we mean within a 10 minute drive of your home?  [1- strongly disagree- 5 strongly agree]
  18. Assuming prior knowledge or understanding To what extent to you agree or disagree that there are enough arts and cultural activities in your local area? [1- strongly disagree- 5 strongly agree]   To what extent to you agree or disagree that there are enough arts and cultural activities in your local area?  By ‘local area’ we mean within a 10 minute drive of your home?  [1- strongly disagree- 5 strongly agree]
  19. Asking two questions in one:   A double-barreled question is a one that has more than one question embedded within it. Participants may answer one but not both, or may disagree with part or all of the question. Double-barreled question: Do you agree that campus parking is a problem and that the administration should be working diligently on a solution? Revised question: Is campus parking a problem? (If the participant responds yes): Should the administration be responsible for solving this problem?
  20. Asking two questions in one:   A double-barreled question is a one that has more than one question embedded within it. Participants may answer one but not both, or may disagree with part or all of the question. Double-barreled question: Do you agree that campus parking is a problem and that the administration should be working diligently on a solution? Revised question: Is campus parking a problem? (If the participant responds yes): Should the administration be responsible for solving this problem?
  21. Asking two questions in one:   A double-barreled question is a one that has more than one question embedded within it. Participants may answer one but not both, or may disagree with part or all of the question. Double-barreled question: Do you agree that campus parking is a problem and that the administration should be working diligently on a solution? Revised question: Is campus parking a problem? (If the participant responds yes): Should the administration be responsible for solving this problem?
  22. Bad Question: How short was Napoleon? The word “short” immediately brings images to the mind of the respondent. If the question is rewritten to be neutral-sounding, it can eliminate the leading bias. Good Question: How would you describe Napoleon’s height? Starting with demographics   Asking two questions in one
  23. Bad Question: How short was Napoleon? The word “short” immediately brings images to the mind of the respondent. If the question is rewritten to be neutral-sounding, it can eliminate the leading bias. Good Question: How would you describe Napoleon’s height? Starting with demographics   Asking two questions in one
  24. Bad Question: How short was Napoleon? The word “short” immediately brings images to the mind of the respondent. If the question is rewritten to be neutral-sounding, it can eliminate the leading bias. Good Question: How would you describe Napoleon’s height?
  25. Inadequate response options Example: How long have you been working in this job?   [1-2 years,  2-5 years, 5 or more years] Alternative: How long have you been working in this job?  [Less than a year, 1-2 years,  2-5 years, 5 or more years].
  26. [Less than a year, 1-2 years,  2-5 years, 5 or more years].
  27. Using negative question wording Negative question wording make respondents do a double take when trying to respond.  They usually include the word ‘not’ in the question itself, then they ask respondents to agree or disagree with the position or statement. Designing your survey questions in positive language gives you one less source of error and bias to worry about, and that’s not a bad thing! Example: Do you agree or disagree that KFC’s Double Down Sandwich is not a healthy option for our growing children? [Agree/ Disagree/ Don’t know] Alternative: Do you think the arts are recognized as a valuable tool in helping to strengthen communities?  [Yes/ No/ Don’t know]
  28. Using negative question wording Negative question wording make respondents do a double take when trying to respond.  They usually include the word ‘not’ in the question itself, then they ask respondents to agree or disagree with the position or statement. Designing your survey questions in positive language gives you one less source of error and bias to worry about, and that’s not a bad thing! Example: Do you agree or disagree that KFC’s Double Down Sandwich is not a healthy option for our growing children? [Agree/ Disagree/ Don’t know] Alternative: Do you think the arts are recognized as a valuable tool in helping to strengthen communities?  [Yes/ No/ Don’t know]
  29. Example 2.7. Anchoring Consider two different wordings for a particular question: Wording 1: Knowing that the population of the U.S. is 316 million, what is the population of Canada? Wording 2: Knowing that the population of Australia is 23 million, what is the population of Canada? This survey was conducted in Stat 100 classes where both wordings of the question were randomly distributed.  The students did not know that there were two versions of this question so each only answered the question that they received. The results for this survey are found in Figure 2.4. Figure 2.4. STAT 100 Survey Results As you can see, the students were influenced by the wording of the question that they were asked to answer. People's perceptions can be severely distorted when they are provided with a reference point or an anchor. People tend to say close to the anchor because of either having limited knowledge about the topic or being distracted by the anchor. You should also consider the following three points: The sample sizes were large enough to detect a difference in the two groups Canada's population is about 35 million The anchor might be less distracting if the following wording were used: "What is the population of Canada, when knowing that the population of the U.S. is 316 million?" but it is best to leave out the anchoring statement altogether.
  30. Using negative question wording Negative question wording make respondents do a double take when trying to respond.  They usually include the word ‘not’ in the question itself, then they ask respondents to agree or disagree with the position or statement. Designing your survey questions in positive language gives you one less source of error and bias to worry about, and that’s not a bad thing! Example: Do you agree or disagree that KFC’s Double Down Sandwich is not a healthy option for our growing children? [Agree/ Disagree/ Don’t know] Alternative: Do you think the arts are recognized as a valuable tool in helping to strengthen communities?  [Yes/ No/ Don’t know]
  31. Using negative question wording Negative question wording make respondents do a double take when trying to respond.  They usually include the word ‘not’ in the question itself, then they ask respondents to agree or disagree with the position or statement. Designing your survey questions in positive language gives you one less source of error and bias to worry about, and that’s not a bad thing! Example: Do you agree or disagree that KFC’s Double Down Sandwich is not a healthy option for our growing children? [Agree/ Disagree/ Don’t know] Alternative: Do you think the arts are recognized as a valuable tool in helping to strengthen communities?  [Yes/ No/ Don’t know]
  32. User Experience Professional Association Website (N=60) 95% confidence level People adjust to the wording and disagree. They tend to adjust for the language being used. Additional work is needed, but it suggests you may want to be careful how you use language in your surveys or else encounter this type of bias.
  33. Too many open-ended questions
  34. The short answer is that 7-point scales are a little better than 5-points—but not by much. The psychometric literature suggests that having more scale points is better but there is a diminishing return after around 11 points (Nunnally 1978).  Having seven points tends to be a good balance between having enough points of discrimination without having to maintain too many response options. So what are the consequences of this?
  35. The short answer is that 7-point scales are a little better than 5-points—but not by much. The psychometric literature suggests that having more scale points is better but there is a diminishing return after around 11 points (Nunnally 1978).  Having seven points tends to be a good balance between having enough points of discrimination without having to maintain too many response options. So what are the consequences of this? A recent article tested the SUS response error by counting the number of times users couldn’t decide between two points. The users were allowed to “interpolate” or pick between points such as 3.5.  In 2.5% of the 858 responses users provided responses between two points (95% CI between 1.6% to 3.9%). In contrast, when a 7-point version of the SUS was used there was no interpolating for any of the 840 ratings. While this seems like compelling evidence to always use 7-point over 5 scales there are two tempering factors. While there was error in the five point SUS it is unclear how much of an impact this actually has on the final SUS score since the study used different systems for each scale. Errors in statistics have a way of cancelling themselves out.  It is likely that many responses that are “forced” into higher numbers will be cancelled out by those forced into lower numbers. There are many things that can go wrong with questions and responses: users misinterpret the question, users select the wrong box or administrators forget to invert the scales. The data suggests this type of error is small but can be addressed in the following four ways.
  36. Too many open-ended questions