SlideShare a Scribd company logo
1 of 27
Download to read offline
Unmoderated,
remote
User Testing.
Get to know it
better.
20/01/2018 © Marc-Oliver | Experience Lead
1. What is unmoderated user testing?
2. Why we should do it?
3. How others do it
4. How we get started
2
What is unmoderated UT?
_
What is unmoderated user testing
Unmoderated usability sessions are completed alone
by the participant with no real-time interaction or
influence from another person.
The main focus of such a study is a few specific
elements, rather than an overall review. It’s great to see
the impact of a minor change. The more ‘moving parts’
you need to test, the more you have to invest in
technology and setup. This is also true for prototype
versus testing on production.
Pros
» Low running cost, once setup.
» Can do last minute
» Potentially with many participants
» Easier to recruit given you have
proper channels set up (Facebook,
twitter, email etc)
» Can go global on a large scale
» Users use their own device and
‘natural environment’
Cons
» Might need a lot of software/setup
» No influence when something goes
wrong
» Cannot control precisely who does it
and when
» No follow up questions – harder to
get the ‘WHY’
» Have to spend more time filtering
the results
» Sensitive data/design can leak
easily
Why we should do it?
_
Why you should do it
testtest
What you (probably) currently do
time
informed or semi-informed tinkering, tinkering, tinkering, tinkering, tinkering ….
Why you should do it
What you should do: Frequently collect new small/relevant data points. Know
what people do right now. Respond immediately.
Test &
Iterate
Test &
Iterate
Test &
Iterate
Test &
Iterate
Test &
Iterate
Test &
Iterate
Test &
Iterate
Test &
Iterate
Test &
Iterate
time
How others do it.
_
How Netflix does it
Netflix often runs 20 to 30 ‘experiments’
at the same time on an ongoing basis,
with different audiences (subscriber
types) and in selected markets. They do
this with a complex setup of tools and live
(web)services.
Resource:
• https://medium.com/netflix-techblog/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation-
and-data-5b0ae9295bdf
• https://medium.com/netflix-techblog/how-we-determine-product-success-980f81f0047e
• https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-2-d9b96aa399f5
• https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429
• https://www.quora.com/profile/Neil-Hunt/answers/A-B-Testing
• https://medium.com/netflix-techblog/optimizing-the-netflix-streaming-experience-with-data-science-725f04c3e834
How etsy does it
etsy – an online retailer – is known for
their “test first – don’t discuss” culture.
They focus on multi-variant and A/B tests
on their live website and can deploy tests
in hours.
Resource:
• https://www.slideshare.net/InfoQ/ab-testing-continuous-delivery-everyday-product-launches?qid=66f85988-52fb-4d28-bece-
b1b8c5465ab4&v=&b=&from_search=4
How Meetup does it
There is never a question as to whether a
product or (new/existing) feature should
be tested at MeetUp … the only question
is “how soon can we fit this in?”
Resource:
• https://www.youtube.com/watch?v=8QO-2olF9M4
• http://making.meetup.com/post/46268864442/how-meetup-hq-does-usability-testing
• https://uxdesign.cc/meetup-a-usability-case-study-e909c33f1e3e
How Obama did it in 2008
Optimisation was the name of the game
for the Obama Digital team. They
optimised just about everything from web
pages to emails. To give you an idea: They
conducted 500 A/B tests on their web
pages in a 20 month period.
Resource:
• https://blog.optimizely.com/2013/01/15/how-obamas-campaign-team-sourced-ab-tests-from-user-feedback/
• https://www.wired.com/2012/04/ff_abtesting/
• https://www.optimizely.com/customers/obama2012/
• https://www.youtube.com/watch?v=71bH8z6iqSc
• http://kylerush.net/blog/optimization-at-the-obama-campaign-ab-testing/
How Spotify is doing it
Spotify strongly believes, data makes you
customer focused.
Resource:
• https://www.slideshare.net/alisarrafi3/ab-testing-at-spotify
• https://www.slideshare.net/dj4b1n/ab-testing-pitfalls-and-lessons-learned-at-spotify-31935130
• https://www.slideshare.net/alisarrafi3/being-a-data-driven-business
How to get started
_
Example 1a: Test understanding/validating
of product benefits
Key Steps
To set a benchmark, isolate two to
three tariffs (leave design untouched)
and set up a preference test. Recruit
participants from Amazon Turk or in-
house (existing and non-existing
customers). Isolate key benefits and
let participants sort them by priority.
Use an online card-sorting tool. Show
one tariff and let participants freely
sort and annotate elements they do
not understand.
Effort & Time
Low effort, 1.5h to 2.5h
Tools needed
optimalworkshop.com or Helio from
Zurb
Recruiting
Per email, social media, etc. Small
sample size >10
Example 1b: Test understanding/
interpretation of icons
Key Steps
Take screenshots of new/ambiguous
icons and embed them in an online
survey. Underneath each icon, allow
people to interpret the meaning of it
via a free form field (or selection of
words). Share it first internally and if
more responses are needed, share
online with a larger group.
Alternatively, kick-things off internally
and just share icons on slack to get
some early/fast responses. Use your
friends and just WhatsApp them the
test-item. Get creative.
Effort & Time
low effort, 0.5h to 1h
Tools needed
Survey-Monkey, Typeform, Google
Docs, WhatsApp
Recruiting
Per email, social media, WhatsApp,
etc. Small sample size >10
Example 2: Test Layout &
UI elements
Key Steps
Create or show people an existing
screenshot. Share it with your
audience and let them annotate/
comment on specific elements. Start
with questions such as “What would
you expect to happen when you click
on …” -or- “Please tell us what …
does” -or- “Can you show us where
you would click to do/go to …” and so
on. Great for initial A/B testing.
Effort & Time
Depending on complexity of screen,
low- to mid- effort, 2h to 8h
Tools needed
Helio from Zurb, SurveyMonkey
Recruiting
Per email, social media, WhatsApp,
etc. Small sample size >25 to 50
Example 3: Test functionality of
features, modules or pages
Key Steps
Create a task, e.g. “Please find tariff
X” -or- keep it more open and set up an
easy-to-grasp scenario. Next, build a
clickable prototype (low/high fidelity),
so participants can run through the
task you gave them. Observe or
automate recording of session(s).
Effort & Time
High effort, 1 to 2 days.
Tools needed
Prototyping tool (Sketch+Invision),
screen-recording setup
(Usertesting.com or Lookback.io)
Recruiting
Per email, social media, WhatsApp,
AmazonTurk, etc. Small sample size
>25 to 50
Example 4: Test functionality
& page performance
Key Steps
Focus on a key functionality or
section/flow of your app/website.
Choose an experiment and set-up
event tracking and visual analysis
tools. This type of testing requires
modifying code and the ability to
deploy the master within hours/
minutes. It is great for getting a large
sample size >1000.
Effort & Time
High effort, multiple days to prep.
Less, once automated testing is setup
Tools needed
Mixpanel or GA, Hotjar, Mousflow, etc.
depending on test-type, Cohort
Analysis
Recruiting
Per email, social media, WhatsApp,
AmazonTurk, etc. Large sample size
>1000
3 Key Take-aways
_
• Test early and often small, but relevant bits
(early feedback is good feedback)
• Automate most steps to speed things up
• Share learnings frequently
Suggested Process
_
Example:
Test
proposal
Heck, more tooling!
_
Cool tools
1. https://zurb.com/helio/
2. https://unbounce.com/
3. https://www.usertesting.com/
4. https://www.invisionapp.com/ in combination with https://lookback.io/
5. https://www.hotjar.com/ or https://www.crazyegg.com/
6. https://www.userzoom.com/
7. https://validately.com/
8. https://www.whatusersdo.com/
9. https://fivesecondtest.com/
10.https://www.optimizely.com/
11.More tools: http://remoteresear.ch/tools/
MANY THANKS.
You can hire me. Send an email ›

More Related Content

What's hot

What's hot (20)

Usability Overview by Craig Tomlin
Usability Overview by Craig TomlinUsability Overview by Craig Tomlin
Usability Overview by Craig Tomlin
 
User Testing- guiding the way to success
User Testing- guiding the way to successUser Testing- guiding the way to success
User Testing- guiding the way to success
 
User testing workshop
User testing workshopUser testing workshop
User testing workshop
 
Usability testing tips for UX Designers
Usability testing tips for UX DesignersUsability testing tips for UX Designers
Usability testing tips for UX Designers
 
Tech. Writing Usability Presentation
Tech. Writing Usability PresentationTech. Writing Usability Presentation
Tech. Writing Usability Presentation
 
Usability 2.0
Usability 2.0Usability 2.0
Usability 2.0
 
Usability of web application
Usability of web applicationUsability of web application
Usability of web application
 
Usability Testing 101 - an introduction
Usability Testing 101 - an introductionUsability Testing 101 - an introduction
Usability Testing 101 - an introduction
 
What is usability testing?
What is usability testing?What is usability testing?
What is usability testing?
 
Chapter 7 the good-enough - tables
Chapter 7   the good-enough - tablesChapter 7   the good-enough - tables
Chapter 7 the good-enough - tables
 
Jan Moons talks at World Usability Day 2015 in Antwerp
Jan Moons talks at World Usability Day 2015 in AntwerpJan Moons talks at World Usability Day 2015 in Antwerp
Jan Moons talks at World Usability Day 2015 in Antwerp
 
Lean Usability
Lean UsabilityLean Usability
Lean Usability
 
Usability testing - have fun and improve your work
Usability testing - have fun and improve your workUsability testing - have fun and improve your work
Usability testing - have fun and improve your work
 
Exploratory testing workshop
Exploratory testing workshopExploratory testing workshop
Exploratory testing workshop
 
Prototyping and Usability Testing your designs
Prototyping and Usability Testing your designsPrototyping and Usability Testing your designs
Prototyping and Usability Testing your designs
 
User Research Fast & Cheap
User Research Fast & Cheap User Research Fast & Cheap
User Research Fast & Cheap
 
Mobile Usability Testing
Mobile Usability TestingMobile Usability Testing
Mobile Usability Testing
 
Usability for all budgets
Usability for all budgetsUsability for all budgets
Usability for all budgets
 
A Tester’s Experience with User Experience Mapping
A Tester’s Experience with User Experience MappingA Tester’s Experience with User Experience Mapping
A Tester’s Experience with User Experience Mapping
 
Organizing Your First Website Usability Test - WordCamp Boston 2016
Organizing Your First Website Usability Test - WordCamp Boston 2016Organizing Your First Website Usability Test - WordCamp Boston 2016
Organizing Your First Website Usability Test - WordCamp Boston 2016
 

Similar to Remote, unmoderated usability and user testing.

Webinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability TestingWebinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability Testing
UserZoom
 
Keeping Pace with Recruitment Technology Trends - Fordyce Forum Presentation
Keeping Pace with Recruitment Technology Trends - Fordyce Forum PresentationKeeping Pace with Recruitment Technology Trends - Fordyce Forum Presentation
Keeping Pace with Recruitment Technology Trends - Fordyce Forum Presentation
Shannon Myers
 

Similar to Remote, unmoderated usability and user testing. (20)

How to do usability testing and eye tracking
How to do usability testing and eye trackingHow to do usability testing and eye tracking
How to do usability testing and eye tracking
 
Usability Testing and QA 12 11-15
Usability Testing and QA 12 11-15Usability Testing and QA 12 11-15
Usability Testing and QA 12 11-15
 
Usability Testing by Rajdeep Gupta, Misys
Usability Testing by Rajdeep Gupta, MisysUsability Testing by Rajdeep Gupta, Misys
Usability Testing by Rajdeep Gupta, Misys
 
Agile methodology - Humanity
Agile methodology  - HumanityAgile methodology  - Humanity
Agile methodology - Humanity
 
Webinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability TestingWebinar: How to Conduct Unmoderated Remote Usability Testing
Webinar: How to Conduct Unmoderated Remote Usability Testing
 
week-11-PROTOTYPE.pptx
week-11-PROTOTYPE.pptxweek-11-PROTOTYPE.pptx
week-11-PROTOTYPE.pptx
 
User Zoom Webinar Monster Aug09 Vf
User Zoom Webinar Monster Aug09 VfUser Zoom Webinar Monster Aug09 Vf
User Zoom Webinar Monster Aug09 Vf
 
Usability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working GroupUsability Primer - for Alberta Municipal Webmasters Working Group
Usability Primer - for Alberta Municipal Webmasters Working Group
 
Keeping Pace with Recruitment Technology Trends - Fordyce Forum Presentation
Keeping Pace with Recruitment Technology Trends - Fordyce Forum PresentationKeeping Pace with Recruitment Technology Trends - Fordyce Forum Presentation
Keeping Pace with Recruitment Technology Trends - Fordyce Forum Presentation
 
Useagility Webinar - Automated User Testing
Useagility Webinar - Automated User TestingUseagility Webinar - Automated User Testing
Useagility Webinar - Automated User Testing
 
"Open" includes users - Leverage their input
"Open" includes users - Leverage their input"Open" includes users - Leverage their input
"Open" includes users - Leverage their input
 
Slides from "Taking an Holistic Approach to Product Quality"
Slides from "Taking an Holistic Approach to Product Quality"Slides from "Taking an Holistic Approach to Product Quality"
Slides from "Taking an Holistic Approach to Product Quality"
 
Usability Testing Basics: What's it All About? at Web SIG Cleveland
Usability Testing Basics: What's it All About? at Web SIG ClevelandUsability Testing Basics: What's it All About? at Web SIG Cleveland
Usability Testing Basics: What's it All About? at Web SIG Cleveland
 
Lean UX
Lean UXLean UX
Lean UX
 
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOpsWinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
WinOps Conf 2016 - Matteo Emili - Development and QA Dilemmas in DevOps
 
5 Lessons Learned in Product Management by Twitch Senior PM
5 Lessons Learned in Product Management by Twitch Senior PM5 Lessons Learned in Product Management by Twitch Senior PM
5 Lessons Learned in Product Management by Twitch Senior PM
 
Software testing
Software testingSoftware testing
Software testing
 
Users are Losers! They’ll Like Whatever we Make! and Other Fallacies.
Users are Losers! They’ll Like Whatever we Make! and Other Fallacies.Users are Losers! They’ll Like Whatever we Make! and Other Fallacies.
Users are Losers! They’ll Like Whatever we Make! and Other Fallacies.
 
Solution Design - The Hidden Side of UX (for Product Managers)
Solution Design - The Hidden Side of UX (for Product Managers)Solution Design - The Hidden Side of UX (for Product Managers)
Solution Design - The Hidden Side of UX (for Product Managers)
 
Solution Design - The Hidden Side of UX (for Product Managers)
Solution Design - The Hidden Side of UX (for Product Managers)Solution Design - The Hidden Side of UX (for Product Managers)
Solution Design - The Hidden Side of UX (for Product Managers)
 

More from Marc-Oliver Gern

UX Tools for Agile Teams – Design Mission
UX Tools for Agile Teams – Design MissionUX Tools for Agile Teams – Design Mission
UX Tools for Agile Teams – Design Mission
Marc-Oliver Gern
 
UX Tools for Agile Teams – The Job Table
UX Tools for Agile Teams – The Job TableUX Tools for Agile Teams – The Job Table
UX Tools for Agile Teams – The Job Table
Marc-Oliver Gern
 
Digital Branding for Hydro Company
Digital Branding for Hydro CompanyDigital Branding for Hydro Company
Digital Branding for Hydro Company
Marc-Oliver Gern
 
Digital Branding for an Airline
Digital Branding for an AirlineDigital Branding for an Airline
Digital Branding for an Airline
Marc-Oliver Gern
 
Corporate Design, Brand Management
Corporate Design, Brand ManagementCorporate Design, Brand Management
Corporate Design, Brand Management
Marc-Oliver Gern
 

More from Marc-Oliver Gern (18)

UX Tools for Agile Teams – Design Mission
UX Tools for Agile Teams – Design MissionUX Tools for Agile Teams – Design Mission
UX Tools for Agile Teams – Design Mission
 
UX Tools for Agile Teams – The Job Table
UX Tools for Agile Teams – The Job TableUX Tools for Agile Teams – The Job Table
UX Tools for Agile Teams – The Job Table
 
THE UX INTERVIEW – 3 Quick Questions. 3 Short Answers.
THE UX INTERVIEW – 3 Quick Questions. 3 Short Answers.THE UX INTERVIEW – 3 Quick Questions. 3 Short Answers.
THE UX INTERVIEW – 3 Quick Questions. 3 Short Answers.
 
Mobile Brand Experiences – Key Findings
Mobile Brand Experiences – Key FindingsMobile Brand Experiences – Key Findings
Mobile Brand Experiences – Key Findings
 
The Impact of Brand User Experience Design
The Impact of Brand User Experience DesignThe Impact of Brand User Experience Design
The Impact of Brand User Experience Design
 
iPad App Development, Concept Stage
iPad App Development, Concept StageiPad App Development, Concept Stage
iPad App Development, Concept Stage
 
Twitter Task Flow Table
Twitter Task Flow TableTwitter Task Flow Table
Twitter Task Flow Table
 
The Future of Travelling: Top Social Media Options for Tour Operators
The Future of Travelling: Top Social Media Options for Tour OperatorsThe Future of Travelling: Top Social Media Options for Tour Operators
The Future of Travelling: Top Social Media Options for Tour Operators
 
Digital Branding for Hydro Company
Digital Branding for Hydro CompanyDigital Branding for Hydro Company
Digital Branding for Hydro Company
 
Digital Branding
Digital BrandingDigital Branding
Digital Branding
 
Digital Branding for an Airline
Digital Branding for an AirlineDigital Branding for an Airline
Digital Branding for an Airline
 
A dynamic Screensaver for Airlines 2
A dynamic Screensaver for Airlines 2A dynamic Screensaver for Airlines 2
A dynamic Screensaver for Airlines 2
 
A dynamic Screensaver for Airlines.
A dynamic Screensaver for Airlines.A dynamic Screensaver for Airlines.
A dynamic Screensaver for Airlines.
 
Social Media Playbook
Social Media PlaybookSocial Media Playbook
Social Media Playbook
 
Digital Portfolio Marc Oliver
Digital Portfolio Marc OliverDigital Portfolio Marc Oliver
Digital Portfolio Marc Oliver
 
Strategic Brand Development
Strategic Brand DevelopmentStrategic Brand Development
Strategic Brand Development
 
Brand Identity, Example for 1st Design Session
Brand Identity, Example for 1st Design SessionBrand Identity, Example for 1st Design Session
Brand Identity, Example for 1st Design Session
 
Corporate Design, Brand Management
Corporate Design, Brand ManagementCorporate Design, Brand Management
Corporate Design, Brand Management
 

Recently uploaded

Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
ChloeMeadows1
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
lolsDocherty
 

Recently uploaded (17)

Thank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsThank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirts
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
 
Statistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfStatistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdf
 
Reggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirts
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of apps
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirt
 
Pvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdfPvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdf
 
The Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case StudyThe Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case Study
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
Premier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfPremier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdf
 
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresenceCyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
 
GOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdfGOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdf
 
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
 
How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?
 
Free scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirtsFree scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirts
 
Bug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideBug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's Guide
 

Remote, unmoderated usability and user testing.

  • 1. Unmoderated, remote User Testing. Get to know it better. 20/01/2018 © Marc-Oliver | Experience Lead
  • 2. 1. What is unmoderated user testing? 2. Why we should do it? 3. How others do it 4. How we get started 2
  • 4. What is unmoderated user testing Unmoderated usability sessions are completed alone by the participant with no real-time interaction or influence from another person. The main focus of such a study is a few specific elements, rather than an overall review. It’s great to see the impact of a minor change. The more ‘moving parts’ you need to test, the more you have to invest in technology and setup. This is also true for prototype versus testing on production.
  • 5. Pros » Low running cost, once setup. » Can do last minute » Potentially with many participants » Easier to recruit given you have proper channels set up (Facebook, twitter, email etc) » Can go global on a large scale » Users use their own device and ‘natural environment’ Cons » Might need a lot of software/setup » No influence when something goes wrong » Cannot control precisely who does it and when » No follow up questions – harder to get the ‘WHY’ » Have to spend more time filtering the results » Sensitive data/design can leak easily
  • 6. Why we should do it? _
  • 7. Why you should do it testtest What you (probably) currently do time informed or semi-informed tinkering, tinkering, tinkering, tinkering, tinkering ….
  • 8. Why you should do it What you should do: Frequently collect new small/relevant data points. Know what people do right now. Respond immediately. Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate Test & Iterate time
  • 10. How Netflix does it Netflix often runs 20 to 30 ‘experiments’ at the same time on an ongoing basis, with different audiences (subscriber types) and in selected markets. They do this with a complex setup of tools and live (web)services. Resource: • https://medium.com/netflix-techblog/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation- and-data-5b0ae9295bdf • https://medium.com/netflix-techblog/how-we-determine-product-success-980f81f0047e • https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-2-d9b96aa399f5 • https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429 • https://www.quora.com/profile/Neil-Hunt/answers/A-B-Testing • https://medium.com/netflix-techblog/optimizing-the-netflix-streaming-experience-with-data-science-725f04c3e834
  • 11. How etsy does it etsy – an online retailer – is known for their “test first – don’t discuss” culture. They focus on multi-variant and A/B tests on their live website and can deploy tests in hours. Resource: • https://www.slideshare.net/InfoQ/ab-testing-continuous-delivery-everyday-product-launches?qid=66f85988-52fb-4d28-bece- b1b8c5465ab4&v=&b=&from_search=4
  • 12. How Meetup does it There is never a question as to whether a product or (new/existing) feature should be tested at MeetUp … the only question is “how soon can we fit this in?” Resource: • https://www.youtube.com/watch?v=8QO-2olF9M4 • http://making.meetup.com/post/46268864442/how-meetup-hq-does-usability-testing • https://uxdesign.cc/meetup-a-usability-case-study-e909c33f1e3e
  • 13. How Obama did it in 2008 Optimisation was the name of the game for the Obama Digital team. They optimised just about everything from web pages to emails. To give you an idea: They conducted 500 A/B tests on their web pages in a 20 month period. Resource: • https://blog.optimizely.com/2013/01/15/how-obamas-campaign-team-sourced-ab-tests-from-user-feedback/ • https://www.wired.com/2012/04/ff_abtesting/ • https://www.optimizely.com/customers/obama2012/ • https://www.youtube.com/watch?v=71bH8z6iqSc • http://kylerush.net/blog/optimization-at-the-obama-campaign-ab-testing/
  • 14. How Spotify is doing it Spotify strongly believes, data makes you customer focused. Resource: • https://www.slideshare.net/alisarrafi3/ab-testing-at-spotify • https://www.slideshare.net/dj4b1n/ab-testing-pitfalls-and-lessons-learned-at-spotify-31935130 • https://www.slideshare.net/alisarrafi3/being-a-data-driven-business
  • 15. How to get started _
  • 16. Example 1a: Test understanding/validating of product benefits Key Steps To set a benchmark, isolate two to three tariffs (leave design untouched) and set up a preference test. Recruit participants from Amazon Turk or in- house (existing and non-existing customers). Isolate key benefits and let participants sort them by priority. Use an online card-sorting tool. Show one tariff and let participants freely sort and annotate elements they do not understand. Effort & Time Low effort, 1.5h to 2.5h Tools needed optimalworkshop.com or Helio from Zurb Recruiting Per email, social media, etc. Small sample size >10
  • 17. Example 1b: Test understanding/ interpretation of icons Key Steps Take screenshots of new/ambiguous icons and embed them in an online survey. Underneath each icon, allow people to interpret the meaning of it via a free form field (or selection of words). Share it first internally and if more responses are needed, share online with a larger group. Alternatively, kick-things off internally and just share icons on slack to get some early/fast responses. Use your friends and just WhatsApp them the test-item. Get creative. Effort & Time low effort, 0.5h to 1h Tools needed Survey-Monkey, Typeform, Google Docs, WhatsApp Recruiting Per email, social media, WhatsApp, etc. Small sample size >10
  • 18. Example 2: Test Layout & UI elements Key Steps Create or show people an existing screenshot. Share it with your audience and let them annotate/ comment on specific elements. Start with questions such as “What would you expect to happen when you click on …” -or- “Please tell us what … does” -or- “Can you show us where you would click to do/go to …” and so on. Great for initial A/B testing. Effort & Time Depending on complexity of screen, low- to mid- effort, 2h to 8h Tools needed Helio from Zurb, SurveyMonkey Recruiting Per email, social media, WhatsApp, etc. Small sample size >25 to 50
  • 19. Example 3: Test functionality of features, modules or pages Key Steps Create a task, e.g. “Please find tariff X” -or- keep it more open and set up an easy-to-grasp scenario. Next, build a clickable prototype (low/high fidelity), so participants can run through the task you gave them. Observe or automate recording of session(s). Effort & Time High effort, 1 to 2 days. Tools needed Prototyping tool (Sketch+Invision), screen-recording setup (Usertesting.com or Lookback.io) Recruiting Per email, social media, WhatsApp, AmazonTurk, etc. Small sample size >25 to 50
  • 20. Example 4: Test functionality & page performance Key Steps Focus on a key functionality or section/flow of your app/website. Choose an experiment and set-up event tracking and visual analysis tools. This type of testing requires modifying code and the ability to deploy the master within hours/ minutes. It is great for getting a large sample size >1000. Effort & Time High effort, multiple days to prep. Less, once automated testing is setup Tools needed Mixpanel or GA, Hotjar, Mousflow, etc. depending on test-type, Cohort Analysis Recruiting Per email, social media, WhatsApp, AmazonTurk, etc. Large sample size >1000
  • 22. • Test early and often small, but relevant bits (early feedback is good feedback) • Automate most steps to speed things up • Share learnings frequently
  • 26. Cool tools 1. https://zurb.com/helio/ 2. https://unbounce.com/ 3. https://www.usertesting.com/ 4. https://www.invisionapp.com/ in combination with https://lookback.io/ 5. https://www.hotjar.com/ or https://www.crazyegg.com/ 6. https://www.userzoom.com/ 7. https://validately.com/ 8. https://www.whatusersdo.com/ 9. https://fivesecondtest.com/ 10.https://www.optimizely.com/ 11.More tools: http://remoteresear.ch/tools/
  • 27. MANY THANKS. You can hire me. Send an email ›