SlideShare a Scribd company logo
1 of 66
Download to read offline
Research and
Discovery Tools for
Experimentation
1
Optimise Or Die
© Optimal Visit 2d018
INTRODUCTION
• UX Research
• Analytics data
• Toolkits
• AB Testing
@OptimiseOrDie
linkd.in/pvrg14
Optimise Or Die
1. TOP TECHNIQUES
2. WHY ARE THEY USEFUL?
3. HOW TO USE THEM
4. CASE STUDY
5. AI AUGMENTATION
Higher Quality Research Inputs:
Optimise Or Die
Behavioural
(Observation)
Attitudinal (Opinion)
Qualitative Quantitative
Analytics
Event Data
Lab
Usability
Testing
Unmoderate
d Remote
Testing
Remote
Moderated
Testing
Diary / Field
Contextual
Shadowing
Guerrilla
Testing
User
Interviews
Participatory
Design
Focus
Groups
Site based
poll or
survey
Email or
Postal
Survey
Intercept
Surveys
NPS, CSAT,
Service
Metrics
Tree
Testing
Session
Replay
Heat, Click,
Scroll maps
Open Ended
Feedback
Device &
Viewport
Testing
Eye / EEG
Tracking
AB/Split
Testing
5 second
Preference
Card
Sorting
Expert
Eval
Optimise Or Die
Behavioural
(Observation)
Attitudinal (Opinion)
Qualitative Quantitative
Analytics
Event Data
Lab
Usability
Testing
Unmoderate
d Remote
Testing
Remote
Moderated
Testing
Diary / Field
Contextual
Shadowing
Guerrilla
Testing
User
Interviews
Participatory
Design
Focus
Groups
Site based
poll or
survey
Email or
Postal
Survey
Intercept
Surveys
NPS, CSAT,
Service
Metrics
Tree
Testing
Session
Replay
Heat, Click,
Scroll maps
Open Ended
Feedback
Device &
Viewport
Testing
Eye / EEG
Tracking
AB/Split
Testing
5 second
Preference
Card
Sorting
Expert
Eval
bit.ly/30QANDQ Optimise Or Die
1
Optimise Or Die
• Why don’t you do more usability testing?
• There’s a poll in the webinar settings (top right)
• Please pick a poll option and tell me more in the Q&A
• What are the biggest reasons?
POLL TIME 
Optimise Or Die
Why don’t you do Usability Testing?
• Amazing product & experiment ideas
• Usually ‘Evaluative’ – we aren’t asking if people like
it, we are testing if they can use it get jobs done
• UT goes where opinions or assumptions cannot
• Your designs are flawed, you just don’t know where
• Once you know what’s wrong, you can fix it
• Baymard* is a good guide to what will go wrong
• There’s no replacement for user testing
Optimise Or Die
* https://baymard.com/
NO – Usability testing will not tell you:
• How much money it will make
• If people like the new thing or interface
• If the feature/product will be a success
• How people would rate it if it was live
• What people like about the design
• Whether the new colours are nice
• If it will work when you make it live
• For this, we need Analytics & AB testing
Optimise Or Die
NO – Usability testing will not tell you:
• Don’t ask participants market research type questions –
it’s usually too small a sample
• ‘How easy was that to use out of 10?’ type questions are
useless here
• So none of these “Rate the product” type questions –
that’s a quant question shoved into a qual study. It’s not a
survey, you’re evaluating an interface!
• Don’t ask for preferences or future behaviour – you get
nothing from this that’s useful
• So what can we get then?
Optimise Or Die
Usability Testing DOES tell you:
• If most people can use the feature or product elements
they encountered during the test
• If they can use it to get tasks, jobs or goals completed
• This is all you are testing – useability
• Can people use the wish list? Can they find stuff? Can
they checkout? Can they use the navigation?
• Good usability is highly correlated with high conversion –
bugs & usability are the biggest drag on potential
conversion, not design
Optimise Or Die
bit.ly/6minuteUsability
Els Aerts
Hot Takes from Craig on Testing:
Optimise Or Die
Unmoderated or
Moderated?
I prefer moderated tests becauseit’s a skilled
lens onto evaluations and insights
Lab or Remote?
Quality of tools is so good,remote works
just fine
Panel or Self
Recruitment?
Quality can be poor, so ruthlesslyscreento
avoid bias or serialtesting
Screening is vital
Screento get quality participants =
value out.
How many people?
For work that’s previously been usability tested, 5
quality participants, per roundof testing are good
for iterating. For new evaluations, 12-20
dependingon key audiencesegments.
Can you use AI?
Yes, to take video interviews and transcribe
them to text. Then you can usea variety of
tools to analyse and map out key topics
and theme clusters from the raw text of
multiple sessions.
No Budget?
If you advertise a car, it’s important to make
sure it works when peoplego to buy one. If
you’re spendingon ads,you need to do the same
with your product.
Saves $$$
Removing defects from products
earlier in the process,before you’ve
shippedthe feature or product, saves
a ton of money!
@OptimiseOrDie
Budget User Research
Concepts:
Usability Hub usabilityhub.com
Optimal Workshop optimalworkshop.com
Five second test fivesecondtest.com
Maze maze.co
Five second, Preference, Prototype, First Click, Card Sort, Tree Test
Lookback (M,U) lookback.com
Userlytics (M,U) userlytics.com
Usertesting (M,U) bit.ly/3ZtzWAO
Userzoom (M,U) bit.ly/464oQV8
Loop11 (U) loop11.com
Userbrain (U) userbrain.com
Ethnio (R) ethn.io/
14
Guerrilla Testing
WHAT IS GT?
maze.co/guides/usability-testing/guerrilla/
TIPS & TRICKS:
https://bit.ly/3LBL1dm
WHAT DO YOU NEED?
• A screen recording tool
• Coffee shop, Bar, Conference, Meetup
• A script you’ve practised
15
120
FIVE SECOND TEST
• Did people ‘get’ key messages?
• What do they remember?
• What is their interpretation?
• Did they read the text?
Optimise Or Die
PREFERENCE TEST
• Which design did they prefer?
• Why?
• Good for competing options
Optimise Or Die
FIRST CLICK TEST
• Can they find something
• Is the control visible?
• On a page with multiple
options, which one would they
start with and why?
Optimise Or Die
CARD SORT
• If you want to design navigation or product assortments from
scratch, this is the way to figure it out
• It’s like mining people’s heads for how they would organise books
in a library, clothing inside a store or groceries in a supermarket
1. Users put “things” into piles and give the piles “names”
2. Using the data from multiple users, we extract the structure
that maps onto human mental models
3. Solve Navigation, Categorisation and Findability from scratch
Optimise Or Die
TREE TEST
• For testing an existing site or product that’s live
• Card sorting to plan a new structure
• Tree sorting to optimise an existing one
Optimise Or Die
PROTOTYPE OR MOCKUP TESTING
• Lots of services allow for mockup testing
• Usability hub allows you to import Figma flows
• Easy way to get quick and cheap feedback
• User test later on, use this for early ideas
• Saves time & expense fixing defects later
Optimise Or Die
NEW EMPLOYEES
Use new employees (e.g. call centre staff) as your usability testing subjects
FRIENDS & FAMILY
Use Friends and Family as testing subjects (if they are largely unaware of what you do)
SCREEN ALL PARTICIPANTS PROPERLY
Make sure you don’t use Web developers, Designers, Journalists, UX people – any ‘content’ or ‘design’ profession
HIRE AN EXPERIENCED STUDENT
A University near you is running an Usability (HCI) degree. Talk to them about a project for students there!
LEARN HOW TO INTERVIEW
Not anyone can be a racing driver or a nuclear scientist. Same for being a moderator or interviewer – it takes lots of
experience and practice. Took me two years to become even passable as a moderator. Hire someone if you think this
is easy to learn or that you are good at it (without critical evaluation).
Low Budget Recruitment:
Optimise Or Die
Interviewing Tips
• Unbiased and Useful Usability Research is HARD
• No – you can’t fly a 777 if you’ve done flight simulator
• No – you can’t just interpret user interviews or tests
• Yes – you need years of experience
• Read these & Practice:
“Don’t Make Me Think” amzn.to/1gIZEJn
“Rocket Surgery Made Easy” amzn.to/1e0hnUL
“Talking to Customers” bit.ly/1e0hT58
“Talking with Participants” bit.ly/1kKL3LE
“Don’t listen to Users” bit.ly/3OVA336
“Interviewing Tips” bit.ly/3yx4aUQ
“More interviewing Tips” bit.ly/3dOHI1o
23
Usability Testing - Summary
• Reach parts of product other tools cannot
• No testing is worse than low budget testing
• Get awesome product & pivot ideas
• Orgs with UT have better experiments!
• Problem Statement -> Hypothesis -> Experiment
• Understand customer problems better
• See flaws from a different perspective
Optimise Or Die
@OptimiseOrDie
WHERE THE ACTION HAPPENS:
x x
User &
Context
Device
Your Design
2
SURVEYS
POLLS
FEEDBACK
Optimise Or Die
• A useful and underrated skill
• If you don’t know what to do, why
not just ask people stuff?
• User inquiry is one of the easiest
ways to remove BS from work
• Triggered polls and feedback
buttons are both incredibly useful
• OPEN ENDED QUESTIONS are
your friend – and really useful to
analyse in GPT
Optimise Or Die
Customer Feedback is GOLD:
The Three Greatest Questions Ever:
“What’s the purpose of your visit today?”
“Were you able to complete your task
today?”
“If you weren’t able to complete your task
today, why not?”
Optimise Or Die
Feedback, Polls & Surveys
“I don’t know what to do” – Why not ask visitors:
1. Big SaaS tool company had a hero landing page
2. Big budget $$$ paid traffic landed here
3. 11 months of AB testing, no shift in Conversion Rate
4. AB tests had failed to find a new winner
5. They had tried nearly everything
6. We added a poll to the page (Purpose of Visit)
7. We figured out they were “Boss Comparison Shopping”
8. We asked them what competitors they were looking at
9. We added a feature comparison for these tools
10. We ran an AB test and it improved CR by 4%
Optimise Or Die
These are the ONLY types you will need:
PURPOSE OF VISIT ON ENTRY Landing page Poll
TASK GAP ANALYSIS ENTRY/EXIT Did you get your tasks done?
LEAVING REASONS SITE EXIT Exit Survey
FUNNEL EXIT ON EXIT Funnel Exit Survey
FUNNEL FEEDBACK ALWAYS ON Funnel Specific Feedback
THANK YOU PAGE ALWAYS ON “Was there one thing that nearly...”
SITE FEEDBACK ALWAYS ON “Any unanswered questions?”
PRODUCT PAGE ALWAYS ON “Is there any information missing?”
Optimise Or Die
Based on
today’s visit,
how would you
rate your site
experience
overall?
Were you able
to complete
the purpose of
your visit
today?
If NO, why
were you not
able to?
If YES, what
did you value
most about
the website?
Thank You Page:
“Was there one thing that nearly stopped you from purchasing a flight today?”
Task Gap Survey:
Optimise Or Die
Carglass Thank You:
1. Tells you about customers who nearly made
converted, but something stopped them
2. The customers who manage to get through, will
also experience the same problems or issues as
those who abandon near the end – their
motivation or patience was higher
3. Asking people to explain potential
abandonment reasons, helps you understand
the group who never made it to the end
4. If you resolve the problems, you’ve moved a
group from abandoning to converting,
permanently
5. Asking for DETAIL or specifics in these questions
means you get more & better feedback*
* See papers Smith & Jones (2018), Johnson et al. (2019)
Optimise Or Die
Aim for Open & Inquiry Based Feedback:
“What other products or service should we offer?”
“How would you describe x to a friend?”
“What other options did you consider before looking at x?”
“Is there some information that you’re missing here?”
“What other products are you comparing?”
“Was there one thing that nearly stopped you from purchasing from
us today? Please be as detailed as possible.”
Optimise Or Die
Els Aerts: The Lost Art of Asking Questions
https://conversionista.com/wp-content/uploads/2019/10/CJAM-2019-ElsAerts-Sthlm.pdf
https://yoast.com/audience-research-expert-interview/
3. Recommended VOC Tools
2.11 Further Reading
Optimise Or Die
@OptimiseOrDie
Surveys That Work:
• Best book on surveys
• Caroline knows her stuff
• Stop wasting time and
• Stop doing biased useless surveys
Surveys Tools & Services:
• Paperform www.paperform.co
• Typeform www.typeform.com
• Google Forms www.google.com/forms/about/
• Surveymonkey www.surveymonkey.co.uk
• Alchemer www.alchemer.com
• Qualtrics www.qualtrics.com
• GfK www.gfk.com
“Surveys seem like “hard to do” but if you get the right help and
iterate them based on data, you get quality insights. If you have no
survey or the same one as last year, you’re not listening.”
Optimise Or Die
@OptimiseOrDie
Competitor Surveys
• Get 500 responses from your site
• Recruit 500 from competitors (ask GFK)
• Analyse data and compare
“One of the most useful tools during a hyper growth
phase was knowing what drove customer delight in
terms of service metrics and how this compared to our
competitors”
Andrew Ground
3 INTERVIEWING
PEOPLE
Optimise Or Die
Support or Customer Services:
1. What are the top 3-5 questions or worries you get from
potential customers?
2. What do you answer when you get these questions?
3. Are there any particular aspects of X that people don’t
understand?
4. What aspects of X do people like the most/least?
5. Did I miss anything important? Got something to add?
6. If you could change anything about X that would help
YOU to delight customers, what would the top issues be?
Optimise Or Die
Sales or Commercial:
1. What are the top 3-5 objections from potential
customers?
2. What do you answer when you get these objections?
3. Are there any aspects of X that people don’t understand?
4. What aspects of X do people like the most/least?
5. Did I miss anything important? Got something to add?
6. If you could change anything about X to help sell it more
easily, what would it be?
Optimise Or Die
UX Levers Interviews (EV New Car):
General Questions
• What has prompted you to consider purchasing an electric vehicle at this point in time?
• Have you owned or considered an EV before? If not, what types of vehicles have you
previously considered or owned?
• How familiar are you with electric vehicles and their technology?
• What are your main sources of information when researching new cars?
Motivation and Preferences
• What are the most important factors for you when choosing a new car (e.g., cost,
performance, environmental impact, brand, etc.)?
• How does an electric vehicle fit into your lifestyle?
• Are there specific features you are looking for in an electric vehicle (e.g., range, charging
time, autopilot capabilities)?
• What concerns do you have about owning an EV (e.g., charging infrastructure, battery life,
maintenance costs)?
Decision-making Process
• How do you plan to compare different EV models and brands?
• Are there any websites that you plan to use for your search?
• What would be a deal breaker for you when selecting an EV?
• Who else is involved in the decision-making process for purchasing your new car?
Optimise Or Die
Insight - Inputs
Customers do not know what to do and need support and advice
• Emphasize the fact that you understand that their situation is stressful
• Emphasize your expertise and leadership in vehicle glazing and that you will help them get the best solution for
their situation
• Explain what they will need to do online and during the call-back so that they know what the next steps will be
• Explain that they will be able ask any other questions they might have during the call-back
Customers do not feel confident in assessing the damage
• Emphasize the fact that you will help them assess the damage correctly online
Customers need to understand the benefits of booking online
• Emphasize that the online booking system is quick, easy and provides all the information they need in regards
with their appointment and general cost information
Customers mistrust insurers and find dealing with their insurance situation very frustrating
• Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for
money for cash customers
• Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their
insurance paperwork for them, freeing them of this burden
Some customers cannot be bothered to take action to fix their car glass
• Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’
UX Levers – Vehicle Glass Repair Example
High Value Customers + Unhappy Customers:
• Reverse engineer delight by studying these groups
• Engage with unhappy customers (reviews, customer
services or special team to handle low review scores)
• Interview loyal or high spending customers
• Find out why they are delighted/unhappy
• Use questions but let them tell you their story:
✔ Ask good questions
✔ Shut up
✔ Listen
✔ Repeat
Optimise Or Die
Interviewing Tips
Want to interview better? Read these:
“Don’t Make Me Think” amzn.to/1gIZEJn
“Rocket Surgery Made Easy” amzn.to/1e0hnUL
“Talking to Customers” bit.ly/1e0hT58
“Talking with Participants” bit.ly/1kKL3LE
“Don’t listen to Users” bit.ly/3OVA336
“Interviewing Tips” bit.ly/3yx4aUQ
44
4 SESSION REPLAY
Optimise Or Die
Session Replay – what’s useful?
1. Good quality recordings, with superb filtering
2. Scroll Maps sometimes useful
3. Heat Maps are not useful
4. Click Maps are useful for only one thing
5. A ‘Half Way House’ between Analytics & User Testing
6. Immersion in the negative side, struggle & frustration
Optimise Or Die
Heat, Scroll and Click Maps
PROS: Models attention
CONS:
• Weak correlation between
mouse & eye gaze
• Not useful for touch devices
PROS:
• Map scroll depth
• Find attention ‘Dead Zones’
CONS
• Does it map to your design?
PROS:
• See clicks that didn’t work
CONS
• Useless for anything else
Session Replay Recordings
PROS
• Get Video recordings of real users
• Spot problems, struggle, bugs, device
issues, errors or confusion
• Useful for checking AB test variants
CONS
• NOT a substitute for analytics IMO
• Hard to ‘view’ thousands of recordings
• Does hit site performance (check)
• It’s always an abstraction of reality*
Optimise Or Die
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
Case Study – Abandonment & Session Replay
Postcode drop
Delivery drop
Payment drop
So what did I do?
1. Pulled 400 “Failure” recordings of the payment page
2. Got a piece of paper and made a list
3. I put people who “Looked and Left” in one pile
4. I put people who “Did something and left” into other piles
5. I put errors into specific piles too
6. 3 hours work – what does data say?
Optimise Or Die
Session Replay Failure Marking:
Reason %
1. An error occurred - click start your subscription again
(PayPal)*
22%
2. Didn't start payment interaction or exited PayPal 21%
3. Payment processing spinner 'hangs’* 17%
4. No idea for the reason 12%
5. Discount removed - already been used (house/account)* 11%
6. Opened order summary & left 5%
7. Email already in use 4%
8. Can't login on payment page 3%
9. Insufficient funds 2%
10. Bank declined payment 1%
11. Animation breaks page 1%
12. Promo code invalid 1%
• The two in orange are people
“Looked and Left”
• The other issues are all ones we
can solve or mitigate
• 40% of exits are down to 2
payment failure bugs
• 11% of failures are because the
promo code is asked for too late
and rejected only at payment
• 4% are rejected because their
email is already in use but there’s
no easy way to log in
• This short piece of work nailed a
a 750k a month loss in payment
fails!
Optimise Or Die
Tip – Failure Marking bit.ly/RainbowTomer
• You can use Tomer
Sharon’s rainbow
spreadsheet for marking
usability sessions or
screen recordings
bit.ly/RainbowTomer
Optimise Or Die
5
DEVICE TESTING
Most companies don’t VALIDATE these:
Easy
Stuff ☺
I can use it
I can read it
Loads and runs quickly
Works on my device
Optimise Or Die
Craig’s Hierarchy of Product Foundations
Easy
Stuff ☺
Usability
Accessibility
Performance
Device Compatibility
Optimise Or Die
Device Testing
• One of the simplest ways to find problems
• Every site has a problem with devices & screens
• Companies pay for ‘Responsive’ websites
• What does this mean though? Stretchy? Fluid? Working?
• Responsive: means ‘responds’ to scale images, text and
design on devices that visitors or customers have
• Evidence – if you say it is responsive, we should prove it
Optimise Or Die
5.4” iPhone (12 mini)
6.1” iPhone (various)
6.2” Android (Samsung S2x)
Mac 13” Laptop (1440x900)
Win 13” Laptop (1366x768)
2 year old iPad
Minimum Viable Testing List (85-95% traffic)
Optimise Or Die
@OptimiseOrDie
£320 iPhone 12 mini (Amazon)
£210 iPhone XR (Amazon)
£230 Samsung S22 (Cex)
£240 Windows Laptop (Amazon)
£410 Mac Laptop (eBay)
Cost of Testing Hardware ~ 1.4K
@OptimiseOrDie
Desktop Browsers:
Crossbrowsertesting.com
Browserstack.com
bit.ly/MobileDeviceLab
Device Testing in the Cloud:
Mobile & Tablet Cloud:
Aws.amazon.com/device-farm
Browserstack.com/app-live
Lambdatest.com
Perfectomobile.com & Kobiton.com
No Hardware?
Friends & Family!
Device Testing Summary:
• Your “dev testing list” is probably wrong or out of date
• Use your analytics data to define what to test
• Don’t use emulators or simulators, not worth it
• For core devices (85-95% of audience) get real kit
• Cloud devices (AWS, Browserstack) are okay for desktop or
non core devices
• Buy the hardware, don’t cut corners
Optimise Or Die
6
AI ASSIST FOR
RESEARCH &
DISCOVERY
Optimise Or Die
AI in the back seat:
HELPING YOU with:
● Market & Product Research
● Ideation & Prioritisation
● Wireframing & Design
● Understanding Markets
● Persuasive copywriting
● LPO & SEO
● What customers are saying
● What words and phrases they
use
Photo by Pexels
Real Use Cases that
work:
● Summarising text (reviews, chat, anything)
● Sentiment Analysis
● Copy, Microcopy, UX writing research
● Market and Product Research
● Competitor Analysis & Feature Comparison
● White paper research (scite, elicit)
● Personas, Journeys, User Stories help
● Moderator scripts, use cases, tasks
● Design patterns or anti-patterns
● Video or Speech transcribing
● Note taking for interviews
● Clustering notes or interview feedback
Photo by Pexels
Photo by Pexels
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4708466
Experimentation AI FTW:
VWO sponsors Experimentation Elite: 11-12 June, Birmingham
• Attend my AI Playbook workshop:
experimentationelite.com/all-events/
• Workshops for Product Managers & Leaders – get in touch
• 12 week Maven Course for Experimenters – on the way!
Optimise Or Die
Thank you!
Craig Sullivan
CEO, Optimise Or Die Ltd.
sullivac@gmail.com
+447711 657315
linkedin.com/in/craigsullivan
@OptimiseOrDie
Optimise Or Die
Will be sent to you!
Slides:

More Related Content

Similar to Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf

How to Do Better Market Research
How to Do Better Market ResearchHow to Do Better Market Research
How to Do Better Market Research
Kissmetrics on SlideShare
 

Similar to Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf (20)

UX Field Research Toolkit - A Workshop at Big Design - 2017
UX Field Research Toolkit - A Workshop at Big Design - 2017UX Field Research Toolkit - A Workshop at Big Design - 2017
UX Field Research Toolkit - A Workshop at Big Design - 2017
 
Matthew Roach, Sanoma (AUS/UK/NL) - Conversion Hotel 2017 - keynote
Matthew Roach, Sanoma (AUS/UK/NL) - Conversion Hotel 2017 - keynoteMatthew Roach, Sanoma (AUS/UK/NL) - Conversion Hotel 2017 - keynote
Matthew Roach, Sanoma (AUS/UK/NL) - Conversion Hotel 2017 - keynote
 
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...
 
How to do usability testing and eye tracking
How to do usability testing and eye trackingHow to do usability testing and eye tracking
How to do usability testing and eye tracking
 
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 20133 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
 
Usability: whats the use? Presented by We are Sigma and PRWD
Usability: whats the use? Presented by We are Sigma and PRWDUsability: whats the use? Presented by We are Sigma and PRWD
Usability: whats the use? Presented by We are Sigma and PRWD
 
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
 
12 reasons your site sucks - InvestNI
12 reasons your site sucks - InvestNI12 reasons your site sucks - InvestNI
12 reasons your site sucks - InvestNI
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig Sullivan
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
 
Understanding User Experience Workshop - Interlink Conference 2012
Understanding User Experience Workshop - Interlink Conference 2012Understanding User Experience Workshop - Interlink Conference 2012
Understanding User Experience Workshop - Interlink Conference 2012
 
體驗劇場_1050524_W14_易用性測試_楊政達
體驗劇場_1050524_W14_易用性測試_楊政達體驗劇場_1050524_W14_易用性測試_楊政達
體驗劇場_1050524_W14_易用性測試_楊政達
 
How to Do Better Market Research
How to Do Better Market ResearchHow to Do Better Market Research
How to Do Better Market Research
 
Usability Testing and QA 12 11-15
Usability Testing and QA 12 11-15Usability Testing and QA 12 11-15
Usability Testing and QA 12 11-15
 
Dual Track Agile & Data Driven Design
Dual Track Agile & Data Driven DesignDual Track Agile & Data Driven Design
Dual Track Agile & Data Driven Design
 
Design process
Design processDesign process
Design process
 
Usability: Whats The Use by PRWD & Sigma
Usability: Whats The Use by PRWD & SigmaUsability: Whats The Use by PRWD & Sigma
Usability: Whats The Use by PRWD & Sigma
 
Patrick McKenzie Opticon 2014: Advanced A/B Testing
Patrick McKenzie Opticon 2014: Advanced A/B TestingPatrick McKenzie Opticon 2014: Advanced A/B Testing
Patrick McKenzie Opticon 2014: Advanced A/B Testing
 
"Open" includes users - Leverage their input
"Open" includes users - Leverage their input"Open" includes users - Leverage their input
"Open" includes users - Leverage their input
 

More from VWO

More from VWO (20)

The Science of Landing Page Messaging.pdf
The Science of Landing Page Messaging.pdfThe Science of Landing Page Messaging.pdf
The Science of Landing Page Messaging.pdf
 
Uncover Insightful User Journey Secrets Using GA4 Reports
Uncover Insightful User Journey Secrets Using GA4 ReportsUncover Insightful User Journey Secrets Using GA4 Reports
Uncover Insightful User Journey Secrets Using GA4 Reports
 
Fueling A_B experiments with behavioral insights (1).pdf
Fueling A_B experiments with behavioral insights (1).pdfFueling A_B experiments with behavioral insights (1).pdf
Fueling A_B experiments with behavioral insights (1).pdf
 
Life between variants by Tomas Voves.pdf
Life between variants by Tomas Voves.pdfLife between variants by Tomas Voves.pdf
Life between variants by Tomas Voves.pdf
 
Message-Market Fit VWO Workshop by Daphne tideman
Message-Market Fit VWO Workshop by Daphne tidemanMessage-Market Fit VWO Workshop by Daphne tideman
Message-Market Fit VWO Workshop by Daphne tideman
 
A_B Testing Personalized Meditation Recommendations.pdf
A_B Testing Personalized Meditation Recommendations.pdfA_B Testing Personalized Meditation Recommendations.pdf
A_B Testing Personalized Meditation Recommendations.pdf
 
Converting with Comedy: Research Parallels for CRO
Converting with Comedy: Research Parallels for CROConverting with Comedy: Research Parallels for CRO
Converting with Comedy: Research Parallels for CRO
 
Unifying feature management with experiments - Server Side Webinar (1).pdf
Unifying feature management with experiments - Server Side Webinar (1).pdfUnifying feature management with experiments - Server Side Webinar (1).pdf
Unifying feature management with experiments - Server Side Webinar (1).pdf
 
Run more experiments with fewer resources
Run more experiments with fewer resourcesRun more experiments with fewer resources
Run more experiments with fewer resources
 
Crafting High-Converting eCommerce Landing Pages
Crafting High-Converting eCommerce Landing PagesCrafting High-Converting eCommerce Landing Pages
Crafting High-Converting eCommerce Landing Pages
 
Diseñando según la regulación actual de cookies para lograr más conversión
Diseñando según la regulación actual de cookies para lograr más conversiónDiseñando según la regulación actual de cookies para lograr más conversión
Diseñando según la regulación actual de cookies para lograr más conversión
 
VWO - Mark de Winter - Run more experiments with fewer resources.pdf
VWO - Mark de Winter - Run more experiments with fewer resources.pdfVWO - Mark de Winter - Run more experiments with fewer resources.pdf
VWO - Mark de Winter - Run more experiments with fewer resources.pdf
 
Turn Website Visits into Conversions (2).pdf
Turn Website Visits into Conversions (2).pdfTurn Website Visits into Conversions (2).pdf
Turn Website Visits into Conversions (2).pdf
 
Principales obstáculos de la experimentación y cómo abordarlos en StartUps...
Principales obstáculos de la experimentación y cómo abordarlos en StartUps...Principales obstáculos de la experimentación y cómo abordarlos en StartUps...
Principales obstáculos de la experimentación y cómo abordarlos en StartUps...
 
Understand user struggles. Optimize mobile app UX.
Understand user struggles. Optimize mobile app UX.Understand user struggles. Optimize mobile app UX.
Understand user struggles. Optimize mobile app UX.
 
Turn Website Visits into Conversions (1).pdf
Turn Website Visits into Conversions (1).pdfTurn Website Visits into Conversions (1).pdf
Turn Website Visits into Conversions (1).pdf
 
Converting with Comedy: Research Parallels for CRO
Converting with Comedy: Research Parallels for CROConverting with Comedy: Research Parallels for CRO
Converting with Comedy: Research Parallels for CRO
 
Optimizing Mobile App UX for better engagement & conversions.pptx
Optimizing Mobile App UX for better engagement & conversions.pptxOptimizing Mobile App UX for better engagement & conversions.pptx
Optimizing Mobile App UX for better engagement & conversions.pptx
 
Optimizing Mobile App UX for better engagement & conversions.pdf
Optimizing Mobile App UX for better engagement & conversions.pdfOptimizing Mobile App UX for better engagement & conversions.pdf
Optimizing Mobile App UX for better engagement & conversions.pdf
 
7 puntos clave y prácticos en todo proyecto de CRO
7 puntos clave y prácticos en todo proyecto de CRO7 puntos clave y prácticos en todo proyecto de CRO
7 puntos clave y prácticos en todo proyecto de CRO
 

Recently uploaded

Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...
Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...
Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...
Spesialis Kandungan Resmi BPOM
 
Top 10 Recommended Fragrances for Father's Day
Top 10 Recommended Fragrances for Father's DayTop 10 Recommended Fragrances for Father's Day
Top 10 Recommended Fragrances for Father's Day
disenylurial
 
obat pelancar haid di apotik dan harganya
obat pelancar haid di apotik dan harganyaobat pelancar haid di apotik dan harganya
obat pelancar haid di apotik dan harganya
infoobataborsi24
 
Licença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdf
Licença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdfLicença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdf
Licença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdf
Lotter Pro Brasil
 
Fungible Tokens on Bitcoin: Runes Protocol
Fungible Tokens on Bitcoin: Runes ProtocolFungible Tokens on Bitcoin: Runes Protocol
Fungible Tokens on Bitcoin: Runes Protocol
nehapardhi711
 

Recently uploaded (20)

Rhys Cater, Precis, The future of media buying with Generative AI.pdf
Rhys Cater, Precis, The future of media buying with Generative AI.pdfRhys Cater, Precis, The future of media buying with Generative AI.pdf
Rhys Cater, Precis, The future of media buying with Generative AI.pdf
 
The Vital Role of Keyword Density in Crafting SEO-Optimized Content
The Vital Role of Keyword Density in Crafting SEO-Optimized ContentThe Vital Role of Keyword Density in Crafting SEO-Optimized Content
The Vital Role of Keyword Density in Crafting SEO-Optimized Content
 
Building Your Customer Base with MailPoet.pdf
Building Your Customer Base with MailPoet.pdfBuilding Your Customer Base with MailPoet.pdf
Building Your Customer Base with MailPoet.pdf
 
Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...
Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...
Klinik Jual Obat Aborsi Di Bandung wa 0851/7541/5434 Misoprostol 200mcg Pfize...
 
Impacts Of Smart Watch & Wearable Technology On Daily Life
Impacts Of Smart Watch & Wearable Technology On Daily LifeImpacts Of Smart Watch & Wearable Technology On Daily Life
Impacts Of Smart Watch & Wearable Technology On Daily Life
 
Top 10 Recommended Fragrances for Father's Day
Top 10 Recommended Fragrances for Father's DayTop 10 Recommended Fragrances for Father's Day
Top 10 Recommended Fragrances for Father's Day
 
buy best digital marketing course in india
buy best digital marketing course in indiabuy best digital marketing course in india
buy best digital marketing course in india
 
Generating Leads with Analyst Content Webinar Slides_SJN Final.pdf
Generating Leads with Analyst Content Webinar Slides_SJN Final.pdfGenerating Leads with Analyst Content Webinar Slides_SJN Final.pdf
Generating Leads with Analyst Content Webinar Slides_SJN Final.pdf
 
Admya Infotech is the Best Digital Marketing Company
Admya Infotech is the Best Digital Marketing CompanyAdmya Infotech is the Best Digital Marketing Company
Admya Infotech is the Best Digital Marketing Company
 
The BoF Brand Magic Index Volume Two — Preview.pdf
The BoF Brand Magic Index Volume Two — Preview.pdfThe BoF Brand Magic Index Volume Two — Preview.pdf
The BoF Brand Magic Index Volume Two — Preview.pdf
 
Top tips for effective SEO copywriting.pdf
Top tips for effective SEO copywriting.pdfTop tips for effective SEO copywriting.pdf
Top tips for effective SEO copywriting.pdf
 
Beyond Silos: How Holistic B2B Digital Strategy Drives Pipeline
Beyond Silos: How Holistic B2B Digital Strategy Drives PipelineBeyond Silos: How Holistic B2B Digital Strategy Drives Pipeline
Beyond Silos: How Holistic B2B Digital Strategy Drives Pipeline
 
SocialMedia Marketing Plan for TheSparksFoundation
SocialMedia Marketing Plan for TheSparksFoundationSocialMedia Marketing Plan for TheSparksFoundation
SocialMedia Marketing Plan for TheSparksFoundation
 
obat pelancar haid di apotik dan harganya
obat pelancar haid di apotik dan harganyaobat pelancar haid di apotik dan harganya
obat pelancar haid di apotik dan harganya
 
Niche Analysis for Client Outreach Outside Marketplace.pptx
Niche Analysis for Client Outreach Outside Marketplace.pptxNiche Analysis for Client Outreach Outside Marketplace.pptx
Niche Analysis for Client Outreach Outside Marketplace.pptx
 
Licença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdf
Licença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdfLicença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdf
Licença Lotter Pro - Conheça o Certificado Oficial da Licença Lotter Pro.pdf
 
The Indian Ocean Tsunami of 2004 Remembering a Catastrophe.pptx
The Indian Ocean Tsunami of 2004 Remembering a Catastrophe.pptxThe Indian Ocean Tsunami of 2004 Remembering a Catastrophe.pptx
The Indian Ocean Tsunami of 2004 Remembering a Catastrophe.pptx
 
Digital marketing guide complete guide for beginners
Digital marketing guide complete guide for beginnersDigital marketing guide complete guide for beginners
Digital marketing guide complete guide for beginners
 
Being a PMM with a multi-product portfolio - Product Marketing Summit
Being a PMM with a multi-product portfolio - Product Marketing SummitBeing a PMM with a multi-product portfolio - Product Marketing Summit
Being a PMM with a multi-product portfolio - Product Marketing Summit
 
Fungible Tokens on Bitcoin: Runes Protocol
Fungible Tokens on Bitcoin: Runes ProtocolFungible Tokens on Bitcoin: Runes Protocol
Fungible Tokens on Bitcoin: Runes Protocol
 

Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf

  • 1. Research and Discovery Tools for Experimentation 1 Optimise Or Die
  • 2. © Optimal Visit 2d018 INTRODUCTION • UX Research • Analytics data • Toolkits • AB Testing @OptimiseOrDie linkd.in/pvrg14 Optimise Or Die
  • 3. 1. TOP TECHNIQUES 2. WHY ARE THEY USEFUL? 3. HOW TO USE THEM 4. CASE STUDY 5. AI AUGMENTATION Higher Quality Research Inputs: Optimise Or Die
  • 4. Behavioural (Observation) Attitudinal (Opinion) Qualitative Quantitative Analytics Event Data Lab Usability Testing Unmoderate d Remote Testing Remote Moderated Testing Diary / Field Contextual Shadowing Guerrilla Testing User Interviews Participatory Design Focus Groups Site based poll or survey Email or Postal Survey Intercept Surveys NPS, CSAT, Service Metrics Tree Testing Session Replay Heat, Click, Scroll maps Open Ended Feedback Device & Viewport Testing Eye / EEG Tracking AB/Split Testing 5 second Preference Card Sorting Expert Eval Optimise Or Die
  • 5. Behavioural (Observation) Attitudinal (Opinion) Qualitative Quantitative Analytics Event Data Lab Usability Testing Unmoderate d Remote Testing Remote Moderated Testing Diary / Field Contextual Shadowing Guerrilla Testing User Interviews Participatory Design Focus Groups Site based poll or survey Email or Postal Survey Intercept Surveys NPS, CSAT, Service Metrics Tree Testing Session Replay Heat, Click, Scroll maps Open Ended Feedback Device & Viewport Testing Eye / EEG Tracking AB/Split Testing 5 second Preference Card Sorting Expert Eval bit.ly/30QANDQ Optimise Or Die
  • 7. • Why don’t you do more usability testing? • There’s a poll in the webinar settings (top right) • Please pick a poll option and tell me more in the Q&A • What are the biggest reasons? POLL TIME  Optimise Or Die
  • 8. Why don’t you do Usability Testing? • Amazing product & experiment ideas • Usually ‘Evaluative’ – we aren’t asking if people like it, we are testing if they can use it get jobs done • UT goes where opinions or assumptions cannot • Your designs are flawed, you just don’t know where • Once you know what’s wrong, you can fix it • Baymard* is a good guide to what will go wrong • There’s no replacement for user testing Optimise Or Die * https://baymard.com/
  • 9. NO – Usability testing will not tell you: • How much money it will make • If people like the new thing or interface • If the feature/product will be a success • How people would rate it if it was live • What people like about the design • Whether the new colours are nice • If it will work when you make it live • For this, we need Analytics & AB testing Optimise Or Die
  • 10. NO – Usability testing will not tell you: • Don’t ask participants market research type questions – it’s usually too small a sample • ‘How easy was that to use out of 10?’ type questions are useless here • So none of these “Rate the product” type questions – that’s a quant question shoved into a qual study. It’s not a survey, you’re evaluating an interface! • Don’t ask for preferences or future behaviour – you get nothing from this that’s useful • So what can we get then? Optimise Or Die
  • 11. Usability Testing DOES tell you: • If most people can use the feature or product elements they encountered during the test • If they can use it to get tasks, jobs or goals completed • This is all you are testing – useability • Can people use the wish list? Can they find stuff? Can they checkout? Can they use the navigation? • Good usability is highly correlated with high conversion – bugs & usability are the biggest drag on potential conversion, not design Optimise Or Die
  • 13. Hot Takes from Craig on Testing: Optimise Or Die Unmoderated or Moderated? I prefer moderated tests becauseit’s a skilled lens onto evaluations and insights Lab or Remote? Quality of tools is so good,remote works just fine Panel or Self Recruitment? Quality can be poor, so ruthlesslyscreento avoid bias or serialtesting Screening is vital Screento get quality participants = value out. How many people? For work that’s previously been usability tested, 5 quality participants, per roundof testing are good for iterating. For new evaluations, 12-20 dependingon key audiencesegments. Can you use AI? Yes, to take video interviews and transcribe them to text. Then you can usea variety of tools to analyse and map out key topics and theme clusters from the raw text of multiple sessions. No Budget? If you advertise a car, it’s important to make sure it works when peoplego to buy one. If you’re spendingon ads,you need to do the same with your product. Saves $$$ Removing defects from products earlier in the process,before you’ve shippedthe feature or product, saves a ton of money!
  • 14. @OptimiseOrDie Budget User Research Concepts: Usability Hub usabilityhub.com Optimal Workshop optimalworkshop.com Five second test fivesecondtest.com Maze maze.co Five second, Preference, Prototype, First Click, Card Sort, Tree Test Lookback (M,U) lookback.com Userlytics (M,U) userlytics.com Usertesting (M,U) bit.ly/3ZtzWAO Userzoom (M,U) bit.ly/464oQV8 Loop11 (U) loop11.com Userbrain (U) userbrain.com Ethnio (R) ethn.io/ 14
  • 15. Guerrilla Testing WHAT IS GT? maze.co/guides/usability-testing/guerrilla/ TIPS & TRICKS: https://bit.ly/3LBL1dm WHAT DO YOU NEED? • A screen recording tool • Coffee shop, Bar, Conference, Meetup • A script you’ve practised 15 120
  • 16. FIVE SECOND TEST • Did people ‘get’ key messages? • What do they remember? • What is their interpretation? • Did they read the text? Optimise Or Die
  • 17. PREFERENCE TEST • Which design did they prefer? • Why? • Good for competing options Optimise Or Die
  • 18. FIRST CLICK TEST • Can they find something • Is the control visible? • On a page with multiple options, which one would they start with and why? Optimise Or Die
  • 19. CARD SORT • If you want to design navigation or product assortments from scratch, this is the way to figure it out • It’s like mining people’s heads for how they would organise books in a library, clothing inside a store or groceries in a supermarket 1. Users put “things” into piles and give the piles “names” 2. Using the data from multiple users, we extract the structure that maps onto human mental models 3. Solve Navigation, Categorisation and Findability from scratch Optimise Or Die
  • 20. TREE TEST • For testing an existing site or product that’s live • Card sorting to plan a new structure • Tree sorting to optimise an existing one Optimise Or Die
  • 21. PROTOTYPE OR MOCKUP TESTING • Lots of services allow for mockup testing • Usability hub allows you to import Figma flows • Easy way to get quick and cheap feedback • User test later on, use this for early ideas • Saves time & expense fixing defects later Optimise Or Die
  • 22. NEW EMPLOYEES Use new employees (e.g. call centre staff) as your usability testing subjects FRIENDS & FAMILY Use Friends and Family as testing subjects (if they are largely unaware of what you do) SCREEN ALL PARTICIPANTS PROPERLY Make sure you don’t use Web developers, Designers, Journalists, UX people – any ‘content’ or ‘design’ profession HIRE AN EXPERIENCED STUDENT A University near you is running an Usability (HCI) degree. Talk to them about a project for students there! LEARN HOW TO INTERVIEW Not anyone can be a racing driver or a nuclear scientist. Same for being a moderator or interviewer – it takes lots of experience and practice. Took me two years to become even passable as a moderator. Hire someone if you think this is easy to learn or that you are good at it (without critical evaluation). Low Budget Recruitment: Optimise Or Die
  • 23. Interviewing Tips • Unbiased and Useful Usability Research is HARD • No – you can’t fly a 777 if you’ve done flight simulator • No – you can’t just interpret user interviews or tests • Yes – you need years of experience • Read these & Practice: “Don’t Make Me Think” amzn.to/1gIZEJn “Rocket Surgery Made Easy” amzn.to/1e0hnUL “Talking to Customers” bit.ly/1e0hT58 “Talking with Participants” bit.ly/1kKL3LE “Don’t listen to Users” bit.ly/3OVA336 “Interviewing Tips” bit.ly/3yx4aUQ “More interviewing Tips” bit.ly/3dOHI1o 23
  • 24. Usability Testing - Summary • Reach parts of product other tools cannot • No testing is worse than low budget testing • Get awesome product & pivot ideas • Orgs with UT have better experiments! • Problem Statement -> Hypothesis -> Experiment • Understand customer problems better • See flaws from a different perspective Optimise Or Die
  • 25. @OptimiseOrDie WHERE THE ACTION HAPPENS: x x User & Context Device Your Design
  • 27. • A useful and underrated skill • If you don’t know what to do, why not just ask people stuff? • User inquiry is one of the easiest ways to remove BS from work • Triggered polls and feedback buttons are both incredibly useful • OPEN ENDED QUESTIONS are your friend – and really useful to analyse in GPT Optimise Or Die Customer Feedback is GOLD:
  • 28. The Three Greatest Questions Ever: “What’s the purpose of your visit today?” “Were you able to complete your task today?” “If you weren’t able to complete your task today, why not?” Optimise Or Die
  • 29. Feedback, Polls & Surveys “I don’t know what to do” – Why not ask visitors: 1. Big SaaS tool company had a hero landing page 2. Big budget $$$ paid traffic landed here 3. 11 months of AB testing, no shift in Conversion Rate 4. AB tests had failed to find a new winner 5. They had tried nearly everything 6. We added a poll to the page (Purpose of Visit) 7. We figured out they were “Boss Comparison Shopping” 8. We asked them what competitors they were looking at 9. We added a feature comparison for these tools 10. We ran an AB test and it improved CR by 4% Optimise Or Die
  • 30. These are the ONLY types you will need: PURPOSE OF VISIT ON ENTRY Landing page Poll TASK GAP ANALYSIS ENTRY/EXIT Did you get your tasks done? LEAVING REASONS SITE EXIT Exit Survey FUNNEL EXIT ON EXIT Funnel Exit Survey FUNNEL FEEDBACK ALWAYS ON Funnel Specific Feedback THANK YOU PAGE ALWAYS ON “Was there one thing that nearly...” SITE FEEDBACK ALWAYS ON “Any unanswered questions?” PRODUCT PAGE ALWAYS ON “Is there any information missing?” Optimise Or Die
  • 31. Based on today’s visit, how would you rate your site experience overall? Were you able to complete the purpose of your visit today? If NO, why were you not able to? If YES, what did you value most about the website? Thank You Page: “Was there one thing that nearly stopped you from purchasing a flight today?” Task Gap Survey: Optimise Or Die
  • 32. Carglass Thank You: 1. Tells you about customers who nearly made converted, but something stopped them 2. The customers who manage to get through, will also experience the same problems or issues as those who abandon near the end – their motivation or patience was higher 3. Asking people to explain potential abandonment reasons, helps you understand the group who never made it to the end 4. If you resolve the problems, you’ve moved a group from abandoning to converting, permanently 5. Asking for DETAIL or specifics in these questions means you get more & better feedback* * See papers Smith & Jones (2018), Johnson et al. (2019) Optimise Or Die
  • 33. Aim for Open & Inquiry Based Feedback: “What other products or service should we offer?” “How would you describe x to a friend?” “What other options did you consider before looking at x?” “Is there some information that you’re missing here?” “What other products are you comparing?” “Was there one thing that nearly stopped you from purchasing from us today? Please be as detailed as possible.” Optimise Or Die
  • 34. Els Aerts: The Lost Art of Asking Questions https://conversionista.com/wp-content/uploads/2019/10/CJAM-2019-ElsAerts-Sthlm.pdf https://yoast.com/audience-research-expert-interview/ 3. Recommended VOC Tools 2.11 Further Reading Optimise Or Die
  • 35. @OptimiseOrDie Surveys That Work: • Best book on surveys • Caroline knows her stuff • Stop wasting time and • Stop doing biased useless surveys
  • 36. Surveys Tools & Services: • Paperform www.paperform.co • Typeform www.typeform.com • Google Forms www.google.com/forms/about/ • Surveymonkey www.surveymonkey.co.uk • Alchemer www.alchemer.com • Qualtrics www.qualtrics.com • GfK www.gfk.com “Surveys seem like “hard to do” but if you get the right help and iterate them based on data, you get quality insights. If you have no survey or the same one as last year, you’re not listening.” Optimise Or Die
  • 37. @OptimiseOrDie Competitor Surveys • Get 500 responses from your site • Recruit 500 from competitors (ask GFK) • Analyse data and compare “One of the most useful tools during a hyper growth phase was knowing what drove customer delight in terms of service metrics and how this compared to our competitors” Andrew Ground
  • 39. Support or Customer Services: 1. What are the top 3-5 questions or worries you get from potential customers? 2. What do you answer when you get these questions? 3. Are there any particular aspects of X that people don’t understand? 4. What aspects of X do people like the most/least? 5. Did I miss anything important? Got something to add? 6. If you could change anything about X that would help YOU to delight customers, what would the top issues be? Optimise Or Die
  • 40. Sales or Commercial: 1. What are the top 3-5 objections from potential customers? 2. What do you answer when you get these objections? 3. Are there any aspects of X that people don’t understand? 4. What aspects of X do people like the most/least? 5. Did I miss anything important? Got something to add? 6. If you could change anything about X to help sell it more easily, what would it be? Optimise Or Die
  • 41. UX Levers Interviews (EV New Car): General Questions • What has prompted you to consider purchasing an electric vehicle at this point in time? • Have you owned or considered an EV before? If not, what types of vehicles have you previously considered or owned? • How familiar are you with electric vehicles and their technology? • What are your main sources of information when researching new cars? Motivation and Preferences • What are the most important factors for you when choosing a new car (e.g., cost, performance, environmental impact, brand, etc.)? • How does an electric vehicle fit into your lifestyle? • Are there specific features you are looking for in an electric vehicle (e.g., range, charging time, autopilot capabilities)? • What concerns do you have about owning an EV (e.g., charging infrastructure, battery life, maintenance costs)? Decision-making Process • How do you plan to compare different EV models and brands? • Are there any websites that you plan to use for your search? • What would be a deal breaker for you when selecting an EV? • Who else is involved in the decision-making process for purchasing your new car? Optimise Or Die
  • 42. Insight - Inputs Customers do not know what to do and need support and advice • Emphasize the fact that you understand that their situation is stressful • Emphasize your expertise and leadership in vehicle glazing and that you will help them get the best solution for their situation • Explain what they will need to do online and during the call-back so that they know what the next steps will be • Explain that they will be able ask any other questions they might have during the call-back Customers do not feel confident in assessing the damage • Emphasize the fact that you will help them assess the damage correctly online Customers need to understand the benefits of booking online • Emphasize that the online booking system is quick, easy and provides all the information they need in regards with their appointment and general cost information Customers mistrust insurers and find dealing with their insurance situation very frustrating • Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for money for cash customers • Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their insurance paperwork for them, freeing them of this burden Some customers cannot be bothered to take action to fix their car glass • Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’ UX Levers – Vehicle Glass Repair Example
  • 43. High Value Customers + Unhappy Customers: • Reverse engineer delight by studying these groups • Engage with unhappy customers (reviews, customer services or special team to handle low review scores) • Interview loyal or high spending customers • Find out why they are delighted/unhappy • Use questions but let them tell you their story: ✔ Ask good questions ✔ Shut up ✔ Listen ✔ Repeat Optimise Or Die
  • 44. Interviewing Tips Want to interview better? Read these: “Don’t Make Me Think” amzn.to/1gIZEJn “Rocket Surgery Made Easy” amzn.to/1e0hnUL “Talking to Customers” bit.ly/1e0hT58 “Talking with Participants” bit.ly/1kKL3LE “Don’t listen to Users” bit.ly/3OVA336 “Interviewing Tips” bit.ly/3yx4aUQ 44
  • 46. Session Replay – what’s useful? 1. Good quality recordings, with superb filtering 2. Scroll Maps sometimes useful 3. Heat Maps are not useful 4. Click Maps are useful for only one thing 5. A ‘Half Way House’ between Analytics & User Testing 6. Immersion in the negative side, struggle & frustration Optimise Or Die
  • 47. Heat, Scroll and Click Maps PROS: Models attention CONS: • Weak correlation between mouse & eye gaze • Not useful for touch devices PROS: • Map scroll depth • Find attention ‘Dead Zones’ CONS • Does it map to your design? PROS: • See clicks that didn’t work CONS • Useless for anything else
  • 48. Session Replay Recordings PROS • Get Video recordings of real users • Spot problems, struggle, bugs, device issues, errors or confusion • Useful for checking AB test variants CONS • NOT a substitute for analytics IMO • Hard to ‘view’ thousands of recordings • Does hit site performance (check) • It’s always an abstraction of reality* Optimise Or Die
  • 49. 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% 50.00% Case Study – Abandonment & Session Replay Postcode drop Delivery drop Payment drop
  • 50. So what did I do? 1. Pulled 400 “Failure” recordings of the payment page 2. Got a piece of paper and made a list 3. I put people who “Looked and Left” in one pile 4. I put people who “Did something and left” into other piles 5. I put errors into specific piles too 6. 3 hours work – what does data say? Optimise Or Die
  • 51. Session Replay Failure Marking: Reason % 1. An error occurred - click start your subscription again (PayPal)* 22% 2. Didn't start payment interaction or exited PayPal 21% 3. Payment processing spinner 'hangs’* 17% 4. No idea for the reason 12% 5. Discount removed - already been used (house/account)* 11% 6. Opened order summary & left 5% 7. Email already in use 4% 8. Can't login on payment page 3% 9. Insufficient funds 2% 10. Bank declined payment 1% 11. Animation breaks page 1% 12. Promo code invalid 1% • The two in orange are people “Looked and Left” • The other issues are all ones we can solve or mitigate • 40% of exits are down to 2 payment failure bugs • 11% of failures are because the promo code is asked for too late and rejected only at payment • 4% are rejected because their email is already in use but there’s no easy way to log in • This short piece of work nailed a a 750k a month loss in payment fails! Optimise Or Die
  • 52. Tip – Failure Marking bit.ly/RainbowTomer • You can use Tomer Sharon’s rainbow spreadsheet for marking usability sessions or screen recordings bit.ly/RainbowTomer Optimise Or Die
  • 54. Most companies don’t VALIDATE these: Easy Stuff ☺ I can use it I can read it Loads and runs quickly Works on my device Optimise Or Die
  • 55. Craig’s Hierarchy of Product Foundations Easy Stuff ☺ Usability Accessibility Performance Device Compatibility Optimise Or Die
  • 56. Device Testing • One of the simplest ways to find problems • Every site has a problem with devices & screens • Companies pay for ‘Responsive’ websites • What does this mean though? Stretchy? Fluid? Working? • Responsive: means ‘responds’ to scale images, text and design on devices that visitors or customers have • Evidence – if you say it is responsive, we should prove it Optimise Or Die
  • 57. 5.4” iPhone (12 mini) 6.1” iPhone (various) 6.2” Android (Samsung S2x) Mac 13” Laptop (1440x900) Win 13” Laptop (1366x768) 2 year old iPad Minimum Viable Testing List (85-95% traffic) Optimise Or Die
  • 58. @OptimiseOrDie £320 iPhone 12 mini (Amazon) £210 iPhone XR (Amazon) £230 Samsung S22 (Cex) £240 Windows Laptop (Amazon) £410 Mac Laptop (eBay) Cost of Testing Hardware ~ 1.4K
  • 59. @OptimiseOrDie Desktop Browsers: Crossbrowsertesting.com Browserstack.com bit.ly/MobileDeviceLab Device Testing in the Cloud: Mobile & Tablet Cloud: Aws.amazon.com/device-farm Browserstack.com/app-live Lambdatest.com Perfectomobile.com & Kobiton.com No Hardware? Friends & Family!
  • 60. Device Testing Summary: • Your “dev testing list” is probably wrong or out of date • Use your analytics data to define what to test • Don’t use emulators or simulators, not worth it • For core devices (85-95% of audience) get real kit • Cloud devices (AWS, Browserstack) are okay for desktop or non core devices • Buy the hardware, don’t cut corners Optimise Or Die
  • 61. 6 AI ASSIST FOR RESEARCH & DISCOVERY Optimise Or Die
  • 62. AI in the back seat: HELPING YOU with: ● Market & Product Research ● Ideation & Prioritisation ● Wireframing & Design ● Understanding Markets ● Persuasive copywriting ● LPO & SEO ● What customers are saying ● What words and phrases they use Photo by Pexels
  • 63. Real Use Cases that work: ● Summarising text (reviews, chat, anything) ● Sentiment Analysis ● Copy, Microcopy, UX writing research ● Market and Product Research ● Competitor Analysis & Feature Comparison ● White paper research (scite, elicit) ● Personas, Journeys, User Stories help ● Moderator scripts, use cases, tasks ● Design patterns or anti-patterns ● Video or Speech transcribing ● Note taking for interviews ● Clustering notes or interview feedback Photo by Pexels
  • 65. Experimentation AI FTW: VWO sponsors Experimentation Elite: 11-12 June, Birmingham • Attend my AI Playbook workshop: experimentationelite.com/all-events/ • Workshops for Product Managers & Leaders – get in touch • 12 week Maven Course for Experimenters – on the way! Optimise Or Die
  • 66. Thank you! Craig Sullivan CEO, Optimise Or Die Ltd. sullivac@gmail.com +447711 657315 linkedin.com/in/craigsullivan @OptimiseOrDie Optimise Or Die Will be sent to you! Slides: