You can utilize various forms of Generative Research to deepen your understanding of how people interact with your product or service. Craig has amassed a vast toolkit of research methods, which he has employed to optimize websites and apps for over 500 companies. He'll share which methods yielded the highest return on investment, identified key customer pain points, and generated the best experiment ideas. By sharing the top inspection methods essential for our work, Craig will provide advice for each technique. Anticipate insights on driving experiment hypotheses from research, a list of essential toolkit components for tomorrow, and additional resources for further reading.

1. Research and
Discovery Tools for
Experimentation
1
Optimise Or Die

2. © Optimal Visit 2d018
INTRODUCTION
• UX Research
• Analytics data
• Toolkits
• AB Testing
@OptimiseOrDie
linkd.in/pvrg14
Optimise Or Die

3. 1. TOP TECHNIQUES
2. WHY ARE THEY USEFUL?
3. HOW TO USE THEM
4. CASE STUDY
5. AI AUGMENTATION
Higher Quality Research Inputs:
Optimise Or Die

4. Behavioural
(Observation)
Attitudinal (Opinion)
Qualitative Quantitative
Analytics
Event Data
Lab
Usability
Testing
Unmoderate
d Remote
Testing
Remote
Moderated
Testing
Diary / Field
Contextual
Shadowing
Guerrilla
Testing
User
Interviews
Participatory
Design
Focus
Groups
Site based
poll or
survey
Email or
Postal
Survey
Intercept
Surveys
NPS, CSAT,
Service
Metrics
Tree
Testing
Session
Replay
Heat, Click,
Scroll maps
Open Ended
Feedback
Device &
Viewport
Testing
Eye / EEG
Tracking
AB/Split
Testing
5 second
Preference
Card
Sorting
Expert
Eval
Optimise Or Die

5. Behavioural
(Observation)
Attitudinal (Opinion)
Qualitative Quantitative
Analytics
Event Data
Lab
Usability
Testing
Unmoderate
d Remote
Testing
Remote
Moderated
Testing
Diary / Field
Contextual
Shadowing
Guerrilla
Testing
User
Interviews
Participatory
Design
Focus
Groups
Site based
poll or
survey
Email or
Postal
Survey
Intercept
Surveys
NPS, CSAT,
Service
Metrics
Tree
Testing
Session
Replay
Heat, Click,
Scroll maps
Open Ended
Feedback
Device &
Viewport
Testing
Eye / EEG
Tracking
AB/Split
Testing
5 second
Preference
Card
Sorting
Expert
Eval
bit.ly/30QANDQ Optimise Or Die

6. 1
Optimise Or Die

7. • Why don’t you do more usability testing?
• There’s a poll in the webinar settings (top right)
• Please pick a poll option and tell me more in the Q&A
• What are the biggest reasons?
POLL TIME 
Optimise Or Die

8. Why don’t you do Usability Testing?
• Amazing product & experiment ideas
• Usually ‘Evaluative’ – we aren’t asking if people like
it, we are testing if they can use it get jobs done
• UT goes where opinions or assumptions cannot
• Your designs are flawed, you just don’t know where
• Once you know what’s wrong, you can fix it
• Baymard* is a good guide to what will go wrong
• There’s no replacement for user testing
Optimise Or Die
* https://baymard.com/

9. NO – Usability testing will not tell you:
• How much money it will make
• If people like the new thing or interface
• If the feature/product will be a success
• How people would rate it if it was live
• What people like about the design
• Whether the new colours are nice
• If it will work when you make it live
• For this, we need Analytics & AB testing
Optimise Or Die

10. NO – Usability testing will not tell you:
• Don’t ask participants market research type questions –
it’s usually too small a sample
• ‘How easy was that to use out of 10?’ type questions are
useless here
• So none of these “Rate the product” type questions –
that’s a quant question shoved into a qual study. It’s not a
survey, you’re evaluating an interface!
• Don’t ask for preferences or future behaviour – you get
nothing from this that’s useful
• So what can we get then?
Optimise Or Die

11. Usability Testing DOES tell you:
• If most people can use the feature or product elements
they encountered during the test
• If they can use it to get tasks, jobs or goals completed
• This is all you are testing – useability
• Can people use the wish list? Can they find stuff? Can
they checkout? Can they use the navigation?
• Good usability is highly correlated with high conversion –
bugs & usability are the biggest drag on potential
conversion, not design
Optimise Or Die

12. bit.ly/6minuteUsability
Els Aerts

13. Hot Takes from Craig on Testing:
Optimise Or Die
Unmoderated or
Moderated?
I prefer moderated tests becauseit’s a skilled
lens onto evaluations and insights
Lab or Remote?
Quality of tools is so good,remote works
just fine
Panel or Self
Recruitment?
Quality can be poor, so ruthlesslyscreento
avoid bias or serialtesting
Screening is vital
Screento get quality participants =
value out.
How many people?
For work that’s previously been usability tested, 5
quality participants, per roundof testing are good
for iterating. For new evaluations, 12-20
dependingon key audiencesegments.
Can you use AI?
Yes, to take video interviews and transcribe
them to text. Then you can usea variety of
tools to analyse and map out key topics
and theme clusters from the raw text of
multiple sessions.
No Budget?
If you advertise a car, it’s important to make
sure it works when peoplego to buy one. If
you’re spendingon ads,you need to do the same
with your product.
Saves $$$
Removing defects from products
earlier in the process,before you’ve
shippedthe feature or product, saves
a ton of money!

14. @OptimiseOrDie
Budget User Research
Concepts:
Usability Hub usabilityhub.com
Optimal Workshop optimalworkshop.com
Five second test fivesecondtest.com
Maze maze.co
Five second, Preference, Prototype, First Click, Card Sort, Tree Test
Lookback (M,U) lookback.com
Userlytics (M,U) userlytics.com
Usertesting (M,U) bit.ly/3ZtzWAO
Userzoom (M,U) bit.ly/464oQV8
Loop11 (U) loop11.com
Userbrain (U) userbrain.com
Ethnio (R) ethn.io/
14

15. Guerrilla Testing
WHAT IS GT?
maze.co/guides/usability-testing/guerrilla/
TIPS & TRICKS:
https://bit.ly/3LBL1dm
WHAT DO YOU NEED?
• A screen recording tool
• Coffee shop, Bar, Conference, Meetup
• A script you’ve practised
15
120

16. FIVE SECOND TEST
• Did people ‘get’ key messages?
• What do they remember?
• What is their interpretation?
• Did they read the text?
Optimise Or Die

17. PREFERENCE TEST
• Which design did they prefer?
• Why?
• Good for competing options
Optimise Or Die

18. FIRST CLICK TEST
• Can they find something
• Is the control visible?
• On a page with multiple
options, which one would they
start with and why?
Optimise Or Die

19. CARD SORT
• If you want to design navigation or product assortments from
scratch, this is the way to figure it out
• It’s like mining people’s heads for how they would organise books
in a library, clothing inside a store or groceries in a supermarket
1. Users put “things” into piles and give the piles “names”
2. Using the data from multiple users, we extract the structure
that maps onto human mental models
3. Solve Navigation, Categorisation and Findability from scratch
Optimise Or Die

20. TREE TEST
• For testing an existing site or product that’s live
• Card sorting to plan a new structure
• Tree sorting to optimise an existing one
Optimise Or Die

21. PROTOTYPE OR MOCKUP TESTING
• Lots of services allow for mockup testing
• Usability hub allows you to import Figma flows
• Easy way to get quick and cheap feedback
• User test later on, use this for early ideas
• Saves time & expense fixing defects later
Optimise Or Die

22. NEW EMPLOYEES
Use new employees (e.g. call centre staff) as your usability testing subjects
FRIENDS & FAMILY
Use Friends and Family as testing subjects (if they are largely unaware of what you do)
SCREEN ALL PARTICIPANTS PROPERLY
Make sure you don’t use Web developers, Designers, Journalists, UX people – any ‘content’ or ‘design’ profession
HIRE AN EXPERIENCED STUDENT
A University near you is running an Usability (HCI) degree. Talk to them about a project for students there!
LEARN HOW TO INTERVIEW
Not anyone can be a racing driver or a nuclear scientist. Same for being a moderator or interviewer – it takes lots of
experience and practice. Took me two years to become even passable as a moderator. Hire someone if you think this
is easy to learn or that you are good at it (without critical evaluation).
Low Budget Recruitment:
Optimise Or Die

23. Interviewing Tips
• Unbiased and Useful Usability Research is HARD
• No – you can’t fly a 777 if you’ve done flight simulator
• No – you can’t just interpret user interviews or tests
• Yes – you need years of experience
• Read these & Practice:
“Don’t Make Me Think” amzn.to/1gIZEJn
“Rocket Surgery Made Easy” amzn.to/1e0hnUL
“Talking to Customers” bit.ly/1e0hT58
“Talking with Participants” bit.ly/1kKL3LE
“Don’t listen to Users” bit.ly/3OVA336
“Interviewing Tips” bit.ly/3yx4aUQ
“More interviewing Tips” bit.ly/3dOHI1o
23

24. Usability Testing - Summary
• Reach parts of product other tools cannot
• No testing is worse than low budget testing
• Get awesome product & pivot ideas
• Orgs with UT have better experiments!
• Problem Statement -> Hypothesis -> Experiment
• Understand customer problems better
• See flaws from a different perspective
Optimise Or Die

25. @OptimiseOrDie
WHERE THE ACTION HAPPENS:
x x
User &
Context
Device
Your Design

26. 2
SURVEYS
POLLS
FEEDBACK
Optimise Or Die

27. • A useful and underrated skill
• If you don’t know what to do, why
not just ask people stuff?
• User inquiry is one of the easiest
ways to remove BS from work
• Triggered polls and feedback
buttons are both incredibly useful
• OPEN ENDED QUESTIONS are
your friend – and really useful to
analyse in GPT
Optimise Or Die
Customer Feedback is GOLD:

28. The Three Greatest Questions Ever:
“What’s the purpose of your visit today?”
“Were you able to complete your task
today?”
“If you weren’t able to complete your task
today, why not?”
Optimise Or Die

29. Feedback, Polls & Surveys
“I don’t know what to do” – Why not ask visitors:
1. Big SaaS tool company had a hero landing page
2. Big budget $$$ paid traffic landed here
3. 11 months of AB testing, no shift in Conversion Rate
4. AB tests had failed to find a new winner
5. They had tried nearly everything
6. We added a poll to the page (Purpose of Visit)
7. We figured out they were “Boss Comparison Shopping”
8. We asked them what competitors they were looking at
9. We added a feature comparison for these tools
10. We ran an AB test and it improved CR by 4%
Optimise Or Die

30. These are the ONLY types you will need:
PURPOSE OF VISIT ON ENTRY Landing page Poll
TASK GAP ANALYSIS ENTRY/EXIT Did you get your tasks done?
LEAVING REASONS SITE EXIT Exit Survey
FUNNEL EXIT ON EXIT Funnel Exit Survey
FUNNEL FEEDBACK ALWAYS ON Funnel Specific Feedback
THANK YOU PAGE ALWAYS ON “Was there one thing that nearly...”
SITE FEEDBACK ALWAYS ON “Any unanswered questions?”
PRODUCT PAGE ALWAYS ON “Is there any information missing?”
Optimise Or Die

31. Based on
today’s visit,
how would you
rate your site
experience
overall?
Were you able
to complete
the purpose of
your visit
today?
If NO, why
were you not
able to?
If YES, what
did you value
most about
the website?
Thank You Page:
“Was there one thing that nearly stopped you from purchasing a flight today?”
Task Gap Survey:
Optimise Or Die

32. Carglass Thank You:
1. Tells you about customers who nearly made
converted, but something stopped them
2. The customers who manage to get through, will
also experience the same problems or issues as
those who abandon near the end – their
motivation or patience was higher
3. Asking people to explain potential
abandonment reasons, helps you understand
the group who never made it to the end
4. If you resolve the problems, you’ve moved a
group from abandoning to converting,
permanently
5. Asking for DETAIL or specifics in these questions
means you get more & better feedback*
* See papers Smith & Jones (2018), Johnson et al. (2019)
Optimise Or Die

33. Aim for Open & Inquiry Based Feedback:
“What other products or service should we offer?”
“How would you describe x to a friend?”
“What other options did you consider before looking at x?”
“Is there some information that you’re missing here?”
“What other products are you comparing?”
“Was there one thing that nearly stopped you from purchasing from
us today? Please be as detailed as possible.”
Optimise Or Die

34. Els Aerts: The Lost Art of Asking Questions
https://conversionista.com/wp-content/uploads/2019/10/CJAM-2019-ElsAerts-Sthlm.pdf
https://yoast.com/audience-research-expert-interview/
3. Recommended VOC Tools
2.11 Further Reading
Optimise Or Die

35. @OptimiseOrDie
Surveys That Work:
• Best book on surveys
• Caroline knows her stuff
• Stop wasting time and
• Stop doing biased useless surveys

36. Surveys Tools & Services:
• Paperform www.paperform.co
• Typeform www.typeform.com
• Google Forms www.google.com/forms/about/
• Surveymonkey www.surveymonkey.co.uk
• Alchemer www.alchemer.com
• Qualtrics www.qualtrics.com
• GfK www.gfk.com
“Surveys seem like “hard to do” but if you get the right help and
iterate them based on data, you get quality insights. If you have no
survey or the same one as last year, you’re not listening.”
Optimise Or Die

37. @OptimiseOrDie
Competitor Surveys
• Get 500 responses from your site
• Recruit 500 from competitors (ask GFK)
• Analyse data and compare
“One of the most useful tools during a hyper growth
phase was knowing what drove customer delight in
terms of service metrics and how this compared to our
competitors”
Andrew Ground

38. 3 INTERVIEWING
PEOPLE
Optimise Or Die

39. Support or Customer Services:
1. What are the top 3-5 questions or worries you get from
potential customers?
2. What do you answer when you get these questions?
3. Are there any particular aspects of X that people don’t
understand?
4. What aspects of X do people like the most/least?
5. Did I miss anything important? Got something to add?
6. If you could change anything about X that would help
YOU to delight customers, what would the top issues be?
Optimise Or Die

40. Sales or Commercial:
1. What are the top 3-5 objections from potential
customers?
2. What do you answer when you get these objections?
3. Are there any aspects of X that people don’t understand?
4. What aspects of X do people like the most/least?
5. Did I miss anything important? Got something to add?
6. If you could change anything about X to help sell it more
easily, what would it be?
Optimise Or Die

41. UX Levers Interviews (EV New Car):
General Questions
• What has prompted you to consider purchasing an electric vehicle at this point in time?
• Have you owned or considered an EV before? If not, what types of vehicles have you
previously considered or owned?
• How familiar are you with electric vehicles and their technology?
• What are your main sources of information when researching new cars?
Motivation and Preferences
• What are the most important factors for you when choosing a new car (e.g., cost,
performance, environmental impact, brand, etc.)?
• How does an electric vehicle fit into your lifestyle?
• Are there specific features you are looking for in an electric vehicle (e.g., range, charging
time, autopilot capabilities)?
• What concerns do you have about owning an EV (e.g., charging infrastructure, battery life,
maintenance costs)?
Decision-making Process
• How do you plan to compare different EV models and brands?
• Are there any websites that you plan to use for your search?
• What would be a deal breaker for you when selecting an EV?
• Who else is involved in the decision-making process for purchasing your new car?
Optimise Or Die

42. Insight - Inputs
Customers do not know what to do and need support and advice
• Emphasize the fact that you understand that their situation is stressful
• Emphasize your expertise and leadership in vehicle glazing and that you will help them get the best solution for
their situation
• Explain what they will need to do online and during the call-back so that they know what the next steps will be
• Explain that they will be able ask any other questions they might have during the call-back
Customers do not feel confident in assessing the damage
• Emphasize the fact that you will help them assess the damage correctly online
Customers need to understand the benefits of booking online
• Emphasize that the online booking system is quick, easy and provides all the information they need in regards
with their appointment and general cost information
Customers mistrust insurers and find dealing with their insurance situation very frustrating
• Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for
money for cash customers
• Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their
insurance paperwork for them, freeing them of this burden
Some customers cannot be bothered to take action to fix their car glass
• Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’
UX Levers – Vehicle Glass Repair Example

43. High Value Customers + Unhappy Customers:
• Reverse engineer delight by studying these groups
• Engage with unhappy customers (reviews, customer
services or special team to handle low review scores)
• Interview loyal or high spending customers
• Find out why they are delighted/unhappy
• Use questions but let them tell you their story:
✔ Ask good questions
✔ Shut up
✔ Listen
✔ Repeat
Optimise Or Die

44. Interviewing Tips
Want to interview better? Read these:
“Don’t Make Me Think” amzn.to/1gIZEJn
“Rocket Surgery Made Easy” amzn.to/1e0hnUL
“Talking to Customers” bit.ly/1e0hT58
“Talking with Participants” bit.ly/1kKL3LE
“Don’t listen to Users” bit.ly/3OVA336
“Interviewing Tips” bit.ly/3yx4aUQ
44

45. 4 SESSION REPLAY
Optimise Or Die

46. Session Replay – what’s useful?
1. Good quality recordings, with superb filtering
2. Scroll Maps sometimes useful
3. Heat Maps are not useful
4. Click Maps are useful for only one thing
5. A ‘Half Way House’ between Analytics & User Testing
6. Immersion in the negative side, struggle & frustration
Optimise Or Die

47. Heat, Scroll and Click Maps
PROS: Models attention
CONS:
• Weak correlation between
mouse & eye gaze
• Not useful for touch devices
PROS:
• Map scroll depth
• Find attention ‘Dead Zones’
CONS
• Does it map to your design?
PROS:
• See clicks that didn’t work
CONS
• Useless for anything else

48. Session Replay Recordings
PROS
• Get Video recordings of real users
• Spot problems, struggle, bugs, device
issues, errors or confusion
• Useful for checking AB test variants
CONS
• NOT a substitute for analytics IMO
• Hard to ‘view’ thousands of recordings
• Does hit site performance (check)
• It’s always an abstraction of reality*
Optimise Or Die

49. 0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
Case Study – Abandonment & Session Replay
Postcode drop
Delivery drop
Payment drop

50. So what did I do?
1. Pulled 400 “Failure” recordings of the payment page
2. Got a piece of paper and made a list
3. I put people who “Looked and Left” in one pile
4. I put people who “Did something and left” into other piles
5. I put errors into specific piles too
6. 3 hours work – what does data say?
Optimise Or Die

51. Session Replay Failure Marking:
Reason %
1. An error occurred - click start your subscription again
(PayPal)*
22%
2. Didn't start payment interaction or exited PayPal 21%
3. Payment processing spinner 'hangs’* 17%
4. No idea for the reason 12%
5. Discount removed - already been used (house/account)* 11%
6. Opened order summary & left 5%
7. Email already in use 4%
8. Can't login on payment page 3%
9. Insufficient funds 2%
10. Bank declined payment 1%
11. Animation breaks page 1%
12. Promo code invalid 1%
• The two in orange are people
“Looked and Left”
• The other issues are all ones we
can solve or mitigate
• 40% of exits are down to 2
payment failure bugs
• 11% of failures are because the
promo code is asked for too late
and rejected only at payment
• 4% are rejected because their
email is already in use but there’s
no easy way to log in
• This short piece of work nailed a
a 750k a month loss in payment
fails!
Optimise Or Die

52. Tip – Failure Marking bit.ly/RainbowTomer
• You can use Tomer
Sharon’s rainbow
spreadsheet for marking
usability sessions or
screen recordings
bit.ly/RainbowTomer
Optimise Or Die

53. 5
DEVICE TESTING

54. Most companies don’t VALIDATE these:
Easy
Stuff ☺
I can use it
I can read it
Loads and runs quickly
Works on my device
Optimise Or Die

55. Craig’s Hierarchy of Product Foundations
Easy
Stuff ☺
Usability
Accessibility
Performance
Device Compatibility
Optimise Or Die

56. Device Testing
• One of the simplest ways to find problems
• Every site has a problem with devices & screens
• Companies pay for ‘Responsive’ websites
• What does this mean though? Stretchy? Fluid? Working?
• Responsive: means ‘responds’ to scale images, text and
design on devices that visitors or customers have
• Evidence – if you say it is responsive, we should prove it
Optimise Or Die

57. 5.4” iPhone (12 mini)
6.1” iPhone (various)
6.2” Android (Samsung S2x)
Mac 13” Laptop (1440x900)
Win 13” Laptop (1366x768)
2 year old iPad
Minimum Viable Testing List (85-95% traffic)
Optimise Or Die

58. @OptimiseOrDie
£320 iPhone 12 mini (Amazon)
£210 iPhone XR (Amazon)
£230 Samsung S22 (Cex)
£240 Windows Laptop (Amazon)
£410 Mac Laptop (eBay)
Cost of Testing Hardware ~ 1.4K

59. @OptimiseOrDie
Desktop Browsers:
Crossbrowsertesting.com
Browserstack.com
bit.ly/MobileDeviceLab
Device Testing in the Cloud:
Mobile & Tablet Cloud:
Aws.amazon.com/device-farm
Browserstack.com/app-live
Lambdatest.com
Perfectomobile.com & Kobiton.com
No Hardware?
Friends & Family!

60. Device Testing Summary:
• Your “dev testing list” is probably wrong or out of date
• Use your analytics data to define what to test
• Don’t use emulators or simulators, not worth it
• For core devices (85-95% of audience) get real kit
• Cloud devices (AWS, Browserstack) are okay for desktop or
non core devices
• Buy the hardware, don’t cut corners
Optimise Or Die

61. 6
AI ASSIST FOR
RESEARCH &
DISCOVERY
Optimise Or Die

62. AI in the back seat:
HELPING YOU with:
● Market & Product Research
● Ideation & Prioritisation
● Wireframing & Design
● Understanding Markets
● Persuasive copywriting
● LPO & SEO
● What customers are saying
● What words and phrases they
use
Photo by Pexels

63. Real Use Cases that
work:
● Summarising text (reviews, chat, anything)
● Sentiment Analysis
● Copy, Microcopy, UX writing research
● Market and Product Research
● Competitor Analysis & Feature Comparison
● White paper research (scite, elicit)
● Personas, Journeys, User Stories help
● Moderator scripts, use cases, tasks
● Design patterns or anti-patterns
● Video or Speech transcribing
● Note taking for interviews
● Clustering notes or interview feedback
Photo by Pexels

64. Photo by Pexels
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4708466

65. Experimentation AI FTW:
VWO sponsors Experimentation Elite: 11-12 June, Birmingham
• Attend my AI Playbook workshop:
experimentationelite.com/all-events/
• Workshops for Product Managers & Leaders – get in touch
• 12 week Maven Course for Experimenters – on the way!
Optimise Or Die

66. Thank you!
Craig Sullivan
CEO, Optimise Or Die Ltd.
sullivac@gmail.com
+447711 657315
linkedin.com/in/craigsullivan
@OptimiseOrDie
Optimise Or Die
Will be sent to you!
Slides: