This document discusses how confirmation bias can negatively impact product experimentation by lowering success rates. It describes how teams often look for data to support their existing hypotheses rather than using discovery-first approaches. This "Forcing Insights and Learnings to Hypotheses" or "FILTH" practice misses opportunities. The document advocates conducting regular discovery research through methods like usability testing to identify problems and opportunities before ideating solutions. Tracking the origin and success rates of ideas can help teams move away from FILTH and improve experimentation outcomes.

1. Adopting a discovery-first
approach to product
experimentation to drive
higher success rates
Slideshare.Net/ChrisGibbins6
@cjgibbins
Chris Gibbins
Creative CX, The Experimentation Consultancy
https://www.linkedin.com/in/chris-gibbins/

3. The flag is waving
in the wind?!

4. The flag is waving
in the wind?!
There are no
stars?!

5. “We Never Went to the Moon” - Bill Kaysing’s conspiracy theory
The flag is waving
in the wind?!
There are no
stars?!

6. Diamonds Are Forever (1971)
It must have been staged
on a film set.

7. “We Never Went to the Moon” - Bill Kaysing’s conspiracy theory
The flag is waving
in the wind?!
There are no
stars?!
Flag has a crossbar
which was a little
too short

8. “We Never Went to the Moon” - Bill Kaysing’s conspiracy theory
The flag is waving
in the wind?!
There are no
stars?!
Flag has a crossbar
which was a little
too short
No visible stars
because of the
camera shutter
speed

9. “We Never Went to the Moon” - Bill Kaysing’s conspiracy theory
The flag is waving
in the wind?!
There are no
stars?!
Flag has a crossbar
which was a little
too short
No visible stars
because of the
camera shutter
speed
EVIDENCE
THEY IGNORED

12. A key contributing factor here is:
Confirmation Bias

13. EVIDENCE
THEY IGNORE
Facts &
Evidence
EVIDENCE
THEY BELIEVE
A crazy
idea

14. But it doesn’t just affect Conspiracy
Theorists and their followers

15. It’s a big problem in
Experimentation too…

16. The confirmation bias we see
everywhere:
1.Team has an
Idea & Hypothesis
2. Look for insights
& learnings to
back-up their idea
3. A/B test

17. “Full screen auto-playing video on product
details pages, wow that would be very cool”

18. “We’re data-driven, so we need to find
some data insights and learnings to
back up our idea”

19. INSIGHT 1
BEHAVIOURAL
ANALYTICS:
People who watch the product
videos have a conversion rate
that is 4x higher than those
that don’t!

20. INSIGHT 1
BEHAVIOURAL
ANALYTICS:
People who watch the product
videos have a conversion rate
that is 4x higher than those
that don’t!
“Reverse
causation”
in action!

21. INSIGHT 2
SURVEY
FINDING:
15% commented that the
videos were most important to
them

22. INSIGHT 2
SURVEY
FINDING:
15% commented that the
videos were most important to
them
What
about the
other 85%?

23. EVIDENCE
THEY IGNORE
Facts &
Evidence
EVIDENCE
THEY BELIEVE
A cool idea
they really
want to
A/B test

24. Why do we see this everywhere?

25. 1. Teams know they
need data to get
their ideas
prioritised!

26. 1. Teams know they
need data to get
their ideas
prioritised!

27. 2. Data is so much more accessible
these days, which is a good thing

28. But it makes this confirmation bias
MUCH more prevalent!

29. 3. Jumping to conclusions is a human
condition (and another bias)

30. 4. Lack of resource for discovery
UX Research teams too busy:
- Validating new ideas
Data teams too busy:
- Creating reports
- Cleaning data
- Finding data insights to support
existing ideas!

31. Consequences?

32. Teams miss out on so many potentially
impactful discoveries

33. They’re not really DISCOVERING
anything new…

34. Chasing their
own tails….
and their
competitors

35. VERY
LOW
SUCCESS
RATES
*Source: RONNY KOHAVI
Based on primary business metric
33%* Microsoft, Intuit Quicken
10%* Booking, Google Ads, Netflix
Bing 15%*
AirBnB Search 8%*
0%
100%
A/B TEST WIN RATES

36. So to help raise awareness of this bias

37. We had a Creative CX competition
to come up with a name

38. Forcing Insights & Learnings To Hypotheses
Tom Painter,
Senior Strategy Consultant

39. Forcing Insights & Learnings To Hypotheses
Tom Painter,
Senior Strategy Consultant

40. FILTH

41. So what’s the answer?
How do teams improve the success
rates of their solutions?

42. Move away from this FILTHy practice:
1.Team has an
Idea & Hypothesis
2. Look for insights
& learnings to
back-up their idea
3. A/B test

43. Take a Discovery-first approach
1.Discovery 2. Ideation 3. A/B test

44. Take a Discovery-first approach
FILTH

45. Example insight from our
Discovery-first approach

46. No-one thought
there was an
opportunity on
the Travelodge
login step

47. Prominent CTA’s
✓
Clear-ish booking summary
✓
Secure checkout padlock
✓
“Guest checkout” available
✓
73% Progress to the next step
18% Go back to basket (previous step)
9% Exit rate
Low drop-off rate
✓

48. After one of our Opportunity Audits

49. Opportunity Audit (Discovery Research)
High
Priority
Medium
Priority
Low
Priority
Total (by
site area)
Sitewide 0 2 1 3
Homepage 0 3 2 5
Search / Nav 1 8 7 16
Search Results 2 9 5 16
HDP 12 16 7 35
Extras 1 10 4 15
Basket 0 1 0 1
Login 1 1 0 2
Checkout 3 2 3 8
Total (by priority) 20 52 29 101
101 Problems /
Opportunities
61 A/B tests
5 Perso ideas
2 Initiatives
26 Fixes / JDIs
Recommendations
User Research,
Data Analysis,
Ideation

50. Opportunity Audit (Discovery Research)
High
Priority
Medium
Priority
Low
Priority
Total (by
site area)
Sitewide 0 2 1 3
Homepage 0 3 2 5
Search / Nav 1 8 7 16
Search Results 2 9 5 16
HDP 12 16 7 35
Extras 1 10 4 15
Basket 0 1 0 1
Login 1 1 0 2
Checkout 3 2 3 8
Total (by priority) 20 52 29 101
101 Problems /
Opportunities
61 A/B tests
5 Perso ideas
2 Initiatives
26 Fixes / JDIs
Recommendations
User Research,
Data Analysis,
Ideation

51. One of the Usability Testing findings
Mark,
participant 7,
who regularly
books budget
hotels

52. The Aha moment!
Mark,
participant 7,
who regularly
books budget
hotels

53. “Checkout as guest”
not the best
terminology for a
hotel site!

54. We wouldn’t have found this without
a Discovery-first approach

55. CONTROL
VARIATION 4 -
Winning Variation
Significant uplift in
Booking Conversion Rate
(99% stat-sig)

56. If you have moonshot ambitions for
experimentation…

57. Avoid this FILTHy practice of
fake-data-drivenness!

58. Summary
Conduct regular
Discovery
Research &
Analysis projects
throughout the
year.
1.
Follow-up with
problem-
focussed Ideation
workshops to
ensure a greater %
ideas are
data-driven.
2.
Make sure your
prioritisation
framework gives
higher scores to
data-driven ideas
than FILTHy ones!
3.
Track where the
ideas come from &
their success rates
when A/B tested.
4.
Origin:
Usability Testing
Supported:
Analytics
Origin:
Call centre
Origin:
Usability Testing

59. UNDERSTAND
your customers, products & services
EXPERIMENT
to optimise, personalise & innovate
THANK YOU
SCALE
experimentation across the organisation
Slideshare.Net/ChrisGibbins6
@cjgibbins
https://www.linkedin.com/in/chris-gibbins/