1. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Experimentation is a reality not a culture,
are you ready for it?
Andrew Furlong, Managing Director RedEye Optimisation

2. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
• A little bit about me & RedEye
• Why AB testing isn’t enough these days
• The displace model & Culture
• Change curve
• Key Takeaways
Agenda

3. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
An introduction…
Andrew Furlong

4. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
CRO UX Design
Customer
Experience

5. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
AB Testing isn’t enough these days…

6. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
uplift in
total revenue
37%
uplift in
revenues
12% increase in
bookings
27%
uplift in submitted
applications
26% uplift in users
process
starting registration
69%
uplift in
81%
sign ups

7. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0

8. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
2017 Today
Inconclusive Tests
Winners & losers

9. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
The basics

10. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Data InsightsAnalysis
UX Research
Ideation &
Prioritisation
Development
Data Insights
Identify trends and areas of opportunity

11. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Data InsightsAnalysis
UX Research
Ideation &
Prioritisation
Development
UX Research
Understanding user behaviour through
insight

12. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Data InsightsAnalysis
UX Research
Ideation &
Prioritisation
Development
Ideation & Prioritisation
Translating research into actions

13. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Data InsightsAnalysis
UX Research
Ideation &
Prioritisation
Development
Development
Implement within Optimizely

14. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Data InsightsAnalysis
UX Research
Ideation &
Prioritisation
Development
Analysis
What did we learn and how do we
iterate?

15. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
User Behaviour
User Testing Groups
Testing
Remote
Testing
Reporting Only Expert Review Competitor
Review
Competitor
Benchmarking
1st Click Tree Testing Card SortingSurveys In-store testing Persona
Development
Wireframing &
prototyping

16. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
SALESCONVERSION
OVERALL SALES CONTRIBUTION
LOW CONVERSION
LOW CONTRIBUTION
HIGH CONVERSION
LOW CONTRIBUTION
HIGH CONVERSION
HIGH CONTRIBUTION
LOW CONVERSION
HIGH CONTRIBUTION
Performs well but doesn’t get a lot of traffic Already performing well so less room for growth
Biggest room for improvement, highest site-wide
impact
Harder to get results from,
requires more complex changes
Ordersofuserswho
haveseenXpageVisitstoXpage/
Orders of users who
have seen X page
Total website
orders/
Conversion vs. Contribution

17. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
60%
Site wide
estimated uplift
25%
Monthly visitors
7.5%
Technical ease
7.5%
Internal ease
Expected site wide result
from test. We take into
account overall revenue
return
Test implementation
complexity given by
the person responsible
for building the test in
the testing tool
How easy would it be
to hardcode the
variation, ideally need
to get a developer
involved if it’s a more
complex build
3 High = ≥ 8%
2 Moderate = 2% to 7.99%
1 Little = ≤ 2%
SITE WIDE ESTIMATED UPLIFT
Consider amount of
traffic to the area of the
website
3 High = ≥ 50,000
2 Moderate = 5,001 to 10,000
1 Little = ≤ 5000
MONTHLY UNIQUE VISITORS
3 Easy = ≤ 10 hrs
2 Moderate = 11 to 19 hrs
1 Difficult = ≥ 20 hrs
TECHNICAL EASE
3 Easy = CMS change
2 Moderate = Change request
1 Difficult = Template amendment
INTERNAL EASE
Example
15%
Prioritisation Calculator

18. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
uplift in
total revenue
37%
uplift in
revenues
12% increase in
bookings
27%
uplift in
excess weight take-up
60%
uplift in submitted
applications
26% uplift in users
process
starting registration
69%
uplift in
81% uplift in add-ons
purchased
12%
sign ups
Results!

19. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Displace Model

20. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Quick wins
Experimentation methodology for optimising what already exists

21. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Culture of Experimentation
Culture not really that important
Culture needs to change
Quick wins

22. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Challenge
Challenging & creative alternatives to transform the experience

23. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
- The UK’s no.1 cashback site
They have shared over
£400 million
in cashback
Over seven million
People in the UK have become
Quidco members since 2005
to offer consumers cashback
4,500+
Partnerships with top brands
every time they shop

24. INSERT INFORMATION CLASSIFICATION HERE Insert Version NumberUNCLASSIFIED PUBLIC
Data and Insight helps to
provide context around
the issue and the scale of
the opportunity captured
in the form of a DIBB
The management team
then asses all of the
submissions and
categorises them ‘Now’,
‘Next’ and ‘Future’
RedEye Optimisation
helps investigate the
‘Next’ and ‘Future’ Bets to
better qualify what is
needed to maximise the
opportunity
Any employee at Quidco
is able to submit an
opportunity to improve
the experience aligned to
their key focuses
The Bets Process

25. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
25
UNCLASSIFIED PUBLIC
483% increase in click throughs
Redesigning the Retailer Pages
Control Variation

26. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Culture of Experimentation
Culture not really that important
Culture needs to change
Challenge

27. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Innovate

28. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Culture of Experimentation
Culture not really that important
Culture needs to change
Innovate

29. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Change curve

30. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Change curve
Realise AB Testing not enough
Denial
Fear
Frustration/stressWhat’s the point in this change!
Am I doing myself out of
a job if decentralised?
Inconclusive tests
Transition
Culture of Experimentation
ConfusionI have never ran an experiment
Enthusiasm
Enablement
Scepticism
Acceptance
Perceived Value
Adapted from William Bridges Managing transitions
Make it easy for people to learn
Late adopters vs Champions
Starting to become BAU
Positive impact on KPIs…
Champion!

31. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
• Tech Stack
Predictive modelling
Project Management
Experimentation
User Analytics
Analytics
Data Management
How do we measure a programme of optimisation?
How do we manage experimentation at scale?
How do we constantly monitor the user experience?

32. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Interview with Project Sponsor(s) Stakeholder interviews
FindingsPre-interview questionnaire
Approach
Used to prep relevant questions for
stakeholder interviews & narrow down
interview pool
Understand goals, motivations, scope,
identification, definition and an internal
overview
5 to 10 face-to-face interviews with
individuals or small groups of people.
Uncover conflicts, good practice and pain
points
Analysis & presentation of findings to
Project Sponsor(s)

33. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
Stakeholder interview example
Are you a “Data Driven” business?
Data is at the heart of our prioritisation methodology.
We always learn from previous tests.
We understand our key customer journeys.

34. INSERT INFORMATION CLASSIFICATION HERE Insert Version Number
Proprietary and Intellectual property · Version 1.0
In conclusion…

35. 35
Key takeaways
UNCLASSIFIED PUBLIC
• Run quick wins while simultaneously challenging & innovating within your optimisation programme.
• An increase in inconclusive results & UX issues on a specific site area, start challenging the status quo.
• Technical capabilities is more than tools and resource. It's about how the team approaches change.
• You need to be creative and think outside of the box when implementing challenges.
• Innovation requires cross-function teams working together.
• Try running regular hackathons with your team or come up with a format like the bets approach.

36. Thank you for your time
www.redeyeoptimisation.com
UNCLASSIFIED PUBLIC
@REOptimisation @redeyeoptimisation