Beyond Simple A/B testing

ratiopartners.co.uk 1
Webinar – Beyond simple A/B
testing

Topics in todays webinar
• ‘Optimising Optimisation’ or how to improve your program performance
over time
• Supporting innovation with Processes and Methodology
• The types of research that will deliver the best experiment hypothesis &
strategy
• The capabilities (people) needed to excel
• Technology and evaluating the suitability of a product for your
organisation

Topics in todays webinar
• ‘Optimising Optimisation’ or how to improve your program
performance over time
• Supporting innovation with Processes and Methodology
• The types of research that will deliver the best experiment
hypothesis & strategy
• The capabilities (people) needed to excel
• Technology and evaluating the suitability of a product for your
organisation

There’s a natural tendency to focus on the outputs of Optimisation…
Like increases in Conversion Rate, AOV, CLV, Customer satisfaction, repeat
purchase, etc…
Topic 1: Optimising Optimisation

But are these outputs levers that we can pull to drive more success from
our program?
Will they tell us what we are doing well or poorly, or where we are going
wrong?
Not really..

A best in class optimisation program needs a set of performance
metrics that will help us to understand where the weak points are in
our program as a whole,
not just report on how any one individual experiment happened to
perform

Metric 1: Test Capacity
The more effective experiments we run, the more results our program
will deliver..
Test capacity helps us to understand the upper limit of how many tests
we can run

52
Average test duration ( in
weeks)
# of simultaneously
testable funnels or
pages

52
5
5
Average test duration
( in weeks)
# of simultaneously testable funnels or
pages
52 tests
(annual capacity)

Metric 2: Test velocity
Where experiment capacity showed how
many experiments can be run,
experiment velocity is a measure of how
many experiments have been run
It helps us understand how well we have
capitalised on our opportunities 0
1
2
3
4
5
6
7
Tests
Program velocity tracker
Test Velocity Test Capacity

Metric 3: Zero test tracker
Traffic that doesn’t see a test is a missed
opportunity – permanently.
To help encourage a culture of “Always be
testing” and minimise lost opportunities,
track how many days it’s been since your
teams last day with zero experiments live.
23 days
Since the last
“Zero experiments” day

Metric 4: Win rate
As well as monitoring and
increasing quantity of experiments
delivered, it’s crucial to
understand the quality of those
tests
Monitoring test win rate keeps our
finger on the quality pulse
0%
20%
40%
60%
80%
Test Win Rate tracker
Win rate Inconclusive Rate
Loss rate

Processes and methodology are a key lever in your quest to improve
experiment velocity
Improving the processes that sit around your experimentation will help you
deliver more output from finite resources
Simply put, robust processes support rather than stifle innovation
Topic 2: Process and Methodology

An Experimentation charter is a document
which lays out in explicit detail why and how
the optimisation program will be run
Amongst other things it clearly defines:
• Roles and responsibilities
• Experimentation process from ideation to go
live
• Which stakeholders are involved at each
step and what their committed turn around
time is
Example 1: Experimentation Charter

Centralise planning &
communications around the
CRO project using a PM tool,
even if it’s a free one like Trello.
Build out a workflow that
matches the process agreed in
your optimisation charter
Example 2: Using a centralised PM tool

A thorough QA checklist will
help you to minimise the
amount of time each
experiment will spend in QA
as well as minimise mistakes
made through human oversight
Example 3: QA checklist
QA Checklist
Goals implemented and
triggering correctly
Only the correct
audience
can enter test
Functionality/design
works across key
browsers
Functionality/design
works across key device
types
New links all point to
working pages
Additional GA tracking
implemented

Example 4: Post test analysis & knowledge repository
Conduct meaningful post test analysis,
communicate it within the organisation &
create a system to record this information
for posterity.
This will maximise the future value
derived from each completed experiment
Post test analysis
What was the hypothesis?
What pages/touch points was it executed
across?
Did it win or lose?
What behaviours were
Were there any other insights
Did any customer segments over or under
perform?
Did VOC tools provide any additional insight?

Example 5: Qualitative research
Qualitative research is an important step in
developing hypothesis that deliver results.
Try to standardise the way you collect this
information and optimise your
questions/process over time to help isolate
the questions which inspire the most
helpful information from customers
What was your reason
for visiting today?
 Option A
 Option B
 Option C
 Option D (Free text)
What was your reason
for visiting today?
(free text input)
0.25% response rate
1% response rate

If your optimisation program is to deliver
meaningful changes in user behaviour, you
need to make meaningful changes to their
user experience, with hypothesis grounded
in qualitative and quantitative data
Move beyond button colour testing!
Topic 3: Research and Strategy

Developing hypothesis the right way
The best test hypothesis will have a
grounding in both the qualitative and
quantitative
In general, digital analytics data can tell
you “what” is happening, qualitative
data will give you insights as to “why” it
happened

Digital analytics
Digital analytics data shines at sizing the
opportunity – for example, “where is the biggest
drop out point in the purchase funnel for
mobile users”?
If you don’t have a “traditional” funnel, try to
think about the objective of the page you’re
looking at and what metrics might help you
evaluate possible headroom for optimisation
Landed on Homepage
Category listing
Product Page
18%
Purchase
75%
80%
82%
25%
20%
Responsive Funnel

Digital analytics
Clicked hamburger nav
Opened top level
category in nav
35% 65%
Landed on Homepage
Implemented well, web analytics products can
also help you to isolate where within that
funnel step users are having the most problems
But at this point, we still don’t know what the
problem is and developing a hypothesis with
no other information will mean we are in
danger of applying our own
assumptions/biases
80% 20%
Click through to
category
80% 20%

Voice of customer
‘VOC’ tools will help you
understand what it is about that
step that users struggle with
For example, session replays or
user testing might reveal that users
do not understand the category
links in the navigation are clickable

Useful VOC tools and common use cases
• User testing
• Test new designs on users before building them out and putting them into production
• Interviews with sales / customer service staff
• Understanding customer frustrations with website
• Understanding common customer objections and how they are overcome
• Surveys (email/on site)
• Identify visitor goals and reasons for non completion
• Uncover the reasons that customers buy to communicate to others
• Visitor replay technology
• See the specific challenges users are having and how they are interacting in real time
• Heat/scroll maps
• Identify if users are missing key information within the page due to position

A cautionary tale
- how not to
approach
optimisation & the
danger of HIPPOs

CEO
The navigation menu is
our priority! I think it’s
a big opportunity
Head of
marketing
The navigation menu
looks dated & is not on
brand
Head of
Product
It doesn’t surface
all of our
categories

Adventure
Creative
Driving
Flying
Getaways
Gourmet
Indulge
Adventure subcategories
Subcategory 1
Photo/link
Subcategory 2
Photo/link
Subcategory 3
Photo/link
Subcategory 4
Photo/link
Subcategory 5
Photo/link
Subcategory 6
Photo/link
(2 months of
design &
development
later)

But the test lost, and we had no demonstrable idea why,
just more of our own assumptions…

Some of the problems with our approach
Our hypothesis sucked - Or rather, there wasn’t one!
The decision to rebuild the navigation was based on Highest paid
persons opinions and not a data driven hypothesis

Learning:
Good test hypothesis should set out to solve
actual customer problems, highlight or clarify
your value proposition, reduce anxiety or
introduce urgency
At a minimum, we should have set metrics to
understand how the existing nav performed
to validate “headroom” for improvement
existed
“I can’t find
pricing
information”
“It’s not clear why I
would take your
service over a
competitors”
“How do I know that
your organisation can
be trusted with my
money?” “I don’t know
what this sizing
means”
“I cant see how
to buy it”

The decision to rebuild the navigation was based on Highest paid
persons opinions and not a data driven hypothesis
We didn’t quantify the opportunity or prioritise this hypothesis
against any others

Learning:
Fundamentally, test hypothesis
should be prioritised based on
expected return vs expected
effort

The decision to rebuild the navigation was based on Highest paid persons
opinions and not a data driven hypothesis
We didn’t quantify or prioritise this hypothesis against
others
We had no way of validating what behaviour was occurring with the
new navigation – back to our own assumptions to interpret the results

Learning:
Make sure you are thinking about
what tracking you will need to
maximise test learnings and iterate
on your hypothesis
It needs to be in place before the
test goes live!

Research/Strategy takeaways:
• Use digital analytics data:
• To quantify and prioritise the key drop out points within your funnel
• To identify possible drivers of this drop out by looking at in page interactions
• Use customer research to:
• Identify customer pain points & UX barriers
• Identify customer objections and the persuasion elements that can help you overcome same
• Identify what actual customers view as your value proposition to inform how you communicate to prospective
customers
• When developing hypothesis:
• The most powerful hypothesis will come from both qualitative + quantitative understanding
• At the most fundamental level, prioritise your hypothesis based on estimated impact and effort to execute
• When executing:
• Make sure you have enough tracking in place that you can get a complete understanding of the behaviours
your test influenced, win or lose

The Strategist
• Strong understanding of the business
context
• User research skills to uncover the
qualitative gold
• Can integrate disparate data sources
(qualitative + quantitative) and formulate
hypothesis
• Must be data driven (identify the
problems) and creative (ideate the
solutions)

The Analyst
• Very strong web analytics platform
knowledge – using it for deep dive analysis of
customer journeys, not just reporting
• Can identify the patterns and stories behind
the numbers
• Good command of statistics (statistical
significance & power, probability, sample
sizes etc)

The Designer
• Translates test hypothesis into designs
• Strong understanding of persuasive design
principles
• Understanding of responsive design
principles

The Copywriter
• Translates user research findings in to
persuasive copy which adheres to brands
tone and style
• Understanding of the conversion
funnel/buyer journey and has an ability to
write according to funnel stages
• Familiarity with industry specific language
• Comfortable with SEO principles

The Developer
• Modern testing platforms allow a lot to be
done without development, but at some point
you will start to exhaust these capabilities
• Look to bring in a front end developer who is
very strong at HTML/css/javascript to execute
on more complex design changes
• They should be comfortable with delivering an
‘MVP’ product to validate the hypothesis is
valid

Design
Front end
Team member BTeam member A
Data
Strategy
Copy-
writing
• There can certainly be
overlap between these
roles, particularly when the
team is just starting out.
But it’s important that they
are all present in some
way!

Questions to help evaluate technology vendors
 Is your priority optimisation of individual touch points or personalisation of the entire customer journey?
 If your priority is personalisation, can the technology link together behaviour between different touch points & devices
over multiple visits and use this knowledge to drive personalisation?
 If your priority is optimisation, how much flexibility will the solution give you to alter design and layout of pages? Will it
allow the optimisation project to be abstracted from existing development efforts? (release cycles, development
resources, etc)
 Can the technology help you identify opportunities?
Eg, through integrated VOC testing tools (VWO) or Machine Learning (Sitecore)
 Can the technology provide you with any other customer intelligence?
 Do you have other data which could enrich personalisation (eg CRM) and if so how readily can this be integrated with
the testing platform?

hello@ratiopartners.co.uk
Get in touch

Beyond Simple A/B testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Beyond Simple A/B testing

Similar to Beyond Simple A/B testing (20)

More from Ratio

More from Ratio (7)

Recently uploaded

Recently uploaded (20)

Beyond Simple A/B testing