Ryan Hutchings of Vacation Roost talks testing and optimization for websites in 2014

1. A Crazy Culture of Testing
1

2. Plethora of Piñatas
Useful
1. Webinars: Marketing Experiments & Site Tuners
Tools
2. Landing Page Optimization: Optimizely
3. User Testing: Usertesting.com
4. Browser Extensions: Cookies tab in Firebug &
Responsive design view in Web developers
5. Tag management – Brighttag.com, Google Tag Manager
Shiny Objects
Landing Page vs. Conversion optimization
2

3. Getting in the game:
some webinars, look
into crazyegg and
Google Experiments
Get me some tools:
Usertesting, Optimiz
ely, more
webinars, analytics
based projects
3
Fuel on the Fire:
statistical
significance, full A/B,
adopted
methodology

4. Phase I &
II
Landing Page optimization
Optimizely/Google Experiments
Version testing – compare time periods
Gut based w/ mix of some research (webinars or
competition
• Seasonality, external factors
(economy, news, weather), time of day, marketing &
advertising efforts
• STATISTICAL SIGNIFICANCE
• MATHMATICAL METHODOLOGY
4

5. The “Next Level”
Phase III
Methodology
Realistic mathematical understanding of site funnel
metrics
Full A/B testing platform
Mix of large scale A/B & LPO – dual track
Full tracking system & statistical baselines
5

6. Methodology - Conversion Heuristic
C = probability of conversion
m = motivation of user (when)
v = clarity of the value proposition (why)
i = incentive to take action
f = friction elements of process
= anxiety about entering information/taking action
6

7. 427% Lift
7

8. Math & the Funnel
1,000 * 40% bounce = 600
1,000 * 30% bounce = 700
3.5% conversion = 21 vs. 25
WIN?
1,000 * 40% bounce * 45% funnel
entries@ 4% & 55%@1% = 13
1,000 * 30% bounce * 45% funnel
entries@ 4% & 55%@1% = 15
8

9. The Basic Funnel
Single
page
LPO
Landing Pages
1.7%
Search Results
3.5%
Property
“rising tide”
A/B
Cart
Billing
Purchase
9
40%

10. A/B & LPO – The Dual Track
PPC/SE
O
Search
Results
Cart
Produc
t
A/B Tests
LPO
Internal system
with server-side
splitting, custom
vars
Optimizley
Full manual
reporting toolset
Software
determines
10
Headline
s/CTA
Lead
Forms
Image
s

11. Determining the Priority & Process
Single
page
LPO
Optimizel
y
Landing Pages
Search Results
Book Now or Web
Lead
Property
“rising tide”
A/B
Checkou
t
Cart
Billing
Purchase
11
Boo
k

12. Cart 1
Results
• 32% Conversion rate increase
• 96% Level of confidence
• $800,000 Additional annual online
sales
Cart 2

15. Rigorous Testing
29.8%
Lift
18.7%
Lift
54% Total Lift

16. Rigorous Testing – Sample of current tests
Site / Page
Lift
Beaver Creek Lodging
29.8%
Breckenridge Lodging
427.4%
Park City Resort
128.9%
Vail Destination
69.1%
Mexico Destination
8.7%
Mexico Lodging Ideas
5.8%
Mexico Travel Ideas
5.6%
And many more…

17. Tidbits from Testing
•
Orange buttons for main CTA
•
One main CTA button vs. multiple options
•
People on pages
•
Security/value icons work every time
•
Consistent value propositions
•
Fold doesn’t matter much
•
Typically the vertical layout works
•
Smaller “hero” images
17

18. PowerPack Punch of Death
1.
2.
3.
4.
5.
6.
Marketing Experiments
•
Webinars, research library, methodology
A/B Split Testing Capabilities
•
Split testing across the whole site, server side traffic
splitting, custom vars
Full Tracking based on vars
•
Online & Offline conversions based on split
Optimizely
•
Ability to deploy on the fly, integrate w/GA & crazyegg
GA Reporting
•
Custom reports & custom segments
Statistical Tracking & “Funnel Math”
•
Excel report and/or online tool (visualwebsiteoptimizer.com/absplit-significance-calculator/ – based on 5% p-value, or 95%
confidence level)
18