A Crazy Culture of Testing

1
Plethora of Piñatas

Useful
1. Webinars: Marketing Experiments & Site Tuners
Tools
2. Landing Page Optimization: Optimizel...
Getting in the game:
some webinars, look
into crazyegg and
Google Experiments

Get me some tools:
Usertesting, Optimiz
ely...
Phase I &
II

Landing Page optimization
Optimizely/Google Experiments
Version testing – compare time periods
Gut based w/ ...
The “Next Level”

Phase III

Methodology
Realistic mathematical understanding of site funnel
metrics
Full A/B testing plat...
Methodology - Conversion Heuristic

C = probability of conversion
m = motivation of user (when)
v = clarity of the value p...
427% Lift

7
Math & the Funnel
1,000 * 40% bounce = 600
1,000 * 30% bounce = 700
3.5% conversion = 21 vs. 25

WIN?

1,000 * 40% bounce ...
The Basic Funnel
Single
page
LPO

Landing Pages

1.7%

Search Results

3.5%

Property
“rising tide”
A/B

Cart
Billing
Purc...
A/B & LPO – The Dual Track
PPC/SE
O
Search
Results

Cart

Produc
t

A/B Tests

LPO

Internal system
with server-side
split...
Determining the Priority & Process
Single
page
LPO

Optimizel
y

Landing Pages
Search Results

Book Now or Web
Lead

Prope...
Cart 1
Results
• 32% Conversion rate increase
• 96% Level of confidence
• $800,000 Additional annual online
sales

Cart 2
Rigorous Testing

29.8%
Lift

18.7%
Lift

54% Total Lift
Rigorous Testing – Sample of current tests

Site / Page

Lift

Beaver Creek Lodging

29.8%

Breckenridge Lodging

427.4%

...
Tidbits from Testing
•

Orange buttons for main CTA

•

One main CTA button vs. multiple options

•

People on pages

•

S...
PowerPack Punch of Death
1.
2.
3.
4.
5.
6.

Marketing Experiments

•

Webinars, research library, methodology

A/B Split T...
Slcsem testing-jan2014
Slcsem testing-jan2014
Upcoming SlideShare
Loading in …5
×

Slcsem testing-jan2014

760 views
673 views

Published on

Ryan Hutchings of Vacation Roost talks testing and optimization for websites in 2014

Published in: Marketing, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
760
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
9
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • **excel file – tracking or que
  • **find good fail picture
  • **insert cart pages
  • Slcsem testing-jan2014

    1. 1. A Crazy Culture of Testing 1
    2. 2. Plethora of Piñatas Useful 1. Webinars: Marketing Experiments & Site Tuners Tools 2. Landing Page Optimization: Optimizely 3. User Testing: Usertesting.com 4. Browser Extensions: Cookies tab in Firebug & Responsive design view in Web developers 5. Tag management – Brighttag.com, Google Tag Manager Shiny Objects Landing Page vs. Conversion optimization 2
    3. 3. Getting in the game: some webinars, look into crazyegg and Google Experiments Get me some tools: Usertesting, Optimiz ely, more webinars, analytics based projects 3 Fuel on the Fire: statistical significance, full A/B, adopted methodology
    4. 4. Phase I & II Landing Page optimization Optimizely/Google Experiments Version testing – compare time periods Gut based w/ mix of some research (webinars or competition • Seasonality, external factors (economy, news, weather), time of day, marketing & advertising efforts • STATISTICAL SIGNIFICANCE • MATHMATICAL METHODOLOGY 4
    5. 5. The “Next Level” Phase III Methodology Realistic mathematical understanding of site funnel metrics Full A/B testing platform Mix of large scale A/B & LPO – dual track Full tracking system & statistical baselines 5
    6. 6. Methodology - Conversion Heuristic C = probability of conversion m = motivation of user (when) v = clarity of the value proposition (why) i = incentive to take action f = friction elements of process = anxiety about entering information/taking action 6
    7. 7. 427% Lift 7
    8. 8. Math & the Funnel 1,000 * 40% bounce = 600 1,000 * 30% bounce = 700 3.5% conversion = 21 vs. 25 WIN? 1,000 * 40% bounce * 45% funnel entries@ 4% & 55%@1% = 13 1,000 * 30% bounce * 45% funnel entries@ 4% & 55%@1% = 15 8
    9. 9. The Basic Funnel Single page LPO Landing Pages 1.7% Search Results 3.5% Property “rising tide” A/B Cart Billing Purchase 9 40%
    10. 10. A/B & LPO – The Dual Track PPC/SE O Search Results Cart Produc t A/B Tests LPO Internal system with server-side splitting, custom vars Optimizley Full manual reporting toolset Software determines 10 Headline s/CTA Lead Forms Image s
    11. 11. Determining the Priority & Process Single page LPO Optimizel y Landing Pages Search Results Book Now or Web Lead Property “rising tide” A/B Checkou t Cart Billing Purchase 11 Boo k
    12. 12. Cart 1 Results • 32% Conversion rate increase • 96% Level of confidence • $800,000 Additional annual online sales Cart 2
    13. 13. Rigorous Testing 29.8% Lift 18.7% Lift 54% Total Lift
    14. 14. Rigorous Testing – Sample of current tests Site / Page Lift Beaver Creek Lodging 29.8% Breckenridge Lodging 427.4% Park City Resort 128.9% Vail Destination 69.1% Mexico Destination 8.7% Mexico Lodging Ideas 5.8% Mexico Travel Ideas 5.6% And many more…
    15. 15. Tidbits from Testing • Orange buttons for main CTA • One main CTA button vs. multiple options • People on pages • Security/value icons work every time • Consistent value propositions • Fold doesn’t matter much • Typically the vertical layout works • Smaller “hero” images 17
    16. 16. PowerPack Punch of Death 1. 2. 3. 4. 5. 6. Marketing Experiments • Webinars, research library, methodology A/B Split Testing Capabilities • Split testing across the whole site, server side traffic splitting, custom vars Full Tracking based on vars • Online & Offline conversions based on split Optimizely • Ability to deploy on the fly, integrate w/GA & crazyegg GA Reporting • Custom reports & custom segments Statistical Tracking & “Funnel Math” • Excel report and/or online tool (visualwebsiteoptimizer.com/absplit-significance-calculator/ – based on 5% p-value, or 95% confidence level) 18

    ×