0
How one company implements an entire testing methodology
every day
The Nuts and Bolts
Ryan Hutchings
Director of Marketing...
Session speaker
2
Ryan Hutchings
Director of Marketing
VacationRoost
Hutchings has more than 10 years of online marketing ...
About VacationRoost
About VacationRoost
• 25 websites
• Always running 2 to 3 tests
• 100+ tests run across landing pages, homepages,
lead for...
Fundamental
Question:
How can marketers systematically run
tests to achieve consistent lifts?
and hundreds of other tests….
An overall testing process …
Build
… to support an iterative optimization cycle
Collecting
data
Planning
experiments
Preparing
treatments
Conducting
tests
An...
The dual testing track
A/B split test
• Internal system with server-side
splitting, custom variables
• Full manual reporti...
The dual testing track
A/B split test
• Internal system with server-side
splitting, custom variables
• Full manual reporti...
How to test small
Decide Identify Hypothesize Design Analyze
Decide what to test
90%
Replace “our recommendations” section with content snippet
for value proposition banner usage
Restyle “most popular de...
How to test small
Decide Identify Hypothesize Design Analyze
Identify the target conversion goal
What’s a good goal for bounce rates?
> 50%
40%-50% 30%-40%
< 30%
What’s a good goal for CTR to the next page
or step?
> 60%
50%-60%30%-50%
< 30%
How to test small
Decide Identify Hypothesize Design Analyze
Create a hypothesis
Small Test: PPC landing pages
17
Small Test: PPC landing pages
18
Small Test: PPC landing pages
19
How can we
reduce friction and
anxiety on this
page?
How to test small
Decide Identify Hypothesize Design Analyze
Wireframe, design and launch treatment
Small Test: PPC landing pages
21
Small Test: PPC landing pages
22
Small Test: PPC landing pages
23
Small Test: PPC landing pages
24
How to test small
Decide Identify Hypothesize Design Analyze
Analyze results and calculate ROI
Small Test: PPC landing pages
26
427% increase in CTR
Small test validation tool
95.2%
ROI template
Let’s take a
closer look…
ROI template
Average ticket
Jan
Feb
Mar.
Apr
May
Jun
Jul.
Aug
Sept
Oct
Nov.
Dec
Close rate
Jan
Feb
Mar.
Apr
May
Jun
Jul.
A...
ROI template
Incremental Increases
New Bounce Rate
Leads
Sales $
Contribution $
53
$ 40,307
$ 3,225
Incremental Increases
...
How do you get the resources to run
large tests?
The dual testing track
A/B split test
• Internal system with server-side
splitting, custom variables
• Full manual reporti...
How to test big
Decide Collaborate Identify Create Design Analyze
Decide on an element to optimize
Looking at the same
elements as the small
tests.
The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Purchase
3.5%
1.7%
“Rising tide”
A/B
(large test)
Sing...
Determining a “good” conversion rate
Source: MarketingSherpa.com/ecommerce
How to test big
Decide Collaborate Identify Create Design Analyze
Work with IT to define variables or events
Large Testing: Variables are the key
38
Conditions
Segment your users and/or their visits to single or multi-visit conditi...
How to test big
Decide Collaborate Identify Create Design Analyze
Identify target conversion goal for control
The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Book now, Web lead,
interactions/events
Checkout, add ...
The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Book Now, web lead,
interactions/events
Checkout, add ...
Conversion goal results
How to test big
Decide Collaborate Identify Create Design Analyze
Create hypothesis and set up validation sheet
Manual testing database
Number of Days
40
Control Treatment
Large Test: Lead forms 1
Security seals
Control
Fail – no lift.
Large Test: Lead forms 1
Treatment
Control Treatment
Large Test: Lead forms 2
Shorter form
Control
Fail – no lift.
Large Test: Lead forms 2
Treatment
Control Treatment
Large Test: Lead forms 3
Option to call
Control Treatment
Fail – no lift.
Large Test: Lead forms 3
How to test big
Decide Collaborate Identify Create Design Analyze
Wireframe, design and launch treatment
Large Test: Lead forms
Large Test: Lead forms
Large Test: Lead forms
19% Lift
How to test big
Decide Collaborate Identify Create Design Analyze
Analyze results and calculate ROI
Manual testing database
Testing roadmap – it’s really:
Search Results
Property Details
Cart
Billing info
Booking
ROI template
Same as a small test.
Setting up your own programmatic testing
1
2
3
4
5
Custom variable tracking system, spreadsheet templates are vital buildi...
Questions?
Thank you
Ryan Hutchings
Marketing Director
VacationRoost
@HutchSEO
Daniel Burstein
Director of Editorial Content
MECLABS
...
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
Upcoming SlideShare
Loading in...5
×

The Nuts and Bolts: How one company implements an entire testing methodology every day

342

Published on

Published in: Marketing, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
342
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "The Nuts and Bolts: How one company implements an entire testing methodology every day"

  1. 1. How one company implements an entire testing methodology every day The Nuts and Bolts Ryan Hutchings Director of Marketing VacationRoost Daniel Burstein Director of Editorial Content MECLABS
  2. 2. Session speaker 2 Ryan Hutchings Director of Marketing VacationRoost Hutchings has more than 10 years of online marketing experience. He leads the marketing team for VacationRoost, a $120 million private company in the travel industry. From advertising to analytics, he organizes a comprehensive and robust online marketing strategy focused on tracking and optimization. His specialties include SEO, PPC, testing, conversion rate optimization, advanced data modeling, analytics and every technical aspect related to online marketing. Hutchings is a Google AdWords certified professional, SEOmoz Pro member, nationally recognized industry speaker and participant of top online marketing communities. He also help found the SLCSEM organization, which is Utah’s largest nonprofit group of online marketing professionals. @HutchSEO
  3. 3. About VacationRoost
  4. 4. About VacationRoost • 25 websites • Always running 2 to 3 tests • 100+ tests run across landing pages, homepages, lead forms and shopping carts • Goal of 10% incremental lift in total revenue generated by testing
  5. 5. Fundamental Question: How can marketers systematically run tests to achieve consistent lifts? and hundreds of other tests….
  6. 6. An overall testing process … Build
  7. 7. … to support an iterative optimization cycle Collecting data Planning experiments Preparing treatments Conducting tests Analyzing results
  8. 8. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  9. 9. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  10. 10. How to test small Decide Identify Hypothesize Design Analyze Decide what to test
  11. 11. 90% Replace “our recommendations” section with content snippet for value proposition banner usage Restyle “most popular destinations” change hierarchy 40%
  12. 12. How to test small Decide Identify Hypothesize Design Analyze Identify the target conversion goal
  13. 13. What’s a good goal for bounce rates? > 50% 40%-50% 30%-40% < 30%
  14. 14. What’s a good goal for CTR to the next page or step? > 60% 50%-60%30%-50% < 30%
  15. 15. How to test small Decide Identify Hypothesize Design Analyze Create a hypothesis
  16. 16. Small Test: PPC landing pages 17
  17. 17. Small Test: PPC landing pages 18
  18. 18. Small Test: PPC landing pages 19 How can we reduce friction and anxiety on this page?
  19. 19. How to test small Decide Identify Hypothesize Design Analyze Wireframe, design and launch treatment
  20. 20. Small Test: PPC landing pages 21
  21. 21. Small Test: PPC landing pages 22
  22. 22. Small Test: PPC landing pages 23
  23. 23. Small Test: PPC landing pages 24
  24. 24. How to test small Decide Identify Hypothesize Design Analyze Analyze results and calculate ROI
  25. 25. Small Test: PPC landing pages 26 427% increase in CTR
  26. 26. Small test validation tool 95.2%
  27. 27. ROI template Let’s take a closer look…
  28. 28. ROI template Average ticket Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Close rate Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Lead conversion Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Contribution % of sales
  29. 29. ROI template Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 53 $ 40,307 $ 3,225 Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 20% 7 5,246 $ 420 $ At 10% At 30% 158 $ 119,256 $ 9,541 25 % 2 d 1,749 $ 140 $
  30. 30. How do you get the resources to run large tests?
  31. 31. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  32. 32. How to test big Decide Collaborate Identify Create Design Analyze Decide on an element to optimize
  33. 33. Looking at the same elements as the small tests.
  34. 34. The basic funnel Landing Pages Search Results Property Cart Billing Purchase 3.5% 1.7% “Rising tide” A/B (large test) Single page LPO (small test) 40% 1.7% Purchase
  35. 35. Determining a “good” conversion rate Source: MarketingSherpa.com/ecommerce
  36. 36. How to test big Decide Collaborate Identify Create Design Analyze Work with IT to define variables or events
  37. 37. Large Testing: Variables are the key 38 Conditions Segment your users and/or their visits to single or multi-visit conditions. inlineLeadForm1 mrSearchPage1 Notest simpleModalLeadForm simpleModalLeadForm4 control test
  38. 38. How to test big Decide Collaborate Identify Create Design Analyze Identify target conversion goal for control
  39. 39. The basic funnel Landing Pages Search Results Property Cart Billing Book now, Web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase “Rising tide” A/B (large test) Single page LPO (small test)
  40. 40. The basic funnel Landing Pages Search Results Property Cart Billing Book Now, web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase Book Now, web lead, interactions/events “Rising tide” A/B (large test) Single page LPO (small test) 10% - 30% minimum targeted gains
  41. 41. Conversion goal results
  42. 42. How to test big Decide Collaborate Identify Create Design Analyze Create hypothesis and set up validation sheet
  43. 43. Manual testing database Number of Days 40
  44. 44. Control Treatment Large Test: Lead forms 1 Security seals
  45. 45. Control Fail – no lift. Large Test: Lead forms 1 Treatment
  46. 46. Control Treatment Large Test: Lead forms 2 Shorter form
  47. 47. Control Fail – no lift. Large Test: Lead forms 2 Treatment
  48. 48. Control Treatment Large Test: Lead forms 3 Option to call
  49. 49. Control Treatment Fail – no lift. Large Test: Lead forms 3
  50. 50. How to test big Decide Collaborate Identify Create Design Analyze Wireframe, design and launch treatment
  51. 51. Large Test: Lead forms
  52. 52. Large Test: Lead forms
  53. 53. Large Test: Lead forms 19% Lift
  54. 54. How to test big Decide Collaborate Identify Create Design Analyze Analyze results and calculate ROI
  55. 55. Manual testing database
  56. 56. Testing roadmap – it’s really: Search Results Property Details Cart Billing info Booking
  57. 57. ROI template Same as a small test.
  58. 58. Setting up your own programmatic testing 1 2 3 4 5 Custom variable tracking system, spreadsheet templates are vital building blocks Dedicate a person and tie performance to testing “Top of the funnel” to the “bottom conversion” may have too much noise in between – keep it simple and isolated Build a reporting system that ties front end and back end together Get buy-in from the management team
  59. 59. Questions?
  60. 60. Thank you Ryan Hutchings Marketing Director VacationRoost @HutchSEO Daniel Burstein Director of Editorial Content MECLABS @DanielBurstein Black & White Headshot Black & White Headshot
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×