The Nuts and Bolts: How one company implements an entire testing methodology every day

626 views
530 views

Published on

Published in: Marketing, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
626
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
26
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

The Nuts and Bolts: How one company implements an entire testing methodology every day

  1. 1. How one company implements an entire testing methodology every day The Nuts and Bolts Ryan Hutchings Director of Marketing VacationRoost Daniel Burstein Director of Editorial Content MECLABS
  2. 2. Session speaker 2 Ryan Hutchings Director of Marketing VacationRoost Hutchings has more than 10 years of online marketing experience. He leads the marketing team for VacationRoost, a $120 million private company in the travel industry. From advertising to analytics, he organizes a comprehensive and robust online marketing strategy focused on tracking and optimization. His specialties include SEO, PPC, testing, conversion rate optimization, advanced data modeling, analytics and every technical aspect related to online marketing. Hutchings is a Google AdWords certified professional, SEOmoz Pro member, nationally recognized industry speaker and participant of top online marketing communities. He also help found the SLCSEM organization, which is Utah’s largest nonprofit group of online marketing professionals. @HutchSEO
  3. 3. About VacationRoost
  4. 4. About VacationRoost • 25 websites • Always running 2 to 3 tests • 100+ tests run across landing pages, homepages, lead forms and shopping carts • Goal of 10% incremental lift in total revenue generated by testing
  5. 5. Fundamental Question: How can marketers systematically run tests to achieve consistent lifts? and hundreds of other tests….
  6. 6. An overall testing process … Build
  7. 7. … to support an iterative optimization cycle Collecting data Planning experiments Preparing treatments Conducting tests Analyzing results
  8. 8. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  9. 9. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  10. 10. How to test small Decide Identify Hypothesize Design Analyze Decide what to test
  11. 11. 90% Replace “our recommendations” section with content snippet for value proposition banner usage Restyle “most popular destinations” change hierarchy 40%
  12. 12. How to test small Decide Identify Hypothesize Design Analyze Identify the target conversion goal
  13. 13. What’s a good goal for bounce rates? > 50% 40%-50% 30%-40% < 30%
  14. 14. What’s a good goal for CTR to the next page or step? > 60% 50%-60%30%-50% < 30%
  15. 15. How to test small Decide Identify Hypothesize Design Analyze Create a hypothesis
  16. 16. Small Test: PPC landing pages 17
  17. 17. Small Test: PPC landing pages 18
  18. 18. Small Test: PPC landing pages 19 How can we reduce friction and anxiety on this page?
  19. 19. How to test small Decide Identify Hypothesize Design Analyze Wireframe, design and launch treatment
  20. 20. Small Test: PPC landing pages 21
  21. 21. Small Test: PPC landing pages 22
  22. 22. Small Test: PPC landing pages 23
  23. 23. Small Test: PPC landing pages 24
  24. 24. How to test small Decide Identify Hypothesize Design Analyze Analyze results and calculate ROI
  25. 25. Small Test: PPC landing pages 26 427% increase in CTR
  26. 26. Small test validation tool 95.2%
  27. 27. ROI template Let’s take a closer look…
  28. 28. ROI template Average ticket Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Close rate Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Lead conversion Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Contribution % of sales
  29. 29. ROI template Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 53 $ 40,307 $ 3,225 Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 20% 7 5,246 $ 420 $ At 10% At 30% 158 $ 119,256 $ 9,541 25 % 2 d 1,749 $ 140 $
  30. 30. How do you get the resources to run large tests?
  31. 31. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  32. 32. How to test big Decide Collaborate Identify Create Design Analyze Decide on an element to optimize
  33. 33. Looking at the same elements as the small tests.
  34. 34. The basic funnel Landing Pages Search Results Property Cart Billing Purchase 3.5% 1.7% “Rising tide” A/B (large test) Single page LPO (small test) 40% 1.7% Purchase
  35. 35. Determining a “good” conversion rate Source: MarketingSherpa.com/ecommerce
  36. 36. How to test big Decide Collaborate Identify Create Design Analyze Work with IT to define variables or events
  37. 37. Large Testing: Variables are the key 38 Conditions Segment your users and/or their visits to single or multi-visit conditions. inlineLeadForm1 mrSearchPage1 Notest simpleModalLeadForm simpleModalLeadForm4 control test
  38. 38. How to test big Decide Collaborate Identify Create Design Analyze Identify target conversion goal for control
  39. 39. The basic funnel Landing Pages Search Results Property Cart Billing Book now, Web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase “Rising tide” A/B (large test) Single page LPO (small test)
  40. 40. The basic funnel Landing Pages Search Results Property Cart Billing Book Now, web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase Book Now, web lead, interactions/events “Rising tide” A/B (large test) Single page LPO (small test) 10% - 30% minimum targeted gains
  41. 41. Conversion goal results
  42. 42. How to test big Decide Collaborate Identify Create Design Analyze Create hypothesis and set up validation sheet
  43. 43. Manual testing database Number of Days 40
  44. 44. Control Treatment Large Test: Lead forms 1 Security seals
  45. 45. Control Fail – no lift. Large Test: Lead forms 1 Treatment
  46. 46. Control Treatment Large Test: Lead forms 2 Shorter form
  47. 47. Control Fail – no lift. Large Test: Lead forms 2 Treatment
  48. 48. Control Treatment Large Test: Lead forms 3 Option to call
  49. 49. Control Treatment Fail – no lift. Large Test: Lead forms 3
  50. 50. How to test big Decide Collaborate Identify Create Design Analyze Wireframe, design and launch treatment
  51. 51. Large Test: Lead forms
  52. 52. Large Test: Lead forms
  53. 53. Large Test: Lead forms 19% Lift
  54. 54. How to test big Decide Collaborate Identify Create Design Analyze Analyze results and calculate ROI
  55. 55. Manual testing database
  56. 56. Testing roadmap – it’s really: Search Results Property Details Cart Billing info Booking
  57. 57. ROI template Same as a small test.
  58. 58. Setting up your own programmatic testing 1 2 3 4 5 Custom variable tracking system, spreadsheet templates are vital building blocks Dedicate a person and tie performance to testing “Top of the funnel” to the “bottom conversion” may have too much noise in between – keep it simple and isolated Build a reporting system that ties front end and back end together Get buy-in from the management team
  59. 59. Questions?
  60. 60. Thank you Ryan Hutchings Marketing Director VacationRoost @HutchSEO Daniel Burstein Director of Editorial Content MECLABS @DanielBurstein Black & White Headshot Black & White Headshot

×