Your SlideShare is downloading. ×
0
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
The Nuts and Bolts: How one company implements an entire testing methodology every day
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The Nuts and Bolts: How one company implements an entire testing methodology every day

307

Published on

Published in: Marketing, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
307
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. How one company implements an entire testing methodology every day The Nuts and Bolts Ryan Hutchings Director of Marketing VacationRoost Daniel Burstein Director of Editorial Content MECLABS
  • 2. Session speaker 2 Ryan Hutchings Director of Marketing VacationRoost Hutchings has more than 10 years of online marketing experience. He leads the marketing team for VacationRoost, a $120 million private company in the travel industry. From advertising to analytics, he organizes a comprehensive and robust online marketing strategy focused on tracking and optimization. His specialties include SEO, PPC, testing, conversion rate optimization, advanced data modeling, analytics and every technical aspect related to online marketing. Hutchings is a Google AdWords certified professional, SEOmoz Pro member, nationally recognized industry speaker and participant of top online marketing communities. He also help found the SLCSEM organization, which is Utah’s largest nonprofit group of online marketing professionals. @HutchSEO
  • 3. About VacationRoost
  • 4. About VacationRoost • 25 websites • Always running 2 to 3 tests • 100+ tests run across landing pages, homepages, lead forms and shopping carts • Goal of 10% incremental lift in total revenue generated by testing
  • 5. Fundamental Question: How can marketers systematically run tests to achieve consistent lifts? and hundreds of other tests….
  • 6. An overall testing process … Build
  • 7. … to support an iterative optimization cycle Collecting data Planning experiments Preparing treatments Conducting tests Analyzing results
  • 8. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  • 9. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  • 10. How to test small Decide Identify Hypothesize Design Analyze Decide what to test
  • 11. 90% Replace “our recommendations” section with content snippet for value proposition banner usage Restyle “most popular destinations” change hierarchy 40%
  • 12. How to test small Decide Identify Hypothesize Design Analyze Identify the target conversion goal
  • 13. What’s a good goal for bounce rates? > 50% 40%-50% 30%-40% < 30%
  • 14. What’s a good goal for CTR to the next page or step? > 60% 50%-60%30%-50% < 30%
  • 15. How to test small Decide Identify Hypothesize Design Analyze Create a hypothesis
  • 16. Small Test: PPC landing pages 17
  • 17. Small Test: PPC landing pages 18
  • 18. Small Test: PPC landing pages 19 How can we reduce friction and anxiety on this page?
  • 19. How to test small Decide Identify Hypothesize Design Analyze Wireframe, design and launch treatment
  • 20. Small Test: PPC landing pages 21
  • 21. Small Test: PPC landing pages 22
  • 22. Small Test: PPC landing pages 23
  • 23. Small Test: PPC landing pages 24
  • 24. How to test small Decide Identify Hypothesize Design Analyze Analyze results and calculate ROI
  • 25. Small Test: PPC landing pages 26 427% increase in CTR
  • 26. Small test validation tool 95.2%
  • 27. ROI template Let’s take a closer look…
  • 28. ROI template Average ticket Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Close rate Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Lead conversion Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Contribution % of sales
  • 29. ROI template Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 53 $ 40,307 $ 3,225 Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 20% 7 5,246 $ 420 $ At 10% At 30% 158 $ 119,256 $ 9,541 25 % 2 d 1,749 $ 140 $
  • 30. How do you get the resources to run large tests?
  • 31. The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  • 32. How to test big Decide Collaborate Identify Create Design Analyze Decide on an element to optimize
  • 33. Looking at the same elements as the small tests.
  • 34. The basic funnel Landing Pages Search Results Property Cart Billing Purchase 3.5% 1.7% “Rising tide” A/B (large test) Single page LPO (small test) 40% 1.7% Purchase
  • 35. Determining a “good” conversion rate Source: MarketingSherpa.com/ecommerce
  • 36. How to test big Decide Collaborate Identify Create Design Analyze Work with IT to define variables or events
  • 37. Large Testing: Variables are the key 38 Conditions Segment your users and/or their visits to single or multi-visit conditions. inlineLeadForm1 mrSearchPage1 Notest simpleModalLeadForm simpleModalLeadForm4 control test
  • 38. How to test big Decide Collaborate Identify Create Design Analyze Identify target conversion goal for control
  • 39. The basic funnel Landing Pages Search Results Property Cart Billing Book now, Web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase “Rising tide” A/B (large test) Single page LPO (small test)
  • 40. The basic funnel Landing Pages Search Results Property Cart Billing Book Now, web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase Book Now, web lead, interactions/events “Rising tide” A/B (large test) Single page LPO (small test) 10% - 30% minimum targeted gains
  • 41. Conversion goal results
  • 42. How to test big Decide Collaborate Identify Create Design Analyze Create hypothesis and set up validation sheet
  • 43. Manual testing database Number of Days 40
  • 44. Control Treatment Large Test: Lead forms 1 Security seals
  • 45. Control Fail – no lift. Large Test: Lead forms 1 Treatment
  • 46. Control Treatment Large Test: Lead forms 2 Shorter form
  • 47. Control Fail – no lift. Large Test: Lead forms 2 Treatment
  • 48. Control Treatment Large Test: Lead forms 3 Option to call
  • 49. Control Treatment Fail – no lift. Large Test: Lead forms 3
  • 50. How to test big Decide Collaborate Identify Create Design Analyze Wireframe, design and launch treatment
  • 51. Large Test: Lead forms
  • 52. Large Test: Lead forms
  • 53. Large Test: Lead forms 19% Lift
  • 54. How to test big Decide Collaborate Identify Create Design Analyze Analyze results and calculate ROI
  • 55. Manual testing database
  • 56. Testing roadmap – it’s really: Search Results Property Details Cart Billing info Booking
  • 57. ROI template Same as a small test.
  • 58. Setting up your own programmatic testing 1 2 3 4 5 Custom variable tracking system, spreadsheet templates are vital building blocks Dedicate a person and tie performance to testing “Top of the funnel” to the “bottom conversion” may have too much noise in between – keep it simple and isolated Build a reporting system that ties front end and back end together Get buy-in from the management team
  • 59. Questions?
  • 60. Thank you Ryan Hutchings Marketing Director VacationRoost @HutchSEO Daniel Burstein Director of Editorial Content MECLABS @DanielBurstein Black & White Headshot Black & White Headshot

×