The Nuts and Bolts: How one company implements an entire testing methodology every day
Upcoming SlideShare
Loading in...5
×
 

The Nuts and Bolts: How one company implements an entire testing methodology every day

on

  • 283 views

 

Statistics

Views

Total Views
283
Views on SlideShare
283
Embed Views
0

Actions

Likes
1
Downloads
20
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The Nuts and Bolts: How one company implements an entire testing methodology every day The Nuts and Bolts: How one company implements an entire testing methodology every day Presentation Transcript

  • How one company implements an entire testing methodology every day The Nuts and Bolts Ryan Hutchings Director of Marketing VacationRoost Daniel Burstein Director of Editorial Content MECLABS
  • Session speaker 2 Ryan Hutchings Director of Marketing VacationRoost Hutchings has more than 10 years of online marketing experience. He leads the marketing team for VacationRoost, a $120 million private company in the travel industry. From advertising to analytics, he organizes a comprehensive and robust online marketing strategy focused on tracking and optimization. His specialties include SEO, PPC, testing, conversion rate optimization, advanced data modeling, analytics and every technical aspect related to online marketing. Hutchings is a Google AdWords certified professional, SEOmoz Pro member, nationally recognized industry speaker and participant of top online marketing communities. He also help found the SLCSEM organization, which is Utah’s largest nonprofit group of online marketing professionals. @HutchSEO
  • About VacationRoost
  • About VacationRoost • 25 websites • Always running 2 to 3 tests • 100+ tests run across landing pages, homepages, lead forms and shopping carts • Goal of 10% incremental lift in total revenue generated by testing
  • Fundamental Question: How can marketers systematically run tests to achieve consistent lifts? and hundreds of other tests….
  • An overall testing process … Build
  • … to support an iterative optimization cycle Collecting data Planning experiments Preparing treatments Conducting tests Analyzing results
  • The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  • The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  • How to test small Decide Identify Hypothesize Design Analyze Decide what to test
  • 90% Replace “our recommendations” section with content snippet for value proposition banner usage Restyle “most popular destinations” change hierarchy 40%
  • How to test small Decide Identify Hypothesize Design Analyze Identify the target conversion goal
  • What’s a good goal for bounce rates? > 50% 40%-50% 30%-40% < 30%
  • What’s a good goal for CTR to the next page or step? > 60% 50%-60%30%-50% < 30%
  • How to test small Decide Identify Hypothesize Design Analyze Create a hypothesis
  • Small Test: PPC landing pages 17
  • Small Test: PPC landing pages 18
  • Small Test: PPC landing pages 19 How can we reduce friction and anxiety on this page?
  • How to test small Decide Identify Hypothesize Design Analyze Wireframe, design and launch treatment
  • Small Test: PPC landing pages 21
  • Small Test: PPC landing pages 22
  • Small Test: PPC landing pages 23
  • Small Test: PPC landing pages 24
  • How to test small Decide Identify Hypothesize Design Analyze Analyze results and calculate ROI
  • Small Test: PPC landing pages 26 427% increase in CTR
  • Small test validation tool 95.2%
  • ROI template Let’s take a closer look…
  • ROI template Average ticket Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Close rate Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Lead conversion Jan Feb Mar. Apr May Jun Jul. Aug Sept Oct Nov. Dec Contribution % of sales
  • ROI template Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 53 $ 40,307 $ 3,225 Incremental Increases New Bounce Rate Leads Sales $ Contribution $ 20% 7 5,246 $ 420 $ At 10% At 30% 158 $ 119,256 $ 9,541 25 % 2 d 1,749 $ 140 $
  • How do you get the resources to run large tests?
  • The dual testing track A/B split test • Internal system with server-side splitting, custom variables • Full manual reporting – spreadsheets and database reports Largetest Search results Cart Products Testing Landing page optimization • Third-party software • Internal analysis determines what to test Smalltest PPC and SEO landing pages Headlines/CTAs Images
  • How to test big Decide Collaborate Identify Create Design Analyze Decide on an element to optimize
  • Looking at the same elements as the small tests.
  • The basic funnel Landing Pages Search Results Property Cart Billing Purchase 3.5% 1.7% “Rising tide” A/B (large test) Single page LPO (small test) 40% 1.7% Purchase
  • Determining a “good” conversion rate Source: MarketingSherpa.com/ecommerce
  • How to test big Decide Collaborate Identify Create Design Analyze Work with IT to define variables or events
  • Large Testing: Variables are the key 38 Conditions Segment your users and/or their visits to single or multi-visit conditions. inlineLeadForm1 mrSearchPage1 Notest simpleModalLeadForm simpleModalLeadForm4 control test
  • How to test big Decide Collaborate Identify Create Design Analyze Identify target conversion goal for control
  • The basic funnel Landing Pages Search Results Property Cart Billing Book now, Web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase “Rising tide” A/B (large test) Single page LPO (small test)
  • The basic funnel Landing Pages Search Results Property Cart Billing Book Now, web lead, interactions/events Checkout, add “cross sell” Book, fill out info Bounce rates, CTR Purchase Book Now, web lead, interactions/events “Rising tide” A/B (large test) Single page LPO (small test) 10% - 30% minimum targeted gains
  • Conversion goal results
  • How to test big Decide Collaborate Identify Create Design Analyze Create hypothesis and set up validation sheet
  • Manual testing database Number of Days 40
  • Control Treatment Large Test: Lead forms 1 Security seals
  • Control Fail – no lift. Large Test: Lead forms 1 Treatment
  • Control Treatment Large Test: Lead forms 2 Shorter form
  • Control Fail – no lift. Large Test: Lead forms 2 Treatment
  • Control Treatment Large Test: Lead forms 3 Option to call
  • Control Treatment Fail – no lift. Large Test: Lead forms 3
  • How to test big Decide Collaborate Identify Create Design Analyze Wireframe, design and launch treatment
  • Large Test: Lead forms
  • Large Test: Lead forms
  • Large Test: Lead forms 19% Lift
  • How to test big Decide Collaborate Identify Create Design Analyze Analyze results and calculate ROI
  • Manual testing database
  • Testing roadmap – it’s really: Search Results Property Details Cart Billing info Booking
  • ROI template Same as a small test.
  • Setting up your own programmatic testing 1 2 3 4 5 Custom variable tracking system, spreadsheet templates are vital building blocks Dedicate a person and tie performance to testing “Top of the funnel” to the “bottom conversion” may have too much noise in between – keep it simple and isolated Build a reporting system that ties front end and back end together Get buy-in from the management team
  • Questions?
  • Thank you Ryan Hutchings Marketing Director VacationRoost @HutchSEO Daniel Burstein Director of Editorial Content MECLABS @DanielBurstein Black & White Headshot Black & White Headshot