Your SlideShare is downloading. ×
Measure Right The First Time
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Measure Right The First Time

6,921
views

Published on

The prevailing culture of “Ship the MVP!” – where sites and apps are built in the cheapest, fastest way possible without regard for the future – needs to end. In this session, Vuurr will show you how …

The prevailing culture of “Ship the MVP!” – where sites and apps are built in the cheapest, fastest way possible without regard for the future – needs to end. In this session, Vuurr will show you how to build web properties right the first time, specifically emphasising analytics and data collection. Learn how to properly configure many analytics tools, track offline interactions like phone calls and tangible marketing efforts, and structure onsite code and content. In addition, you will learn what to do after data has been collected and it’s time to iterate. We’ll show you how these strategies can lead to better decisions by you and your clients.

Published in: Technology

0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,921
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
0
Comments
0
Likes
13
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. MEASURE RIGHTTHE FIRST TIME Scott Yacko & Jonathan Kressaty BOLO 2012 vuurr.com 1
  • 2. WHAT WE’LL COVER_ What you need_ How to track_ How to design experiments_ What to experiment on_ Putting it all together 2
  • 3. INTERNET MARKETING IS THE INTERSECTION OF COPYWRITING AND MATHEMATICS. 3
  • 4. YOUR HIRING DECISIONSSHOULD REFLECT THIS. 4
  • 5. TOOLS 5
  • 6. TOOLS YOU MUST HAVE_ Google Analytics (or SiteCatalyst, or similar)_ Optimizely (or similar, but you need killer statistics)_ Call tracking => analytics platform of choice_ Link builder appropriate for your analytics platform of choice_ AdWords/AdCenter/Facebook properly configured (if advertising) 6
  • 7. DETERMINE YOUR MODEL CPA ROI MARGIN Minimize the Maximize Minimize ad cost of each revenue for spend as a customer every dollar percent of acquisition spent revenue 7
  • 8. DETERMINE YOUR MODEL CPA ROI MARGIN Ad Spend ($) Revenue - Spend Revenue - Spend Conversion (qty) Spend Revenueanswer is in dollars ($) per conversion all in dollars ($), answer is % all in dollars ($), answer is % 8
  • 9. DETERMINE YOUR MODEL -FINDING ROI & MARGIN MARGIN ROI 9
  • 10. DETERMINE YOUR MODEL - FINDING CPA (MANUALLY) CompletedAdvertising => AdWords => Campaigns => Site Usage Conversions Advertising => AdWords => Campaigns => Clicks Total Cost 10
  • 11. DETERMINE YOUR MODEL -FINDING CPA (AUTOMATICALLY) _ Use the Google Analytics API to query for total number of conversion by source & medium, and then properly divide them into the total marketing expense for each source & medium. _ Use a third party tool 11
  • 12. TRACKING 12
  • 13. INSTALLING ANALYTICS_ Follow your vendor’s instructions_ Don’t just dump the tracking code in and “go”_ Setup proper funnels, events, goals, and filters_ Know the difference between accounts, properties, and profiles_ Attach GWT and AdWords (if applicable)_ Track ALL OF THE THINGS 13
  • 14. GOALS & ECOMMERCE_ Assign relevant goal values - video view != purchase_ Track eCommerce as accurately as possible_ Explicit funnels are only useful if the supporting steps are relevant and provide information - use sparingly 14
  • 15. TRACKING CAMPAIGNS _ Every click is valuable Name/ Content _ Every traffic source matters _ Proper campaign tracking is Campaign crucial to adequately gaugingSource Medium traffic value 15
  • 16. PROPER URL TAGGING 16
  • 17. TRACKING RICH MEDIA:VIDEOS_ Instantiate video with YouTube/Vendor JavaScript API_ Track state changes, namely Plays and Completes_ On these changes, push GA events 17
  • 18. TRACKING RICH MEDIA:SOCIAL BUTTONS_gaq.push([_trackSocial, network, socialAction, opt_target,opt_pagePath]); Instantiate by subscribing to edge.create and message.send Google Analytics handles for you Instantiate by subscribing to the ‘tweet’ intent event 18
  • 19. TRACKING RICH MEDIA:FORM ABANDONMENT _ Track full and empty exits as events _ Determine fields causing abandonment 19
  • 20. TRACKING OFFLINE:PHONE CALLS _ Build GA gif request and HTTP GET on phone call _ Push calls as pageviews ( /call/+14805551234/abcdefg ) _ Use Twilio & Twimlbin or another 3rd party service _ Consider filtering out those pageviews from main profile _ Setup goals in Google Analytics to fire on calls _ Give each ad, landing page, social source, different number 20
  • 21. TRACKING PAID CLICKS _ Properly configure AdWords to push to Analytics _ Verify all cost data is applied to Analytics account _ Put MSN AdCenter {AdId} & {Keyword} in destination URL, then use AdCenter API to pull the cost and associate with relevant clicks from Analytics API 21
  • 22. EXPERIMENT DESIGN 22
  • 23. INTERNET MARKETING ISABOUT EXPERIMENTATIONy = f( x1, x2, x3, . . . . xn )y= CTR x1 = Page or Ad Copy CR x2 = Page Layout Goals x3 = Button Color & Size xn = Other Factors... 23
  • 24. DESIGNING EXPERIMENTS - ONE FACTOR AT A TIMELet’s run an experiment where we test two different button colors. Run Button Color ( x1 ) CTR ( y ) 1 Green 5.2% 2 Orange 7.7%Factors are tested in series, one at a time. This is extremely timeintensive, requiring many tests for very gradual improvement. 24
  • 25. DESIGNING EXPERIMENTS - FULL FACTORIAL DESIGNLet’s run an experiment where we test two different button colors,button positions, and button labels simultaneously. 3 FACTORS WITH 2 VARIATIONS EACH Number of Unique Tests =23=8This tests multiple changes in parallel to find optimal combination offactors (x variables) to optimize results (y variable). 25
  • 26. DESIGNING EXPERIMENTS -FULL FACTORIAL DESIGN Run Color (x1) Position (x2) Text (x3) CTR (y) 1 Green Left Try Now 5.2% 2 Orange Left Try Now 7.7% 3 Green Right Try Now 9.3% 4 Orange Right Try Now 12.7% 5 Green Left Buy Today 21.2% 6 Orange Left Buy Today 19.3% 7 Green Right Buy Today 14.9% 8 Orange Right Buy Today 17.7% 26
  • 27. DESIGNING EXPERIMENTS -OFAT VS. FULL FACTORIALONE FACTOR AT A TIME FULL FACTORIAL Iterative More Complex to Setup Simple to Setup VS. Requires more visitor dataEasy to Measure Results to test all combinations SLOW & GRADUAL QUICK & DRASTIC IMPROVEMENT IMPROVEMENT 27
  • 28. DESIGNING EXPERIMENTS - STATISTICAL SIGNIFICANCEPreliminary results - is orange better?Button Color ( x1 ) Visitors Clicks CTR ( y ) Green 38 2 5.2% Orange 39 3 7.7%Too small of a sample size leads to incorrect conclusions. 28
  • 29. DESIGNING EXPERIMENTS - STATISTICAL SIGNIFICANCEIt turns out our original conclusion of CTR was incorrect.Button Color ( x1 ) Visitors Clicks CTR ( y ) Green 8,238 486 5.9% Orange 7,893 734 9.3% Too large of a sample size wastes time, effort, & money. 29
  • 30. SO WHAT QUANTITY OF VISITORS IS ENOUGH? 30
  • 31. DETERMINING MINIMUM SAMPLE SIZEx = Sample meanµ = Population mean E=x-µZ = Number of standarddeviations above meanα = Confidence distance σfrom 100%σ = Standard deviation E = Z α/2 √nn = Minimum sample size 31
  • 32. DETERMINING MINIMUMSAMPLE SIZE - α/2 Z α/2 32
  • 33. DETERMINING MINIMUMSAMPLE SIZE (CONTINUED) σ Standard E = Z α/2 √n Deviation Minimum Sample Size Z σ n= [ α/2 E ] 33
  • 34. A/B TESTING SOFTWARE 34
  • 35. TESTING 35
  • 36. EVERY SINGLE CLICK HAS ACOST ASSOCIATED WITH IT 36
  • 37. KNOW YOUR LEVERS _ Product Pricing _ Organic Rankings _ Ad bids, placement, content _ Email subject lines & content _ Social content _ Site Content 37
  • 38. FUNNEL ANALYSIS &OPTIMIZATION IMPRESSIONS Traffic Sources drive impressions & clicks CTR _ SEM, SEO, Social, Email, & More Optimizing those sources get more clicks CLICKS _ Improve Quality Score (Site Visits) _ Split Test Ad Copy _ Adjust Keyword Bids _ Increase Organic Rankings _ Increase Social Reach CR Optimized landing pages & site content increase conversions _ Split Test Landing Page Copy _ Improve Call to Action _ Experiment with Deep Linking CONVERSIONS _ Look at Bounce Rates _ Look at Cart Abandonment (Sales/Leads) 38
  • 39. FUNNEL ANALYSIS &OPTIMIZATION ORGANIC OFFLINE EMAIL SEARCH MARKETING AFFILIATES PPC SOCIAL PHONE CALLS ALL OF THESE SOURCES HAVE LEVERS THAT MANIPULATE THEIR QUALITY & LIKELIHOOD TO BECOME BUYERS 39
  • 40. EMAIL OPTIMIZATION LEVERSEMAILS SENT LEVER 1: List Size Increase List Size _ Completely _ Optimize CR on dependent on subscription page conversion rate of _ Increase sales to lure OPENED subscription form repeat buyers EMAILS LEVER 2: Open Rate Test Subject Line LEVER 3: CTR Test Email Copy LINKS CLICKED 40
  • 41. EMAIL OPTIMIZATION LEVERSEMAILS SENT LEVER 1: List Size Increase List Size LEVER 2: Open Rate Test Subject Line Quantity Opened _ Use an email application OPEN = OPENED with A/B testing RATE Quantity Sent _ Call to action is EMAILS important LEVER 3: CTR Test Email Copy LINKS CLICKED 41
  • 42. EMAIL OPTIMIZATION LEVERSEMAILS SENT LEVER 1: List Size Increase List Size LEVER 2: Open Rate Test Subject Line LEVER 3: CTR Test Email Copy OPENED Clicks from Email _ Treat as a landing page EMAILS CTR = Quantity Opened _ Test layout, buttons, content, etc. LINKS CLICKED 42
  • 43. PPC OPTIMIZATION LEVERSAD IMPRESSIONS LEVER 1: Budget Increase Budget _ Directly drive _ Optimize for maximum conversions factoring in conversions at current CPC conversion rate & CPC LEVER 2: Bids Optimize for KPI AD CLICKS LEVER 3: Quality Score Maximize QS LEVER 4: CTR Test Ad Copy 43
  • 44. PPC OPTIMIZATION LEVERSAD IMPRESSIONS LEVER 1: Budget Increase Budget LEVER 2: Bids Optimize for KPI Ranking Score of Position Below Actual Cost = Quality Score of Your Ad + $0.01 AD CLICKS LEVER 3: Quality Score Maximize QS LEVER 4: CTR Test Ad Copy 44
  • 45. PPC OPTIMIZATION LEVERSAD IMPRESSIONS LEVER 1: Budget Increase Budget LEVER 2: Bids Optimize for KPI LEVER 3: Quality Score Maximize QS _ Function of ad content, _ Relevant ad AD CLICKS landing page content, _ Relevant landing page historical CTR, & more _ Display URL LEVER 4: CTR Test Ad Copy 45
  • 46. PPC OPTIMIZATION LEVERSAD IMPRESSIONS LEVER 1: Budget Increase Budget LEVER 2: Bids Optimize for KPI LEVER 3: Quality Score Maximize QS AD CLICKS LEVER 4: CTR Test Ad Copy Clicks from Ad _ Continuously A/B test ad CTR = Ad Impressions _ Keep highest CTR ad _ Delete lower CTR ads _ Repeat until returns diminish 46
  • 47. SOCIAL CONTENTOPTIMIZATION LEVERS SOCIAL IMPRESSIONS LEVER 1: Network Add Networks _ Be there in the first place _ Pinterest _ Reddit _ OthersSOCIAL CONTENT CLICKS LEVER 2: Content Quantity/Quality LEVER 3: Interactions Followers/Frequency 47
  • 48. SOCIAL CONTENTOPTIMIZATION LEVERS SOCIAL IMPRESSIONS LEVER 1: Network Add Networks LEVER 2: Content Quantity/Quality _ Links to deep content _ Frequency _ Polite, relevant posts _ Better content drivesSOCIAL CONTENT more shares/retweets CLICKS LEVER 3: Interactions Followers/Frequency 48
  • 49. SOCIAL CONTENTOPTIMIZATION LEVERS SOCIAL IMPRESSIONS LEVER 1: Network Add Networks LEVER 2: Content Quantity/Quality LEVER 3: Interactions Followers/FrequencySOCIAL CONTENT _ More interactions drives _ Increase quantity and CLICKS more clicks value of followers _ Don’t “buy” followers 49
  • 50. WEBSITE LAYOUT & CONTENT VISITS LEVER 1: Layout Forms, Phone #s LEVER 2: Content Quantity, Quality LEVER 3: Click Areas Size, Color, PositionCONVERSIONS LEVER X: etc..... 50
  • 51. EXECUTION 51
  • 52. THE WEB IS A SYSTEM_ Your marketing strategy is a system of equations_ Some levers are independent of others_ Some levers manipulate others_ Some levers have a larger impact on KPI than othersAd Bid SEO Email Social Spend Subject Effort 52
  • 53. THE WEB IS A SYSTEM_ Think about moving several levers at once_ Always consider what a single and multiple-lever move will do tothe system, and accordingly, your end result_ Design a good experiment, continue to measure, and repeaty = f( x1, x2, x3, . . . . xn ) 53
  • 54. EXAMPLE SCENARIO_ Current Budget: $7,500 / month ( $250 / day )_ Conversions: 48 / month ( 1.6 / day )_ Average CPA = $158.17_ Average CPC = $2.29_ Landing Page CR = 1.45%New Target: 10 Conversions / Day AND Lower CPA 54
  • 55. SCENARIO ONE:ADJUSTING BUDGET_ Average CPA at $158.17 means 10 conversions per day is $1581.70_ $47,000 per month_ Easy, but too expensive 55
  • 56. SCENARIO TWO:ADJUSTING CPC_ Budget stays at $250 / day, conversion rate steady at 1.45%_ Need 689 clicks to get 10 conversions_ Average CPC must be $0.36_ Improbable to bring $2.29 down to $0.36 in this space whilemaintaining volume 56
  • 57. SCENARIO THREE:ADJUST CONVERSION RATE_ Budget stays at $250 / day, CPC at $2.29_ Can purchase 110 clicks for $250_ To get 10 conversions, landing page needs to convert at 9.1%_ Improbable to quickly jump from 1.45% to 9.1% 57
  • 58. FINAL SCENARIO:MOVE ALL THREE LEVERS_ Double the budget to $15,000_ Reduce average CPC by 50% ( $2.29 to $1.15 ) via QS & Keywords_ Improve landing page CR from 1.45% to 2.3% via A/B testing_ ( $15k / $1.15 per click ) at 2.3% drives 300 conversions monthly ata CPA of $50_ Used 3 levers simultaneously to improve conversions6x and reduce average CPA by 68% 58
  • 59. ITERATING 59
  • 60. CREATE A CULTURE OFCONTINUOUS IMPROVEMENT_ Define success_ Measure current performance and compare to targets_ Determine biggest levers and how they contribute to KPIs_ Design an experiment_ Test repeatedly_ Finalize improvements based on adequate data_ Repeat 60
  • 61. QUESTIONS? Scott Yacko - scott@vuurr.com - @scottmyacko Jonathan Kressaty - jonathan@vuurr.com - @kressaty Code & Presentation at vuurr.com/bolo 61