Measure Right the First Time - Infusionsoft Partnercon

1. MEASURE RIGHT THE FIRST TIME Infusionsoft Partnercon 2012 vuurr.com
2. WHAT WE’LL COVER • What you need • How to track • How to design experiments • What to experiment on • Putting it all together
3. TOOLS
4. TOOLS YOU MUST HAVE • Google Analytics (or SiteCatalyst, or similar) • Optimizely (or similar, but you need killer statistics) • Call tracking => analytics platform of choice • Link builder appropriate for your analytics platform of choice • AdWords/AdCenter/Facebook properly configured (if advertising)
5. DETERMINE YOUR MODEL CPA ROI MARGIN Minimize Maximize Minimize ad the cost of revenue for spend as a each every dollar percent of customer spent revenue acquisition
6. DETERMINE YOUR MODEL CPA ROI MARGIN Ad Spend ($) Revenue - Spend Revenue - Spend Conversion (qty) Spend Revenue answer is in dollars ($) per all in dollars ($), answer is % all in dollars ($), answer is % conversion
7. DETERMINE YOUR MODEL - FINDING ROI & MARGIN MARGIN ROI
8. DETERMINE YOUR MODEL - FINDING CPA (MANUALLY) Advertising => AdWords Completed => Campaigns => Site Usage Conversions Total Cost Advertising => AdWords => Campaigns => Clicks
9. DETERMINE YOUR MODEL - FINDING CPA (AUTOMATICALLY) • Use the Google Analytics API to query for total number of conversion by source & medium, and then properly divide them into the total marketing expense for each source & medium. • Use a third party tool
10. TRACKING
11. INSTALLING ANALYTICS • Follow your vendor’s instructions • Don’t just dump the tracking code in and “go” • Setup proper funnels, events, goals, and filters • Know the difference between accounts, properties, and profiles • Attach GWT and AdWords (if applicable) • Track ALL OF THE THINGS
12. GOALS & ECOMMERCE • Assign relevant goal values. A video view != purchase • Track eCommerce as accurately as possible • Explicit funnels are only useful if the supporting steps are relevant and provide information - use sparingly
13. TRACKING CAMPAIGNS • Every click is valuable Name/ • Every traffic source Content matters • Proper campaign Campai tracking is crucial to adequately gauging gn traffic value Source Medium
14. PROPER URL TAGGING
15. TRACKING RICH MEDIA: VIDEOS • Instantiate video with YouTube/Vendor JavaScript API • Track state changes, namely Plays and Completes • On these changes, push GA events
16. TRACKING RICH MEDIA: SOCIAL BUTTONS _gaq.push(['_trackSocial', network, socialAction, opt_target, opt_pagePath]); Instantiate by subscribing to edge.create and message.send Google Analytics handles for you Instantiate by subscribing to the ‘tweet’ intent event
17. TRACKING RICH MEDIA: FORM ABANDONMENT • Track full and empty exits as events • Determine fields causing abandonment
18. TRACKING OFFLINE: PHONE CALLS • Build GA gif request and HTTP GET on phone call • Push calls as pageviews ( /call/ +14805551234/abcdefg ) • Use Twilio & Twimlbin or another 3rd party service • Consider filtering out those pageviews from main profile • Setup goals in Google Analytics to fire on calls • Give each ad, landing page, social source, different number
19. TRACKING PAID CLICKS • Properly configure AdWords to push to Analytics • Verify all cost data is applied to Analytics account • Put MSN AdCenter {AdId} & {Keyword} in destination URL, then use AdCenter API to pull the cost and associate with relevant clicks from Analytics API
20. EXPERIMENT DESIGN
21. INTERNET MARKETING IS ABOUT EXPERIMENTATION y = f( x1, x2, x3, . . . . xn ) y CTR CTR X1= Page or Ad Copy Page or Ad Copy CR CR X2= Page Layout Page Layout X3= Button Color & Size Button Color & Size Goals Goals Xn= Other Factors Other Factors
22. DESIGNING EXPERIMENTS - ONE FACTOR AT A TIME Let’s run an experiment where we test two different button colors. Run Button Color ( x1 ) CTR ( y ) 1 Green 5.2% 2 Orange 7.7% Factors are tested in series, one at a time. This is extremely time intensive, requiring many tests for very gradual improvement.
23. DESIGNING EXPERIMENTS - FULL FACTORIAL DESIGN Let’s run an experiment where we test two different button colors, button positions, and button labels simultaneously. 3 FACTORS WITH 2 VARIATIONS EACH Number of Unique Tests =2 =8 3 This tests multiple changes in parallel to find optimal combination of factors (x variables) to optimize results (y variable).
24. DESIGNING EXPERIMENTS: FULL FACTORIAL DESIGN Run Color (x1) Position (x2) Text (x3) CTR (y) 1 Green Left Try Now 5.2% 2 Orange Left Try Now 7.7% 3 Green Right Try Now 9.3% 4 Orange Right Try Now 12.7% 5 Green Left Buy Today 21.2% 6 Orange Left Buy Today 19.3% 7 Green Right Buy Today 14.9% 8 Orange Right Buy Today 17.7%
25. DESIGNING EXPERIMENTS: OFAT VS. FULL FACTORIAL ONE FACTOR FULL FACTORIAL AT A TIME More Complex to Setup Iterative Simple to Setup VS. Requires more visitor data to test all Easy to Measure Results combinations SLOW & GRADUAL QUICK & DRASTIC IMPROVEMENT IMPROVEMENT
26. DESIGNING EXPERIMENTS: STATISTICAL SIGNIFICANCE Preliminary results - is orange better? Button Color ( Visitors Clicks CTR ( y ) x1 ) Green 38 2 5.2% Orange 39 3 7.7% Too small of a sample size leads to incorrect conclusions.
27. DESIGNING EXPERIMENTS: STATISTICAL SIGNIFICANCE It turns out our original conclusion of CTR was incorrect. Button Color ( Visitors Clicks CTR ( y ) x1 ) Green 8,238 486 5.9% Orange 7,893 734 9.3% Too large of a sample size wastes time, effort, & money.
28. SO WHAT QUANTITY OF VISITORS IS ENOUGH?
29. DETERMINING MINIMUM SAMPLE SIZE x = Sample mean E=x-μ μ = Population mean Z = Number of standard σ deviations above mean α = Confidence distance from 100% E=Z α/2 σ = Standard deviation n = Minimum sample size √n
30. DETERMINING MINIMUM SAMPLE SIZE - α/2 Z α/2
31. DETERMINING MINIMUM SAMPLE SIZE (CONTINUED) σ Standard Deviation E=Z α/2 √n Minimum Sample Size n= [Z α/2 σ] E
32. A/B TESTING SOFTWARE
33. TESTING
34. EVERY SINGLE CLICK HAS A COST ASSOCIATED WITH IT
35. KNOW YOUR LEVERS • Product Pricing • Organic Rankings • Ad bids, placement, content • Email subject lines & content • Social content • Site Content
36. FUNNEL ANALYSIS & OPTIMIZATION IMPRESSIONS Traffic Sources drive impressions & clicks CT • SEM, SEO, Social, Email, & More R CLICKS Optimizing those sources get more clicks (Site Visits) • • Improve Quality Score Split Test Ad Copy • Adjust Keyword Bids • Increase Organic Rankings • Increase Social Reach CR Optimized landing pages & site content increase conversions • Split Test Landing Page Copy • Improve Call to Action • Experiment with Deep CONVERSION Linking S • • Look at Bounce Rates Look at Cart Abandonment (Sales/Leads)
37. FUNNEL ANALYSIS & OPTIMIZATION ORGANIC OFFLINE S EARC H MARKETING AFFILIATES EMAIL PPC S OC IAL PHONE C ALLS ALL OF THESE SOURCES HAVE LEVERS THAT MANIPULATE THEIR QUALITY & LIKELIHOOD TO BECOME BUYERS
38. EMAIL OPTIMIZATION LEVERS EMAILS SENT LEVER 1: List Size Increase List Size • Completely • Optimize CR on dependent on subscription page conversion rate of • Increase sales to lure subscription form repeat buyers OPENED EMAILS LEVER 2: Open Rate Test Subject Line LEVER 3: CTR Test Email Copy LINKS CLICKED
39. EMAIL OPTIMIZATION LEVERS EMAILS SENT LEVER 1: List Size Increase List Size LEVER 2: Open Rate Test Subject Line Quantity • Use an email OPEN = application with A/B RATE Opened testing OPENED EMAILS Quantity Sent • Call to action is important LEVER 3: CTR Test Email Copy LINKS CLICKED
40. EMAIL OPTIMIZATION LEVERS EMAILS EMAILS SENT SENT LEVER 1: List Size Increase List Size LEVER 2: Open Rate Test Subject Line LEVER 3: CTR Test Email Copy OPENED • Treat as a landing EMAILS Clicks from CTR = Email page • Test layout, buttons, Quantity content, etc. Opened LINKS CLICKED
41. PPC OPTIMIZATION LEVERS AD IMPRESSIONS LEVER 1: Budget Increase Budget • Directly drive • Optimize for maximum conversions factoring in conversions at current conversion rate & CPC CPC LEVER 2: Bids Optimize for KPI AD CLICKS LEVER 3: Quality Score Maximize QS LEVER 4: CTR Test Ad Copy
42. PPC OPTIMIZATION LEVERS AD IMPRESSIONS LEVER 1: Budget Increase Budget LEVER 2: Bids Optimize for KPI Actual + Ranking Score of Position Below Cost = + $0.01 Quality Score of Your Ad AD CLICKS LEVER 3: Quality Score Maximize QS LEVER 4: CTR Test Ad Copy
43. PPC OPTIMIZATION LEVERS AD IMPRESSIONS LEVER 1: Budget Increase Budget LEVER 2: Bids Optimize for KPI LEVER 3: Quality Score Maximize QS • Function of ad content, • Relevant ad landing page content, • Relevant landing page AD CLICKS historical CTR, & more • Display URL LEVER 4: CTR Test Ad Copy
44. PPC OPTIMIZATION LEVERS AD IMPRESSIONS LEVER 1: Budget Increase Budget LEVER 2: Bids Optimize for KPI LEVER 3: Quality Score Maximize QS LEVER 4: CTR Test Ad Copy AD CLICKS • Continuously A/B test CTR = Clicks from Ad ad Ad Impressions • Keep highest CTR ad • Delete lower CTR ads • Repeat until returns diminish
45. SOCIAL CONTENT OPTIMIZATION LEVERS SOCIAL IMPRESSIONS LEVER 1: Network Add Networks • Be there in the first • Pinterest place • Reddit • Others SOCIAL CONTENT LEVER 2: Content Quantity/Quality CLICKS LEVER 3: Interactions Followers/Frequency
46. SOCIAL CONTENT OPTIMIZATION LEVERS SOCIAL IMPRESSIONS LEVER 1: Network Add Networks LEVER 2: Content Quantity/Quality • Links to deep content • Frequency • Polite, relevant posts • Better content drives more shares/retweets SOCIAL CONTENT CLICKS LEVER 3: Interactions Followers/Frequency
47. SOCIAL CONTENT OPTIMIZATION LEVERS SOCIAL IMPRESSIONS LEVER 1: Network Add Networks LEVER 2: Content Quantity/Quality LEVER 3: Interactions Followers/Frequency SOCIAL CONTENT • More interactions • Increase quantity and CLICKS drives more clicks value of followers • Don’t “buy” followers
48. WEBSITE LAYOUT & CONTENT VISITS LEVER 1: Layout Forms, Phone #s LEVER 2: Content Quantity, Quality LEVER 3: Click Areas Size, Color, Position LEVER X: etc..... CONVERSIONS
49. EXECUTION
50. THE WEB IS A SYSTEM • Your marketing strategy is a system of equations • Some levers are independent of others • Some levers manipulate others • Some levers have a larger impact on KPI than others Ad Bid SEO Email Social Spend Subject Effort
51. THE WEB IS A SYSTEM • Think about moving several levers at once • Always consider what a single and multiple-lever move will do to the system, and accordingly, your end result • Design a good experiment, continue to measure, and repeat y = f( x1, x2, x3, . . . . xn )
52. EXAMPLE SCENARIO • Current Budget: $7,500 / month ( $250 / day ) • Conversions: 48 / month ( 1.6 / day ) • Average CPA = $158.17 • Average CPC = $2.29 • Landing Page CR = 1.45% New Target: 10 Conversions / Day AND Lower CPA
53. SCENARIO ONE: ADJUSTING BUDGET • Average CPA at $158.17 means 10 conversions per day is $1581.70 • $47,000 per month • Easy, but too expensive
54. SCENARIO TWO: ADJUSTING CPC • Budget stays at $250 / day, conversion rate steady at 1.45% • Need 689 clicks to get 10 conversions • Average CPC must be $0.36 • Improbable to bring $2.29 down to $0.36 in this space while maintaining volume
55. SCENARIO THREE: ADJUST CONVERSION RATE • Budget stays at $250 / day, CPC at $2.29 • Can purchase 110 clicks for $250 • To get 10 conversions, landing page needs to convert at 9.1% • Improbable to quickly jump from 1.45% to 9.1%
56. FINAL SCENARIO: MOVE ALL THREE LEVERS • Double the budget to $15,000 • Reduce average CPC by 50% ( $2.29 to $1.15 ) via QS & Keywords • Improve landing page CR from 1.45% to 2.3% via A/B testing • ( $15k / $1.15 per click ) at 2.3% drives 300 conversions monthly at a CPA of $50 • Used 3 levers simultaneously to improve conversions 6x and reduce average CPA by 68%
57. ITERATING
58. CREATE A CULTURE OF CONTINUOUS IMPROVEMENT • Define success • Measure current performance and compare to targets • Determine biggest levers and how they contribute to KPIs • Design an experiment • Test repeatedly • Finalize improvements based on adequate data • Repeat
59. Questions? Scott Yacko - scott@vuurr.com - @scottmyacko Jonathan Kressaty - jonathan@vuurr.com - @kressaty

Editor's Notes

Jonathan Analytics platform of choice is irrelevant, as long as it has the major features of something like Google analytics. We ’ ll be acting as if you use Google Analytics exclusively. It ’ s our preferred application. Optimizely is our favorite A/B testing tool - Adobe ’ s offering is great but complicated and expensive There are many other tools Call tracking is crucial if there ’ s a phone number on the website. We roll our own using Twimlbin & Twilio Just get the data into Analytics. You must get in the habit of tagging links so they track in analytics. Google has a utility that does this which we ’ ll review later If you ’ re doing online advertising, configure those tools to properly push into Analytics. Cost and all Campaign data can be automatically configured to push into analytics, but you ’ ll need to properly tag your AdCenter and Facebook advertising URLs.
Jonathan It ’ s crucial that whether you ’ re working on your own or a client ’ s site, you understand the revenue model in play. CPA is minimizing the cost of customer acquisition on a per customer basis. ROI is maximizing the return on every advertising dollar spent. Margin is maximizing advertising dollars spent as taken from revenue to the point of diminishing marginal returns - it ’ s minimizing ad spend as a percent of revenue while increasing revenue.
Jonathan
Jonathan Advertising section, AdWords, Campaigns, and click on the “ Clicks ” subsection This is only useful for AdWords. Apply the formula mentioned previously to all other traffic sources
Jonathan
Jonathan
Jonathan
Jonathan Goals are everything Clients or previous consultants have often setup goals that are totally irrelevant. You need to track ecommerce as accurately as possible. Consult external system for accuracy If advertising, you MUST completely understand their costs. Simply asking what the target margin is is unacceptable, and often times inaccurate. When setting up goals, funnels are often put in place to show drop off in the steps. Be careful - if someone can complete your goal or purchase without completing the funnel steps, your goal will not fire correctly, and your metrics will be off.
Jonathan Everything belongs to a campaign - SO TRACK IT Source and medium are self explanatory Name/Content can be many things - keyword, twitter campaign, hashtag, email campaign, etc. Don ’ t just fill out the url builder - keep it CONSISTENT
Jonathan
Jonathan To track video plays, bind an event to the "Play" action of the player. You can do this with almost every embedable player, our favorite is YouTube.
Jonathan Track social events with _trackSocial() method Treats social interactions as a special event in Google Analytics Attributed to engagement and conversion
Jonathan Knowing how your form converts is crucial - we go down to individual fields Track blur on each field and hit an event if it ’ s empty or full Determine which form fields cause problems
Jonathan
Jonathan
Scott This is the simplest way to experiment - one factor at a time Make sure your Y-value (what you ’ re measuring) is aligned with what ’ s critical to the client In this case, CTR of a button
Scott This is where you test multiple factors at one time in combination with one another This requires more data because you need to test every combination of factors More Factors = more unique tests in order to gain statistically significant data More Variations also = more unique tests “” If you ’ re short on time, decide what ’ s more important - the number of factors you ’ re testing, or number of variations of each factor
Scott
Scott: We recommend one factor for simple cases where you ’ re trying to influence one output Often times this is with a single drastic change We recommend full factorial for things like designing a new landing page, where you want to test copy, images, buttons, layout, and more all at the same time
Scott
Scott
Scott This is the data that ’ s available out of either a one factor or full factorial test Trying to get “ N ” value, which is the minimum number of test participants necessary to achieve statistical significance Sample mean is the arithmetic average number of your output of your new variant U is the arithmetic average of all of your traffic (it ’ s the Y value) E is the difference between those - is your test variant different from your typical traffic? Z is a measure of how confident you want to be in your results (90%, 95%, 98%, etc) Alpha is the distance from 100% Sigma is standard deviation - a measure of how much variation there is in your output
Scott This is a normal curve Important to determine how confident you want to be such that your change is actually significant As Za/2 creates a narrower range, you are less confident in your results
Scott To make this valuable, need to solve for n in terms of the other variables What ’ s important here isn ’ t the actual algebra, but how each factor drives the number of unique visitors necessary to achieve valid test results As Za/2 increases (this means you want to have greater confidence) it requires more test participants in order to meet that confidence requirement As sigma increases (there is more variation in your output) it requires more tests to prove that any change is more than just a statistical anomaly As E increases (there is a larger gap between the output of your test and the output of the rest of your traffic) it requires less participants since the difference between the test and the baseline is so great
Scott This is what this looks like in A/B testing software such as optimizely - it does most of the work for you The conversion rate of each variation is your x-bar, and the change to beat baseline is your confidence (which is correlated to Za/2) Usually 95% is acceptable to declare a winning variation.
Scott
Scott Remember to think of your digital marketing such that Y is a function of X ’ s, which are variables Some of these you can control, some you can ’ t These are some typical X ’ s that impact the typical outputs: CPA, Margin, ROI
Scott A typical digital marketing model is a funnel where a portion of impressions become clicks and a portion of those clicks become conversions Between impressions and clicks we measure CTR, and between clicks and conversions we measure CR Each of these Y-values are controlled by different levers - things you can change to influence the output
Scott Each traffic source has its own performance level in terms of the output Y The overall output of your marketing system is the weighted average of all your traffic sources You can increase or decrease marketing spend on each traffic source to influence the weighting and thus the overall output
Scott
Scott
Scott
Scott
Scott
Scott
Scott
Scott
Scott
Scott
Scott
Scott Not all levers are created equal Some have a much larger impact on output than others Some have zero impact on output Some only influence other levers It is important to know which levers have large and small impacts so you can budget time and effort accordingly
Scott It is important to always think about the entire system - moving one lever doesn ’ t only change one metric, often times it influences many or all metrics The further down in the funnel you go (i.e. the closer to conversion you are), the more drastic an effect the lever has on output In order to ensure that all changes to the funnel are trackable and accounted for, you need to design great experiments
Scott We ’ re going to run through three scenarios, each moving only one lever at a time We will then move all three at once to maximize for our outcome
Scott
Scott
Scott
Scott
Jonathan

Measure Right the First Time - Infusionsoft Partnercon

You are reading a preview.

Check these out next

Measure Right the First Time - Infusionsoft Partnercon

More Related Content

Similar to Measure Right the First Time - Infusionsoft Partnercon (20)

Measure Right the First Time - Infusionsoft Partnercon

Editor's Notes