Five Steps to Better Metrics
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Five Steps to Better Metrics

  • 1,343 views
Uploaded on

Learn more about Web clinics here: http://bit.ly/LnULEI ...

Learn more about Web clinics here: http://bit.ly/LnULEI

More in: Business , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,343
On Slideshare
1,339
From Embeds
4
Number of Embeds
1

Actions

Shares
Downloads
80
Comments
0
Likes
2

Embeds 4

https://twitter.com 4

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Five Steps to Better Metrics:How one marketer leveraged web analytics for an annualrevenue increase of $500,000 #webclinic
  • 2. Join the conversation on Twitter #webclinic #webclinic
  • 3. Today’s team Dr. Flint McGlaughlin Jon Powell Managing Director Senior Manager Research and Strategy #webclinic
  • 4. The ChallengeQ: Are there metrics your organization does NOT monitor, only because they arenot set up properly? 4 #webclinic
  • 5. Background and Test Design Experiment ID: REGOnline Homepage Test Location: MarketingExperiments Research Library Test Protocol Number: TP1427Research Notes: Background: REGOnline is event management software that lets users create online registration forms and event websites to manage their events. Goal: To increase number of completed leads on homepage. Primary research question: Which page will generate the greatest number of leads? Approach: A/B multifactor split test #webclinic
  • 6. Experiment: Control Control - Homepage • Our researchers hypothesized that we could increase the appeal associated with the value proposition of this offer by focusing more on the product and its specific features and benefits. #webclinic
  • 7. Experiment: Treatment Treatment - Homepage• Headline was written to focus more on the product.• Specific features and benefits are utilized to express the value.• The page emphasizes “Free Access.”• Also, ensured that this value was being communicated in subsequent steps. 7 #webclinic
  • 8. Experiment: Side-by-side Control Treatment 8 #webclinic
  • 9. Experiment: Results 24.5% Decrease in Conversion The Treatment generated 24.5% less completed leads Conversion Versions Rel. diff Rate Control – Two-step homepage 2.3% - Treatment – Three-step homepage 1.7% -24.5%  What youthe amount of form fieldsspite of first step,clearer valuestill reducing need to understand: In in the having a the control and outperformed the treatment. #webclinic
  • 10. Experiment #2: Background Homepage from Previous Test • Before we could get a lift, we needed to learn more about the prospects coming to this site. 24.5% Decrease in Conversion • We decided to use one of their SEO pages as a research window into the cognitive psychology of the customer’s motivation. 10 #webclinic
  • 11. Experiment #2: Background Experiment ID: REGonline SEO landing page test Location: MarketingExperiments Research Library Test Protocol Number: TP3055Research Notes: Background: A technology and media company specializing in online registration and event management software. Goal: To increase the amount of leads generated online. Primary research question: Which online capture process will generate the higher addressable lead rate? Approach: A/B multifactor split test 11 #webclinic
  • 12. Experiment #2: Control SEO Landing Page • This landing page was offering the same product as the home page but dealt with a smaller subset of visitors who matched the profile of those coming to the homepage. • Our researchers could test here without the negative consequences of hurting conversion on the homepage. 12 #webclinic
  • 13. Experiment #2: Treatment Treatment SEO Landing Page• For our first test on this page, we tested focusing on how this product made the process of creating registration forms easier and could cut the prospects’ time in half…• …and yet it still had a robust functionality. #webclinic
  • 14. Experiment #2: Side-by-side Control Treatment Which copy language will generate the most leads? 14 #webclinic
  • 15. Experiment #2: Results 548% Increase in Complete Leads The new page’s conversion rate increased by 548.46% Conversion Rate Relative Statistical Level Design (%) Difference of Confidence Original Page 0.7% - - Treatment 4.8% 548% 99%  What you need to understand: Bythe treatment wasthis product made creating registration forms easier, focusing on how able to increase step-level clickthrough rate by 1,312%, and completed leads captured by 548%. 15 #webclinic
  • 16. Experiment #2: Final Results Original Homepage New Homepage Test SEO Page Test 548% Learning Learning 90% • We were able to take what we learned about the motivations of their customers from testing on the SEO landing page and apply it to the homepage, which generated a 90% increase in leads captured. 16 #webclinic
  • 17. What we discoveredF Key Principles 1. The goal of all customer research is to enable the marketer to predict customer behavior. 2. Therefore, the primary usefulness of metrics is not in answering “how many?” but rather in answering, “why so?” 3. Ultimately, metrics enable the marketer to see the cognitive trail left by the visitor’s mind. #webclinic
  • 18. How do we cut through it all? • The problem is not typically getting sufficient data from you metrics software. Rather, the challenge is making sense of it. 18 #webclinic
  • 19. Online Testing Heuristic Online Testing Heuristic: u = 2q + t + m + 2v + i © u = Utility q = Research Question t = Treatment m = Metric System v = Validity Factor i = Interpretation 19 #webclinic
  • 20. Today, we will walk through a simple 5-step process for translating raw testing data into predictive power 20#webclinic
  • 21. Translating Raw Data to Predictive PowerF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: 21 #webclinic
  • 22. STEP 1: Establish Visibility Types of Analytics – Visual Page views referrers search terms visitor sessions languages Amount Source returning visitors organizations impressions geographic location Entry pages Sign-ups Orders exit pages browsers Number of page views Nature Results Screen resolution time on page Click trails Load errors Most requested pages 22 #webclinic
  • 23. Translating Raw Data to Predictive PowerF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: Amount – How many instances of a particular action are occurring? Source – Where are prospects coming from? Nature – What are prospects experiencing on your site? Results – What are prospects doing on your site? 23 #webclinic
  • 24. Translating Raw Data to Predictive PowerF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results. 2. Determine Objective – Determine the exact research question you are setting out to answer with your metrics. 24 #webclinic
  • 25. STEP 2: Determine the ObjectiveThe Research Question 1. Whether you are running a live test or conducting a forensics metrics analysis, your research and metrics analysis must be grounded in a properly framed Research Question. 2. A properly framed Research Question is a question of “which” and sets out to identify an alternative (treatment) that performs better than the control. Example: Not this.. What is the best price for product X? But this… Which of these three price points is best for product X? * Depending on the data available, forensics data is often grounded in a research question of “what?” rather than “which”. 25 #webclinic
  • 26. STEP 2: Determine the ObjectiveAudience Exercise ? How would you refine the following three research questions? 1. What is the best headline for my landing page? 2. Why do I have such a high bounce rate on my offer page? 3. How many objectives should I have on my homepage? 26 #webclinic
  • 27. STEP 2: Determine the Objective The Research Question1. Often, metrics can also Unique visits be utilized to Flights 40,607,893 determine the most Hotels 32% effective research 14,185,646 Autos Not all visitors go questions you should 7,729,403 through each of these steps be asking. Activities 9,167,901 60% Travelers2. Metrics can be a 73% 12,883,177 Summary window into key gaps 58% 7,717,122 into your customer Login 5,665,020 76% theory and ultimately Contact 3,260,292 into the highest 71% Payment potential revenue 2,484,236 opportunities for Completion Rate once process begins 4% 1,766,609 marketing efforts. 27 #webclinic
  • 28. STEP 2: Determine the ObjectiveExample Case Study – Experiment Background Experiment ID: (Protected) Location: MarketingExperiments Research Library Test Protocol Number: TP1305Research Notes: Background: A website that sells retail and wholesale collector items Goal: To increase conversion rate Primary research question: Which version of second step in the conversion funnel will produce the highest conversion rate? Approach: A/B variable cluster split test that focused on reducing anxiety through credibility indicators, copy, and re-organization of existing page elements 28 #webclinic
  • 29. STEP 2: Determine the ObjectiveExample Case Study – Experiment Background Fallout Report: New Customers • When we analyzed the metrics, we realized there were leaks throughout the checkout process, the credit card submission page stood out as low cost opportunity for immediate return. • When we analyzed the metrics even further, we saw that this step also had the highest lost revenue per cart (more than double of any other step). • From this, we hypothesized that optimizing this step would have the highest potential return on our efforts. 29 #webclinic
  • 30. STEP 2: Determine the ObjectiveExample Case Study – Experiment Control Control What might be causing the fallout? • It is unclear why the credit card is required when payment method is different. • The complexity of the Purchase Agreement Terms’ causes confusion and concern. • There is no indication that my credit card information is secure. 30 #webclinic
  • 31. STEP 2: Determine the ObjectiveExample Case Study – Experiment Treatment Treatment How we addressed the issues: • Third-party security indicators have been added. • Clearer explanation of why a credit card is required and that it will not be charged. • “Satisfaction Guaranteed” promise is emphasized. 31 #webclinic
  • 32. STEP 2: Determine the ObjectiveExample Case Study – Experiment Results 5% Increase in total conversion The new credit card page increased conversion by 4.51% Design Conversion Rate Control 82.33% Treatment 86.04% Relative Difference 4.51%  What youthis specific step in theWhile it mighttoseem resulted in a projected choosing need to understand: sales funnel test like a small increase, $500,000+ increase in revenue per year. This underscores the potential impact of a properly identified research question. 32 #webclinic
  • 33. Translating Raw Data to Predictive PowerF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results. 2. Determine Objective – Determine the exact research question you are setting out to answer with your metrics. 3. Track and Measure – Track and measure the appropriate metrics that will provide you with the answer to your determined research question. 33 #webclinic
  • 34. STEP 3: Track and MeasurePrimary and Secondary Metrics 1. Primary “Test” Metrics: The essential metrics that enable you to answer the research question Primary Metrics 2. Secondary Metrics: The additional metrics you can utilize to help interpret the Secondary results of your primary metrics Metrics 34 #webclinic
  • 35. STEP 3: Track and MeasurePrimary Metrics – ExamplesExample #1: Research Question: Which headline will generate the most subscriptions? Primary Metrics: Visits, subscriptions  subscription rate (%)Example #2: Research Question: Which PPC ad will generate the most qualified traffic? Primary Metrics: Ad spend, conversions  cost per acquisition ($) Example #3: Research Question: Which page will generate the most Facebook fans? Primary Metrics: Visitors, clicks on the “Like” button  fans per visitor (%) 35 #webclinic
  • 36. STEP 3: Track and MeasureSecondary Metrics – Examples Secondary Metric Potential Insights Are visitors engaged with the content? Time on page Are they confused with the process? What are visitors interested in? Click tracking Are they confused with the process? Is there a lack of relevance to visitors? Bounce rate Are there too many distractions? Is there too much (or little) information? What motivates individual visitor types? Segment-level data Where are the deeper optimization opportunities? Form event tracking What form fields cause anxiety or confusion? How much friction will your visitor put up with? Traffic patterns Who is coming and where are they coming from? Can we be more relevant to the visitor? 36 #webclinic
  • 37. STEP 3: Track and MeasureExample Case Study – Experiment Background Experiment ID: (Protected) Location: MarketingExperiments Research Library Test Protocol Number: TP1341Research Notes: Background: A company offering dedicated hosting services Goal: To increase the number of leads Primary research question: Which page design will generate the greater number of leads? Approach: A/B multi-factor split test (radical redesign) 37 #webclinic
  • 38. STEP 3: Track and MeasureExample Case Study – Experiment Treatments Control Treatment Let’s consider both the primary and secondary metrics utilized for this test… 38 #webclinic
  • 39. STEP 3: Track and MeasureExample Case Study – Experiment Metrics Control Treatment Research Question: Which page design will generate the greater number of leads? Primary Metrics Primary Metrics Visits = 31,400* Visits = 30,560* leads = 628* Leads = 1,764* CR = 2.0% CR = 5.7% Answer: The treatment design will generate 188% more leads. * Numbers have been anonymized 39 #webclinic
  • 40. STEP 3: Track and MeasureExample Case Study – Experiment Metrics • In addition to tracking the primary metrics, the research analysts installed some secondary event tracking metrics. • On this page, there were six expandable sections of copy featuring different elements of the product value proposition. • By monitoring the specific clicks of visitors on this page, we were better able to understand what aspect of this product’s value proposition was most appealing to the visitor. 40 #webclinic
  • 41. Translating Raw Data to Predictive PowerF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results. 2. Determine Objective – Determine the exact research question you are setting out to answer with your metrics. 3. Track and Measure – Track and measure the appropriate metrics that will provide you with the answer to your determined research question. 4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a validity threat. 41 #webclinic
  • 42. STEP 4: Monitor Anomalies Audience Question ? What wrong with this test data set? 19.00% 17.00% 15.00%Conversion Rate 13.00% 11.00% Control 9.00% Treatment 3 7.00% 5.00% 3.00% Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Test Duration 42 #webclinic
  • 43. #webclinic campaign rates, etc.) Validity Threats to a specific online rates, sales, average • A more subtle clue is a of response visitors are temporary spikes in the having to a specific online amount of traffic or views campaign (e.g., conversion • Monitor for unexplainable purchase amounts, bounce noticeable shift in the kind 3.000 0.000 1.000 2.000 4.000 (3.000) (2.000) (1.000) Saturday, October 11, 2008 Sunday, October 12, 2008 Monday, October 13, 2008 Tuesday, October 14, 2008 Wednesday, October 15, 2008 Thursday, October 16, 2008 Friday, October 17, 2008 STEP 4: Monitor Anomalies YES Saturday, October 18, 2008 Sunday, October 19, 2008 Monday, October 20, 2008 Tuesday, October 21, 2008 Wednesday, October 22, 2008 Thursday, October 23, 2008 Friday, October 24, 2008 Saturday, October 25, 2008 Sunday, October 26, 2008 Monday, October 27, 2008 Tuesday, October 28, 2008 NO Wednesday, October 29, 2008 Thursday, October 30, 2008 Friday, October 31, 2008 Saturday, November 01, 2008 Sunday, November 02, 2008 Monday, November 03, 2008 Tuesday, November 04, 2008 Wednesday, November 05, 2008 Standardized Conversion Rate Thursday, November 06, 2008 NO Friday, November 07, 2008 Saturday, November 08, 2008 Sunday, November 09, 2008 Monday, November 10, 2008 Tuesday, November 11, 2008 Wednesday, November 12, 2008 Thursday, November 13, 2008 Graphed results of a 4-week email test with an ecommerce retailer: Normalized Normalized B Normalized Traffic Normalized Traffic B 43
  • 44. STEP 4: Monitor AnomaliesValidity Threats Anomalies in your metrics can indicate that there may be validity threats in your tests and data. Be sure to check for the following validity threats should you encounter any anomaly. History Effect – when a test variable is affected by an extraneous variable associated with the passage of time Instrumentation Effect – when a test variable is affected by a change in the measurement instrument Selection Effect – when a test variable is affected by different types of subjects not being properly distributed among experimental treatments For more on validity threats, see our previous Web clinic replay: “Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed).” 44 #webclinic
  • 45. Translating Raw Data to Predictive PowerF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results. 2. Determine Objective – Determine the exact research question you are setting out to answer with your metrics. 3. Track and Measure – Track and measure the appropriate metrics that will provide you with the answer to your determined research question. 4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a validity threat. 5. Interpret Data – Interpret the data by moving from “Which?” to “Why?” to “What?” to “Where?”. 45 #webclinic
  • 46. STEP 5: Interpret DataFrom Customer Behavior to Customer Theory Which? Why? What? Customer Behavior Customer Theory Which headline will Why this headline? What does my customer generate a higher want the most? response? Why this testimonial? Which testimonial will What makes my customer generate the most especially anxious? response? Which call to action will Why this call-to-action? What is my customer’s position in generate a higher the sequence of micro-yeses? response? 46 #webclinic
  • 47. STEP 5: Interpret DataExample Case Study Again, test results are interpreted and the next round of testing is started for this page 201% 2% 29% Test results are interpreted and Test is again interpreted and second test was created based on transferrable principles are the analyst’s observations applied to other offer pages 47 #webclinic
  • 48. STEP 5: Interpret Data Where else can we apply this data?• The discoveries and insights about 451% customer motivation from the three prior tests were applied to other landing pages and used to optimize PPC campaigns.• The purposeful effort to identify 302% and selectively apply these transferrable insights led to widespread optimization gains . 257% 28% 603% 48 #webclinic
  • 49. Baltimore Training Week Save $100 off any workshop Promo Code: 284-WS-2022 July 30 - August 1 www.meclabs.com/BTW 49 #webclinic
  • 50. Summary: Putting it all togetherF Key Principles 1. The goal of all customer research is to enable the marketer to predict customer behavior. 2. Therefore, the primary usefulness of metrics is not in answering “how many?” but rather in answering, “why so?” 3. Ultimately, metrics enable the marketer to see the cognitive trail left by the visitor’s mind. 50 #webclinic
  • 51. Summary: Putting it all togetherF Key Steps 1. Establish Visibility – Ensure that your metric platforms are able to track the four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results. 2. Determine Objective – Determine the exact research question you are setting out to answer with your metrics. 3. Track and Measure – Track and measure the appropriate metrics that will provide you with the answer to your determined research question. 4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a validity threat. 5. Interpret Data – Interpret the data by moving from “Which?” to “Why?” to “What?” to “Where?”. 51 #webclinic
  • 52. Audience Question How can I track and integrate social media metrics ? into my web analytics? -Anne 52 #webclinic
  • 53. Audience Question Is Google Analytics "good enough“ to measure ? everything I need? -Lou 53 #webclinic
  • 54. Audience Question What is the best method for calculating ? incremental click costs for low volume keywords? - Don 54 #webclinic
  • 55. Audience How should I interpret bounce rates? ? - Steve 55 #webclinic
  • 56. MarketingExperiments.com/subscribe 56#webclinic