Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measure and Learn

429 views

Published on

"Measure and Learn" Slides for the Communication Design Lab from IED Master Madrid.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Measure and Learn

  1. 1. MEASURE & LEARN IED - COMMUNICATION DESIGN LABS
  2. 2. HELLO !
  3. 3. about.me Psychologist, full-stack digital marketer, data native, founder and much more buzz words! connect about.me/andykaiser
  4. 4. “Data Scientist:The Sexiest Job of the 21st Century”Perhaps NOT! We all will work as Data-Specialist. https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/
  5. 5. • Most startups don’t know what they’ll be when they grow up. • Markets get disrupted by Data • Evaluate data to understand what’s working and what’s not.
  6. 6. first build for Palmpilots
  7. 7. was a database company
  8. 8. was written by experts only
  9. 9. made desktop automation
  10. 10. was going to be a MMO
  11. 11. was a podcasting company
  12. 12. –Avinash Kaushik “Analytics is the analysis of qualitative and quantitative data from your business and competition to drive continual improvement of the online experience that your customers and potential customers have which translates to your desired outcomes, both online and offline.”
  13. 13. A DAY ATTHE RACES Data Driven Decisions
  14. 14. –Pat Symonds “Bloody hell, it's so different these days. I used to be able to have dinner with the drivers on Friday night! Nowadays they spend evenings mulling over graphs and spreadsheets to inform their race preparation.”
  15. 15. • 150 sensors per car • 1.000 data points per lap • 200 Gigabytes move from track and factory every race weekend • 50 Data Engineers per race
  16. 16. –Kenny O’Donnell “You will make the best decision you can make.”
  17. 17. Gut Feeling Data-driven DecisionIntuition Scientific
  18. 18. METRICS 📈
  19. 19. A good metric is… • comparable to another time period, group, competitor,… • understandable for the target audience. • a ratio or rate • targeted to the right audience (developers, marketers, business development,…) • a behaviour changer.
  20. 20. • Qualitative metrics are unstructured, anecdotal, revealing, hard to aggregate. • Quantitative metrics are numbers and stats, hard facts but less insights. Discover Proof
  21. 21. https://www.google.com/trends/
  22. 22. https://www.youtube.com/trendsdashboard
  23. 23. VANITY METRICS • Vanity Metrics makes you feel good but doesn’t change how you’ll act. • Vanity Metrics are bad!
  24. 24. • Hits • PageViews • Visits • Users • Followers/friends/likes • Logins Only for Ad Inventory Count People Cross it with People Why they stuck or left? Count actions! What are the actions and value?
  25. 25. “STATISTICS” 🎲
  26. 26. CORRELATION • Understand Correlation/Causality • What is the relationship between two variables? • Correlation does not imply causation.
  27. 27. chart courtesy fromTylerVigen | www.tylervigen.com
  28. 28. chart courtesy fromTylerVigen | www.tylervigen.com
  29. 29. chart courtesy fromTylerVigen | www.tylervigen.com
  30. 30. CORRELATION • Use R, Excel, Numbers or online tools to calculate the linear relationship between two variables. • Interpret a Correlation Coefficient r • +- 1:A perfect linear relationship • +- 0.70 strong linear relationship • +- 0.50 moderate linear relationship • +- 0.30: weak linear relationship • +- 0: no linear relationship
  31. 31. STANDARD DEVIATION • Standard Deviation (SD) is a measure that is used to quantify the amount of variation or dispersion of a set of data values. • A standard deviation close to 0 indicates that the data points tend to be very close to the mean of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values.
  32. 32. STATISTICAL SIGNIFICANCE • Statistical significance is fundamental to statistical hypothesis testing (determine whether a null hypothesis should be rejected or retained) • In any experiment or observation that involves drawing a sample from a population, there is always the possibility that an observed effect would have occurred due to sampling error alone.
  33. 33. A/BTESTING 🎲
  34. 34. Find Correlation Test Causality Optimize the causal factor Correlation lets you predict the future Causality lets you change the future
  35. 35. • A/B testing (also known as split testing) is a method of comparing two versions against each other to determine which one performs better. • Testing takes the guesswork out of optimization and enables data-backed decisions that shift business conversations from “we think” to “we know.”
  36. 36. Goal:
 Drive traffic to theVodafone website. Winner: +6%
  37. 37. Goal:
 Increase sales. Winner: +38%
  38. 38. Goal:
 Drive traffic to theVodafone website. Winner!
  39. 39. Goal:
 Increase Insurance Leads Winner!
  40. 40. Goal:
 Increase hotel bookings Winner!
  41. 41. Goal:
 Increase unique opens/clicks of email Winner: +86%
  42. 42. Goal:
 Reduce Bounce Rate Winner!
  43. 43. • Define a Hypothesis and Goal. • Define the minimum sample size. • Wait until the minimum sample size have run through the test. • Calculate if the differences between versions are statistical significant.
  44. 44. WHICH METRIC
 TO FOCUS ON?
  45. 45. start with WHY? How great leaders inspire action: bit.ly/OX0t5a
  46. 46. follow with WHAT? HOW? Metrics Tools, Persons,…
  47. 47. THE ABCD FRAMEWORK … HOWTO ANSWER “WHAT”
  48. 48. ACQUISITION
  49. 49. BEHAVIOUR
  50. 50. CONVERSION
  51. 51. DIMENSION & DATASOURCE
  52. 52. A B C dimension datasource Why? visit easyabcd.com for more information and analysis tools acquisition behaviour conversion easy|A|B|C|D| Audience
  53. 53. THE AARRR FRAMEWORK
  54. 54. Acquisition Activation Retention Referral Revenue Customer Intent Fulfilment of Customer Intent How to users find you? Do users have a great first experience? Do users come back? Do users tell others? How do you make money?
  55. 55. ONE METRICTHAT MATTERS
  56. 56. Stickiness Virality Price Approach Math that
 matters Keep people coming back Gest customers faster than you lose them Make people invite friends How many they tell, how fast they tell them Spend revenue getting customers Customers are worth more than they cost to get THREE ENGINES AARRR
 Metric Retention Referral Revenue
  57. 57. Facebook on early days had only 150k users, little revenue and many superior competitors …but 75% of users visit one or more times per day. And within one month of launching on a new campus, could acquire 90% of the students.
  58. 58. COHORT ANALYSIS
  59. 59. • Cohort analysis is a subset of behavioural analytics that takes the data from a product and rather than looking at all users as one unit, it breaks them into related groups for analysis. • These related groups, or cohorts, usually share common characteristics or experiences within a defined timespan.
  60. 60. CohortTrend LongitudinalTrendOnboardingTrend Percentage of Monthly Active Users (MAU) each week for 12 weeks
  61. 61. DON’T FORGET
 THE CONTEXT
  62. 62. Weather, Day of the week, Calendar Day, etc influence the User Behaviour and so the Metrics.
  63. 63. THANKYOU! 😊

×