Metrics and Analytics, Guest Lecture, UCLA


Published on

Provided hour lecture on metrics and analytics for Jaime Levy's UX Design course at UCLA Extension.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • You rarely here of companies going out of business because technology was to difficult to build. Usually it’s about figuring out a business model and pivoting, which comes from understanding metrics.
  • No, it’s your job , knowledge is power.
  • When you guys go to the doctor, what metrics do you pay attention to? Anyone watch sports….what metrics do you guys pay attention to?Is the product working, viable or is it a useless piece of crap?
  • You know these guys? The big metric here was the purity of their meth.
  • …for the purpose of this session, we’re gonna touch on anlalytics but the focus will be on metrics.
  • It’s not that GA is bad, it’s actually incredibly powerful, you just have to realize it’s a great place to start, but shouldn’t do everything.
  • If you read lean analytics, chapter 1 is about “we’re all liars” delusional entrepreneurs. It’s good for entrepreneurs to be able to make a strong pitch, but stop selling to yourself, pay attention to the data and learn when you’re full of it.
  • You could probably say Ali G is lying to himself, and he might be. His data is all assertions. Is this a bad product?
    Who knows, maybe he should get something on the market and see.
  • So where does metrics fit in the lean startup “build measure learn” – it’s the Data that you learn from, enabling you to pivot or stay the course.
  • Traffic – one day it’s 10000, next day it’s 10001, so whattime on site – is that good or bad, is site broken?Overall conversion rate – not really so useful especially when prices are a major factor
  • Especially for startups, less so far larger businessesone month focus is on acquiring and optimizing traffic, then reg, then optimizing activation
  • Starting at top of the funnel with acquisition, you can continue to push focus as you progress.
  • LTV must be greater than CPA, ARPU can be less than CPA (but hurts cashflow)
  • Add a little image of a price test (similar to the AB test)
  • Now add a spreadsheet showing the $6 as a $4 option but showing the $6 winning in terms of ARPU and Net, but executive winning on Gross
  • Show groups of things that are testable (from AB testing book)
  • November 2012 Amazon quietly switched their pricing to offer a monthly and annual, then switching it off several days later.
  • November 2012 Amazon quietly switched their pricing to offer a monthly and annual, then switching it off several days later.
  • Quantitative can’t tell you your web site looks like shit; ask them if they ever speak to their customers.
  • Now add a spreadsheet showing the $6 as a $4 option but showing the $6 winning in terms of ARPU and Net, but executive winning on Gross
  • Show groups of things that are testable (from AB testing book)
  • Metrics and Analytics, Guest Lecture, UCLA

    1. 1. Analytics & Metrics UCLA Extension (Guest Lecture) Darren Levy - Senior Director of Revenue, Retention at - Founder of
    2. 2. True or False: Technology is the hardest part of building an online business?
    3. 3. True or False: Knowing the metrics is someone else’s job.
    4. 4. Breakdown of today’s discussion - Part 1 – Definitions, why they are so critical - Part 2 – What metrics to focus on? - Part 3 – Optimizing metrics - Part 4 – Case Studies
    5. 5. I. Why do we need metrics?
    6. 6. What are metrics, why do we need them? - Small units of measurement, typically ratio’s. - How are we doing? - Is business healthy? - Is the customer happy, is the product working?
    7. 7. 99.1% Pure
    8. 8. …and what about analytics? - Combining metrics, finding patterns of behavior. - What should we be working on? - Should we pivot?
    9. 9. Google Analytics – be cautious - Storing all of your proprietary data with Google Limitations on customizations When shit hits the fan, getting customer support You NEED to slice and dice and query your data
    10. 10. Why are metrics so important for YOU?! - Be the Expert, be the Hero, the “go to” person - Don’t rely so much on others - Because we’re all Liars
    11. 11. Ice Cream Glove
    12. 12. Tightly ingrained within Lean Startup Process Validation Hypothesis Experiment Metrics
    13. 13. II. What metrics to focus on?
    14. 14. Vanity Metrics - Not actionable - Not understandable - Examples: • traffic • time on site • cumulative registrations • # of visits • Overall conversion rate
    15. 15. Single Key Metric OMTM - Analysis Paralysis (don’t overdo it!) - For startups, business rally around a metric - Changes over time as business progresses
    16. 16. Customer Lifecycle – 5 Steps to Success AARRR!
    17. 17. Dave McClure’s Pirate Metrics AARRR!
    18. 18. AARRR Lifecycle Category Lifecycle Experience Metrics * CPC * Keyword Rank * Ad CTR Est Value Acquisition Visit Site (or landing page, or widget) Activation Jump through initial hoop, signup (“a ha” moment) * CTR Retention Repeat Visitor, Open up emails * Avg # of Visits * Engagement Score CTR ~2% Value=$5 Customer generates revenue * RPV * AOV * ARPU CTR ~1% Value=$25 Revenue CTR ~100% Value=.01 CTR ~5% Value=$3
    19. 19. Lean Analytics Growth Stages (“Gates”)
    20. 20. Single Key Metrics For Startups How much do we spend to get a customer? Cost per Acquisition ($) CPA How much do our customers pay us? Average Revenue per User ($) ARPU At what rate are customers cancelling? Churn (%) Churn How much is a customer worth over time? Lifetime Value per Customer ($) LTV
    21. 21. Subscription v. Transactional Models Subscription Business VPR (value per reg) Churn / LTV / ARPU Activation & Engagement Transactional Business Shopping Cart Abandonment Average Order Value (AOV) Upsell & Product Page CTR
    22. 22. III. How do we do this?
    23. 23. How do we get Metrics? - Google Analytics DIY – recording events, activities in a SQL db AB testing Other tools – Heatmaps
    24. 24. Types of AB Testing - Simple AB testing - Multivariate testing - Price testing
    25. 25. Simple AB Testing - Changing 1 element at a time? - Effecting CTR at any point in a funnel
    26. 26. Multivariate Testing - More simultaneous variables - Takes longer to test - Better results than “simple”
    27. 27. Price Testing - Testing different prices or terms Not measuring CTR, measuring RPV Pay attention to “Accumulation of Revenue” Pricing Strategies (freemium, subsc., bundling)
    28. 28. Ugly Friend Concept
    29. 29. Famous Price Test (Ugly Friend) - Decoy (ugly friend) pushes price selection - In this example, decoy pushed sales by 30%
    30. 30. Apple iPhone Pricing ~67% ~20% ~13%
    31. 31. 10 Things you MUST Test 1. Calls to Action (CTA) 2. Propositions 3. Copy & Content 4. Visual Media 5. Funnel testing 6. Forms 7. Emails 8. Pricing 9. Shipping 10. Personalization
    32. 32. AB Test Example
    33. 33. Cohort Analysis Measuring engagement and value over time
    34. 34. Optimization via Heatmaps
    35. 35. Quantitative v. Qualitative Metrics - Example of qualitative (fire alarm in middle of night, 99.97% up time, yet pissed off customers) - Why you need both. (user testing, surveys, call your customers)
    36. 36. IV. Case Studies
    37. 37. Price Test @ Control Test 4 4.4% higher 24 hr Net 7 7% higher CLTV
    38. 38. Case Study: Netflix & House of Cards - Goal to be “HBO” of online TV - Cost of $100m on production - Analytics told them * Most customers watched David Fincher’s social network all the way * British version of “House of Cards” was highly watched * Significant overlap of British House of Cards watch Kevin Spacy films and Fincher films. - Data so far * Q1 2013, brought in 3m new custmers * in Q1 alone, paid for cost of production * Increasing CLTV
    39. 39. Q&A