This document summarizes a guest lecture on analytics and metrics given by Darren Levy. The lecture covered: 1) Definitions of metrics and why they are important for measuring business performance and customer satisfaction. 2) What specific metrics to focus on, including vanity metrics to avoid and key startup metrics like CPA, ARPU, churn, and LTV. 3) Methods for collecting and optimizing metrics, including A/B testing, cohort analysis, and heatmaps. 4) Case studies demonstrating how Dollar Shave Club and Netflix used metrics and analytics to optimize pricing and content.

1. Analytics & Metrics
UCLA Extension (Guest Lecture)
Darren Levy
- Senior Director of Revenue, Retention at Mylife.com
- Founder of Gatherspace.com
darrenjlevy@gmail.com

2. True or False:
Technology is the
hardest part of building
an online business?

3. True or False:
Knowing the metrics is
someone else’s job.

4. Breakdown of today’s discussion
- Part 1 – Definitions, why they are so critical
- Part 2 – What metrics to focus on?
- Part 3 – Optimizing metrics
- Part 4 – Case Studies

5. I. Why do we need metrics?

6. What are metrics, why do we need them?
- Small units of measurement, typically ratio’s.
- How are we doing?
- Is business healthy?
- Is the customer happy, is the product working?

7. 99.1% Pure

8. …and what about analytics?
- Combining metrics, finding patterns of behavior.
- What should we be working on?
- Should we pivot?

9. Google Analytics – be cautious
-
Storing all of your proprietary data with Google
Limitations on customizations
When shit hits the fan, getting customer support
You NEED to slice and dice and query your data

10. Why are metrics so important
for YOU?!
- Be the Expert, be the Hero, the “go to” person
- Don’t rely so much on others
- Because we’re all Liars

11. Ice Cream Glove

12. Tightly ingrained within
Lean Startup Process
Validation
Hypothesis
Experiment
Metrics

13. II. What metrics to focus on?

14. Vanity Metrics
- Not actionable
- Not understandable
- Examples:
• traffic
• time on site
• cumulative registrations
• # of visits
• Overall conversion rate

15. Single Key Metric
OMTM
- Analysis Paralysis (don’t overdo it!)
- For startups, business rally around a metric
- Changes over time as business progresses

16. Customer Lifecycle – 5 Steps to Success
AARRR!

17. Dave McClure’s Pirate Metrics
AARRR!

18. AARRR Lifecycle
Category
Lifecycle Experience
Metrics
* CPC
* Keyword Rank
* Ad CTR
Est Value
Acquisition
Visit Site
(or landing page, or widget)
Activation
Jump through initial hoop,
signup (“a ha” moment)
* CTR
Retention
Repeat Visitor,
Open up emails
* Avg # of Visits
* Engagement
Score
CTR ~2%
Value=$5
Customer generates revenue
* RPV
* AOV
* ARPU
CTR ~1%
Value=$25
Revenue
CTR ~100%
Value=.01
CTR ~5%
Value=$3

19. Lean Analytics
Growth Stages (“Gates”)

20. Single Key Metrics
For Startups
How much do we spend to get a customer?
Cost per Acquisition ($)
CPA
How much do our customers pay us?
Average Revenue per User ($)
ARPU
At what rate are customers cancelling?
Churn (%)
Churn
How much is a customer worth over time?
Lifetime Value per Customer ($)
LTV

21. Subscription v. Transactional
Models
Subscription Business
VPR (value per reg)
Churn / LTV / ARPU
Activation & Engagement
Transactional Business
Shopping Cart Abandonment
Average Order Value (AOV)
Upsell & Product Page CTR

22. III. How do we do this?

23. How do we get Metrics?
-
Google Analytics
DIY – recording events, activities in a SQL db
AB testing
Other tools – Heatmaps

24. Types of AB Testing
- Simple AB testing
- Multivariate testing
- Price testing

25. Simple AB Testing
- Changing 1 element at a time?
- Effecting CTR at any point in a funnel

26. Multivariate Testing
- More simultaneous variables
- Takes longer to test
- Better results than “simple”

27. Price Testing
-
Testing different prices or terms
Not measuring CTR, measuring RPV
Pay attention to “Accumulation of Revenue”
Pricing Strategies (freemium, subsc., bundling)

28. Ugly Friend Concept

29. Famous Price Test (Ugly Friend)
- Decoy (ugly friend) pushes price selection
- In this example, decoy pushed sales by 30%

30. Apple iPhone Pricing
~67%
~20%
~13%

31. 10 Things you MUST Test
1. Calls to Action (CTA)
2. Propositions
3. Copy & Content
4. Visual Media
5. Funnel testing
6. Forms
7. Emails
8. Pricing
9. Shipping
10. Personalization

32. AB Test Example

33. Cohort Analysis
Measuring engagement and value over time

34. Optimization via Heatmaps

35. Quantitative v. Qualitative Metrics
- Example of qualitative (fire alarm in middle of night,
99.97% up time, yet pissed off customers)
- Why you need both. (user testing, surveys,
call your customers)

36. IV. Case Studies

37. Price Test @ Dollarshaveclub.com
Control
Test
4
4.4% higher 24 hr Net
7
7% higher CLTV

38. Case Study: Netflix & House of Cards
- Goal to be “HBO” of online TV
- Cost of $100m on production
- Analytics told them
* Most customers watched David Fincher’s
social network all the way
* British version of “House of Cards” was highly
watched
* Significant overlap of British House of Cards
watch Kevin Spacy films and Fincher films.
- Data so far
* Q1 2013, brought in 3m new custmers
* in Q1 alone, paid for cost of production
* Increasing CLTV

39. Q&A