How to measure what matters before Product/Market Fit

1. ACTIONABLE METRICS
HOW TO MEASURE WHAT
MATTERS BEFORE PRODUCT/MARKET FIT
ASH MAURYA
@ashmaurya
http://www.ashmaurya.com

2. What is an Actionable Metric?

3. Ties specific, repeatable actions to
observed results

4. Vanity Metrics

5. 3 Stages of a Lean Startup
Customer Discovery Customer Validation Customer Creation
(Problem/Solution Fit) (Product/Market Fit) (Scale)

6. How You Measure Matters

7. 3 Rules for Actionable Metrics
1. Measure the “Right” Macro
2. Create Simple Reports
3. Metrics are People too

8. 3 Rules for Actionable Metrics
1. Measure the “Right” Macro
2. Create Simple Reports
3. Metrics are People too

9. Identify Key Metrics
Acquisition How do users find you?
Activation Do users have a great first experience?
Retention Do users come back?
Revenue How do you make money?
Referral Do users tell others?

10. Identify Key Metrics
Acquisition How do users find you?
Activation Do users have a great first experience?
Retention Do users come back?
Revenue How do you make money?
Referral Do users tell others?
Before Product/Market Fit

11. Map Metrics to Actions

12. Activation Actions
Signup Download Share Stuff Tell Friends

13. Retention Actions
1. Revisits
2. Churn - cancellations, no activity
3. Loyalty - days since last visit
4. Key Activity - Shared 1 new album/movie per month

14. 3 Rules for Actionable Metrics
1. Measure the “Right” Macro
2. Create Simple Reports
3. Metrics are People too

15. Funnel Analysis - The Good
Conversion Funnel for June
Signed-up
4200 (100%) 1. Simple
Downloaded 2.Visual
3400 (81%)
3. Maps well to Activation Flow
Did key activity
1880 (45%)
Purchased
375 (9%)

16. Funnel Analysis - The Bad
Conversion Funnel for June
Signed-up
4200 (100%)
How do you:
Downloaded
1. Track long lifecycle events
3400 (81%)
Did key activity
1880 (45%)
Purchased
375 (9%)

17. Funnel Analysis - The Bad
Conversion Funnel for June
Signed-up
4200 (100%)
How do you:
Downloaded
1. Track long lifecycle events
3400 (81%)
2. Handle split tests
Did key activity
1880 (45%)
Purchased
375 (9%)

18. Funnel Analysis - The Bad
Conversion Funnel for June
Signed-up
4200 (100%)
How do you:
Downloaded
1. Track long lifecycle events
3400 (81%)
2. Handle split tests
Did key activity
1880 (45%) 3. Measure Retention
Purchased
375 (9%)

19. Funnels Alone Are Not Enough.
Say Hello to the Cohort.

20. What is Cohort Analysis
A cohort is a group of people who share a common
characteristic over a period of time e.g. join date.

21. Tracking Long Lifecycle Events
Conversion Funnel for June
Signed-up
4200 (100%)
Downloaded
3400 (81%)
Did key activity
1880 (45%)
Purchased
375 (9%)

22. Tracking Long Lifecycle Events
Reporting Period: June 1 - June 30 2010
Week 1 Cohort Week 2 Cohort Week 3 Cohort Week 4 Cohort
Signed-up Signed-up Signed-up Signed-up
900 (100%) 1000 (100%) 1100 (100%) 1200 (100%)
Downloaded Downloaded Downloaded Downloaded
750 (83%) 800 (80%) 900 (81%) 950 (80%)
Did key activity Did key activity Did key activity Did key activity
380 (42%) 500 (50%) 650 (60%) 350 (30%)
Purchased Purchased Purchased Purchased
95 (11%) 100 (10%) 180 (16%) 0 (0 %)

23. Tracking Split Tests
Reporting Period: June 1 - June 30 2010
Control - Free Trial Hypothesis A - Freemium
Signed-up Signed-up
1000 (100%) 1500 (100%)
Downloaded Downloaded
800 (80%) 1200 (80%)
Did key activity Did key activity
400 (40%) 450 (30%)
Purchased Purchased
100 (10%) 75 (5%)

24. Tracking Split Tests
Reporting Period: June 1 - June 30 2010
SEO Cohort: “Photo Sharing” SEO Cohort: “Cloud Photo Backup”
Signed-up Signed-up
1000 (100%) 1500 (100%)
Downloaded Downloaded
800 (80%) 1200 (80%)
Did key activity Did key activity
400 (40%) 750 (50%)
Purchased Purchased
100 (10%) 225 (15%)

25. Tracking Retention
Month 1 Month 2 Month 3 Month 4 Month 5 Month 6 Month 7
(Joined in)
100% 10% 9% 9% 7% 7% 7%
Week 1
Week 2 100% 12% 10% 10% 8% 7% ?
Week 3 100% 16% 14% 13% 12% ?
Week 4 100% 17% 15% 14% ?
Week 5 100% 20% 19% ?
Week 6 100% 22% ?
… … … … … … … …

26. 3 Rules for Actionable Metrics
1. Measure the “Right” Macro
2. Create Simple Reports
3. Metrics are People too

27. Validate Qualitatively,Verify Quantitatively

28. Who are these people?
Reporting Period: June 1 - June 30 2010
Cohort - Free Trial
Signed-up
1000 (100%)
Downloaded
800 (80%)
Did key activity
400 (40%)
Purchased
100 (10%)

29. Who are these people?
Reporting Period: June 1 - June 30 2010
Cohort - Free Trial
Signed-up
1000 (100%)
Failed Downloads List
Downloaded
800 (80%) Email
Did key activity john.doe@example.com
400 (40%)
peter@abc.com
Purchased
100 (10%)
mary.jane@example.com
mark@acme.com
…
jack_jill@hill.com

30. How do I create these
reports?

31. 3rd Party Tools versus Homegrown
Funnel Retention Funnel Metrics to
3rd Party
Analysis Cohorts Cohorts People
KISSMetrics Ad-hoc No No Not easy
MixPanel Static Yes No Not easy
Google Static/
No No No
Analytics Limited

32. 3rd Party Tools versus Homegrown
Funnel Retention Funnel Metrics to
3rd Party
Analysis Cohorts Cohorts People
KISSMetrics Ad-hoc No No Not easy
MixPanel Static Yes No Not easy
Google Static/
No No No
Analytics Limited
Homegrown Comments
Index Cards Manual and time consuming process
SQL + Excel Complex queries and pivot tables
Events Database Separate database plus additional code

33. What’s Next?
Acquisition How do users find you?
Activation Do users have a great first experience?
Retention Do users come back?
Revenue How do you make money?
Referral Do users tell others?
After Product/Market Fit