This document discusses measuring user growth and retention for online products and services. It explains how to account for different factors that influence monthly active user (MAU) growth, such as new users, resurrected former users, and churned users. Examining these metrics at a cohort level over time can provide insights into the drivers of growth and retention. Hiring a data science team requires finding people strong in product sense, engineering skills, and math/statistics, with the goal of achieving business objectives.

1. Measuring for Growth:
Understanding Your Data & Next Steps
WMD 2015
Jonathan Hsu
Head of Data Science
The Social+Capital Partnership

2. whoami
Past Current

3. Analytics @ FB
• Pre-2009 : Growth and Data
• IPO FB (mid-2012) : ~50 distributed data scientists/analysts
• Sometimes centralized, other times distributed
• Current FB: ~250 analysts
Unit of Product Development
1x PM: Mini-CEO
1x Data: Evidence
1x Design: Aesthetics/UEX
3-10x Eng: Feasibility
Pro Con
Centralized
Uniform
expectations/c
ulture
Product mis-
alignment
Distributed
Product
Alignment
Uneven
expectations/c
ulture
No “right” answer.

4. Accounting for MAU Growth
• MAU growth alone can be misleading.
• Account for the deltas:
MAU(t+∆t) = MAU(t) + new(t) + resurrected(t) - churned(t)
MAU(t) = retained(t+∆t) + churned(t+∆t)
-
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
200,000
1 2 3 4 5 6 7 8 9 10 11 12
MAU growing at 5% per month

5. Accounting for MAU Growth
• Very different companies underneath!
• Both high retention (90%+) but very different growth
dynamics and implied courses of action.
• Typical consumer MAU added/MAU lost ≳ 1
-20,000
-10,000
-
10,000
20,000
30,000
1 2 3 4 5 6 7 8 9 10 11 12
Low churn, low acq  Ramp up new
churned
resurrected
new
-20,000
-10,000
-
10,000
20,000
30,000
1 2 3 4 5 6 7 8 9 10 11 12
High churn, high user acq  Fix churn
churned
resurrected
new

6. Accounting for MRR Growth
• Also works for subscription/saas revenue.
MRR(t+∆t) = MRR(t) + new(t) + expansion(t) - churned(t) - contraction(t)
MRR(t) = retained(t+∆t) + churned(t+∆t) + contraction(t+∆t)
“Quick Ratio” = (Expansion + New)/(Cancelled + Contraction)
= 1.6x = 4.5x *See Mamoon Hamid’s deck

7. Understanding it at the cohort level
• Phenomena at fixed calendar date vs. fixed
age.

8. • Can be used for things other than retention.
Cumulative revenue for instance..

9. What are the
drivers?
Be Bayesian
P(X | E) = P(E | X) P(X)
P(E)

10. • Tests are great for micro-optimization, but
often not appropriate for macro questions.
• Correlation is not the best, but it’s also not
valueless.
• Example:
– At FB, “10 friends in
14 days”
– Notion of “time-to-
value” in enterprise
is similar.

11. How to hire this function
• Hire to achieve objectives, which may or may not
include building features.
• Three areas, above the line on all, great in at least one.
– Product/Business sense
– Engineering/Hacking
– Math/Statistics
• Interviews should cross boundaries as much as
possible.
• Reporting chain based on objectives.
• Very hard to hire experienced folks, consider the
Insight Data Science Fellowship.

12. Questions?