This document discusses using data to guide product development. It recommends analyzing user data and key performance indicators to identify features that correlate with retention. The document advocates continually running A/B tests to evaluate potential product features, noting tests should be fast to implement with minimal impact and visible results. Failure of tests is acceptable as getting results now is better than waiting.

1. Using data to guide product development
Mat Clayton
Co-Founder

2. What is Mixcloud?

3. The Plan

4. The Plan

5. What does failure look like?

6. turntable.fm - MAU
Text

7. turntable.fm - DAU

8. Competitor analysis
• Sitemaps are an amazing resource
• Google - “site:competitor.com“
• OpenSiteExplorer, can be used to find links and any growth
tactics being used
• graph.facebook.com/<app_id> can give you monthly Facebook
actives for FB connected apps
• API’s

10. Key Performance Indicators
(KPI’s)

11. Pick a single KPI
• Easily understood
• Provides focus and align the team
• Represent the core of the product, and usually isn’t uniques
Mixcloud - Listener minutes / 28 days
AirBnb - Nights stayed / 28 days
Eventbrite - Tickets Booked / 28 days
SendGrid - Mail Sent / 28 days

12. Dashboards
• Distribute information amongst the team
• Make trends observable
• Give the team a solid understanding in
top level metrics
• We’re big fans of graphite and grafana
• Alert on changes

13. User Accounting

15. User accounting
Site A Site B
Active Users 1000 1000
Born +800 +100
Die -100 0
Sleep -300 -50
Awaken +100 +450
Next Period’s Active Users 1500 1500

16. Retention Curves

17. Identifying key
product features for retention

18. The Brainstorm

19. What correlates with retention?

20. First week Listens

21. Key User Metrics - Twitter

22. Twitter’s data

23. Correlation does not imply Causation

24. Cause or effect?

25. How do you choose
what to build?

26. What facts do we know about our users?
• What information did they give us? name, email, Facebook?
• Do they use mobile web/desktop/iOS/Android?
• Do they have email notifications enabled?
• Which core features have they used and to what degree?
• Which features did they miss?
• What was their last interaction with the service?
• Where did they come from? (Referrer)
• Do they use competitor products?

28. Evaluating potential product features
• Do we have the resources to build it?
• How many users will the feature be relevant to?
• How frequently will they interact with it?
• What is the conversion rate against our primary goal?

29. AB Testing
(Always Be Testing)

30. AB Testing Examples

32. Big changes == Big results

34. How big is big?

35. AB testing
• Continually run AATests
• Requirements
- Fast to implement, a single line of code
- Negligible cost to product performance, including SEO
- Results need to be available to the entire team
• Tools
- Django Experiments (we wrote it)
- A/Bingo (Ruby on Rails)
- Optimizely
- Google Website Optimizer

36. Failure is acceptable

37. “A good plan, violently executed now, is better than
a perfect plan next week.” - George S. Patton

38. Thanks for listening!
Any Questions?
@matclayton
mat@mixcloud.com