This document discusses how data can be leveraged for product management. It outlines how data can be used as the core of a product, to optimize unit economics by ensuring lifetime value exceeds customer acquisition costs, for marketing optimizations by testing channels and optimizing return on marketing investment, and for product optimizations through A/B testing. However, it notes that most A/B tests are not statistically valid or impactful. It also discusses using data for personalization, including basic personalization triggers not requiring data science. Overall, the document advocates using data to understand customers, test changes, and optimize performance.

1. Data-driven
product management

2. Coming out
I’m really a bad product manager.
My areas of stupidity are:
• empathy and understanding customers;
• UX and UI fuss;
• making stakeholders happy.
At the same time i’m good in data crunching.

3. How do we leverage from
data?
1. Data as product core.
2. Data for product economics (LTV > CAC).
3. Data for marketing optimizations (ROMI ↑).
4. Data for product optimizations (A/B tests).
5. Data for customer happiness (personalization).

4. Data as product core
1. Data is what we suggest:
- Google Analytics;
- Mixpanel;
- Captiv8.io.
2. Data is what stands under the hood:
- Netflix;
- Yandex.Radio;
- any ad network.
Not for every product.

5. Data for unit economics
No chance you can avoid it.
LTV = ARPU * LT
LT = hmmm, here is a trick.
LTV = N * CAC
if N >= 3, you’re probably fine
if N <=1, you’re rushing into failure.
• http://www.forentrepreneurs.com/saas-metrics-2/ Unit economics (SaaS focused)

6. Data for marketing
optimization
User Acquisition for Dummies
1. Find a channel;
2. Test it;
3. Optimize it;
4. Spend more money there while you have positive
ROMI from it;
5. Goto 1.

7. Data for product
optimization
A/B tests may seem sexy, but most of them are not.
• stupid test just waste part of your expensive traffic;
• most tests aren’t statistically valid;
• small companies can’t do A/B tests;
medium companies can do a few;

8. How to fuck up with A/B test
• ignore confidence intervals;
• ignore seasonal, trend, cycle variations;
• ignore false positives from numerous tests;
• do the black magic trying to solve a problem of
small dataset.
Simulator for your own fake correlations and imperfect A/B tests
https://gist.github.com/cec363e533029535450f

11. • https://en.wikipedia.org/wiki/Multiple_comparisons_problem

12. Personalization
• There are many personalization-related activities
that need no data science but just triggers
(birthday discounts, name in communications);
• Next step: make clusters of your customers
(personas) and create unique strategy for each.
• Finally you may try to solve the generic problem.

13. High-level setup
• What do I know about the customers (features);
• Where can I change the communication;
• What customer behavior is appreciated.

14. High-level setup
• What do I know about the customers (features);
Demographic, marketing, product usage metrics…
• Where can I change the communication;
Emails, in-app welcome screens, push notifications…
• What customer behavior is appreciated.
Clicking the ad, upgrading payment plan…

15. High-level setup
Deus Ex Machina job:
Predict best action/message for each customer for each
goal. It’s (almost) classic supervised learning algorithm.
• Prototype can be coded during the week.
• Tuning may take years.
• Operations may be more complicated than coding.

16. How to make a decision
• Make up a hypothesis (‘we need a green button’);
• Design an experiment;
• Try to validate with existing data;
• Deliver a feature;
• Validate a posteriori.

17. Data health
• Set up reserve data source;
• Seriously, set up reserve data source;
• Be sure you have single point of truth;
• Google Analytics is not single point of truth;

18. 3rd party tool heuristics
• Can I manipulate with raw data not aggregated
pieces?
• Can I create custom aggregates?
• Can I export all my raw data?
• Can I import data from other source?

19. Thank you for the attention!
Feel free to ping me with your questions
me@arseny.info
facebook.com/arseny.info