Metrics @ App Academy


Published on

I've been teaching AppCademy teams about metrics several times since 2013. This is the latest slide deck.
Main goal: dispel convenient default metrics, instead focus on your own business problems and derive metrics to solve them.

AppCademy is a 4-week accelerator camp run by AppCampus, a training program for Windows Phone dev teams.

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Metrics @ App Academy

  1. 1. Metrics January 2015 Niko Vuokko, Sharper Shape
  2. 2. What are metrics ?  Metrics are the eyes of the business  Eyes are for seeing where you’re stepping and where you want to go  Metrics are not for looking cool on the lobby screen
  3. 3. Which metrics should I follow ?  You don’t pick metrics, you pick business problems  Visible change in a metric  visible change in the business  Business problems change and evolve  Seeing problems is not enough  Metrics should point out the root cause and hint at the solution
  4. 4. Example: New subscription-based app  Most effective user acquisition channel ?  Most efficient organic growth mechanism ?  How to fix onboarding ?  What features are unused ?  Should we make a ”special offer” after 2 or 5 days ?
  5. 5. Example: Older IAP-based app  Where are under-penetrated segments remaining ?  What makes users leave ?  What type of content drives monetization ?  Is there content saturation ?
  6. 6. What is my problem ?
  7. 7. User acquisition: example metrics  New users  Active users  Magnet features  Acquisition cost, per channel, country, user revenue, etc.  Channel traffic quality (this is tricky)
  8. 8. Engagement: example metrics  Back in X days after first use  Session length and its relation to revenue/retention  Feature coverage and popularity  Funnels, onboarding effectiveness
  9. 9. Retention: example metrics It’s way cheaper to keep a user than to find a new one  Active after X days since first use  Time between visits  Weekly churn  Core features, what keeps users coming back?
  10. 10. Monetization: example metrics Most freemium apps get a 2 % monetization rate  Monetizing features, what kind to introduce next?  Content saturation, i.e., spending walls  Promotion success, which hooks work?  Time of first monetization
  11. 11. To action
  12. 12. Treat users as ”somewhat” individual  Analysis and optimization across the whole userbase is not worth it  Analysis and optimization of individual users is not worth it  Find criteria that produce noticeable differences between groups  This may vary from metric to metric
  13. 13. Subgroup examples  ”Impact of app localization varies wildly between countries”  ”Users who installed during a weekend can be converted more aggressively”  ”Users with an animal avatar react great to this promotion”  ”Launching the new version made user count go up, but conversion rates suffered”  ”Feature X is very popular in average, but very little among paying users”
  14. 14. Practical issues with metrics  Data quality is absolutely horrible in many cases  Special doom pits: timestamps, IDs  The product and the users change => data changes  Long term aggregates go wrong  Metrics lose their meaning
  15. 15. Statistical significance  Humans are by nature horrible at interpreting statistics  Things get even worse when lots of data and no clear goal  You are not an exception Guidelines  Be wary of any signals other than the painfully obvious ones  Always verify  Even service providers screw up multiple hypothesis testing
  16. 16. Service providers vs. DIY  Collecting and analyzing is expensive to a small team => stay with service providers until you can’t  Decent services: GameAnalytics, Omniata, MixPanel, KissMetrics  Collect as much as you can, the use cases will emerge  Your data is almost certainly tiny => don’t overdo the tools  Getting data collection right MUCH more difficult than you expect  Getting the numbers right is MUCH more difficult than you expect
  17. 17. Power laws The new normal
  18. 18. Things are not normal  School teaches you that everything is a Gaussian  That’s just not true  Most things follow a power law, not a normal distribution  People don’t act the way you think
  19. 19. This is what most revenue/engagement/whatever metrics look like Next, remove the non-paying users
  20. 20. But the result will not be like this normal distribution
  21. 21. This is the actual form The numbers are highly concentrated and go pretty high
  22. 22. The curve follows the power law Log axes produce a straight line
  23. 23. Another example Number of users Revenue per user
  24. 24. Power law  Follows from principle: “Whoever has will be given more”  Example: Web pages get links in proportion to their popularity => virtuous cycle  Characterized by 1) huge whales 2) huge mass at the bottom
  25. 25. Implications of power laws  Averages are worse than useless  Your userbase has very diverse subsets, treat them that way  More users means more users in the future (App store ”Featured” actually works)  => Only two relevant factors: new users and especially retention  Network effects are very powerful
  26. 26. Thank you! REMEMBER! You’re solving business problems, NOT watching cool charts