This document discusses using analytics and data to improve game performance and retention. It emphasizes that vanity metrics are not actionable and that deeper analysis of user behavior and testing hypotheses through experiments is needed. Daily retention cohorts only show symptoms, not causes of churn. Good analysis requires exploring where users are pooling and dropping out, and what differs between progressing and non-progressing users. The document provides an example of hypothesizing that early achievement earning reduces churn, and testing that hypothesis through targeted experiments without changing game code. Measuring experiment results can indicate if changes reduced churn. It recommends tools for direct SQL access to rich event data and easy experimentation on mobile.

1. WHAT YOU DON’T KNOW CAN KILL YOU
ACTIONABLE ANALYTICS – USING BETTER DATA BETTER
Reach  Engage  Earn
JAMES GWERTZMAN
CEO, PLAYFAB
DAMON DANIELI
CEO, APPURI

2. About Us
• Former Founder/CTO, Z2
(acquired by King.com in 2015)
• Core Designer of Xbox Live (Gamertag
‘d’)
• Filed over 20 patents
• Masters in Computer Science, University
of Washington
• BSEE, University of Washington
James Gwertzman
CEO, PlayFab
• Former Founder, Sprout Games
(acquired by PopCap in 2005)
• Launched and led PopCap’s 1st F2P studio
in China
• At Code.org, led Hour of Code campaign
• Named on several patents for server
technology developed while at Microsoft
• A.B. CS, Harvard
Damon Danieli
CEO, Appuri

3. Vanity metrics are not actionable
Game Events Metrics
?

4. Good Acquisition
Stagnant
Cohort
Performance
Poor
Retention
Daily retention cohorts only show the symptoms
Now what?

5. Using metrics more effectively is about learning
Source: Lean Startup Framework
Idea
Try
Data

6. 21,340
46,953
982 1,273 484 1,162 4,881 5,495 3,846 3,728 3,651
14,200
-44,257
-91,368
-12,092 -11,355
-3,308
-7,042
-19,343
-9,996
-2,423 -891 -543 -852
Acquired Level 1 Level 2 Level 3 Level 4 Level 5 Level 6+ Level 10+ Level 20+ Level 30+ Level 40+ Level 50+
Population Churned
“Ideas” takes work – explore the data
Where are
users pooling
and dropping
out?
What is different
between users who
progress and those
who don’t?
Idea
Try
Data

7. 21,340
46,953
982 1,273 484 1,162 4,881 5,495 3,846 3,728 3,651
14,200
-44,257
-91,368
-12,092 -11,355
-3,308
-7,042
-19,343
-9,996
-2,423 -891 -543 -852
Acquired Level 1 Level 2 Level 3 Level 4 Level 5 Level 6+ Level 10+ Level 20+ Level 30+ Level 40+ Level 50+
Population Churned
Come up with a hypothesis Idea
Try
Data
“Users who progress are earning
achievements early”

11. Test your idea with experiments Idea
Try
Data
Targeted push
notification
campaign

12. Test your idea with experiments Idea
Try
Data
Changing
game behavior
in real time

13. Test your idea with experiments Idea
Try
Data
A/B test to try
multiple tests at
the same time

14. 21,340
46,953
1,212 1,354 578 1,239 5,388 6,323 4,231 3,923 4,123
15,300
-44,257
-76,368
-12,092 -10,447
-3,043
-6,479
-17,796
-9,196
-2,229 -820 -500 -784
Acquired Level 1 Level 2 Level 3 Level 4 Level 5 Level 6+ Level 10+ Level 20+ Level 30+ Level 40+ Level 50+
Population Churned
Measure the results of your change Idea
Try
Data
Did the change
reduce the
churn?

15. Requirements
• Deeper access to your data than many analytics tools allow
– Consider Amazon Redshift – provides direct SQL access
• Rich set of events – you can’t measure what you don’t capture
• Tools to easily run experiments w/o writing new code
– Especially on mobile… don’t resubmit through Apple!

16. Questions?
Tweet @playfabnetwork
Learn more:
https://playfab.com
http://appuri.com
Follow us:
@playfabnetwork, @appuricorp
https://playfab.com/blog/