ML is helping a large Russian game developer and publisher called WebGames analyze data from their free-to-play games. They collect over 80 million records daily from their 400k daily players across various platforms. They use ML for tasks like churn prediction, revenue prediction, user classification, A/B testing, balance, and recommendations. Specifically, they build 30 different models to predict LTV for users based on their behavior in the first 30 days. They also use kNN and cohort-based approaches for user classification and Bayesian A/B testing to dynamically adjust testing over time. Rule-based modeling and midgame support based on classification help balance games. Content recommendations are done through static and dynamic clustering.

1. Game analytics: ML to the rescue!
Evgenii Tsymbalov, Data Scientist

2. Who we are
WebGames(“WG”) is one of Russia’s largest developers and
publishers of free-to-play games
Platforms: FB, iOS/ Android, gamesocial platforms (VK, OK,
MW, Congregate, Steam)
Daily audience of over 400K players
Data: ~80M records per day

3. Game analytics
Marketing analytics
In-game analytics
Churn prediction
Retargeting
Revenue prediction
User classification
A/B testing
Balance
Recommendations
User/content classification

4. Churn and retargeting
Churn
Retargeting
Find users who are about to stop playing
Give them bonuses
…
PROFIT!
Channels: app-to-user notifications, messages, mail
Find and support users for retargeting
Channels: traffic control

5. Revenue prediction
Costumer LTV (Lifetime Value) - estimate of overall
profit from the entire future relationship with a
customer.
Applications:
indicator of project healthiness;
advertising actions planning;
in-game events planning.
It is important to estimate LTV not in general (platform
of project), but for different cohorts or even every
individual player.

6. LTV: methodology and assumptions
Estimating LTV-100 (may vary for project).
User’s actions determined by his or her behavior in first
30 days after registration =>
30 different models, kth for users who plays k days;
Tracking only last year data
General multistage model:
Classification (going to pay or not)
Regression (revenue prediction)
Additional low-level classifiers, such as events, holidays, etc.

7. LTV: accuracy metrics
TA (total accuracy) = . TA = 1 for perfect
predictor.
RAE-d (relative absolute error on day d) =
. This equals to zero for
perfect predictor.
Here, - LTV-100 forecast for i-th player, – total
revenue on day d for i-th player.
TA is a main indicator for marketing department, while
RAE is widely used to compare models’ performance on
different days.

8. LTV models: kNN + cohorts

9. User classification

10. User classification

11. A/B testing
Classic approach: fixed group size, results after
full filling.
Bayesian approach: prior distribution changes over time
with test results using Bayes theorem.
Bayesian A/B Testing at VWO, Chris Stucchio, 2015

12. Balance and recommendations
Classic approach: game-designers with Google Spreadsheets.
Better approach: modeling.
Rule-based approach
Midgame support based on classification
Content recommendations.
Static case
Dynamic case

13. Balance: rule-based approach

14. Balance: midgame support

15. Content clustering

16. Instead of conclusion: what helps us

17. Instead of conclusion: what helps us

18. Questions?
evgenii.tsymbalov@corpwebgames.com

19. Вопрос из зала: конкретные цифры

20. Вопрос из зала: какие алгоритмы?