3. Who am I?
Senior Data Scientist at SuperScale
4 years of experience with data in gaming
Started in Pixel Federation
• Objective feedback to production team
• Innovative statistical methods to improve the games
Studied applied mathematics
2020-05-28 3
7. How does F2P business work?
You can play for free
Game is monetized with
• In-app purchases (only 2%)
• Ads
8. We need a lot of players to be successful
How does F2P business work?
Game
Organic searchMarketing
9. Cost per player (CPI)
Revenue per player - customer lifetime value (LTV)
Goal of marketing is to grow while being profitable
LTV > CPI
How does F2P business work?
10. LTV > CPI
CPI is easily calculated
But what about LTV?
• You must wait months or years to calculate it
• But that is not the only problem…
How does F2P business work?
11. How much can we spend?
Optimization between scale and profitability
new players
CPI
LTV
PROFIT
profitperplayer
12. How much can we spend?
Optimization between scale and profitability
new players
CPI
LTV
PROFIT
13. How much can we spend?
Optimization between scale and profitability
new players
CPI
LTV
MAXIMUM
PROFIT
14. Spend optimization
Optimization between scale and profitability
• Spend too slow lose profit
• Spend too fast lose profit
When spending millions you want to know your LTV
15. Spend optimization
It is not just one optimization problem
You have several marketing channels
• Facebook
• Google
• Instagram
• And many more…
Granularity can go much deeper
• This leads to small samples and high variance
18. Google cloud infrastructure
Takes care of automatic scaling
Game raw data
Marketing data
Data warehouse
BigQuery
Batch preprocessing
Dataflow
Machine Learning
ML Engine
AI Platform
Model storage
Cloud Storage
Scheduler
Cloud Functions
Analytics & BI
Data Studio
Colab
19. Define the prediction problem
The most important step in modelling
• What specifically are we predicting?
• What are the training/testing data?
• How do we evaluate the predictions?
To answer these, we must specify the final product
20. Define the prediction problem
Important questions
• How will the model be used?
• Will the predictions change anything? What?
Answers determine
• What kind of model should be used
• How should the model be tuned and evaluated
21. Possible use-cases for LTV prediction
Company financial planning
• Simple aggregated model based on averages is enough
22. Possible use-cases for LTV prediction
Company financial planning
Prioritize customer support
• Identification of VIP players as soon as possible
• Classification is more appropriate than regression
23. Possible use-cases for LTV prediction
Company financial planning
Prioritize customer support
Budget allocation between marketing channels
• Absolute values are not as important as relative comparisons
• Variation is more important than bias
24. Possible use-cases for LTV prediction
Company financial planning
Prioritize customer support
Budget allocation between marketing channels
Optimize scale of marketing
• Spend more or not?
• Absolute error is important since it is compared to costs
25. Optimize scale of marketing
We start a campaign in US on Facebook
How is the campaign performing?
• Should we scale up?
• Or cut back on spending?
26. Optimize scale of marketing
Check data
• Each date has different distribution of players
• Not very useful
36. How can we predict this?
Best case scenario
• We have more than 1 year of data
• The game has stable influx of similar players
We can estimate the curve using old cohorts and fit it to new players
37. Best case scenario
We can estimate the curve using old cohorts and fit it to new players
43. Easy, right?
Well, usually not so much
• We do not have enough data
• The curve changes in time
• There is huge variance in cohorts
44. The model works but
You need big cohorts to counter the variance
You need time (data points) to get stable prediction
We want to do better!
45. How to improve?
Variance will always be there unless we sacrifice granularity
But we did not use all available data yet
• We have detailed behavioral data for each player
ML models can be useful here
46. How to train useful ML model
Before you dump everything in a deep NN…
Think about the prediction problem and its use case
• Predict 1-year revenue using first days (e.g. 7 days)
Straightforward approach
• One cohort – one row in dataset
• All data from 7 days are columns of feature matrix
• 1-year revenue is the target
• Train the model
• ???
• Profit!
47. How to train useful ML model
Even if you get a good model, there is a problem
You had to exclude all cohorts from last year
Gaming industry is changing all the time
Try explaining that to the end user
What now?
48. How to train useful ML model
One possible solution
• Change the target to shorter horizon
• Add the prediction to your original data and fit the curve again
• This can improve your early prediction performance
data
ML model result
Improved fitted curve
49. How to communicate results
You have a great model, but you must convince others
• Different stakeholders have different interests
How did we convince EA that our model is better?
50. How to communicate results
Definitely not like this
Electronics Arts SuperScale
RMSE 195 036 28 487
51. How to communicate results
We visualize the usefulness leading to better business decisions
30% change compared to day 7 prediction
10% change compared to day 7 prediction
52. How to communicate results
Accuracy is not enough in real applications
• Bias
• Variance
• Stability in time
53. Final notes
We simplified a lot
• There are a lot questions, uncertainties and practical difficulties
• You must make a lot of impactful decisions along the whole process
This leads to numerous variations of the final model
54. How we look at it
There are
• a lot of different models with different advantages
• a lot different games with different monetization
One model cannot incorporate everything
Our approach to LTV modelling
• Create an ensemble of models that works on any game
• Use the models to make automatic recommendations for marketing optimization
55. Conclusion
LTV predictions can save you a lot of money
Basic predictions are easy, but it is hard to use the full potential
Making useful ML model is never straightforward
Think first about how the final model will be used
• This leads to appropriate model definition and evaluation strategy