Recommended
More Related Content
What's hot
What's hot (20)
Similar to UK GIAF Summer 2015 - From data science to data impact
Similar to UK GIAF Summer 2015 - From data science to data impact (20)
Recently uploaded
Recently uploaded (20)
UK GIAF Summer 2015 - From data science to data impact
- 3. ‹#› 2015 © All rights reserved to From Data Science to Data Impact: On many ways to segment your players Volodymyr (Vlad) Kazantsev Head of Data Science at Product Madness jobs@productmadness.com volodymyrk
- 4. ‹#› What we do? jobs@productmadness.com volodymyr.kazantsev@productmadness.com volodymyrk
- 5. ‹#› Heart of Vegas in (public) Numbers iPad US - #13 top grossing iPhone US - #32 top grossing Android - #44 top grossing US (games) Australia iPad - #1 top grossing iPhone - #1 top grossing Android -#3 top grossing jobs@productmadness.com volodymyrk
- 6. ‹#› Data Impact Team Ad-hoc analytics; dashboards Deep dive analysis; Predictive analytics ETL, R&D jobs@productmadness.com volodymyrk
- 7. ‹#› Data Impact Team 7 people; 4 in London office jobs@productmadness.com volodymyrk Ad-hoc analytics; dashboards Deep dive analysis; Predictive analytics ETL, R&D
- 8. ‹#› Ad-hoc analytics; dashboards Deep dive analysis; Predictive analytics ETL, R&D Data Impact Team 7 people; 4 in London office jobs@productmadness.com volodymyrk
- 9. ‹#› Technology Stack ETL orchestration Transformation & Aggregation SQL Data Products Reports Dashboards +
- 10. ‹#› few examples .. A B A/B TestsCustomer Lifetime Value days $value Segmentation group 1 group 2 group 3 group 4
- 12. ‹#› Successful segmentation is the product of a detailed understanding of your market and will therefore take time - Market Segmentation: How to Do it and Profit from it, 4th edition: Malcolm McDonald
- 13. ‹#› Basics Customers have different needs and means Segmentation can help to understand those differences Which can help to deliver on those needs And drive higher profitability
- 14. ‹#› What is a Player Segment? A segment is a group of customers who display similarities to each other... Customers move in and out of segments over time
- 15. ‹#› How many segments are there? There is no one right way to segment (not should there be): Many different approaches and techniques Mix of art, science, common sense, experience and practical knowledge Depends on business needs and availability of data Don’t aim to build one holistic model to meet all needs
- 16. ‹#› Strategic Management Product Development Marketing Operations Comments Geography /Demographics Loyalty / Length of Relationship Behavioural Needs-based Value Based
- 17. ‹#› Strategic Management Product Development Marketing Operations Comments Geography /Demographics ✭✭ ✭✭ ✭✭ Separates players by country, city, city-district, distance from land-based casinos. By generational profile: boomers, Gen-Y, Gen-X. Loyalty / Length of Relationship ✭✭✭ ✭ ✭✭✭ New players, on-boarding, engaged, lapsed, re- engaged, cross-promoted. Behavioural ✭ ✭✭✭ ✭✭✭ Based on identifying player’s behaviour characteristics that help to understand why customer behave the way they do Needs-based ✭ ✭✭✭ ✭ Divide customers based on needs which are being fulfilled by playing Online Slots Value Based ✭✭✭ ✭ ✭✭ Based on present and future value of the customer (RFM/CLV)
- 18. ‹#› Land-based Slots Player segmentation <10% >50%
- 19. ‹#› Segmentation = building a taxonomy All Players New (<28 days) Established (>28d) Payer Non Payer0-2 days 3-7d 8-27 <30 spins >30 … High V Med V Low V Engaged Casual… VIP Concierge
- 20. ‹#› ..and simplifying it daily use All Players New (<28 days) Established (>28d) Payer Non Payer0-2 days 3-7d 8-27 <30 spins >30 … High V Med V Low V Casual… New High Value Med Value Low Value Engaged Casual Engaged
- 21. ‹# › How to profit from Segmentation? 2
- 22. ‹#› Clients of Segmentation ○ Strategy and Finance ○ Product development ○ Marketing operations
- 23. ‹#› Strategy and Finance This Month high-value med-value low-value super free-rider casual slotter recently lapsed high-value 55.27% 30.06% 4.81% 5.54% 2.00% 2.32% med-value 11.11% 42.50% 25.25% 10.92% 6.20% 4.02% low-value 0.59% 7.72% 36.02% 30.59% 17.12% 7.96% super free-rider 0.04% 0.30% 2.76% 70.50% 22.22% 4.18% casual slotter 0.01% 0.10% 0.96% 8.98% 51.37% 38.58% recently lapsed 0.05% 0.22% 1.01% 8.93% 13.00% n/a New 0.01% 0.08% 0.67% 3.22% 31.05% 64.97% LastMonth
- 24. ‹#› Strategy and Finance This Month high-value med-value low-value super free-rider casual slotter recently lapsed high-value 6.80% -0.45% -1.66% -2.39% -1.07% -1.24% med-value 3.09% 2.60% -2.81% -2.12% -0.60% -0.16% low-value 0.11% 0.90% -1.63% 1.99% -0.54% -0.82% super free-rider 0.01% 0.05% -0.05% -2.05% 2.58% -0.54% casual slotter 0.00% 0.02% 0.05% -1.26% 2.71% -1.54% recently lapsed -0.01% -0.05% -0.35% -4.21% -8.43% N/A New 0.01% 0.04% 0.36% 1.59% 16.17% 1.21%Manage transitions, not churn LastMonth
- 25. ‹#› Product Development New Slot Game Released Coins Spent
- 26. ‹#› Product Development Geo: Australia Value: Low-value Behaviour: Prefer Medium bet New Slot Game Released Coins Spent
- 27. ‹#› Marketing Objective Behavioral RFM/CLV geo/demographic Lifecycle Sale Events Monetization campaigns Retention campaigns Re-engagement VIP management
- 28. ‹# › How to actually do segmentation? 3
- 29. ‹#› Pillars of Successful Segmentation Project Business knowledge Data knowledge Analytical skills People Process Technology ETL Machine Learning Business Intelligence Product Integration Marketing Product Data Services
- 30. ‹#› Top-down approach to segmentation 1. Define objectives and therefore customer characteristics a.dd 2. Choice method to split users a.d 3. Prioritise segments to target a.d 4. Operationalise segmentation a.s 5. ‘land’ the segmentation within the organization
- 31. ‹#› Bottom-up approach 360o player view Segmentation Player transitions Tailored interventions Prioritisation and testing ● Build database to provide 360o view of the customer ● Demographic, behavioural, payments, etc. ● Add predictive attributes, such as conversion probability, churn risk, LTV, etc. ● Segment customers by desired attributes: more than one approach ● Use robust statistical techniques for clustering or validation of empirical segmentation ● Ensure segmentation is intuitive for the business and can be used across business functions ● Identify how players are moving from one segment to another (segment transition matrix) ● Determine value levers and identify potential improvement ideas ● Create tailored interventions (CRM, push ..), aimed at moving customers to more valuable segments ● Build predictive models to detect best offer and prevent undesirable transitions ● Prioritise interventions based on expected LTV uplift and ease of implementation ● Test interventions through experimentation
- 32. ‹#› How to actually do segmentation? Just Look at Data Clustering Decision Trees Player Attributes de-correlate Normalise Scale
- 33. ‹#› de-correlate and normalise Player 1 more similar to Player 2 ? Player 3 more similar to Player 2 ? Weekly Play Summary
- 34. ‹#› de-correlate and normalise Weekly Play Summary
- 35. ‹#› de-correlate and normalise Player 1 more similar to Player 2 !
- 36. ‹#› de-correlate and normalise Player 1 more similar to Player 2 !
- 37. ‹#› de-correlate and normalise Player 1 more similar to Player 2, isn’t he?
- 38. ‹#› de-correlate and normalise Player 3 more similar to Player 2 !
- 39. ‹#› What now? K-means Hierarchical Clustering Decision Trees .. and many more
- 40. ‹#› Decision Tree for Clustering All Payers 500 (next month>$100): 4.7% 10000 did not: 95.3% Last_months_dollars <=$2 2 (next month>$100): 0.04% 5000 did not: 99% Last_months_dollars >$2 498 (next month>$100) > $100: 9% 5000 did not: 91% Transactions <=10 243 (next month>$100): 5.5% 4200 did not: 94.5% Transactions > 10 255 (next month>$100): 24% 800 did not: 76%
- 41. ‹#› Decision Tree for Clustering All Payers 500 (next month>$100): 4.7% 10000 did not: 95.3% Last_months_dollars <=$2 2 (next month>$100): 0.04% 5000 did not: 99% Last_months_dollars >$2 498 (next month>$100) > $100: 9% 5000 did not: 91% Transactions <=10 243 (next month>$100): 5.5% 4200 did not: 94.5% Transactions > 10 255 (next month>$100): 24% 800 did not: 76%
- 42. ‹# › Segmentation at Product Madness 4
- 43. ‹#› Lifestage Segmentation On-Boarding Disengaged Engaged not played game Churned
- 44. ‹#› On-boarding segment On-Boarding Disengaged Engaged not played game Churned
- 45. ‹#› On-boarding segment On-Boarding Disengaged Engaged not played game Churned
- 46. ‹#› Lifestage Segmentation On-Boarding Disengaged Engaged low risk high risk low risk high risk low risk high risk not played game churned churned Churned
- 47. ‹#› Behavioural Segmentation Average Bet Gifts per Day Bonuses per Day Machine Stickiness Days Played Spins per Day Preference for New Machines %% of spin on High-Roller machines Big Win Stickiness etc. Hierarchical Clustering
- 49. ‹#› Infrastructure Data Warehouse Segmentation Engine CRM Email GAME Reporting Ad Hoc Analytics Predictive Analytics
- 50. ‹#› Segmentation for A/B tests A B
- 51. ‹#› Segmentation for A/B tests A B
- 52. ‹#› Bonferroni correction: Bayesian Hierarchical Model Combine stats with Market Intuition! Adjustment for multiple testing 𝛼adjustted = 𝛼desired/M
- 53. ‹#› 2015 © All rights reserved to Thank You! jobs@productmadness.com volodymyr.kazantsev@productmadness.com volodymyrk
- 54. JOIN IN THE CONVERSATION PARTICIPATE IN THE NEXT GIAF Analytics for Games www.deltadna.com/giaf events@deltadna.com #UKGIAF
Editor's Notes
- \/ \/ \/ \/ \/ \/ \/ US (games) iPad US - #13 top grossing iPhone US - #32 top grossing Android - #44 top grossing Australia Who cares about Australia. We do iPad - #1 top grossing iPhone - #1 top grossing Android -#3 top grossing Overall, not only games
- Ad-hoc analytics, daily fires, dashboards, Insights Deep dive analysis - reports that take few weeks to complete; Predictive analytics, Machine learning, statistical modelling Data Pipeline, platform for machine learning and modelling
- Insights Data Science Data Engineering 7 people; 4 in London office
- We Are Hiring ! jobs@productmadness.com
- Events are generated on server-side. This way we control data quality. We are processing 350 Million events per day They got ingested into Amazon Cloud to S3, with the help of Python and Spark. And then got copied to Amazon Redshift - Columnar parallel database. Currently we have 12 nodes, with total capacity of 24TB Once the data is in there - we do all heavy aggregations and transformations. We have moved from Hadoop more than a year ago and haven’t looked back since. We perform most of interactive analysis in Python Notebooks. For trivial things we are using re.dash, which is similar to Mode and Periscope. It is Web-based SQL client with integrated plotting and collaboration functionality. You can even create dashboards with re.dash, but for production dashboards we prefer to use Tableau or our own D3.js-based application. All our web applications are using Python backend, Flask framework. We use scikit-learn for machine learning and predictive analytics. As you have probably guessed, we like Python.
- What we do: AB tests bread and butter of data science teams yet controversial and often misunderstood Customer Lifetime Value modelling knowing how much your customer worth, shortly after you acquite them is a holy-grail of User Acqusition can easily spend next 40 minutes talking about customer lifetime value modelling, but .. So - segmentation
- In this presentation I will not be taking in details about classification algorithm, dimensionality reduction or machine learning. Instead, we will be looking at segmentation from Product Marketing perspective.
- Successful segmentation is the product of a detailed understanding of your market and will therefore take time Segmentation is not a two-weeks task you assign to your analytics department
- Customers have different needs and means. Some players play for fun, others got a kick from competition. We all know that players have very different willingness to pay. Most of you know how rare it is to find a Normal Distribution among our players - our games are played by outliers. If you remove outliers from any analysis - you will probably miss the point of it. Segmentation can help to understand those differences Which can help to deliver on those needs And drive higher profitability
- A segment is a group of customers who display similarities to each other... They may react similarly in a product/service offering They may provide comparable values (profitability) to the company They may bear the same needs or behave in alike ways Customers move in and out of segments over time
- There is no one right way to segment (not should there be): Many different approaches and techniques. I will cover few techniques in the following slides. Mix of art, science, common sense, experience and practical knowledge You need to take business needs into account, but also what data is available and can be used, operationally. Don’t aim to build one holistic taxonomy to meet all needs, So what are different types of segmentation? How do you approach a problem like this?
- Multiple way to segment users And there are different use cases for segmentation. You can segment on: geography and basic demographics. In our case - Australian players are very important, and usually behave quite differently from the rest of the world. You can segment based on stage in a player’s Lifecycle - new players behave differently to someone who have been playing your game for the last two years. Also, knowing users who are showing signs of disengagement is very important. You can also segment on Behaviour, Needs (if you can identify them, possibly based on observed behaviour) and, of course, based on Player Value. But different parts of the business are interested in different segmentations.
- Product Managers and Marketing teams might be very interested in Behavioural Segments. But CEO may be more interested to track retention metric for your most valuable players (whales).
- The actual segmentation might be hybrid. This is the segmentation of the Land-based Slot players. First layer - by frequency of play, e.g. engagement Second layers - bahavioural
- But of course, it is important to understand why segmentation is useful for a business. What decisions can it help to make? And how it can affect daily operations and possibly product?
- Clients of Segmentation Strategy and Finance Product development Marketing operations
- Strategy and Finance
- When we looked at data after launch, amount of coints spent has actually dropped on the day of the launch! But was it even a real drop?
- But for a specific segment, that day was very positive.
- Business knowledge: - high-level segments goals - product/marketing strategy Data knowledge: - how to access 360 view? - what are segments definitions? Stats/Analytical skills - how to profile various segments? ETL - recalculated daily or real-time - regular reviews Integration with back-office and game - segmentation engine + ETL Dashboards Reporting Marketing: - day-2-day campaigns for segments - reporting (monthly and daily) Product - review feature success for segment Analytics and data engineering - ongoing support and refinement
- What are business objectives and therefore customer characteristics we should use to profile the market? What approach should we take to ensure segments accurately represent the market and actionable? What criteria should we use to prioritise segments and select targets? How can we ensure segmentation is operational and can be deployed? How to do ‘land’ the segmentation within the organization and ensure it gains traction?
- K-means Hierarchical Clustering Decision Trees .. and many more