Download to read offline
More Related Content
Similar to UK GIAF Summer 2015 - From data science to data impact
UK GIAF Summer 2015 - From data science to data impact
- 1.
- 3. ‹#› 2015 © Allrights reserved to From Data Science to Data Impact: On many ways to segment your players Volodymyr (Vlad) Kazantsev Head of Data Science at Product Madness jobs@productmadness.com volodymyrk
- 4. ‹#› What we do? jobs@productmadness.comvolodymyr.kazantsev@productmadness.com volodymyrk
- 5. ‹#› Heart of Vegasin (public) Numbers iPad US - #13 top grossing iPhone US - #32 top grossing Android - #44 top grossing US (games) Australia iPad - #1 top grossing iPhone - #1 top grossing Android -#3 top grossing jobs@productmadness.com volodymyrk
- 6. ‹#› Data Impact Team Ad-hocanalytics; dashboards Deep dive analysis; Predictive analytics ETL, R&D jobs@productmadness.com volodymyrk
- 7. ‹#› Data Impact Team 7people; 4 in London office jobs@productmadness.com volodymyrk Ad-hoc analytics; dashboards Deep dive analysis; Predictive analytics ETL, R&D
- 8. ‹#› Ad-hoc analytics; dashboards Deep diveanalysis; Predictive analytics ETL, R&D Data Impact Team 7 people; 4 in London office jobs@productmadness.com volodymyrk
- 9. ‹#› Technology Stack ETL orchestration Transformation & Aggregation SQL DataProducts Reports Dashboards +
- 10. ‹#› few examples .. AB A/B TestsCustomer Lifetime Value days $value Segmentation group 1 group 2 group 3 group 4
- 11.
- 12. ‹#› Successful segmentation isthe product of a detailed understanding of your market and will therefore take time - Market Segmentation: How to Do it and Profit from it, 4th edition: Malcolm McDonald
- 13. ‹#› Basics Customers have differentneeds and means Segmentation can help to understand those differences Which can help to deliver on those needs And drive higher profitability
- 14. ‹#› What is aPlayer Segment? A segment is a group of customers who display similarities to each other... Customers move in and out of segments over time
- 15. ‹#› How many segmentsare there? There is no one right way to segment (not should there be): Many different approaches and techniques Mix of art, science, common sense, experience and practical knowledge Depends on business needs and availability of data Don’t aim to build one holistic model to meet all needs
- 16. ‹#› Strategic Management Product Development Marketing Operations Comments Geography /Demographics Loyalty / Lengthof Relationship Behavioural Needs-based Value Based
- 17. ‹#› Strategic Management Product Development Marketing Operations Comments Geography /Demographics ✭✭ ✭✭ ✭✭ Separatesplayers by country, city, city-district, distance from land-based casinos. By generational profile: boomers, Gen-Y, Gen-X. Loyalty / Length of Relationship ✭✭✭ ✭ ✭✭✭ New players, on-boarding, engaged, lapsed, re- engaged, cross-promoted. Behavioural ✭ ✭✭✭ ✭✭✭ Based on identifying player’s behaviour characteristics that help to understand why customer behave the way they do Needs-based ✭ ✭✭✭ ✭ Divide customers based on needs which are being fulfilled by playing Online Slots Value Based ✭✭✭ ✭ ✭✭ Based on present and future value of the customer (RFM/CLV)
- 18. ‹#› Land-based Slots Playersegmentation <10% >50%
- 19. ‹#› Segmentation = buildinga taxonomy All Players New (<28 days) Established (>28d) Payer Non Payer0-2 days 3-7d 8-27 <30 spins >30 … High V Med V Low V Engaged Casual… VIP Concierge
- 20. ‹#› ..and simplifying itdaily use All Players New (<28 days) Established (>28d) Payer Non Payer0-2 days 3-7d 8-27 <30 spins >30 … High V Med V Low V Casual… New High Value Med Value Low Value Engaged Casual Engaged
- 21. ‹# › How to profitfrom Segmentation? 2
- 22. ‹#› Clients of Segmentation ○Strategy and Finance ○ Product development ○ Marketing operations
- 23. ‹#› Strategy and Finance ThisMonth high-value med-value low-value super free-rider casual slotter recently lapsed high-value 55.27% 30.06% 4.81% 5.54% 2.00% 2.32% med-value 11.11% 42.50% 25.25% 10.92% 6.20% 4.02% low-value 0.59% 7.72% 36.02% 30.59% 17.12% 7.96% super free-rider 0.04% 0.30% 2.76% 70.50% 22.22% 4.18% casual slotter 0.01% 0.10% 0.96% 8.98% 51.37% 38.58% recently lapsed 0.05% 0.22% 1.01% 8.93% 13.00% n/a New 0.01% 0.08% 0.67% 3.22% 31.05% 64.97% LastMonth
- 24. ‹#› Strategy and Finance ThisMonth high-value med-value low-value super free-rider casual slotter recently lapsed high-value 6.80% -0.45% -1.66% -2.39% -1.07% -1.24% med-value 3.09% 2.60% -2.81% -2.12% -0.60% -0.16% low-value 0.11% 0.90% -1.63% 1.99% -0.54% -0.82% super free-rider 0.01% 0.05% -0.05% -2.05% 2.58% -0.54% casual slotter 0.00% 0.02% 0.05% -1.26% 2.71% -1.54% recently lapsed -0.01% -0.05% -0.35% -4.21% -8.43% N/A New 0.01% 0.04% 0.36% 1.59% 16.17% 1.21%Manage transitions, not churn LastMonth
- 25. ‹#› Product Development New SlotGame Released Coins Spent
- 26. ‹#› Product Development Geo: Australia Value:Low-value Behaviour: Prefer Medium bet New Slot Game Released Coins Spent
- 27. ‹#› Marketing Objective Behavioral RFM/CLVgeo/demographic Lifecycle Sale Events Monetization campaigns Retention campaigns Re-engagement VIP management
- 28. ‹# › How to actuallydo segmentation? 3
- 29. ‹#› Pillars of SuccessfulSegmentation Project Business knowledge Data knowledge Analytical skills People Process Technology ETL Machine Learning Business Intelligence Product Integration Marketing Product Data Services
- 30. ‹#› Top-down approach tosegmentation 1. Define objectives and therefore customer characteristics a.dd 2. Choice method to split users a.d 3. Prioritise segments to target a.d 4. Operationalise segmentation a.s 5. ‘land’ the segmentation within the organization
- 31. ‹#› Bottom-up approach 360o player view Segmentation Player transitions Tailored interventions Prioritisation andtesting ● Build database to provide 360o view of the customer ● Demographic, behavioural, payments, etc. ● Add predictive attributes, such as conversion probability, churn risk, LTV, etc. ● Segment customers by desired attributes: more than one approach ● Use robust statistical techniques for clustering or validation of empirical segmentation ● Ensure segmentation is intuitive for the business and can be used across business functions ● Identify how players are moving from one segment to another (segment transition matrix) ● Determine value levers and identify potential improvement ideas ● Create tailored interventions (CRM, push ..), aimed at moving customers to more valuable segments ● Build predictive models to detect best offer and prevent undesirable transitions ● Prioritise interventions based on expected LTV uplift and ease of implementation ● Test interventions through experimentation
- 32. ‹#› How to actuallydo segmentation? Just Look at Data Clustering Decision Trees Player Attributes de-correlate Normalise Scale
- 33. ‹#› de-correlate and normalise Player1 more similar to Player 2 ? Player 3 more similar to Player 2 ? Weekly Play Summary
- 34. ‹#› de-correlate and normalise WeeklyPlay Summary
- 35. ‹#› de-correlate and normalise Player1 more similar to Player 2 !
- 36. ‹#› de-correlate and normalise Player1 more similar to Player 2 !
- 37. ‹#› de-correlate and normalise Player1 more similar to Player 2, isn’t he?
- 38. ‹#› de-correlate and normalise Player3 more similar to Player 2 !
- 39. ‹#› What now? K-means Hierarchical Clustering DecisionTrees .. and many more
- 40. ‹#› Decision Tree forClustering All Payers 500 (next month>$100): 4.7% 10000 did not: 95.3% Last_months_dollars <=$2 2 (next month>$100): 0.04% 5000 did not: 99% Last_months_dollars >$2 498 (next month>$100) > $100: 9% 5000 did not: 91% Transactions <=10 243 (next month>$100): 5.5% 4200 did not: 94.5% Transactions > 10 255 (next month>$100): 24% 800 did not: 76%
- 41. ‹#› Decision Tree forClustering All Payers 500 (next month>$100): 4.7% 10000 did not: 95.3% Last_months_dollars <=$2 2 (next month>$100): 0.04% 5000 did not: 99% Last_months_dollars >$2 498 (next month>$100) > $100: 9% 5000 did not: 91% Transactions <=10 243 (next month>$100): 5.5% 4200 did not: 94.5% Transactions > 10 255 (next month>$100): 24% 800 did not: 76%
- 42. ‹# › Segmentation at ProductMadness 4
- 43.
- 44.
- 45.
- 46. ‹#› Lifestage Segmentation On-Boarding Disengaged Engaged low risk highrisk low risk high risk low risk high risk not played game churned churned Churned
- 47. ‹#› Behavioural Segmentation Average Bet Giftsper Day Bonuses per Day Machine Stickiness Days Played Spins per Day Preference for New Machines %% of spin on High-Roller machines Big Win Stickiness etc. Hierarchical Clustering
- 48.
- 49. ‹#› Infrastructure Data Warehouse Segmentation Engine CRMEmail GAME Reporting Ad Hoc Analytics Predictive Analytics
- 50. ‹#› Segmentation for A/Btests A B
- 51. ‹#› Segmentation for A/Btests A B
- 52. ‹#› Bonferroni correction: Bayesian HierarchicalModel Combine stats with Market Intuition! Adjustment for multiple testing 𝛼adjustted = 𝛼desired/M
- 53. ‹#› 2015 © Allrights reserved to Thank You! jobs@productmadness.com volodymyr.kazantsev@productmadness.com volodymyrk
- 54. JOIN IN THECONVERSATION PARTICIPATE IN THE NEXT GIAF Analytics for Games www.deltadna.com/giaf events@deltadna.com #UKGIAF
Editor's Notes
- #6 \/ \/ \/ \/ \/ \/ \/ US (games) iPad US - #13 top grossing iPhone US - #32 top grossing Android - #44 top grossing Australia Who cares about Australia. We do iPad - #1 top grossing iPhone - #1 top grossing Android -#3 top grossing Overall, not only games
- #7 Ad-hoc analytics, daily fires, dashboards, Insights Deep dive analysis - reports that take few weeks to complete; Predictive analytics, Machine learning, statistical modelling Data Pipeline, platform for machine learning and modelling
- #8 Insights Data Science Data Engineering 7 people; 4 in London office
- #9 We Are Hiring ! jobs@productmadness.com
- #10 Events are generated on server-side. This way we control data quality. We are processing 350 Million events per day They got ingested into Amazon Cloud to S3, with the help of Python and Spark. And then got copied to Amazon Redshift - Columnar parallel database. Currently we have 12 nodes, with total capacity of 24TB Once the data is in there - we do all heavy aggregations and transformations. We have moved from Hadoop more than a year ago and haven’t looked back since. We perform most of interactive analysis in Python Notebooks. For trivial things we are using re.dash, which is similar to Mode and Periscope. It is Web-based SQL client with integrated plotting and collaboration functionality. You can even create dashboards with re.dash, but for production dashboards we prefer to use Tableau or our own D3.js-based application. All our web applications are using Python backend, Flask framework. We use scikit-learn for machine learning and predictive analytics. As you have probably guessed, we like Python.
- #11 What we do: AB tests bread and butter of data science teams yet controversial and often misunderstood Customer Lifetime Value modelling knowing how much your customer worth, shortly after you acquite them is a holy-grail of User Acqusition can easily spend next 40 minutes talking about customer lifetime value modelling, but .. So - segmentation
- #12 In this presentation I will not be taking in details about classification algorithm, dimensionality reduction or machine learning. Instead, we will be looking at segmentation from Product Marketing perspective.
- #13 Successful segmentation is the product of a detailed understanding of your market and will therefore take time Segmentation is not a two-weeks task you assign to your analytics department
- #14 Customers have different needs and means. Some players play for fun, others got a kick from competition. We all know that players have very different willingness to pay. Most of you know how rare it is to find a Normal Distribution among our players - our games are played by outliers. If you remove outliers from any analysis - you will probably miss the point of it. Segmentation can help to understand those differences Which can help to deliver on those needs And drive higher profitability
- #15 A segment is a group of customers who display similarities to each other... They may react similarly in a product/service offering They may provide comparable values (profitability) to the company They may bear the same needs or behave in alike ways Customers move in and out of segments over time
- #16 There is no one right way to segment (not should there be): Many different approaches and techniques. I will cover few techniques in the following slides. Mix of art, science, common sense, experience and practical knowledge You need to take business needs into account, but also what data is available and can be used, operationally. Don’t aim to build one holistic taxonomy to meet all needs, So what are different types of segmentation? How do you approach a problem like this?
- #17 Multiple way to segment users And there are different use cases for segmentation. You can segment on: geography and basic demographics. In our case - Australian players are very important, and usually behave quite differently from the rest of the world. You can segment based on stage in a player’s Lifecycle - new players behave differently to someone who have been playing your game for the last two years. Also, knowing users who are showing signs of disengagement is very important. You can also segment on Behaviour, Needs (if you can identify them, possibly based on observed behaviour) and, of course, based on Player Value. But different parts of the business are interested in different segmentations.
- #18 Product Managers and Marketing teams might be very interested in Behavioural Segments. But CEO may be more interested to track retention metric for your most valuable players (whales).
- #19 The actual segmentation might be hybrid. This is the segmentation of the Land-based Slot players. First layer - by frequency of play, e.g. engagement Second layers - bahavioural
- #22 But of course, it is important to understand why segmentation is useful for a business. What decisions can it help to make? And how it can affect daily operations and possibly product?
- #23 Clients of Segmentation Strategy and Finance Product development Marketing operations
- #24 Strategy and Finance
- #26 When we looked at data after launch, amount of coints spent has actually dropped on the day of the launch! But was it even a real drop?
- #27 But for a specific segment, that day was very positive.
- #30 Business knowledge: - high-level segments goals - product/marketing strategy Data knowledge: - how to access 360 view? - what are segments definitions? Stats/Analytical skills - how to profile various segments? ETL - recalculated daily or real-time - regular reviews Integration with back-office and game - segmentation engine + ETL Dashboards Reporting Marketing: - day-2-day campaigns for segments - reporting (monthly and daily) Product - review feature success for segment Analytics and data engineering - ongoing support and refinement
- #31 What are business objectives and therefore customer characteristics we should use to profile the market? What approach should we take to ensure segments accurately represent the market and actionable? What criteria should we use to prioritise segments and select targets? How can we ensure segmentation is operational and can be deployed? How to do ‘land’ the segmentation within the organization and ensure it gains traction?
- #40 K-means Hierarchical Clustering Decision Trees .. and many more