Nick Lim Casual Connect CEO Online Social Games Track Sonamine  July 21, 2011 (Revised: DRS, July 28) Beyond Metrics: Pred...
Agenda <ul><li>Chapter 1 – Predictives and metrics </li></ul><ul><li>Chapter 2 – How to identify a predictive </li></ul><u...
© 2009-11 Sonamine LLC.  Confidential and proprietary Chapter 1 What are predictives?
Predictives help you avoid spamming your players © 2009-11 Sonamine LLC.  Confidential and proprietary Source:  “ Capital ...
From  metrics  (reporting) to  predictions © 2009-11 Sonamine LLC.  Confidential and proprietary Source:  Competing on Ana...
<ul><li>Metrics –    Measuring  a characteristic of the past.  </li></ul><ul><ul><li>Engagement: play sessions (frequency,...
<ul><li>Metric:    The outstanding balance on your credit card </li></ul><ul><li>Predictive:   Your FICO credit score </li...
<ul><li>Mobile phone companies   </li></ul><ul><ul><li>Predict which customers will cancel, or buy a new data plan </li></...
© 2009-11 Sonamine LLC.  Confidential and proprietary Chapter 2 How to identify a predictive
<ul><li>Metrics:   If you are looking at data from the  past </li></ul><ul><ul><li>“ARPU last month = $0.25;  DAU yesterda...
<ul><li>Metrics: If you  can  be 100% sure </li></ul><ul><ul><li>“Server was down  for 10 min from 12:00-12:10”  (You can ...
<ul><li>Metrics: If you view correlations between  few variables </li></ul><ul><ul><li>“ Number of invitees  correlates wi...
© 2009-11 Sonamine LLC.  Confidential and proprietary Chapter 3  Using Predictives (A Crash Course)
© 2009-11 Sonamine LLC.  Confidential and proprietary Behavior Random selection Without predictives, every segment has equ...
© 2009-11 Sonamine LLC.  Confidential and proprietary Behavior Random selection Predictive Predictives allow segmentations...
<ul><li>Base rate :   Overall behavior rate found in the population </li></ul><ul><ul><li>Example: daily conversion rate =...
<ul><li>True-positive rate:   % of behaviors correctly predicted  </li></ul><ul><li>False-positive rate:   % of non-behavi...
<ul><li>Lift  over  4x  in first  decile  is very good. </li></ul><ul><li>High lift means: </li></ul><ul><ul><li>More beha...
© 2009-11 Sonamine LLC.  Confidential and proprietary Object for which you’re predicting  (In games: usually the  player )...
© 2009-11 Sonamine LLC.  Confidential and proprietary <ul><li>Most likely to convert   </li></ul><ul><li>Turn off 3 rd  pa...
© 2009-11 Sonamine LLC.  Confidential and proprietary Chapter 4 Myths, Lessons, & Tips
<ul><li>Data warehouse:  A centralized, organized data repository. </li></ul><ul><li>Predictives use data, but it need not...
<ul><li>Try for maximum lift from minimum work. </li></ul><ul><ul><li>An 80/20 rule applies </li></ul></ul><ul><li>You can...
<ul><li>Ad-hoc campaigns are ok, but are not scalable </li></ul><ul><li>Plan for ongoing, customized promotions </li></ul>...
<ul><li>Build  scalable use of predictives  into your games: </li></ul><ul><li>Player-communications to target specific pl...
© 2009-11 Sonamine LLC.  Confidential and proprietary Chapter 5 Case Studies
<ul><li>Opportunity </li></ul><ul><li>Upsell a product to existing customers. </li></ul><ul><li>Solution </li></ul><ul><li...
© 2009-11 Sonamine LLC.  Confidential and proprietary <ul><li>Conversion rates for promotional  campaign. Effect of segmen...
© 2009-11 Sonamine LLC.  Confidential and proprietary Case Study #2:  Korean Social Game
<ul><li>Opportunity </li></ul><ul><li>Coax borderline converters to 1 st -time purchase </li></ul><ul><li>Solution </li></...
© 2009-11 Sonamine LLC.  Confidential and proprietary 63% Higher Case Study #2:  Conversion rates
© 2009-11 Sonamine LLC.  Confidential and proprietary Case Study #3: Portal/developer of multiplayer casual social games
<ul><li>Opportunity </li></ul><ul><li>Get more borderline converters  to make first-time-purchase </li></ul><ul><li>Soluti...
© 2009-11 Sonamine LLC.  Confidential and proprietary Case Study #3: Conversion rates (160% higher than  no  promotion)
© 2009-11 Sonamine LLC.  Confidential and proprietary 150% higher Case Study #3: Conversion rates (150% higher than  rando...
<ul><li>Automated continuous campaigns  are expected to increase new conversions by 10% </li></ul><ul><li>Incremental reve...
© 2009-11 Sonamine LLC.  Confidential and proprietary A final note Games as a service will  profit  from  predictive   pla...
<ul><li>Technical Introduction </li></ul><ul><ul><li>http:// en.wikipedia.org/wiki/Predictive_analytics </li></ul></ul><ul...
© 2009-11 Sonamine LLC.  Confidential and proprietary For more about predictives  and Sonamine’s free trial program
Upcoming SlideShare
Loading in …5
×

Sonamine casual connect july 2011 beyond metrics - predictives

2,827 views

Published on

Sonamine presentation at Casual Connect July 2011. Beyond Metrics - Predictives with lessons and perspectives from other industries.

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,827
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
90
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • 3 weeks’ worth of CapitalOne credit-card offers! This diagram is taken from a recent blog post on consumerist.com http://consumerist.com/2011/07/capital-one-sends-you-over-20-credit-card-offers-in-3-weeks.html Predictives make sure you don’t get sent mail you don’t want.
  • Metrics = measuring things. After you measure things, you have to tell people what you measured. Then you can ask questions about what you measured, and you can be notified regarding measured things. This diagram is taken from the popular book Competing on Analytics by Tom Davenport and Jeanne Harris. http://www.amazon.com/Competing-Analytics-New-Science-Winning/dp/1422103323
  • If that example top decile has 4X lift, it will therefore have 40% of the total player-conversions. For more on lift, including a curve showing the front-loading effect (the cumulative gains) of predictions, and a lift curve: http://www2.cs.uregina.ca/~dbd/cs831/notes/lift_chart/lift_chart.html
  • True-positive rate = sensitivity False positive rate (α) = FP / (FP + TN) = 1 − specificity False negative rate (β) = FN / (TP + FN) = 1 − sensitivity
  • Sonamine casual connect july 2011 beyond metrics - predictives

    1. 1. Nick Lim Casual Connect CEO Online Social Games Track Sonamine July 21, 2011 (Revised: DRS, July 28) Beyond Metrics: Predictives With lessons and perspectives from other industries
    2. 2. Agenda <ul><li>Chapter 1 – Predictives and metrics </li></ul><ul><li>Chapter 2 – How to identify a predictive </li></ul><ul><li>Chapter 3 – Crash course: Using predictives </li></ul><ul><li>Chapter 4 – Myths, lessons, and tips </li></ul><ul><li>Chapter 5 – Case studies </li></ul><ul><ul><li>10% more revenue, 160% more conversions. </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary
    3. 3. © 2009-11 Sonamine LLC. Confidential and proprietary Chapter 1 What are predictives?
    4. 4. Predictives help you avoid spamming your players © 2009-11 Sonamine LLC. Confidential and proprietary Source: “ Capital One…over 20 credit card offers in 3 weeks” Consumerist July, 2011
    5. 5. From metrics (reporting) to predictions © 2009-11 Sonamine LLC. Confidential and proprietary Source: Competing on Analytics Davenport & Harris
    6. 6. <ul><li>Metrics – Measuring a characteristic of the past. </li></ul><ul><ul><li>Engagement: play sessions (frequency, length) </li></ul></ul><ul><ul><li>Brand preference – on a scale of 1-5 do you like it? </li></ul></ul><ul><ul><li>DAU & MAU; conversion & churn rates </li></ul></ul><ul><li>Predictives – Estimating what will happen in the future </li></ul><ul><ul><li>Which players are the most likely to convert ? </li></ul></ul><ul><ul><li>Which item is this player most likely to buy ? </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Predictives are not metrics
    7. 7. <ul><li>Metric: The outstanding balance on your credit card </li></ul><ul><li>Predictive: Your FICO credit score </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Metric or predictive? (Finance)
    8. 8. <ul><li>Mobile phone companies </li></ul><ul><ul><li>Predict which customers will cancel, or buy a new data plan </li></ul></ul><ul><li>Insurance </li></ul><ul><ul><li>Predict which cars & drivers will get into accidents (to charge them higher premiums accordingly) </li></ul></ul><ul><li>Financial services </li></ul><ul><ul><li>Predict which transaction is fraudulent </li></ul></ul><ul><ul><li>Predict likelihood of default on a loan or mortgage </li></ul></ul><ul><li>Online advertising </li></ul><ul><ul><li>Predict which ad you will click on (to select the ads you see) </li></ul></ul><ul><li>Search engines </li></ul><ul><ul><li>Predict which page is most relevant to your search query </li></ul></ul><ul><li>Public service </li></ul><ul><ul><li>Predict which criminal offenders will again commit that crime (to give them longer sentences) </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary How other industries use predictives
    9. 9. © 2009-11 Sonamine LLC. Confidential and proprietary Chapter 2 How to identify a predictive
    10. 10. <ul><li>Metrics: If you are looking at data from the past </li></ul><ul><ul><li>“ARPU last month = $0.25; DAU yesterday = 45K” </li></ul></ul><ul><li>Predictives: If you are looking into the future </li></ul><ul><ul><li>“ In this group, about 2% will convert within 14 days” </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Past vs. Future
    11. 11. <ul><li>Metrics: If you can be 100% sure </li></ul><ul><ul><li>“Server was down for 10 min from 12:00-12:10” (You can check server logs) </li></ul></ul><ul><li>Predictives: If you cannot be sure now, and are guessing </li></ul><ul><ul><li>“ In this group, about 2% will convert within 14 days” (We don’t now know) </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Certainty vs. Guessing
    12. 12. <ul><li>Metrics: If you view correlations between few variables </li></ul><ul><ul><li>“ Number of invitees correlates with acceptance rate ”. </li></ul></ul><ul><li>Predictives: If you look at 50-100 variables at the same time </li></ul><ul><ul><li>Sonamine’s ConvertSoon predictions derive a predictive score from over 50 different variables </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Few variables vs. Many variables
    13. 13. © 2009-11 Sonamine LLC. Confidential and proprietary Chapter 3 Using Predictives (A Crash Course)
    14. 14. © 2009-11 Sonamine LLC. Confidential and proprietary Behavior Random selection Without predictives, every segment has equal concentration of a future behavior
    15. 15. © 2009-11 Sonamine LLC. Confidential and proprietary Behavior Random selection Predictive Predictives allow segmentations that concentrate the target behavior
    16. 16. <ul><li>Base rate : Overall behavior rate found in the population </li></ul><ul><ul><li>Example: daily conversion rate = 0.1% </li></ul></ul><ul><li>Deciles: Breaking a population into ten ranked segments </li></ul><ul><li>Lift: (Top segment’s behavior rate) ÷ (Base rate) </li></ul><ul><ul><li>Example: If the #1 predictive decile has 0.4% conversion, then this prediction’s lift = 4X </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary How do you know it’s working? (First: some terms) 0 100
    17. 17. <ul><li>True-positive rate: % of behaviors correctly predicted </li></ul><ul><li>False-positive rate: % of non-behaviors incorrectly predicted </li></ul><ul><li>Machine learning algorithms </li></ul><ul><ul><li>E.g. neural networks, support vector machines </li></ul></ul><ul><li>Social network analysis: Using connections to find dense communities and to simulate or predict diffusion of behavior </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary More terminology…
    18. 18. <ul><li>Lift over 4x in first decile is very good. </li></ul><ul><li>High lift means: </li></ul><ul><ul><li>More behaviors concentrated into top segment </li></ul></ul><ul><ul><li>In other words: more true positives </li></ul></ul><ul><ul><li>Also, fewer false positives (fewer non-behaviors) </li></ul></ul><ul><li>Business results: </li></ul><ul><ul><li>Better results from a smaller campaign </li></ul></ul><ul><ul><li>More conversions, less churn </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary How do you know it’s working? Part 2: Lift
    19. 19. © 2009-11 Sonamine LLC. Confidential and proprietary Object for which you’re predicting (In games: usually the player ) Score that ranks the players Higher score  player is more likely to behave in a certain way (Examples: convert, churn, buy) What predictive output looks like
    20. 20. © 2009-11 Sonamine LLC. Confidential and proprietary <ul><li>Most likely to convert </li></ul><ul><li>Turn off 3 rd party ads </li></ul><ul><li>Offer a promotion </li></ul><ul><li>Least likely to convert </li></ul><ul><li>Turn on ads </li></ul><ul><li>Turn on cross-promo bar </li></ul>How to use the predictive score?
    21. 21. © 2009-11 Sonamine LLC. Confidential and proprietary Chapter 4 Myths, Lessons, & Tips
    22. 22. <ul><li>Data warehouse: A centralized, organized data repository. </li></ul><ul><li>Predictives use data, but it need not come from a data warehouse. </li></ul><ul><li>Lesson : don’t wait for a 12-month data warehouse project, before you start predictives </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Myth 1: Predictives require a data warehouse
    23. 23. <ul><li>Try for maximum lift from minimum work. </li></ul><ul><ul><li>An 80/20 rule applies </li></ul></ul><ul><li>You can always get better accuracy if you: (a) spend more time, or (b) spend more money …but is it worth it? </li></ul><ul><li>Lesson : If you have 3-4x lift in top decile, that is good enough to start. </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Myth 2: Accuracy is everything
    24. 24. <ul><li>Ad-hoc campaigns are ok, but are not scalable </li></ul><ul><li>Plan for ongoing, customized promotions </li></ul><ul><li>Example: </li></ul><ul><ul><li>Have user-profile fields, to store predictive ratings for conversion, up-sell, and churn </li></ul></ul><ul><ul><li>Have a schedule of promotions and virtual items </li></ul></ul><ul><ul><li>Build into your game: </li></ul></ul><ul><ul><ul><li>Automated checking of player profile </li></ul></ul></ul><ul><ul><ul><li>Conditional display of current promotions </li></ul></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Tip 1: Plan for multiple, simultaneous, automated campaigns
    25. 25. <ul><li>Build scalable use of predictives into your games: </li></ul><ul><li>Player-communications to target specific players </li></ul><ul><li>Change game-behavior based on player id </li></ul><ul><li>Change context (eg. ads) based on player id </li></ul><ul><li>Design with player-communication in mind </li></ul><ul><li>Test the results across different segments: </li></ul><ul><li>Have A-B testing system that’s easy to re-target </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Tip 2: Customize user experiences. Test and learn
    26. 26. © 2009-11 Sonamine LLC. Confidential and proprietary Chapter 5 Case Studies
    27. 27. <ul><li>Opportunity </li></ul><ul><li>Upsell a product to existing customers. </li></ul><ul><li>Solution </li></ul><ul><li>22 Predictive segments created. Based on: </li></ul><ul><ul><li>Loyalty, usage, social-network characteristics. </li></ul></ul><ul><li>Mail campaign (promoting the new product) was customized for each segment </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Case Study #1: AT&T Upsell
    28. 28. © 2009-11 Sonamine LLC. Confidential and proprietary <ul><li>Conversion rates for promotional campaign. Effect of segmentation: </li></ul><ul><li>by behavior , and/or </li></ul><ul><li>by social-network analysis </li></ul>Case Study #1: Conversion Rates
    29. 29. © 2009-11 Sonamine LLC. Confidential and proprietary Case Study #2: Korean Social Game
    30. 30. <ul><li>Opportunity </li></ul><ul><li>Coax borderline converters to 1 st -time purchase </li></ul><ul><li>Solution </li></ul><ul><li>Analyzed available game-play data </li></ul><ul><li>Grouped players into 2 key conversion segments </li></ul><ul><li>Showed in-game promotion to key segments. </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Case Study #2: Korean Social Game
    31. 31. © 2009-11 Sonamine LLC. Confidential and proprietary 63% Higher Case Study #2: Conversion rates
    32. 32. © 2009-11 Sonamine LLC. Confidential and proprietary Case Study #3: Portal/developer of multiplayer casual social games
    33. 33. <ul><li>Opportunity </li></ul><ul><li>Get more borderline converters to make first-time-purchase </li></ul><ul><li>Solution </li></ul><ul><li>Analyzed available game play data </li></ul><ul><li>Grouped players into 20 conversion segments </li></ul><ul><li>Email promo to top segment, with A-B test </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Case Study #3: Portal/developer of multiplayer casual social games
    34. 34. © 2009-11 Sonamine LLC. Confidential and proprietary Case Study #3: Conversion rates (160% higher than no promotion)
    35. 35. © 2009-11 Sonamine LLC. Confidential and proprietary 150% higher Case Study #3: Conversion rates (150% higher than random promotion)
    36. 36. <ul><li>Automated continuous campaigns are expected to increase new conversions by 10% </li></ul><ul><li>Incremental revenue 5x greater than investment </li></ul>© 2009-11 Sonamine LLC. Confidential and proprietary Case Study #3: Additional benefits
    37. 37. © 2009-11 Sonamine LLC. Confidential and proprietary A final note Games as a service will profit from predictive player relationship management
    38. 38. <ul><li>Technical Introduction </li></ul><ul><ul><li>http:// en.wikipedia.org/wiki/Predictive_analytics </li></ul></ul><ul><li>Trade show for learning </li></ul><ul><ul><li>http:// www.predictiveanalyticsworld.com / </li></ul></ul><ul><li>Myths and pitfalls </li></ul><ul><ul><li>http://www.information-management.com/specialreports/20050503/1026882-1.html </li></ul></ul><ul><li>Sonamine information, and whitepaper </li></ul><ul><ul><li>http://www.Sonamine.com/ </li></ul></ul>© 2009-11 Sonamine LLC. Confidential and proprietary
    39. 39. © 2009-11 Sonamine LLC. Confidential and proprietary For more about predictives and Sonamine’s free trial program

    ×