• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
13 jun13 gaming-webinar
 

13 jun13 gaming-webinar

on

  • 3,168 views

 

Statistics

Views

Total Views
3,168
Views on SlideShare
764
Embed Views
2,404

Actions

Likes
3
Downloads
27
Comments
0

10 Embeds 2,404

http://www.revolutionanalytics.com 2245
http://yonniedev.devcloud.acquia-sites.com 87
http://www.linkedin.com 37
http://localhost 22
http://yonnie.devcloud.acquia-sites.com 4
http://yonnietest.devcloud.acquia-sites.com 4
http://www.facebook.com 2
https://www.facebook.com 1
https://twitter.com 1
https://www.google.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    13 jun13 gaming-webinar 13 jun13 gaming-webinar Presentation Transcript

    • Knowing How People are Playing Your GameGives You the Winning HandBill Grosso(bill@scientificrevenue.com)© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • This session will present the state of the art of predictiveanalytics for gaming, focusing on basic ideas to advancedconcepts such as machine learning that can be usedprofitably by almost every gaming studio. We’ll cover:• An overview of the use of analytics in gaming• Predicting churn: why and how• Identifying unusual behavior• Gold farming, fraud analysis, and unlocking gifting• What’s next in gaming analyticsAbstract© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • About Bill• Serial Entrepreneur in SiliconValley– Platform-as-a-Service for 12years– Analytics for 8 years– Gaming for the past 6 years• Currently head of aconsulting practice (Osolog)and CEO of a stealth modestartup (Scientific Revenue)© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential• http://www.linkedin.com/in/williamgrosso• https://twitter.com/wgrosso
    • © 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Agenda• The Shift in Gaming and the State of GamingAnalytics• Six Use Cases for Predictive Analytics• The Importance of Data and UserSegmentation• Parting Thoughts© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Game DeliveryBefore 2000• Most games sold via“packaged goods” model• Game companies made thegame and handed it off toretail partners.• They were done at that pointToday• Download model• Everything gets “patched” continuously(no such thing as done)• Most games have an incrementalmonetization model– Subscriptions– DLC– Free to play© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Multiplayer GamingBefore 2000• “Compare High Scores” model(from arcades)• Conversation and shared tips(social model)• Multiplayer meant “4 peoplearound a console”Today• Social on steroids– Tweeting– Facebook games– Shared videos• “Always online” games becoming the norm– PaaS model for gaming infrastructure• True multiplayer is still pretty rare– I have 60+ games on my phone– <10 involve simultaneous activity by people not in thesame room© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • AnalyticsBefore 2000• Entirely “historical” in nature– You shipped the game, then youfound out how it did in the widermarket• Mostly financial– Sales reports– Very little telemetry or gameplayanalyticsNow• Growing trend of “record everything, sort itout later”– Though there is some skepticism about thevalue of doing so• Not a lot of QA on the telemetry data• Most reporting is still financial or “top line”metrics– People are very suspicious of having “toomany” metrics© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • http://wgrosso.tumblr.com/post/23556458152/metrics-that-game-developers-use-correlations-and© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • The Way We Manage Games isStarting to Change• If the game is always online and frequentlyupdated, there’s an opportunity to change theway we make decisions• To the extent we can make the game into a“data driven application” which use server-side logic for key decisions, we can managethe individual user experience– Game developer: To make better games– Business side: To make more money© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • The Next Step is A Shift In Analytics• Pre-2000 metrics tell you how you did– Mostly from a financial perspective• Current metrics tell you how you’re doing– ARPDAU doesn’t help you get better• Next step is predicting what will happen if youdon’t do something– And enabling you to take action to avoid badoutcomes– Really the point of “predictive analytics” – if you knowabout bad things early enough, you can take action toprevent them© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Major Companies Starting toAnnounce Predictive InitiativesYou can google and find videos of the talks, and the slides© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Agenda• The Shift in Gaming and the State of GamingAnalytics• Six Use Cases For Predictive Analytics• The Importance of Data and UserSegmentation• Parting Thoughts© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Use Case 1: Noob Churn Prediction• Goal: predict whether a newplayer will return for a second day• Idea: Carefully record playerevents and actions during the firstday. Record whether users comeback. Then use supervisedlearning to build a predictivemodel.• What would you do if you knew:– Cancel bad advertising campaigns– Pro-actively cross-market othergames– Try harder to get a purchase– Show more ads© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Use Case 2: Vet Churn Prediction• Goal: predict whether aveteran player is losing interest• Idea: Track simple time-basedengagement in many differentways using multiplemathematical transformationsand then use supervisedlearning• What would you do if youknew:– Pro-actively cross-marketother games– Try harder to get a purchase(the “gosh I loved this game”nostalgia buy)© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Use Case 3: Survival Analysis• Goal: get a simple, top-level,measure of populationdynamics• Idea:– Installing a game is like gettinga terminal disease– Stopping playing is like dying– Fit ether Cox PH or a Weibulldistribution to the data• What would you do if youknew:– Use the survival curves toperform top-line assessmentof game changes© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Use Case 4: Gameplay Analysis• Goal: figure out if a user is gettingstuck, failing to learn a crucial skill, orpicking up bad habits• Secondary goal: finding cheaters orother sources of gameplay imbalance.• Idea:– Carefully record player events andactions.– Measure increasing-skill / success ofplayers.– Then perform supervised learning to seeif there are early predictors of skill-acquisition or success• What would you do if you knew:– Focused tutorials and interventions– Different pairings during matchingalgorithms– Dynamically adjusting the game© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Use Case 5: Fraud Detection• Goal: Spot people who areperforming a fraudulent action(gold-farming, selling goodsoutside of game, …)• Idea: Apply anti-moneylaundering techniques(unsupervised learning todetect outliers, usually viasupport vector machines)• What would you do if youknew:– Shut down the gold-farmers– Open up gifting and third-party gifting channels© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Use Case 6: Whale Spotting• Goal: Sport high spendingcustomers as soon as possible• Idea:– Carefully record player eventsand actions.– Measure the life-time spendof players.– Then perform supervisedlearning to see if there areearly predictors of significantspend.• What would you do if youknew:– Lots of possibilities here© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • R is the Right Tool• The six use casesmentioned– Supervised learning– Unsupervised learning– Anti-money laundering– Cox-PH / Weibull models– Mathematical datatransformations• R is the single best tool forexploratory data analysisusing a wide-variety ofmachine learningalgorithms© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • <<Poll Question>>
    • Agenda• The Shift in Gaming and the State of GamingAnalytics• Six Use Cases for Predictive Analytics• The Importance of Data and UserSegmentation• Parting Thoughts© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • The Basic Idea• Create a detailed model of the user– A user is a point in 2000 dimensionalspace• Create a set of questions– You would like the answers to thesepredictively– You can measure the answers to thesehistorically• Create a training data set– Historical Users x Measured Outcomes• Sample your data and use standardalgorithms– CRAN is your friend– This is just supervised learning. RandomForests and Neural Nets rule the world.© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Historically Measured Outcomes• Supervised learning means you need to knowthe outcome– You can’t learn much on day 1• Not a bad thing. This lets you– Baseline behavior– See how far “Two standard deviations” stylereasoning can take you– Then mix in machine learning© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • The Literature is Extremely Helpful© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Growing Body of Research Papers• Started with Dmitri Williams in 1990’s• Small body of productive researchers– Dmitri Williams– Anders Drachen– Christian Thurau– Alessandro Canossa– Jeremy Gow© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • There’s a Very Good Book Too© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • And, of Course, Thousands of Papersfrom the Telcos• Literally thousands offairly straightforwardand easy tounderstand papers© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • There’s a Lot of Data to Record• Events and activities.• Outcomes– Win / lose subgame– Made it to dungeon level 4– Completed tutorial– Achieved goal• Model skills acquisition– People generally get better asthey play games– But the idea of “level” is oftenmuch too coarse– Leveling up is frequently not thefull story– User satisfaction is often stronglyrelated to skills acquisition© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Sample, Summarize, Model• You’re going to be storing a ton of data• Player activity streams are verbose– Every chat generates an event– Every time a gun gets fired, an event gets generated.• And data about them (headshot!)• Nobody actually reasons about individual events– Store counts and ratios at the “session” level– Choose representative samples of users to reason about• Generally speaking, we’re talking about >100 fine-grained measurements that you use to build a model© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Data Collection inCloudGameMessBusSummarizationLarge ScaleStorage• Game is fully instrumented• Messages sent to server viamessage bus architecture (inbackground and batched)DataIngestion• Standard “NoSQL” stack (storagewith Hadoop/Pig layered on top)• Streaming event aggregation oftendone by aggregating into in-memory caching (memcached orelasticcache).• Often the large-scale storage issimply json files stored in S3• Large scale storage can run intotens of gigabytes a day. Can besignificant cost over time• What we know andloveExample Architecture
    • Agenda• The Shift in Gaming and the State of GamingAnalytics• Six Use Cases for Predictive Analytics• The Importance of Data and UserSegmentation• Parting Thoughts© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • The World is Non-Stationary• Games are now online serviceswith constant tuning• The people who install a gameare constantly changing– Early adopters  massmarket• Social and environmentalexpectations changeconstantly• You can’t create a model, tuneit, get some answers, andmove on• And you really really reallycan’t do this on the betaversion of your game© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Cross-Game Doesn’t Work Well• Games are very different• There is no generic“across all games” modelthat will work perfectlywithout game-specifictuning• Figure that “data scientistper game” (more forlarger games) plus a teamto build data pipelines iswhat you’re looking at© 2013, Scientific Revenue. All Materials Company Proprietary and Confidential
    • Technology is No Longer the Reasonfor Failure• You might fail– You might not have budget– You might not have data– You might not get executivebuy-in– You might run into “culturalissues” or find out that you’reupsetting someone’sapplecart– But ….© 2013, Scientific Revenue. All Materials Company Proprietary and ConfidentialIf you have budget and data and buy-inYou will be able to deliver results
    • <<Poll Question>>