Your SlideShare is downloading. ×
Optimization Technologies for Lifting KPIs in Games
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Optimization Technologies for Lifting KPIs in Games


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • In order of complexityBasic AnalyticsA/B testingA Priori SegmentationAndClustering segmentation I’ll venture to guess that most of you are using analytics (whether developed in-house or purchased as a service) to gauge how well you game is doingAnd many of you are you doing A/B testing to select which element options to select as defaultsAnd some of you are doing A Priori segmentation and serving different options to different pre-defined segmentsAnd few, if any, are doing Clustering segmentation to determine which are the relevant segments for which option. But I trust all of you believe that much more can be achieved by optimization if it was only easy and effective. [switch]
  • So, let’s beginYou have a game, and naturally you have many of the “standard” game elements like Payment page Promotions Tutorial To name but a few elements….(and please note that we’ll interchangeably call game events= elements and vice versa, and each such element may have multiple options or variants)
  • To better understand the concept of optimization, let’s take a look at a typical payment page:There are several itemson this page that can have more than one option:For example the payment range: here we a low range and a high rangeAnd the question is which one of these (when set as default) will produce more revenue? And the answer is, we can say with some certainty that the lower range option will produce more transactions (but with a lower avg. transaction), and the higher range option will produce fewer transactions but with a higher avg. But we don’t know which option will produce more revenue till we test, right?So Hold on to that thought, as we’ll soon expand on the right answer.Note also other possible options: like the bonus (here shown with % bonus, but you can obviously have an absolute number instead ), even the default selection can vary, not to speak of the art, the wording, the call to action button, etc.
  • HereI’m showing 3 options of a virtual good Promotion pageWhich one will perform better ?
  • And here is game flow element : longer or shorter or no tutorialWhich of the options will produce a lower abandon rate?
  • Sofirst let’s look at analytics as an optimization tool:This is a manual method in which you make a selection of a specific option Measure, display and analyze the performance resultsMake changes and go for another loop.Pretty basic stuff but can work.
  • So,skip that and get a tad more sophisticated and invoke a commonly used method called A/B testing, also know as split testing, or even A/B/C/D/E testing, and feel free to attach the rest of the alphabet as you like.And we’re Back to the payment page.So Prepare two options (again we’ll use , low and high ranges), split the traffic, measure the results, compare them, select the better one, and deploy as the defaultSimple, right?
  • Make sure to listen to good advise, like here from Quora:A very important one is to Make sure the test runs long enough with enough trafficAnd repeat test often, as your user base may change over time
  • Bottom line:A/B testing is conceptuallysimple, relatively easy to deploy and can often achieve good resultsThe downside is that at the end, you are still serving one-size-fits-all option, and as we already mentioned you need to repeat the tests often, as user preferences change over time.Please note that sometimes A/B testing produces no clear winner when an equal number of users prefer option A as option B
  • So, lets get even more sophisticated and start serving more than one option. and now segment our user base using criteria based on our intuition, gut-feeling, experience, - you name it.And then serve each segment a different optionTypical A Priori segmentation criteria are: Geographic – countries, regionsDemographic - age, gender, educationPsychographic - lifestyle, personality, valuesSession – time of day, day of weekNow Assuming that your segmentation had merit, you can expect some lift over A/B testing.As you can see in the added green area for every option (point), and your total lift will be a weighted average of these results according to the traffic split.
  • Bottom line:Can be effective and produce lift especially if your segmentation had meaning vis-a-vis the optionsThe downside:Hard to fine-tune - refining segmentation for one element can break the rule base for other elementsIn terms of level of efforts, I’d term this medium to heavy
  • So let’s get a whole lot more sophisticated and do segmentation differently:Instead of selecting the segments intuitively, let’s let the machine select the segments via regression analysisSo, Let’s first define the options, properly run the A/B tests, run regression analysis on the results, and extract the correlations between user attributes and options. Of course, you need to repeat the process for every option for each element.Then, when a user comes in to your game, quickly determine which attribute bucket they belong to, and serve them the appropriate option. The results can be expected to gain additional lift to the A-Priori segmentation method as you can see in the lighter green shade.
  • Bottom line on clustering segmentation:The highest lift, new insightsDownside:Tedious and laboriousEnormous amount of data – (this is where you start meeting the concept of big data)The process is sophisticated and possibly endless (continuous) as you need to run the many tests, collect the data, analyze it, run the statistical enginesOnce you’ve found the right segments, you need to go back to the game and make the needed modifications. Then test it again to make sure it works and repeat on a regular basis.So in terms of effort, you are about to fly off the scale.
  • But what if you can have the best of both worlds:the highest lift possible but without the huge pain that comes along with it?This is where I’d like to introduce you to Predictive best-fit Which is an advanced implementation of clustering segmentation
  • And that brings us to the end of our discussion.If you have any question Yaron and I will gladly answer any questions you haveThank you.
  • Transcript

    • 1. Optimization Technologies for Lifting KPIs in Games Alan Avidan, PhD, MBA Exec. Director & Chief BeesDev
    • 2. Optimization Technologies We’ll Cover • Analytics • A/B Testing • A Priori Segmentation • Clustering Segmentation 2012© Bees and Pollen
    • 3. Game Elements/EventsPayment Page Promotions Tutorial Landing page Full tutorial Level 2 2012© Bees and Pollen 3
    • 4. Payment PageLow Range High RangeWhich one produces more revenue? 2012© Bees and Pollen 4
    • 5. Promotions Percent Discount Absolute DiscountWhich one performs better? 2012© Bees and Pollen 5
    • 6. Tutorial/Game FlowOption 1 Option 2 Option 3Landing page Landing page Landing pageFull tutorial Short tutorial No tutorial Level 2 Level 2 Level 1 2012© Bees and Pollen 6
    • 7. More Elements Can be Optimized• Payment Page• Game flow/tutorial• Promotions and offers• Sharing Messages• Virtual Goods Selections• Payouts (Casino Games)• Themes/Arts/Creatives• Landing Pages• Emails (Subject, Body, Submission)• Special Deals Closing Time 2012© Bees and Pollen
    • 8. Analytics DecideChange Measure Analyze Display 2012© Bees and Pollen
    • 9. A/B TestingDefine Split traffic Measure results Deploy winner Resultoptions high rangeLow rangehigh range 2012© Bees and Pollen
    • 10. Experts Weigh In: A/B Testing Q: What are the most unexpected things people have learned from A/B tests? Make sure that the test is statistically significant - run it for long enough, and with enough traffic to make it count I have learned how dramatically, and ridiculously wrong my most basic assumptions were Its empirically proven that you should let the data tell you what works or not and you should constantly be testing That the devil is in the detail - a minor change can generate a significant result 2012© Bees and Pollen
    • 11. A/B Testing – Bottom line Upside• Simple; understandable Can achieve good results Downside:• One-size-fits-all• Results deteriorate over time 2012© Bees and Pollen
    • 12. A Priori SegmentationDefine segments Define Options and rule Result base Low range high range 2012© Bees and Pollen
    • 13. A Priori Segmentation Upside• Can be effective if segmentation was meaningful Downside• Segments are predefined and remain unchanged during the analysis• Different elements might require different segments• Hard to scale in terms of data-set and number of elements• Hard to fine-tune 2012© Bees and Pollen
    • 14. Clustering Segmentation Define A/B test Segment users Deploy winner options options based on resultLow rangeHigh range 2012© Bees and Pollen
    • 15. Clustering Segmentation Upside:• Highest Lift• Discover correlations never knew existed Downside:• Requires storage of terabytes of data• Need really smart people• Effort = High 2012© Bees and Pollen
    • 16. Clustering Segmentation Predictive Best-FitReal-time and automated predictive algorithmictechnology that serves each user the page optionsthey are most likely to convert on 2012© Bees and Pollen
    • 17. Predictive Best-Fit Advanced algorithms find correlationsbetween user data DNA and conversions User Social and User DNA Predictive Best-Fit User Behavioral Data Generation Algorithms 2012© Bees and Pollen
    • 18. Thank You! 2012© Bees and Pollen
    • 19. Backup slides 2012© Bees and Pollen 19
    • 20. Attribute Sources Facebook attributes: Friends, Likes, Interests, Posts, EventsBehavioral attributes: level, spending, score, progress, custom Session attributes: time of day, day, durationGeo-Demographic attributes: age, gender, education, country Proprietary attributes: novice, high-bidder, risk-averse 3rd Party attributes: income level, education 2012© Bees and Pollen
    • 21. Predictive Best-Fit All the gain without the pain• Automated end-to-end solution• Machine self-learning• Real-time• No game history required• Numerous data sources• Dashboard – Easy to swap-in options• Deep new insights (identify discriminators) 2012© Bees and Pollen