In order of complexityBasic AnalyticsA/B testingA Priori SegmentationAndClustering segmentation I’ll venture to guess that most of you are using analytics (whether developed in-house or purchased as a service) to gauge how well you game is doingAnd many of you are you doing A/B testing to select which element options to select as defaultsAnd some of you are doing A Priori segmentation and serving different options to different pre-defined segmentsAnd few, if any, are doing Clustering segmentation to determine which are the relevant segments for which option. But I trust all of you believe that much more can be achieved by optimization if it was only easy and effective. [switch]
So, let’s beginYou have a game, and naturally you have many of the “standard” game elements like Payment page Promotions Tutorial To name but a few elements….(and please note that we’ll interchangeably call game events= elements and vice versa, and each such element may have multiple options or variants)
To better understand the concept of optimization, let’s take a look at a typical payment page:There are several itemson this page that can have more than one option:For example the payment range: here we a low range and a high rangeAnd the question is which one of these (when set as default) will produce more revenue? And the answer is, we can say with some certainty that the lower range option will produce more transactions (but with a lower avg. transaction), and the higher range option will produce fewer transactions but with a higher avg. But we don’t know which option will produce more revenue till we test, right?So Hold on to that thought, as we’ll soon expand on the right answer.Note also other possible options: like the bonus (here shown with % bonus, but you can obviously have an absolute number instead ), even the default selection can vary, not to speak of the art, the wording, the call to action button, etc.
HereI’m showing 3 options of a virtual good Promotion pageWhich one will perform better ?
And here is game flow element : longer or shorter or no tutorialWhich of the options will produce a lower abandon rate?
Sofirst let’s look at analytics as an optimization tool:This is a manual method in which you make a selection of a specific option Measure, display and analyze the performance resultsMake changes and go for another loop.Pretty basic stuff but can work.
So,skip that and get a tad more sophisticated and invoke a commonly used method called A/B testing, also know as split testing, or even A/B/C/D/E testing, and feel free to attach the rest of the alphabet as you like.And we’re Back to the payment page.So Prepare two options (again we’ll use , low and high ranges), split the traffic, measure the results, compare them, select the better one, and deploy as the defaultSimple, right?
Make sure to listen to good advise, like here from Quora:A very important one is to Make sure the test runs long enough with enough trafficAnd repeat test often, as your user base may change over time
Bottom line:A/B testing is conceptuallysimple, relatively easy to deploy and can often achieve good resultsThe downside is that at the end, you are still serving one-size-fits-all option, and as we already mentioned you need to repeat the tests often, as user preferences change over time.Please note that sometimes A/B testing produces no clear winner when an equal number of users prefer option A as option B
So, lets get even more sophisticated and start serving more than one option. and now segment our user base using criteria based on our intuition, gut-feeling, experience, - you name it.And then serve each segment a different optionTypical A Priori segmentation criteria are: Geographic – countries, regionsDemographic - age, gender, educationPsychographic - lifestyle, personality, valuesSession – time of day, day of weekNow Assuming that your segmentation had merit, you can expect some lift over A/B testing.As you can see in the added green area for every option (point), and your total lift will be a weighted average of these results according to the traffic split.
Bottom line:Can be effective and produce lift especially if your segmentation had meaning vis-a-vis the optionsThe downside:Hard to fine-tune - refining segmentation for one element can break the rule base for other elementsIn terms of level of efforts, I’d term this medium to heavy
So let’s get a whole lot more sophisticated and do segmentation differently:Instead of selecting the segments intuitively, let’s let the machine select the segments via regression analysisSo, Let’s first define the options, properly run the A/B tests, run regression analysis on the results, and extract the correlations between user attributes and options. Of course, you need to repeat the process for every option for each element.Then, when a user comes in to your game, quickly determine which attribute bucket they belong to, and serve them the appropriate option. The results can be expected to gain additional lift to the A-Priori segmentation method as you can see in the lighter green shade.
Bottom line on clustering segmentation:The highest lift, new insightsDownside:Tedious and laboriousEnormous amount of data – (this is where you start meeting the concept of big data)The process is sophisticated and possibly endless (continuous) as you need to run the many tests, collect the data, analyze it, run the statistical enginesOnce you’ve found the right segments, you need to go back to the game and make the needed modifications. Then test it again to make sure it works and repeat on a regular basis.So in terms of effort, you are about to fly off the scale.
But what if you can have the best of both worlds:the highest lift possible but without the huge pain that comes along with it?This is where I’d like to introduce you to Predictive best-fit Which is an advanced implementation of clustering segmentation
And that brings us to the end of our discussion.If you have any question Yaron and I will gladly answer any questions you haveThank you.
Transcript of "Optimization Technologies for Lifting KPIs in Games"
Optimization Technologies for Lifting KPIs in Games Alan Avidan, PhD, MBA Exec. Director & Chief BeesDev