A/B testing of ad monetisation based on experience from Outfit7 tests (9 billions+ downloads). Which aspects of ad monentisation are easier to test which ones are more difficult? How to look for hypothesis, ideas to test? Which points you should start A/B testing. Which ones require more resources, which ones require less. What kind of improvements can we expect?
4. Analytics
infrastructure
● BigQuery
● Python
● R
Introduction
AB testing
What do you need to do?
Randomly put users into two
or more groups
Have enough
users
BE
User level data
Configuration
Collect user level
data
Change their
settings from BE
Analyse data
Analyse a lot of
data
Understand
scientific
methodology
5. Introduction
Ad monetisation
What it is and why do you care?
● Rewarded videos, banners, interstitials
● Generating opportunity to show ads
● Ad mediation of partners
● Big revenue source for specific type of games
6. Introduction
What are we optimizing
Ad mediation
AD TECH SIDE
● Showing the most valuable ad
● Improving data
● Creating algorithms
Generating opportunity
PRODUCT SIDE
● right position and frequency of
banners
● right timing and frequency for
interstitials
● right motivation of users for
rewarded videos
8. Generating opportunity
AB testing
● Number of generated
opportunities
● Requests per user
● Impressions per user
● Percent users with
impressions
● Retention
● Session duration
● Overall monetisation
Focus
“Easy” because user level data is available
Big impact
Good statistical power for most metrics
10. Ad Mediation
AB testing
Hard because user level data is NOT available
Big impact when you start
Testing revenue is tricky
● Waterfall algorithms
● Data processing
algorithms
● ARPDAU
Focus
11. Problems
Ad Mediation
No user level data means you depend on reports from partners
Not real AB tests but testing revenue on segments
Time series analysis
Find pragmatic solutions to move quickly
12. Data processing algorithms
Ad Mediation
Use simple logic to understand and solve problems Simulate Implement and test
Data is messy and comes in late.
What do we base our algorithm on?
Predicting current ECPMs
Irregularities on the partner’s side
make your data messy and volatile.
Algorithms should not overreact.
Volatility algorithm
13. Ad mediation algorithms
Ad Mediation
Waterfall algorithm, calculating the
expected value
In which order should we ask partners
for ads?
Impressions
Revenue
Requests
Country
Platform
...
Algorithm Waterfall
15. Where to start?
Get yourself some data fast
Test opportunity first
Be pragmatic
16. Be careful!
All of this is closely connected to a whole set of metrics
Measure the big picture
Assumptions are not facts, even if you really really believe in them
If you cannot explain it to a kid, you do not understand it
17. What to expect?
Without analytics and AB tests you are not aware of the potential
Without AB tests you can do harm and not even know it
If there is one aspect where you can do huge impact it is
monetisation