This document discusses A/B testing and how it can be used to optimize user experience and monetization for games and other digital products. It defines A/B testing as a randomized experiment with two variants, a control and treatment. It explains that almost anything can be A/B tested, from colors and layouts to features, content, and pricing. Both step-by-step and multi-variant testing are covered, along with challenges like requiring large data sets and determining which metrics to focus on. The document provides examples of how A/B testing can be used to test levels, bonuses, content, peer-to-peer assignments, and pricing to improve engagement, retention, and monetization.

1. GIAF 3 OCTOBER 2013
A/B TESTS
andy.toben@productmadness.com

2. What is an A/B test ?
“A/B
testing
is
a
methodology in advertising of
using
randomized
experiments with two variants,
A and B, which are the control
and treatment in the controlled
experiment.”
- wikipedia.org

3. What Can We A/B Test?
Basically anything:
• Page Colour
• Layout
• Call to action
• Images
• If you can change it, you can test it!

4. Step by Step Optimization
Testing one variable at the time works, but
can create issues:
- Time consuming:
Needs a lot of traffic or data
- Local verses Global Optimum:
Can lead to accepting the best variant for
that test, but it is not optimised against all
possible variants

5. Multi Variant Testing
This allows you to test all combinations at once however:
- Requires massive data sets – much more than Step by Step
- Requires mathematical tools
- No intuitive / building insights

6. How do we know who’s winning ?

7. How do we know who’s winning ?

8. Advertising is a game, so how about games ?

9. Strategies
What can we use A/B testing to discover?
•
•
•
•
Search - Option A / Option B
Optimize - Option A / Option A+ / Option AChange management - treatment verses control
Measure - No treatment verses treatment

10. A/B Testing User Experience
When A/B testing user experience you can
test out the following and create a big
impact… for better or worse!
- Order of game levels / features
- Bonus system
- Available content
- P2P assignment
- Pricing and Economy

11. Challenges We Face
Some A/B tests are very expensive
- You may need to commit development or art resource for
something that may not work.
Variant assignment needs to persistent
- In some cases you can’t just change a player’s track to the
“winning” branch
What to measure:
• Responsiveness (CTR)
• Engagement
• Retention
• Monetization

12. Branching user experience

13. Understanding Results
Which metric is the best one to look at, and which result should we take
action on?
Feature cost
1st day retention
7th day retention User value
10,000 coins
37%
10%
10 USD
25,000 coins
36%
6%
12 USD

14. But after all, It’s just a tool
Is there an
active
Experiment?
No
No
Change
No
Yes
User
Should we test
the User?
Allocate Variant
Yes

15. Questions ?
andy.toben@productmadness.com