1
@PaulCutsinger
Russ Carroll
A|B Testing
2
3
4
Surface the Value
Image from Diner Dash by PlayFirst
Source: Amazon Appstore, January 2013
+75%
ARPPU
5
Show Users How to “Consume”
Image from Cut the Rope by ZeptoLab
Source: Amazon Appstore, January 2013
+65%
repeat orders
6
Experiment and react quickly
7 day retention
Time
Active
Players
7
A|B Testing
8
This is Air Patriots
Case Study: Store Optimization
10
• Goal – Easier to access store content
• Platform – Android & iOS
• Test – Make it possible to unlock content on Main ...
11
LOCKED ON MAIN
MENU
UNLOCKABLE >
<
12
STORE OPTIMIZATION
Test Case
Conversion to
Customer
7-Day Retention
Locked on Main Menu
(Control)
– –
Unlockable on Mai...
13
• Goal – Drive players to our other games
• Platform – Android
• Test – Which image is clicked on more
CROSS PROMOTION
CROSS PROMOTION
15
CROSS PROMOTION
16
Air Patriots: Cross Promotion
B
A 1.77%
0.93%
ClickThrough
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. ...
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. ...
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. ...
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. ...
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. ...
22
Retrieve a variation
Objective C
23
Record a view and conversion
Objective C
bit.ly/ABTestXPlat
bit.ly/ABTestiOS
bit.ly/ABTestJava
More details at
Case Study: Junk Race by GameNug
Junk Race Overview
26
Junk Race Overview
27
Tips:
Producers in control: “With two-way services you can request a value from the
server that you control. Two-way se...
28
Case Study: Difficulty Level
29Amazon Services Confidential
GAME DIFFICULTY TEST
7 day retention
Time
Active
Players
30Amazon Services Confidential
GAME DIFFICULTY TEST
• Goal – A Dramatic Increase!
• Platform – Android & iOS
• Test – A/B/...
31Amazon Services Confidential
GAME DIFFICULTY TEST
32Amazon Services Confidential
Test 1:
A
Control
B
2% easier
C
4% easier
D
6% easier
E
8% easier
7-Day Retention – tbd tbd...
33Amazon Services Confidential
GAME DIFFICULTY TEST
• How much would you pay to increase retention by 20%?
• How much time...
34Amazon Services Confidential
A|B Testing
• Monetization – minimize luck and maximize
skill
• Delight your customers
• te...
Upcoming SlideShare
Loading in...5
×

A/B Testing with Air Patriots - GDC 2014

310

Published on

A/B Testing is about using data to challenge assumptions and test new ideas, often leading to increased engagement and monetization. In addition to showing you the latest cross-platform testing services at your disposal, we'll discuss the "happy accident" that the Air Patriots team made. It was unexpected and led to an increase in both retention and monetization.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
310
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A/B Testing with Air Patriots - GDC 2014

  1. 1. 1 @PaulCutsinger Russ Carroll A|B Testing
  2. 2. 2
  3. 3. 3
  4. 4. 4 Surface the Value Image from Diner Dash by PlayFirst Source: Amazon Appstore, January 2013 +75% ARPPU
  5. 5. 5 Show Users How to “Consume” Image from Cut the Rope by ZeptoLab Source: Amazon Appstore, January 2013 +65% repeat orders
  6. 6. 6 Experiment and react quickly 7 day retention Time Active Players
  7. 7. 7 A|B Testing
  8. 8. 8 This is Air Patriots
  9. 9. Case Study: Store Optimization
  10. 10. 10 • Goal – Easier to access store content • Platform – Android & iOS • Test – Make it possible to unlock content on Main Menu STORE OPTIMIZATION
  11. 11. 11 LOCKED ON MAIN MENU UNLOCKABLE > <
  12. 12. 12 STORE OPTIMIZATION Test Case Conversion to Customer 7-Day Retention Locked on Main Menu (Control) – – Unlockable on Main Menu 20.93% 4.17%
  13. 13. 13 • Goal – Drive players to our other games • Platform – Android • Test – Which image is clicked on more CROSS PROMOTION
  14. 14. CROSS PROMOTION
  15. 15. 15 CROSS PROMOTION
  16. 16. 16 Air Patriots: Cross Promotion B A 1.77% 0.93% ClickThrough
  17. 17. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.89% +3.07% 43,120 The Producer’s Perspective Ads on menu or in game
  18. 18. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo Ads The Producer’s Perspective
  19. 19. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  20. 20. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  21. 21. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  22. 22. 22 Retrieve a variation Objective C
  23. 23. 23 Record a view and conversion Objective C bit.ly/ABTestXPlat bit.ly/ABTestiOS bit.ly/ABTestJava More details at
  24. 24. Case Study: Junk Race by GameNug
  25. 25. Junk Race Overview
  26. 26. 26 Junk Race Overview
  27. 27. 27 Tips: Producers in control: “With two-way services you can request a value from the server that you control. Two-way services let you make changes to your live game instantly. Free: Amazon is the only free two-way service that we have found.” Multi-variable: “We don't test variables, we test user experiences.” Watch sample size: “If a couple users can change your conclusion, you need a larger sample” Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of skipping the week long review period ” -John Goforth
  28. 28. 28 Case Study: Difficulty Level
  29. 29. 29Amazon Services Confidential GAME DIFFICULTY TEST 7 day retention Time Active Players
  30. 30. 30Amazon Services Confidential GAME DIFFICULTY TEST • Goal – A Dramatic Increase! • Platform – Android & iOS • Test – A/B/n Test – 5 Difficulty levels! OR?
  31. 31. 31Amazon Services Confidential GAME DIFFICULTY TEST
  32. 32. 32Amazon Services Confidential Test 1: A Control B 2% easier C 4% easier D 6% easier E 8% easier 7-Day Retention – tbd tbd tbd tbd Conversion to customer – -1.71% 16.11% 19.23% 4.84% GAME DIFFICULTY TEST
  33. 33. 33Amazon Services Confidential GAME DIFFICULTY TEST • How much would you pay to increase retention by 20%? • How much time would you invest? Our Take-Aways! • Never launch a game in future w/o AB difficulty test! • Low-hanging fruit! • Opportunities abound!
  34. 34. 34Amazon Services Confidential A|B Testing • Monetization – minimize luck and maximize skill • Delight your customers • test customer experiences • AB testing tools are easy to use • Cross Platform • Deploy changes instantly • Free • Low-hanging fruit is waiting to be harvested Paul Cutsinger Russell Carroll Find me at: • @PaulCutsinger • @AmazonAppDev • developer.amazon.com/blog • bit.ly/ABTestXPlat
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×