A/B Testing with Air Patriots - GDC 2014
 

A/B Testing with Air Patriots - GDC 2014

on

  • 394 views

A/B Testing is about using data to challenge assumptions and test new ideas, often leading to increased engagement and monetization. In addition to showing you the latest cross-platform testing ...

A/B Testing is about using data to challenge assumptions and test new ideas, often leading to increased engagement and monetization. In addition to showing you the latest cross-platform testing services at your disposal, we'll discuss the "happy accident" that the Air Patriots team made. It was unexpected and led to an increase in both retention and monetization.

Statistics

Views

Total Views
394
Views on SlideShare
394
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

A/B Testing with Air Patriots - GDC 2014 A/B Testing with Air Patriots - GDC 2014 Presentation Transcript

  • 1 @PaulCutsinger Russ Carroll A|B Testing
  • 2
  • 3
  • 4 Surface the Value Image from Diner Dash by PlayFirst Source: Amazon Appstore, January 2013 +75% ARPPU
  • 5 Show Users How to “Consume” Image from Cut the Rope by ZeptoLab Source: Amazon Appstore, January 2013 +65% repeat orders
  • 6 Experiment and react quickly 7 day retention Time Active Players
  • 7 A|B Testing
  • 8 This is Air Patriots
  • Case Study: Store Optimization
  • 10 • Goal – Easier to access store content • Platform – Android & iOS • Test – Make it possible to unlock content on Main Menu STORE OPTIMIZATION
  • 11 LOCKED ON MAIN MENU UNLOCKABLE > <
  • 12 STORE OPTIMIZATION Test Case Conversion to Customer 7-Day Retention Locked on Main Menu (Control) – – Unlockable on Main Menu 20.93% 4.17%
  • 13 • Goal – Drive players to our other games • Platform – Android • Test – Which image is clicked on more CROSS PROMOTION
  • CROSS PROMOTION
  • 15 CROSS PROMOTION
  • 16 Air Patriots: Cross Promotion B A 1.77% 0.93% ClickThrough
  • 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.89% +3.07% 43,120 The Producer’s Perspective Ads on menu or in game
  • 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo Ads The Producer’s Perspective
  • 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  • 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  • 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  • 22 Retrieve a variation Objective C
  • 23 Record a view and conversion Objective C bit.ly/ABTestXPlat bit.ly/ABTestiOS bit.ly/ABTestJava More details at
  • Case Study: Junk Race by GameNug
  • Junk Race Overview
  • 26 Junk Race Overview
  • 27 Tips: Producers in control: “With two-way services you can request a value from the server that you control. Two-way services let you make changes to your live game instantly. Free: Amazon is the only free two-way service that we have found.” Multi-variable: “We don't test variables, we test user experiences.” Watch sample size: “If a couple users can change your conclusion, you need a larger sample” Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of skipping the week long review period ” -John Goforth
  • 28 Case Study: Difficulty Level
  • 29Amazon Services Confidential GAME DIFFICULTY TEST 7 day retention Time Active Players
  • 30Amazon Services Confidential GAME DIFFICULTY TEST • Goal – A Dramatic Increase! • Platform – Android & iOS • Test – A/B/n Test – 5 Difficulty levels! OR?
  • 31Amazon Services Confidential GAME DIFFICULTY TEST
  • 32Amazon Services Confidential Test 1: A Control B 2% easier C 4% easier D 6% easier E 8% easier 7-Day Retention – tbd tbd tbd tbd Conversion to customer – -1.71% 16.11% 19.23% 4.84% GAME DIFFICULTY TEST
  • 33Amazon Services Confidential GAME DIFFICULTY TEST • How much would you pay to increase retention by 20%? • How much time would you invest? Our Take-Aways! • Never launch a game in future w/o AB difficulty test! • Low-hanging fruit! • Opportunities abound!
  • 34Amazon Services Confidential A|B Testing • Monetization – minimize luck and maximize skill • Delight your customers • test customer experiences • AB testing tools are easy to use • Cross Platform • Deploy changes instantly • Free • Low-hanging fruit is waiting to be harvested Paul Cutsinger Russell Carroll Find me at: • @PaulCutsinger • @AmazonAppDev • developer.amazon.com/blog • bit.ly/ABTestXPlat