Adding Knobs to your App – Modifying &
Measuring Behavior in the Wild (without
Republishing)
Cheng	
  Luo,	
  Amazon	
  Ap...
Agenda
• 
• 
• 
• 

Why Amazon Mobile SDK
How to use A|B testing 
Add your A|B testing in 10 minutes
Demo

#mceconf

#mce-...
Active Users
Monthly Unique
Visitors (US):

Active Global Customer
Accounts:

200M

1

Age Composition of Amazon
Unique Vi...
Simplicity
•  Amazon	
  Appstore	
  runs	
  Android	
  apps	
  

•  75%	
  of	
  Android	
  apps	
  work	
  without	
  any...
The latest Kindle Fire HDX

Kindle	
  Fire	
  HDX	
  7"	
  Tablet	
  $229,	
  with	
  24x7	
  May	
  Day	
  support	
  	
 ...
What A|B Testing can do for you

Precisely	
  control	
  
who	
  is	
  affected	
  

#mceconf

#mce-addingknobs

Safely	
  ...
Case Study - Air Patriots
Used A/B Testing to decide:
•  Include in-game advertising
•  Provide purchasing short-cuts
•  N...
Case 1 – Image

#mceconf

#mce-addingknobs

11 Jan 2014
Case 2 – Ads placing 

#mceconf

#mce-addingknobs

11 Jan 2014
Case 3 - Notification
Notification appears 
•  x1 day
•  x 3 days → Sweet spot
•  x 7 days

#mceconf

#mce-addingknobs

11 J...
Case 4 - User Retention 

Level:	
  1	
  
Difficulty:	
  Easy	
  
#mceconf

#mce-addingknobs

Level:	
  5	
  
Difficulty:	
  M...
Air Patriots Sample Results

#mceconf

#mce-addingknobs

11 Jan 2014
Add A/B Testing to Your App

#mceconf

#mce-addingknobs

11 Jan 2014
Add A/B Testing to Your App

#mceconf

#mce-addingknobs

11 Jan 2014
Add A/B Testing to Your App

#mceconf

#mce-addingknobs

11 Jan 2014
Initialize A/B Testing in Your App
private static final String APP_KEY = "876c809ba7084b9fbe2c08a22e5acda1";
private stati...
Create an A/B Testing Project

#mceconf

#mce-addingknobs

11 Jan 2014
Create an A/B Testing Project

#mceconf

#mce-addingknobs

11 Jan 2014
Create an A/B Testing Project

#mceconf

#mce-addingknobs

11 Jan 2014
Add a New Launch

#mceconf

#mce-addingknobs

11 Jan 2014
Add a New Launch

#mceconf

#mce-addingknobs

11 Jan 2014
Add a New Launch

#mceconf

#mce-addingknobs

11 Jan 2014
Multiple Projects Are Allowed

private static final String AB_PRJ_ACC = "Acceleration";
private static final String AB_VAR...
Load Variation Variables
private static final int DEF_MAXACC = 5;
private static final int DEF_MAXVEL = 50;
private int ma...
Changing Values on the Fly

#mceconf

#mce-addingknobs

11 Jan 2014
Changing Values on the Fly

#mceconf

#mce-addingknobs

11 Jan 2014
Changing Values on the Fly

#mceconf

#mce-addingknobs

11 Jan 2014
Changing Values on the Fly

#mceconf

#mce-addingknobs

11 Jan 2014
Add a Segment

#mceconf

#mce-addingknobs

11 Jan 2014
Add a Segment

#mceconf

#mce-addingknobs

11 Jan 2014
Add a Segment

#mceconf

#mce-addingknobs

11 Jan 2014
Use the Segment 
Use	
  App	
  Segngs	
  to	
  idenTfy	
  your	
  user	
  segments	
  
        //get user age from the app...
Demo

#mceconf

#mce-addingknobs

11 Jan 2014
Wrap up 
•  A/B Testing
–  Free cross platform services 
–  Precise measuring behaviours using custom
segments
–  Modifyin...
Thank you

@AmazonAppDev

#mceconf

#mce-addingknobs

11 Jan 2014
Upcoming SlideShare
Loading in …5
×

Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

1,085 views

Published on

Learning about Amazon free cross platform service - AB Testing. And adding knobs to your apps, and fine tune the user experiences and get the max results out of UI, monetization and etc.

Published in: Technology, Business
2 Comments
6 Likes
Statistics
Notes
No Downloads
Views
Total views
1,085
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
14
Comments
2
Likes
6
Embeds 0
No embeds

No notes for slide

Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

  1. 1. Adding Knobs to your App – Modifying & Measuring Behavior in the Wild (without Republishing) Cheng  Luo,  Amazon  Appstore   11th  January,  2014     @chengluo   #mceconf #mce-addingknobs All  rights  reserved.  May  not  be  copied,  modified,  or  distributed  in  whole  or  in  part  without  the  express  consent  of  Amazon.com,  Inc.   ©  2013  Amazon.com,  Inc.  and  its  affiliates.   11 Jan 2014
  2. 2. Agenda •  •  •  •  Why Amazon Mobile SDK How to use A|B testing Add your A|B testing in 10 minutes Demo #mceconf #mce-addingknobs 11 Jan 2014
  3. 3. Active Users Monthly Unique Visitors (US): Active Global Customer Accounts: 200M 1 Age Composition of Amazon Unique Visitors2 97M Avg. Daily Unique Visitors (US): 2 Household Income of Amazon Unique Visitors2 18-­‐34   % of Amazon Shoppers Who have Made an Online Purchase2 38%   32%   45%   30%   18%   0-­‐18   10%   35-­‐44   6.6 2 52%   8%   42%   12.9M 2 Avg. Visits per Visitor per Month (US): 45+   <$25K   $25K-­‐$75K   $75K  +   <$50   25%   $50-­‐$200   Amount Spent $200+   Note: Active Customer Accounts are unique email addresses that have placed an order in the last 12 months Sources: 1. Amazon Investor Relations; 2. ComScore PlanMetrix January, 2013. #mceconf #mce-addingknobs 11 Jan 2014
  4. 4. Simplicity •  Amazon  Appstore  runs  Android  apps   •  75%  of  Android  apps  work  without  any  need  for  customisaTon   •  Simple  APIs  to  publish  free-­‐to-­‐user,  freemium  and  premium  apps   Device  Messaging  API In-­‐App  Purchasing  API #mceconf #mce-addingknobs A/B  TesTng  Service  API Maps  API GameCircle  API 11 Jan 2014
  5. 5. The latest Kindle Fire HDX Kindle  Fire  HDX  7"  Tablet  $229,  with  24x7  May  Day  support     #mceconf #mce-addingknobs 11 Jan 2014
  6. 6. What A|B Testing can do for you Precisely  control   who  is  affected   #mceconf #mce-addingknobs Safely  test   back-­‐end  load   Change  in  the  wild;   No  new  APK  needed   11 Jan 2014
  7. 7. Case Study - Air Patriots Used A/B Testing to decide: •  Include in-game advertising •  Provide purchasing short-cuts •  Notification frequency •  Level difficulty •  Promotional imagery #mceconf #mce-addingknobs 11 Jan 2014
  8. 8. Case 1 – Image #mceconf #mce-addingknobs 11 Jan 2014
  9. 9. Case 2 – Ads placing #mceconf #mce-addingknobs 11 Jan 2014
  10. 10. Case 3 - Notification Notification appears •  x1 day •  x 3 days → Sweet spot •  x 7 days #mceconf #mce-addingknobs 11 Jan 2014
  11. 11. Case 4 - User Retention Level:  1   Difficulty:  Easy   #mceconf #mce-addingknobs Level:  5   Difficulty:  Medium   11 Jan 2014
  12. 12. Air Patriots Sample Results #mceconf #mce-addingknobs 11 Jan 2014
  13. 13. Add A/B Testing to Your App #mceconf #mce-addingknobs 11 Jan 2014
  14. 14. Add A/B Testing to Your App #mceconf #mce-addingknobs 11 Jan 2014
  15. 15. Add A/B Testing to Your App #mceconf #mce-addingknobs 11 Jan 2014
  16. 16. Initialize A/B Testing in Your App private static final String APP_KEY = "876c809ba7084b9fbe2c08a22e5acda1"; private static final String PRIVATE_KEY = "9nxxk4//qL89ymXl/P5aL7w8uhs3DVDrLcyYViG6yn0="; AmazonInsights insightsInstance; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); . . . InsightsCredentials credentials = AmazonInsights.newCredentials(APP_KEY, PRIVATE_KEY); insightsInstance = AmazonInsights.newInstance(credentials, getApplicationContext()); } #mceconf #mce-addingknobs 11 Jan 2014
  17. 17. Create an A/B Testing Project #mceconf #mce-addingknobs 11 Jan 2014
  18. 18. Create an A/B Testing Project #mceconf #mce-addingknobs 11 Jan 2014
  19. 19. Create an A/B Testing Project #mceconf #mce-addingknobs 11 Jan 2014
  20. 20. Add a New Launch #mceconf #mce-addingknobs 11 Jan 2014
  21. 21. Add a New Launch #mceconf #mce-addingknobs 11 Jan 2014
  22. 22. Add a New Launch #mceconf #mce-addingknobs 11 Jan 2014
  23. 23. Multiple Projects Are Allowed private static final String AB_PRJ_ACC = "Acceleration"; private static final String AB_VAR_MAXACC = "maxAcceleration"; //Project name //Variable name private static final String AB_PRJ_VEL = "Velocity"; private static final String AB_VAR_MAXVEL = "maxVelocity"; //Project name //variable name #mceconf #mce-addingknobs 11 Jan 2014
  24. 24. Load Variation Variables private static final int DEF_MAXACC = 5; private static final int DEF_MAXVEL = 50; private int maxAcc; private int maxVel; private void loadTreatment() { ABTestClient abClient = insightsInstance.getABTestClient(); abClient.getVariations(AB_PRJ_VEL, AB_PRJ_ACC).setCallback(new InsightsCallback<VariationSet>() { @Override public void onComplete(VariationSet variations) { Variation variation = variations.getVariation(AB_PRJ_ACC); maxAcc = variation.getVariableAsInt(AB_VAR_MAXACC, DEF_MAXACC); variation = variations.getVariation(AB_PRJ_VEL); maxVel = variation.getVariableAsInt(AB_VAR_MAXVEL, DEF_MAXVEL); } }); } #mceconf #mce-addingknobs 11 Jan 2014
  25. 25. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  26. 26. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  27. 27. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  28. 28. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  29. 29. Add a Segment #mceconf #mce-addingknobs 11 Jan 2014
  30. 30. Add a Segment #mceconf #mce-addingknobs 11 Jan 2014
  31. 31. Add a Segment #mceconf #mce-addingknobs 11 Jan 2014
  32. 32. Use the Segment Use  App  Segngs  to  idenTfy  your  user  segments           //get user age from the app settings         SharedPreferences myPrefs;         myPrefs = getSharedPreferences("com.example.testingapp", Context.MODE_PRIVATE);         int userAge = myPrefs.getInt("com.example.testingapp.USER_AGE", ""); //Initial the segments here UserProfile userProfile = insightsInstance.getUserProfile(); userProfile.addDimensionAsNumber("age", userAge); #mceconf #mce-addingknobs 11 Jan 2014
  33. 33. Demo #mceconf #mce-addingknobs 11 Jan 2014
  34. 34. Wrap up •  A/B Testing –  Free cross platform services –  Precise measuring behaviours using custom segments –  Modifying app behaviors without republishing #mceconf #mce-addingknobs 11 Jan 2014
  35. 35. Thank you
 @AmazonAppDev #mceconf #mce-addingknobs 11 Jan 2014

×