Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

1,024 views

Published on

Learning about Amazon free cross platform service - AB Testing. And adding knobs to your apps, and fine tune the user experiences and get the max results out of UI, monetization and etc.

Published in: Technology, Business
2 Comments
6 Likes
Statistics
Notes
No Downloads
Views
Total views
1,024
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
14
Comments
2
Likes
6
Embeds 0
No embeds

No notes for slide

Amazon AB Testing - Modifying & Measuring App Behaviors Without Republishing

  1. 1. Adding Knobs to your App – Modifying & Measuring Behavior in the Wild (without Republishing) Cheng  Luo,  Amazon  Appstore   11th  January,  2014     @chengluo   #mceconf #mce-addingknobs All  rights  reserved.  May  not  be  copied,  modified,  or  distributed  in  whole  or  in  part  without  the  express  consent  of  Amazon.com,  Inc.   ©  2013  Amazon.com,  Inc.  and  its  affiliates.   11 Jan 2014
  2. 2. Agenda •  •  •  •  Why Amazon Mobile SDK How to use A|B testing Add your A|B testing in 10 minutes Demo #mceconf #mce-addingknobs 11 Jan 2014
  3. 3. Active Users Monthly Unique Visitors (US): Active Global Customer Accounts: 200M 1 Age Composition of Amazon Unique Visitors2 97M Avg. Daily Unique Visitors (US): 2 Household Income of Amazon Unique Visitors2 18-­‐34   % of Amazon Shoppers Who have Made an Online Purchase2 38%   32%   45%   30%   18%   0-­‐18   10%   35-­‐44   6.6 2 52%   8%   42%   12.9M 2 Avg. Visits per Visitor per Month (US): 45+   <$25K   $25K-­‐$75K   $75K  +   <$50   25%   $50-­‐$200   Amount Spent $200+   Note: Active Customer Accounts are unique email addresses that have placed an order in the last 12 months Sources: 1. Amazon Investor Relations; 2. ComScore PlanMetrix January, 2013. #mceconf #mce-addingknobs 11 Jan 2014
  4. 4. Simplicity •  Amazon  Appstore  runs  Android  apps   •  75%  of  Android  apps  work  without  any  need  for  customisaTon   •  Simple  APIs  to  publish  free-­‐to-­‐user,  freemium  and  premium  apps   Device  Messaging  API In-­‐App  Purchasing  API #mceconf #mce-addingknobs A/B  TesTng  Service  API Maps  API GameCircle  API 11 Jan 2014
  5. 5. The latest Kindle Fire HDX Kindle  Fire  HDX  7"  Tablet  $229,  with  24x7  May  Day  support     #mceconf #mce-addingknobs 11 Jan 2014
  6. 6. What A|B Testing can do for you Precisely  control   who  is  affected   #mceconf #mce-addingknobs Safely  test   back-­‐end  load   Change  in  the  wild;   No  new  APK  needed   11 Jan 2014
  7. 7. Case Study - Air Patriots Used A/B Testing to decide: •  Include in-game advertising •  Provide purchasing short-cuts •  Notification frequency •  Level difficulty •  Promotional imagery #mceconf #mce-addingknobs 11 Jan 2014
  8. 8. Case 1 – Image #mceconf #mce-addingknobs 11 Jan 2014
  9. 9. Case 2 – Ads placing #mceconf #mce-addingknobs 11 Jan 2014
  10. 10. Case 3 - Notification Notification appears •  x1 day •  x 3 days → Sweet spot •  x 7 days #mceconf #mce-addingknobs 11 Jan 2014
  11. 11. Case 4 - User Retention Level:  1   Difficulty:  Easy   #mceconf #mce-addingknobs Level:  5   Difficulty:  Medium   11 Jan 2014
  12. 12. Air Patriots Sample Results #mceconf #mce-addingknobs 11 Jan 2014
  13. 13. Add A/B Testing to Your App #mceconf #mce-addingknobs 11 Jan 2014
  14. 14. Add A/B Testing to Your App #mceconf #mce-addingknobs 11 Jan 2014
  15. 15. Add A/B Testing to Your App #mceconf #mce-addingknobs 11 Jan 2014
  16. 16. Initialize A/B Testing in Your App private static final String APP_KEY = "876c809ba7084b9fbe2c08a22e5acda1"; private static final String PRIVATE_KEY = "9nxxk4//qL89ymXl/P5aL7w8uhs3DVDrLcyYViG6yn0="; AmazonInsights insightsInstance; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); . . . InsightsCredentials credentials = AmazonInsights.newCredentials(APP_KEY, PRIVATE_KEY); insightsInstance = AmazonInsights.newInstance(credentials, getApplicationContext()); } #mceconf #mce-addingknobs 11 Jan 2014
  17. 17. Create an A/B Testing Project #mceconf #mce-addingknobs 11 Jan 2014
  18. 18. Create an A/B Testing Project #mceconf #mce-addingknobs 11 Jan 2014
  19. 19. Create an A/B Testing Project #mceconf #mce-addingknobs 11 Jan 2014
  20. 20. Add a New Launch #mceconf #mce-addingknobs 11 Jan 2014
  21. 21. Add a New Launch #mceconf #mce-addingknobs 11 Jan 2014
  22. 22. Add a New Launch #mceconf #mce-addingknobs 11 Jan 2014
  23. 23. Multiple Projects Are Allowed private static final String AB_PRJ_ACC = "Acceleration"; private static final String AB_VAR_MAXACC = "maxAcceleration"; //Project name //Variable name private static final String AB_PRJ_VEL = "Velocity"; private static final String AB_VAR_MAXVEL = "maxVelocity"; //Project name //variable name #mceconf #mce-addingknobs 11 Jan 2014
  24. 24. Load Variation Variables private static final int DEF_MAXACC = 5; private static final int DEF_MAXVEL = 50; private int maxAcc; private int maxVel; private void loadTreatment() { ABTestClient abClient = insightsInstance.getABTestClient(); abClient.getVariations(AB_PRJ_VEL, AB_PRJ_ACC).setCallback(new InsightsCallback<VariationSet>() { @Override public void onComplete(VariationSet variations) { Variation variation = variations.getVariation(AB_PRJ_ACC); maxAcc = variation.getVariableAsInt(AB_VAR_MAXACC, DEF_MAXACC); variation = variations.getVariation(AB_PRJ_VEL); maxVel = variation.getVariableAsInt(AB_VAR_MAXVEL, DEF_MAXVEL); } }); } #mceconf #mce-addingknobs 11 Jan 2014
  25. 25. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  26. 26. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  27. 27. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  28. 28. Changing Values on the Fly #mceconf #mce-addingknobs 11 Jan 2014
  29. 29. Add a Segment #mceconf #mce-addingknobs 11 Jan 2014
  30. 30. Add a Segment #mceconf #mce-addingknobs 11 Jan 2014
  31. 31. Add a Segment #mceconf #mce-addingknobs 11 Jan 2014
  32. 32. Use the Segment Use  App  Segngs  to  idenTfy  your  user  segments           //get user age from the app settings         SharedPreferences myPrefs;         myPrefs = getSharedPreferences("com.example.testingapp", Context.MODE_PRIVATE);         int userAge = myPrefs.getInt("com.example.testingapp.USER_AGE", ""); //Initial the segments here UserProfile userProfile = insightsInstance.getUserProfile(); userProfile.addDimensionAsNumber("age", userAge); #mceconf #mce-addingknobs 11 Jan 2014
  33. 33. Demo #mceconf #mce-addingknobs 11 Jan 2014
  34. 34. Wrap up •  A/B Testing –  Free cross platform services –  Precise measuring behaviours using custom segments –  Modifying app behaviors without republishing #mceconf #mce-addingknobs 11 Jan 2014
  35. 35. Thank you
 @AmazonAppDev #mceconf #mce-addingknobs 11 Jan 2014

×