Your SlideShare is downloading. ×
0
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Free A/B testing for Android platform
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Free A/B testing for Android platform

173

Published on

A/B test is a great way to measure new and old features and take your app/game toward perfection, however implementing it and understanding the results is a fair challenge. There are several platforms …

A/B test is a great way to measure new and old features and take your app/game toward perfection, however implementing it and understanding the results is a fair challenge. There are several platforms that offers SDK and server side solution to AB testing in Android, one of the most interesting platform is Amazon AB test platform, offering a free and widely customize. Drippler made an open-source library, easily to integrate and handle Amazon AB test platform. This lecture will tell you all about AB testing, how to read the results and how to integrate Drippler's open source library in your code.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
173
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. A/B Testing in Android Nir Hartmann Drippler Droidcon Tel-Aviv 2014
  • 2. Why do we need A/B Testing? • Tests takes the guesswork out • Enables data-backed decisions • Enhances engagement and retention
  • 3. Road map • What is an A/B test ? • Segmentation • Multiple Experiments
  • 4. What is A/B Test ? • Case study – onboarding screen
  • 5. Define the test • Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users • Goal – A registered user (the user can skip registration) • View event - Login fragment onCreate(), setRetainInstance(true) - Login activity onCreate(), null savedInstanceState • Variables – Facebook button position (left or right) • Participants – New users
  • 6. Amazon A/B Testing SDK Like Android: • Very customizable, you can do just about anything as long as you know what it is you want to do • It’s free • Drippler created an open source library that simplify the process
  • 7. Setup the A/B test • Setup identifier – https://developer.amazon.com/al/index.html
  • 8. Setup the A/B test • Create a project
  • 9. Setup the A/B test • Create the test
  • 10. Dive into the code • https://github.com/Drippler/ABTester public class MyApplication extends Application { @Override public void onCreate() { super.onCreate(); ABTester.init(getApplicationContext(), "my_public_key", "my_private_key"); } }
  • 11. private void initLoginActivityTest() { try { ABTester.syncPreFetch( TimeUnit.SECONDS.toMillis(15), new ABTest("Login page test", false, "Facebook is first”) ); } catch (TimeoutException e) { // Couldn't reach amazon servers } } Fetch the test
  • 12. Login Fragment @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setRetainInstance(true); ABTester.recordEvent( "Login fragment shown", false); }
  • 13. Login Fragment @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) { boolean shouldShowFacebookFirst = ABTester.getBoolean("Login page test", "Facebook is first", false); if (shouldShowFacebookFirst) return inflater.inflate(R.layout. facebook_first, null); else return inflater.inflate(R.layout. google_first, null); }
  • 14. Report goal event public void onUserLoggedIn() { ABTester.recordEvent("Sign in", false); } @Override protected void onPause() { super.onPause(); /* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ ABTester.submitEvents(); }
  • 15. Analyze the results Variation Views Conversions Conversion rate Change Google+ first 1064 320 30.08% Facebook first 1043 250 23.97% -20.30%
  • 16. Dice experiment Goal: maximize the amount of 3’s we get in a 100 dice roll
  • 17. Dice experiment Hypothesis: wearing a hat will increase the chance to roll a 3
  • 18. Analyze the results Variation Views Conversions Conversion rate Change Hat off 100 31 31.00% Hat on 100 38 38.00% +22.58% Conversion is never a single number.
  • 19. Confidence level • Measure the reliability of an estimate – The confidence levels help us understand if the results are different merely by chance or by reason • 95% confidence level is considered good
  • 20. Analyze the results Variation Views Conversions Conversion rate Change Google First 1064 320 30.08% ± 2.32% Facebook First 1043 250 23.97% ± 2.18% -20.30% • Confidence level of 99%
  • 21. Analyze the results https://developer.amazon.com/public/apis/manage/ab- testing/doc/math-behind-ab-testing
  • 22. Choose the best variation • Launch – Choose the winning variation – Control the percentage of customers that receive a new feature
  • 23. Road map • What is an A/B test ? • Segmentation • Multiple Experiments
  • 24. Segmentation
  • 25. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button’s click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
  • 26. Create the test
  • 27. Rate us DialogFragment public class RateUsDialog extends DialogFragment { public static void show(FragmentManager fm, int color) { RateUsDialog rateUs = new RateUsDialog(); Bundle extras = new Bundle(); extras.putInt(“color”, color); rateUs.setArguments(extras); rateUs.show(fm, “my tag”); ABTester.recordEvent("Rate us dialog shown", false); }
  • 28. Rate us DialogFragment @Override public Dialog onCreateDialog(Bundle savedInstanceState) { int color = getArguments().getInt("color"); return createColoredDialog(color); }
  • 29. Rate us DialogFragment private Dialog createColoredDialog(int color) { ... .setPositiveButton("Rate", new OnClickListener() { @Override public void onClick(DialogInterface dialog, int which) { ABTester.recordEvent("Rate button click", false); } }); return myDialog;
  • 30. Rate us test 1) Asynchronously prefetching SplashActivity Default timeout is 60 seconds, and can be overriden by using preFetch(long timeout, ABTest... Test) ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color") );
  • 31. Rate us test 2) Show the dialog String fetchedColor = ABTester.getString( "Rate us test", "Rate button color", "#F5F5F5"); int color = Color.parseColor(fetchedColor); RateUsDialog.show(getFragmentManager(), color); 3) Submitting the results onPause() ABTester.submitEvents();
  • 32. Analyze the results Variation Views Conversions Conversion rate Change Confidence Control (white) 865 234 27.05% ± 1.51% Variation A (green) 904 250 27.65% ± 1.49% -0.2% Variation B (red) 830 230 27.71% ± 1.55% +2.4% 51% What can I do with these results?
  • 33. Segmentation Variation Views Conversions Conversion rate Change Confidence Control (white) 432 92 21.30% ± 1.97% Variation A (green) 464 165 35.56% ± 2.22% +66.9% 98.7% Variation B (red) 420 120 28.57% ± 2.20% +34.1% Variation Views Conversions Conversion rate Change Confidence Control (white) 433 142 32.79% ± 2.26% +22.2% 97.1% Variation A (green) 440 85 19.32% ± 1.88% -27.9% Variation B (red) 410 110 26.83% ± 2.19% -18.8% Under 40 Over 40
  • 34. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
  • 35. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – Age specific tests
  • 36. Setup the A/B test • Create a segment
  • 37. Create the segment
  • 38. Assign the segment • In your code before the fetch – ABTester.addDimension(”age", myAge); • Dimension will not change during the experiment • ABTester library will automatically add a “percentile” dimension – ABTester.addDimension(“percentile”, new Random().nextInt(100));
  • 39. Road map • What is an A/B test ? • Segmentation • Multiple Experiments
  • 40. Multiple Experiments
  • 41. Multiple Experiments
  • 42. Multiple Experiments • Serial tests – Run the tests one after the other, without the need to redistribute your app – More accurate but takes more time ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color", "Rate actionbar icon") );
  • 43. Multiple Experiments • Parallel tests – Run the tests together, increasing the ‘noise’ for dependent tests – Faster ABTester.preFetch( new ABTest("Rate us button", false, "Rate button color”) new ABTest("Rate us actionbar", false, "Rate button icon”) );
  • 44. Not a replacement for common sense
  • 45. Thanks, any questions ? Don’t overthink it, use Drippler’s A/B test library https://github.com/Drippler/ABTester Nir Hartmann, nhartmann@drippler.com Drippler Droidcon Tel-Aviv 2014

×