Presentation about Drippler's A/B test library for Android devices. presented at DroidCon - Tel Aviv 2014

1. A/B Testing in Android
Nir Hartmann
Drippler
Droidcon Tel-Aviv 2014

2. Why do we need A/B Testing?
• Tests takes the guesswork out
• Enables data-backed decisions
• Enhances engagement and retention

3. Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments

4. What is an A/B Test ?
• Case study – onboarding screen

5. Define the test
• Hypothesis – The layout with the Google+ button at
the left will increase the number of total registered
users.
• Goal – A registered user (the user can skip
registration).
• View event
- Login fragment onCreate().
- Login activity onCreate().
• Variables – Facebook button position (left or right).
• Participants – New users.

6. Amazon A/B Testing SDK
• Very customizable, you can do just about anything as
long as you know what it is you want to do
• It’s free
• Drippler created an open source library that simplify
the process

7. Setup the A/B test
• Setup identifier
– https://developer.amazon.com/al/index.html

8. Setup the A/B test
• Create a project

9. Setup the A/B test
• Create the test

10. Dive into the code
• https://github.com/Drippler/ABTester
public class MyApplication extends Application {
@Override
public void onCreate() {
super.onCreate();
ABTester.init(getApplicationContext(),
"my_public_key", "my_private_key");
}
}

11. private void initLoginActivityTest() {
try {
ABTester.syncPreFetch(
TimeUnit.SECONDS.toMillis(15),
new ABTest("Login page test", false, "Facebook is
first”) );
} catch (TimeoutException e) {
// Couldn't reach amazon servers
}
}
Fetch the test

12. Login Fragment
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setRetainInstance(true);
ABTester.recordEvent(
"Login fragment shown",
false);
}

13. Login Fragment
@Override
public View onCreateView(LayoutInflater inflater,
ViewGroup container, Bundle savedInstanceState) {
boolean shouldShowFacebookFirst =
ABTester.getBoolean("Login page test", "Facebook is
first", false);
if (shouldShowFacebookFirst)
return inflater.inflate(R.layout.facebook_first, null);
else
return inflater.inflate(R.layout.google_first, null);
}

14. Report goal event
public void onUserLoggedIn() {
ABTester.recordEvent("Sign in", false);
}
@Override
protected void onPause() {
super.onPause();
/* Submit the events that were previously stored locally.
* asynchronously
* call it in the onPause() method of an activity */
ABTester.submitEvents();
}

15. Analyze the results
Variation Views Conversions Conversion
rate
Change
Google+
first
1064 320 30.08%
Facebook
first
1043 250 23.97% -20.30%

16. Dice experiment
Goal: maximize the amount of 3’s we get in a 200
dice roll

17. Dice experiment
Hypothesis: wearing a hat will increase the
chance to roll a 3

18. Analyze the results
Variation Views Conversions Conversion
rate
Change
Hat off 200 31 15.50%
Hat on 200 38 19.00% +22.58%
Conversion is never a single number.

19. Confidence level
• Measure the reliability of an estimate
– The confidence levels help us understand if the
results are different merely by chance or by
reason
• 95% confidence level is considered good

20. Analyze the results
Variation Views Conversions Conversion
rate
Change
Google
First
1064 320 30.08% ±
2.32%
Facebook
First
1043 250 23.97% ±
2.18%
-20.30%
• Confidence level of 99%

21. Analyze the results
https://developer.amazon.com/public/apis/manage/ab-
testing/doc/math-behind-ab-testing

22. Choose the best variation
• Launch
– Choose the winning variation
– Control the percentage of customers that receive
a new feature

23. Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png

24. Segmentation

25. Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button’s click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – All users

26. Create the test

27. Rate us DialogFragment
public class RateUsDialog extends DialogFragment {
public static void show(FragmentManager fm,
int color) {
RateUsDialog rateUs = new RateUsDialog();
Bundle extras = new Bundle();
extras.putInt(“color”, color);
rateUs.setArguments(extras);
rateUs.show(fm, “my tag”);
ABTester.recordEvent("Rate us dialog shown", false);
}

28. Rate us DialogFragment
@Override
public Dialog onCreateDialog(Bundle
savedInstanceState) {
int color = getArguments().getInt("color");
return createColoredDialog(color);
}

29. Rate us DialogFragment
private Dialog createColoredDialog(int color) {
...
.setPositiveButton("Rate", new OnClickListener() {
@Override
public void onClick(DialogInterface dialog, int
which) {
ABTester.recordEvent("Rate button click", false);
}
});
return myDialog;

30. Rate us test
1) Asynchronously prefetching SplashActivity
Default timeout is 60 seconds, and can be overriden by
using preFetch(long timeout, ABTest... Test)
ABTester.preFetch(
new ABTest("Rate us test", false, "Rate button color")
);

31. Rate us test
2) Show the dialog
String fetchedColor = ABTester.getString(
"Rate us test", "Rate button color", "#F5F5F5");
int color = Color.parseColor(fetchedColor);
RateUsDialog.show(getFragmentManager(), color);
3) Submitting the results onPause()
ABTester.submitEvents();

32. Analyze the results
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
865 234
27.05%
± 1.51%
Variation
A (green)
904 250
27.65%
± 1.49%
-0.2%
Variation
B (red) 830 230
27.71%
± 1.55% +2.4% 51%
What can I do with these results?

33. Segmentation
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
432 92 21.30% ± 1.97%
Variation A
(green)
464 165 35.56% ± 2.22% +66.9% 98.7%
Variation B
(red)
420 120 28.57% ± 2.20%
+34.1%
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
433 142 32.79% ± 2.26% +22.2% 97.1%
Variation A
(green)
440 85 19.32% ± 1.88% -27.9%
Variation B
(red)
410 110 26.83% ± 2.19% -18.8%
Under 40
Over 40

34. Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – All users

35. Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – Age specific tests

36. Setup the A/B test
• Create a segment

37. Create the segment

38. Assign the segment
• In your code before the fetch
– ABTester.addDimension(”age", myAge);
• Dimension will be remembered forever
• ABTester library will automatically add a
“percentile” dimension
– ABTester.addDimension(“percentile”,
new Random().nextInt(100));

39. Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://unbounce.com/a-b-testing/shocking-results/

40. Multiple Experiments

41. Multiple Experiments

42. Multiple Experiments
• Serial tests
– Run the tests one after the other, without the
need to redistribute your app
– More accurate but takes more time
ABTester.preFetch(
new ABTest("Rate us test", false, "Rate button
color", "Rate actionbar icon")
);

43. Multiple Experiments
• Parallel tests
– Run the tests together, increasing the ‘noise’ for
dependent tests
– Faster
ABTester.preFetch(
new ABTest("Rate us button", false,
"Rate button color”)
new ABTest("Rate us actionbar", false,
"Rate button icon”)
);

44. Not a replacement for common sense

45. Thanks, any questions ?
Don’t overthink it, use Drippler’s A/B test library
https://github.com/Drippler/ABTester
Nir Hartmann, nhartmann@drippler.com
Drippler
Droidcon Tel-Aviv 2014