Your SlideShare is downloading. ×
0
A/B Testing in Android
Nir Hartmann
Drippler
Droidcon Tel-Aviv 2014
Why do we need A/B Testing?
• Tests takes the guesswork out
• Enables data-backed decisions
• Enhances engagement and rete...
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
What is an A/B Test ?
• Case study – onboarding screen
Define the test
• Hypothesis – The layout with the Google+ button at
the left will increase the number of total registered...
Amazon A/B Testing SDK
• Very customizable, you can do just about anything as
long as you know what it is you want to do
•...
Setup the A/B test
• Setup identifier
– https://developer.amazon.com/al/index.html
Setup the A/B test
• Create a project
Setup the A/B test
• Create the test
Dive into the code
• https://github.com/Drippler/ABTester
public class MyApplication extends Application {
@Override
publi...
private void initLoginActivityTest() {
try {
ABTester.syncPreFetch(
TimeUnit.SECONDS.toMillis(15),
new ABTest("Login page ...
Login Fragment
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setRetainIn...
Login Fragment
@Override
public View onCreateView(LayoutInflater inflater,
ViewGroup container, Bundle savedInstanceState)...
Report goal event
public void onUserLoggedIn() {
ABTester.recordEvent("Sign in", false);
}
@Override
protected void onPaus...
Analyze the results
Variation Views Conversions Conversion
rate
Change
Google+
first
1064 320 30.08%
Facebook
first
1043 2...
Dice experiment
Goal: maximize the amount of 3’s we get in a 200
dice roll
Dice experiment
Hypothesis: wearing a hat will increase the
chance to roll a 3
Analyze the results
Variation Views Conversions Conversion
rate
Change
Hat off 200 31 15.50%
Hat on 200 38 19.00% +22.58%
...
Confidence level
• Measure the reliability of an estimate
– The confidence levels help us understand if the
results are di...
Analyze the results
Variation Views Conversions Conversion
rate
Change
Google
First
1064 320 30.08% ±
2.32%
Facebook
First...
Analyze the results
https://developer.amazon.com/public/apis/manage/ab-
testing/doc/math-behind-ab-testing
Choose the best variation
• Launch
– Choose the winning variation
– Control the percentage of customers that receive
a new...
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://visualwebsiteoptimizer.com/split-testing-blo...
Segmentation
Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button’s click rate
• Goal – Click event on t...
Create the test
Rate us DialogFragment
public class RateUsDialog extends DialogFragment {
public static void show(FragmentManager fm,
int ...
Rate us DialogFragment
@Override
public Dialog onCreateDialog(Bundle
savedInstanceState) {
int color = getArguments().getI...
Rate us DialogFragment
private Dialog createColoredDialog(int color) {
...
.setPositiveButton("Rate", new OnClickListener(...
Rate us test
1) Asynchronously prefetching SplashActivity
Default timeout is 60 seconds, and can be overriden by
using pre...
Rate us test
2) Show the dialog
String fetchedColor = ABTester.getString(
"Rate us test", "Rate button color", "#F5F5F5");...
Analyze the results
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
865 234
27.05%
± 1.51%
V...
Segmentation
Variation Views Conversions Conversion rate Change Confidence
Control
(white)
432 92 21.30% ± 1.97%
Variation...
Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button click rate
• Goal – Click event on the...
Define the test
• Hypothesis – Coloring the “Rate” button, will
increase the button click rate
• Goal – Click event on the...
Setup the A/B test
• Create a segment
Create the segment
Assign the segment
• In your code before the fetch
– ABTester.addDimension(”age", myAge);
• Dimension will be remembered f...
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://unbounce.com/a-b-testing/shocking-results/
Multiple Experiments
Multiple Experiments
Multiple Experiments
• Serial tests
– Run the tests one after the other, without the
need to redistribute your app
– More ...
Multiple Experiments
• Parallel tests
– Run the tests together, increasing the ‘noise’ for
dependent tests
– Faster
ABTest...
Not a replacement for common sense
Thanks, any questions ?
Don’t overthink it, use Drippler’s A/B test library
https://github.com/Drippler/ABTester
Nir Hartm...
Upcoming SlideShare
Loading in...5
×

Drippler's A/B test library

214

Published on

Presentation about Drippler's A/B test library for Android devices. presented at DroidCon - Tel Aviv 2014

Published in: Software, Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
214
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Drippler's A/B test library"

  1. 1. A/B Testing in Android Nir Hartmann Drippler Droidcon Tel-Aviv 2014
  2. 2. Why do we need A/B Testing? • Tests takes the guesswork out • Enables data-backed decisions • Enhances engagement and retention
  3. 3. Road map • What is an A/B test ? • Segmentation • Multiple Experiments
  4. 4. What is an A/B Test ? • Case study – onboarding screen
  5. 5. Define the test • Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users. • Goal – A registered user (the user can skip registration). • View event - Login fragment onCreate(). - Login activity onCreate(). • Variables – Facebook button position (left or right). • Participants – New users.
  6. 6. Amazon A/B Testing SDK • Very customizable, you can do just about anything as long as you know what it is you want to do • It’s free • Drippler created an open source library that simplify the process
  7. 7. Setup the A/B test • Setup identifier – https://developer.amazon.com/al/index.html
  8. 8. Setup the A/B test • Create a project
  9. 9. Setup the A/B test • Create the test
  10. 10. Dive into the code • https://github.com/Drippler/ABTester public class MyApplication extends Application { @Override public void onCreate() { super.onCreate(); ABTester.init(getApplicationContext(), "my_public_key", "my_private_key"); } }
  11. 11. private void initLoginActivityTest() { try { ABTester.syncPreFetch( TimeUnit.SECONDS.toMillis(15), new ABTest("Login page test", false, "Facebook is first”) ); } catch (TimeoutException e) { // Couldn't reach amazon servers } } Fetch the test
  12. 12. Login Fragment @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setRetainInstance(true); ABTester.recordEvent( "Login fragment shown", false); }
  13. 13. Login Fragment @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) { boolean shouldShowFacebookFirst = ABTester.getBoolean("Login page test", "Facebook is first", false); if (shouldShowFacebookFirst) return inflater.inflate(R.layout.facebook_first, null); else return inflater.inflate(R.layout.google_first, null); }
  14. 14. Report goal event public void onUserLoggedIn() { ABTester.recordEvent("Sign in", false); } @Override protected void onPause() { super.onPause(); /* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ ABTester.submitEvents(); }
  15. 15. Analyze the results Variation Views Conversions Conversion rate Change Google+ first 1064 320 30.08% Facebook first 1043 250 23.97% -20.30%
  16. 16. Dice experiment Goal: maximize the amount of 3’s we get in a 200 dice roll
  17. 17. Dice experiment Hypothesis: wearing a hat will increase the chance to roll a 3
  18. 18. Analyze the results Variation Views Conversions Conversion rate Change Hat off 200 31 15.50% Hat on 200 38 19.00% +22.58% Conversion is never a single number.
  19. 19. Confidence level • Measure the reliability of an estimate – The confidence levels help us understand if the results are different merely by chance or by reason • 95% confidence level is considered good
  20. 20. Analyze the results Variation Views Conversions Conversion rate Change Google First 1064 320 30.08% ± 2.32% Facebook First 1043 250 23.97% ± 2.18% -20.30% • Confidence level of 99%
  21. 21. Analyze the results https://developer.amazon.com/public/apis/manage/ab- testing/doc/math-behind-ab-testing
  22. 22. Choose the best variation • Launch – Choose the winning variation – Control the percentage of customers that receive a new feature
  23. 23. Road map • What is an A/B test ? • Segmentation • Multiple Experiments http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png
  24. 24. Segmentation
  25. 25. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button’s click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
  26. 26. Create the test
  27. 27. Rate us DialogFragment public class RateUsDialog extends DialogFragment { public static void show(FragmentManager fm, int color) { RateUsDialog rateUs = new RateUsDialog(); Bundle extras = new Bundle(); extras.putInt(“color”, color); rateUs.setArguments(extras); rateUs.show(fm, “my tag”); ABTester.recordEvent("Rate us dialog shown", false); }
  28. 28. Rate us DialogFragment @Override public Dialog onCreateDialog(Bundle savedInstanceState) { int color = getArguments().getInt("color"); return createColoredDialog(color); }
  29. 29. Rate us DialogFragment private Dialog createColoredDialog(int color) { ... .setPositiveButton("Rate", new OnClickListener() { @Override public void onClick(DialogInterface dialog, int which) { ABTester.recordEvent("Rate button click", false); } }); return myDialog;
  30. 30. Rate us test 1) Asynchronously prefetching SplashActivity Default timeout is 60 seconds, and can be overriden by using preFetch(long timeout, ABTest... Test) ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color") );
  31. 31. Rate us test 2) Show the dialog String fetchedColor = ABTester.getString( "Rate us test", "Rate button color", "#F5F5F5"); int color = Color.parseColor(fetchedColor); RateUsDialog.show(getFragmentManager(), color); 3) Submitting the results onPause() ABTester.submitEvents();
  32. 32. Analyze the results Variation Views Conversions Conversion rate Change Confidence Control (white) 865 234 27.05% ± 1.51% Variation A (green) 904 250 27.65% ± 1.49% -0.2% Variation B (red) 830 230 27.71% ± 1.55% +2.4% 51% What can I do with these results?
  33. 33. Segmentation Variation Views Conversions Conversion rate Change Confidence Control (white) 432 92 21.30% ± 1.97% Variation A (green) 464 165 35.56% ± 2.22% +66.9% 98.7% Variation B (red) 420 120 28.57% ± 2.20% +34.1% Variation Views Conversions Conversion rate Change Confidence Control (white) 433 142 32.79% ± 2.26% +22.2% 97.1% Variation A (green) 440 85 19.32% ± 1.88% -27.9% Variation B (red) 410 110 26.83% ± 2.19% -18.8% Under 40 Over 40
  34. 34. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
  35. 35. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – Age specific tests
  36. 36. Setup the A/B test • Create a segment
  37. 37. Create the segment
  38. 38. Assign the segment • In your code before the fetch – ABTester.addDimension(”age", myAge); • Dimension will be remembered forever • ABTester library will automatically add a “percentile” dimension – ABTester.addDimension(“percentile”, new Random().nextInt(100));
  39. 39. Road map • What is an A/B test ? • Segmentation • Multiple Experiments http://unbounce.com/a-b-testing/shocking-results/
  40. 40. Multiple Experiments
  41. 41. Multiple Experiments
  42. 42. Multiple Experiments • Serial tests – Run the tests one after the other, without the need to redistribute your app – More accurate but takes more time ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color", "Rate actionbar icon") );
  43. 43. Multiple Experiments • Parallel tests – Run the tests together, increasing the ‘noise’ for dependent tests – Faster ABTester.preFetch( new ABTest("Rate us button", false, "Rate button color”) new ABTest("Rate us actionbar", false, "Rate button icon”) );
  44. 44. Not a replacement for common sense
  45. 45. Thanks, any questions ? Don’t overthink it, use Drippler’s A/B test library https://github.com/Drippler/ABTester Nir Hartmann, nhartmann@drippler.com Drippler Droidcon Tel-Aviv 2014
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×