A/B testing applies the scientific method and principles of randomized experimentation to digital marketing and product design. In this presentation, Zac Aghion from Splitforce will share his experience with A/B testing in the unique context of mobile applications, as well as two case studies featuring Marks & Spencer - among the UK's largest retailers, and Spot the Difference - a kids' game for iPhone and iPad.

1. Big Data, Little Devices
Mobile A/B testing in theory and practice
Thursday, October 23, 14

2. 2 truths, 1 lie
1. Co-founder of Splitforce, Inc. mobile A/B
testing software
2. Previously product manager @ Google
3. Speak Mandarin better than Zuck
Thursday, October 23, 14

3. 2 truths, 1 lie
1. Co-founder of Splitforce, Inc. mobile A/B
testing software
2. Previously product manager @ Google
3. Speak Mandarin better than Zuck
Thursday, October 23, 14

4. 2 truths, 1 lie
1. Co-founder of Splitforce, Inc. mobile A/B
testing software
2. Previously product manager @ Google
3. Speaks Mandarin better than Zuck
Thursday, October 23, 14

5. 2 truths, 1 lie
1. Co-founder of Splitforce, Inc. mobile A/B
testing software
2. Previously product manager @ Google
3. Speaks Mandarin better than Zuck
Thursday, October 23, 14

6. What is A/B testing?
• Why? To confirm or reject a (null) hypothesis
• What? Randomized experiment in which two
variants, A and B, represent the control and
treatment groups
• Synonyms: bucket testing, split testing
• People also mean... multivariate testing, A/B/n
Thursday, October 23, 14

7. +500%
increase in
‘A/B testing’
as a search
term on
Google since
2009
Google Trends
Thursday, October 23, 14

8. Testing in Web vs. App
Web Mobile App
Javascript Obj-C, Java
Responsive, lightweight Native UI objects,
UX conventions
(Near) instant collection Blind spots
More context
Data Design Tech
Thursday, October 23, 14

9. How Marks & Spencer Boosted
Engagement by +20%
Thursday, October 23, 14

10. Goal: Increase usage of ‘Cooking Mode’
Thursday, October 23, 14

11. Test: Call to action copy on a button
50% Original
(control group)
50% Treatment
(test group)
Thursday, October 23, 14

12. Result: +19.56% usage rate
Thursday, October 23, 14

13. As an agile product team within a larger company, we are
constantly treading the line between moving quickly and
making well informed decisions.
-Neil Kimmett, Marks & Spencer Labs
Thursday, October 23, 14

14. How MyBook Increased
In-App Revenue +25%
Thursday, October 23, 14

15. Goal: Increase monthly subscriptions
• There are free books, and paid books
• When a user arrives at a paywall, she is
presented with an two subscription offers
• Three subscription types: Weekly, Monthly, or
Annual
Thursday, October 23, 14

16. Test: Secondary option as price anchor
50% Original
(Weekly price)
50% Treatment
(Yearly price)
Thursday, October 23, 14

17. Result: +15.2% conversion rate
Thursday, October 23, 14

18. We saw a measurable improvement in payments, and revenue
from the winning variation represented a 25-35% increase
when measured against the baseline.
-Ivan Pestunov, MyBook
Thursday, October 23, 14

19. Matt Boosted Conversions +125%
Matt Mayer
Founder, ReignDesign
Thursday, October 23, 14

20. Users Buy & Spend ‘Stars’ to Play...
...Matt Makes $ $ $
Thursday, October 23, 14

21. Only 0.5% of users purchased...
In-app message prompting to buy
Thursday, October 23, 14

22. Matt decided to test something new...
50%
Thursday, October 23, 14

23. Matt decided to test something new...
50% 50%
Thursday, October 23, 14

24. Matt decided to test something new...
50% 50%
Original = 0.5%
New = 1.13%
Thursday, October 23, 14

25. Thank You!
Zac Aghion
CEO & Co-Founder
Splitforce, Inc.
zac@splitforce.com
Thursday, October 23, 14