This presentation talks about interesting stuff around A/B testing, such as how to generate an hypothesis and handle the conclusion, and also how to use A/B testing as a tool for improving many aspects of your app, including crash reduction and UX.

1. A/B TESTING
IN MOBILE
MAURICIO T ZAQUIA
@mtzaquia

2. MAURICIO T
ZAQUIA
APPLE DISTINGUISHED EDUCATOR, CLASS OF 2015
iOS DEVELOPER @

3. MOBILE
MARKET

4. NEW USER RETENTION
0x
1x
2x
Web Apps
MOBILE TRANSACTIONS
Apps
54 %
Web
46 %

5. MOBILE CONVERSION
Web
Apps
0x 1x 2x 3x

6. ”A/B TESTING IS A TERM FOR A
RANDOMIZED EXPERIMENT WITH TWO
VARIANTS, A AND B, WHICH ARE THE
CONTROL AND VARIATION IN THE
CONTROLLED EXPERIMENT.”
Wikipedia, 2017

8. HYPOTHESIS EXPECTATION
IMPLEMENTATION
EXECUTION
ANALYSIS
DECISION

9. /* code */

10. button.setTitle("Buy", for: .normal)
button.setTitleColor(UIColor.green, for: .normal)
// controller code...
// controller code...

11. // controller code...
if (Experiments.ios_button_urgency.variant == 1) {
button.setTitle("Buy before it's too late!", for: .normal)
button.setTitleColor(UIColor.red, for: .normal)
} else {
button.setTitle("Buy", for: .normal)
button.setTitleColor(UIColor.green, for: .normal)
}
// controller code...
35% 42%

12. button.setTitle("Buy before it's too late!", for: .normal)
button.setTitleColor(UIColor.red, for: .normal)
// controller code...
// controller code...

13. Experiments.ios_button_urgency.variant

14. FEATURE
FLAGS

15. Experiments.ios_button_urgency.variant

16. WHAT
HAPPENS IF
A WEBPAGE
CRASHES?

17. WHAT
HAPPENS IF
AN APP
CRASHES?

18. ”47% OF THE APPS CRASH AT LEAST 1% OF
THE TIME, AND 32% OF THE APPS HAVE A
CRASH RATE OVER 2%”
Crittercism, 2016

19. EVERYBODY BUGS

20. FINDING CRASHES IS ”EASY”…

21. …BUT WITH THE RIGHT TOOLS
NETWORK
LOAD
SCREEN
ISSUES
MEMORY
WARNING

22. NETWORK
LOAD
Experiments.ios_deep_search
DATAEXCHANGEOVERTIME
BASE VARIANT

23. NETWORK
LOAD
Experiments.ios_deep_search
APPABANDONMENTOVERTIME
BASE VARIANT

24. MEMORY
WARNING
Experiments.ios_revamped_card
MEMORYWARNINGSOVERTIME
BASE VARIANT

25. SCREEN
ISSUES
Experiments.ios_table_refactor
1.26% ± 1.64%
BASE
VARIANT
SCREEN FREEZE OVER 1,5 SECONDS

26. SCREEN
ISSUES
Experiments.ios_layout_flow
-95.78% ± 7.19%
BASE
VARIANT
SCREEN VIEWS

27. SCREEN
FREEZE
Experiments.ios_table_refactor
BASE VARIANT

28. FRAMEWORKS
HTTPS://GITHUB.COM/KEEPSAFE/SWITCHBOARD HTTPS://GITHUB.COM/MATTT/SKYLAB

29. COMPANIES

30. Q&A

31. THANK YOU
MAURICIO T ZAQUIA
@mtzaquia