Introduction to AB testing and some personal learnings. Presented on the Munich iOS Developers Meetup 03.09.2018.

1. A look into AB testing
Aydar Mukhametzyanov, Freeletics

2. Agenda
•What is the “Feature Flag”
•Feature Flags at Freeletics
•Learnings

3. What is the “Feature Flag”

4. Idea!
What is the “Feature Flag”

5. What is the “Feature Flag”

6. What is the “Feature Flag”
👎
👎👎
👎
👎
👎
👎
👎
👎
👎
👎 👎
👎

7. What is the “Feature Flag”

8. What is the “Feature Flag”
?50% — true
50% — false
isEnabled

9. What is the “Feature Flag”
Analysis
Decision

10. Feature Flags at Freeletics

11. Feature Flags at Freeletics

12. Feature Flags at Freeletics

13. Feature Flags at Freeletics

14. Feature Flags at Freeletics

15. Feature Flags at Freeletics

16. Feature Flags at Freeletics
remoteConfig = RemoteConfig.remoteConfig()
remoteConfig.setDefaults(fromPlist: "RemoteConfigDefaults")
…
remoteConfig.fetch { (status, error) -> Void in
if status == .success {
print("Config fetched!")
self.remoteConfig.activateFetched()
} else {
print("Config not fetched")
}
}

17. Feature Flags at Freeletics
14

18. Learnings

19. Learnings
Clarify future scenarios

20. Learnings
Clarify future scenarios
• How long should an AB test be alive?
• If the test is successful, then …
• If the test failed, then …

21. Clarify future scenarios
Learnings
Code quality

22. Learnings
Learn fast

23. Learnings
Learn fast
?

24. Learnings
t
100%
DELIVERY
INTEGR.
AB TESTDEV
Learn fast

25. Learnings
t
100%
DELIVERY
DEV
PROT.
AB TEST
INTEGR.
DEV
INTEGR.
Learn fast

26. Learnings
Prototype
• Targeted to one locale and specific group
• Looks like designed
• Tracks user behavior
Learn fast

27. Learnings
Prototype by images
Learn fast

28. Learnings
Prototype by images
imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25,
topCapHeight: 0)
Learn fast

29. Learnings
Prototype by images
imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25,
topCapHeight: 0)
Learn fast

30. Learnings
Prototype by images
imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25,
topCapHeight: 0)
Learn fast

31. Learnings
Prototype by images
Learn fast

32. Learnings
Cleanup ASAP

33. Learnings
Cleanup ASAP

34. Learnings
Test

35. Learnings
Test

36. Learnings
Tips for coding

37. Learnings
Tips for coding
Nothing bad in copy-pasting

38. Learnings
Tips for coding
Nothing bad in copy-pasting
v1
v2

39. Be ready for feature flag changes
Learnings
Tips for coding

40. Learnings
Tips for coding
Be ready for feature flag changes
TableViewModelFeatureFlag
Storage numberOfRows()isEnabled()
false
100
cellForRow(99)isEnabled()
true
User
scrolls
down
Choose
extended
model
Choose
compact
model
Out of bounds
Feature
Flag
update

41. Learnings
• Clarify future scenarios
• Learn fast
• Build prototypes
• Cleanup ASAP
• Test
• Debug panel
• Tips for coding
• Nothing bad in copy-pasting
• Be ready for feature flag changes

42. Thank you!
aydar.mukh
aydar.mukh@gmail.com
aydarin
🎉
Questions