A look into AB testing
Aydar Mukhametzyanov, Freeletics
Agenda
•What is the “Feature Flag”
•Feature Flags at Freeletics
•Learnings
What is the “Feature Flag”
Idea!
What is the “Feature Flag”
What is the “Feature Flag”
What is the “Feature Flag”
👎
👎👎
👎
👎
👎
👎
👎
👎
👎
👎 👎
👎
What is the “Feature Flag”
What is the “Feature Flag”
?50% — true
50% — false
isEnabled
What is the “Feature Flag”
Analysis
Decision
Feature Flags at Freeletics
Feature Flags at Freeletics
Feature Flags at Freeletics
Feature Flags at Freeletics
Feature Flags at Freeletics
Feature Flags at Freeletics
Feature Flags at Freeletics
remoteConfig = RemoteConfig.remoteConfig()
remoteConfig.setDefaults(fromPlist: "RemoteConfigDefaults")
…
remoteConfig.fetch { (status, error) -> Void in
if status == .success {
print("Config fetched!")
self.remoteConfig.activateFetched()
} else {
print("Config not fetched")
}
}
Feature Flags at Freeletics
14
Learnings
Learnings
Clarify future scenarios
Learnings
Clarify future scenarios
• How long should an AB test be alive?

• If the test is successful, then … 

• If the test failed, then …
Clarify future scenarios
Learnings
Code quality
Learnings
Learn fast
Learnings
Learn fast
?
Learnings
t
100%
DELIVERY
INTEGR.
AB TESTDEV
Learn fast
Learnings
t
100%
DELIVERY
DEV
PROT.
AB TEST
INTEGR.
DEV
INTEGR.
Learn fast
Learnings
Prototype
• Targeted to one locale and specific group

• Looks like designed

• Tracks user behavior
Learn fast
Learnings
Prototype by images
Learn fast
Learnings
Prototype by images
imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25,
topCapHeight: 0)
Learn fast
Learnings
Prototype by images
imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25,
topCapHeight: 0)
Learn fast
Learnings
Prototype by images
imageView.image = image.stretchableImage(withLeftCapWidth: Int(image.size.width) - 25,
topCapHeight: 0)
Learn fast
Learnings
Prototype by images
Learn fast
Learnings
Cleanup ASAP
Learnings
Cleanup ASAP
Learnings
Test
Learnings
Test
Learnings
Tips for coding
Learnings
Tips for coding
Nothing bad in copy-pasting
Learnings
Tips for coding
Nothing bad in copy-pasting
v1
v2
Be ready for feature flag changes
Learnings
Tips for coding
Learnings
Tips for coding
Be ready for feature flag changes
TableViewModelFeatureFlag
Storage numberOfRows()isEnabled()
false
100
cellForRow(99)isEnabled()
true
User
scrolls
down
Choose
extended
model
Choose
compact
model
Out of bounds
Feature
Flag
update
Learnings
• Clarify future scenarios

• Learn fast

• Build prototypes

• Cleanup ASAP

• Test

• Debug panel

• Tips for coding

• Nothing bad in copy-pasting

• Be ready for feature flag changes
Thank you!
aydar.mukh
aydar.mukh@gmail.com
aydarin
🎉
Questions

A look into AB testing