This document discusses A/B testing that was implemented at WeTransfer using Elixir and Phoenix. It describes the requirements, solution, and outcome. The key requirements were to experiment with pricing, social elements, and help pages. The solution involved building a CMS and assigner using Phoenix and React. The assigner uses rules to assign users to variants in real-time. This has been running successfully in production since 2017 with response times under 12 milliseconds. Future plans include improved documentation and potentially open sourcing the solution.

1. A/B testing in Elixir

2. Agenda
• What is A/B testing?
• What were the requirements?
• How did we solve it?
• What was the outcome?
WeTransfer - A/B testing

3. What is A/B testing?

4. What were the
requirements?

5. WeTransfer - A/B testing
Experiment with
Pricing block
• Copy
• Size

6. WeTransfer - A/B testing
Add social element
• Social proof

7. WeTransfer - A/B testing
Create awareness
• Tooltips while uploading

8. WeTransfer - A/B testing
Help page
• Copy
• CTA

9. How did we solve
it?

10. WeTransfer - A/B testing
Tools we used
• Elixir Phoenix
• React

11. WeTransfer - A/B testing
Structure
•CMS - manage experiments: CRUD, start/stop
operations.
• Assigner - sever which assigns clients to
experiments based on their segments.
CMS
✦ Phoenix API
✦ React app
Assigner
★ Phoenix API
★ Elixir GenServers

12. WeTransfer - A/B testing
CMS
• Phoenix application (version 1.3 rc2)
• React app (Webpack)
Application User
Experiment
Variants
Rules
Statistics

13. WeTransfer - A/B testing
Rules
• Parameter: any string, e.g. “language”, “region”
• Type: “string”, “boolean”, “number”, “regex”
• Operator: “==”, “!=”, “>”, “<”, “=>”, “<=”, “~=”
• Value: any string, e.g. “en”, “NL”, etc.

14. WeTransfer - A/B testing
Rules Example

15. WeTransfer - A/B testing
Variant
• Name
• Allocation
• Control group
• Description
• Payload

16. WeTransfer - A/B testing
Variant payload

17. WeTransfer - A/B testing
Payload example

18. WeTransfer - A/B testing
Supervision Tree
Structure Init Sup
Assigner
Experiment
Registry
Priority
Registry
EventSubscriber
Experiment
Supervisor
Experiment #1 Experiment #NExperiment #2

19. WeTransfer - A/B testing
Ready To Be Distributed
All messages to manipulate experiments going
through Phoenix Pub/Sub

20. WeTransfer - A/B testing
Protocol
• Client makes a call with user data
• Based on it, server looks for a matching
experiment
• Returns a variant to assign
• Client makes an impression call if the experiment
was shown to the user

21. WeTransfer - A/B testing
Client Request

22. WeTransfer - A/B testing
Response

23. What was the
outcome?

24. WeTransfer - A/B testing
Usage statistics
• In production from March 2017
• Peak load might be about 10.000/min (166/sec)
• Response from the application - maximum about 800 microseconds
• Response from a AWS ALB - maximum about 11 milliseconds
1

25. WeTransfer - A/B testing
Usage statistics

26. WeTransfer - A/B testing
Future plans
• Finish all important features
• Create documentation
• Split into two applications (Umbrella)
• Add deployment options (e.g. Docker)
• Open source it!

27. THANKS!
Dmitry / Manpreet