The document discusses dynamic A/B testing using a service called AB/CD. It explains that traditional A/B testing divides users into static groups to test different buttons or options, but this is costly if a large number of users is needed. Dynamic A/B testing addresses this by initially giving 10% of users a random option and then routing 90% of users to the best performing option based on metrics like views, clicks and click-through rate. This allows conclusions to be drawn more quickly using fewer users. AB/CD is a service that handles the calculations behind dynamic A/B testing, allowing options to be requested via API and results reported back in real time.

1. Dynamic A/B Testing
with AB/CD
Frontend Conference Zurich
September 2. 2016
Memi Beltrame

2. A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups
A B

3. A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups
2. Each group gets a button
A B

4. A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups
2. Each group gets a button
3. See what works and what not
A B

5. Example with 1000 users
Buy Now! Buy for $1’000
500 500

6. Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion
500 500

7. Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion
15 Sales 60 Sales
500 500

8. Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion
15 Sales 60 Sales
$15’000 $60’000
500 500

9. Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion
15 Sales 60 Sales
$15’000 $60’000
500 500
A difference of $45’000

10. Finding out which
button works
just cost you $45’000

11. Is there a way to bring
down that number?

12. Dynamic A/B Tests

13. 2 Rules
Buy Now! Buy for $1’000

14. 2 Rules
Buy Now! Buy for $1’000
1. 10% get a button at random

15. 2 Rules
Buy Now! Buy for $1’000
1. 10% get a button at random
2. 90% get the best button

16. Find the best button
By tracking 3 values for each one

17. Find the best button
1. Views
By tracking 3 values for each one

18. Find the best button
1. Views
2. Clicks
By tracking 3 values for each one

19. Find the best button
1. Views
2. Clicks
3. Ratio of Clicks / Views
By tracking 3 values for each one

20. How are Options Tracked?
Start of Test

21. How are Options Tracked?
After first user

22. How are Options Tracked?
After 1000 users
Resulting in $111’000 for the test

23. Finding out which
button works
just cost you $9’000

24. Finding out which
button works
just cost you $9’000
That’s $36’000 less.

25. AB/CD

27. A service to set up dynamic A/B
tests that handles the math behind it.

28. AB/CD
http://abcd.lessamess.ch

29. Setup

30. After Setup
1. Request an option via an API

31. After Setup
1. Request an option via an API
2. Store the option in a session

32. After Setup
1. Request an option via an API
2. Store the option in a session
3. Report back to AB/CD when
an option performs

33. API

34. What you get
Results in real time

35. Thanks!
Memi Beltrame
@bratwurstkomet