Giving an introduction about A/B testing and its application in product management.

1. A/B TESTING
INTRODUCTION
Ahmed Khaled

2. AGENDA
Definitions
Process
Examples
Tips and tricks
​Summary​

3. DEFINITION
A/B testing introduction 3
allows you to test one variable at a time. It’s used to pick the
better out of two variables. for instance, if you are testing the
background color of your landing pages to determine which one
helps you generate the most conversions, you would use an A/b
test. You can also test on the page level, which means that you
are looking at the entire page as the variable.
A/B testing can tell you whether or not where are you standing now is better
or worse than where you were. Even if, you’ve redesigned everything.

4. A/B testing 4
Not like that
How it works?

5. A/B testing 5
To prevent this
We A/B test

6. A/B testing 6
Why do we need A/B testing ?
- Testing and research historically has been difficult because:
*Expensive.
*Time consuming.
*Unreliable.
*Competitively complicated.
- Experimentation program success needs speed and infrastructure.
- Old approach wasn’t efficient:
*Mailing 20% offers.
*That’s what users love.

7. HOW IT WORKS?
A/B testing 7

8. A/B testing 8

9. A/B testing 9
This is not an A/B testing:
- Too many variants which means that we aren’t sure about what affected
the metrics.
- Bike Size changed.
- Font changed.
- Text changed.
- Order of features is different too.

10. A/B testing 10

11. A/B testing 11
What to experiment? (Experimentation 101)
- We don’t work directly on North Star metrics.
For example: we don’t go to MRR but we go to conversion which
affects MRR directly.
-Primary Metrics: When you create a primary metric, it has to be
conclusive in order to accept that there a statistical significance in you
assumption.
Primary Metric = Conclusive Positive or Conclusive Negative.
- Supporting Metrics: Not as conclusive as primary metrics but gives
you an indication about the direction whether it’s positive or negative.
(Customer service tickets or product returns).
- Health Metrics: Related to the performance of your platform to make
sure that any changes doesn’t impact unexpected metrics (Backend
errors- Queries).

12. A/B testing 12
Binominal Goals: Binominal goals are all the little breadcrumbs users
are leaving while navigating your platform.
Like: Bounce Rate from page- Category page visits.

13. A/B testing 13

14. OUR SUCCESS AT AMAZON IS
A FUNCTION OF HOW MANY
EXPERIMENTS WE DO PER
YEAR, PER MONTH, PER
WEEK, PER DAY
14
“
Jeff Bezos ”

15. A/B TESTING IN ACTION
15

16. A/B testing 16
- Ideate:
Building your backlog of A/B tests.
How to prioritize?
RICE framework
-Reach
-Impact
-Confidence
-Ease

17. A/B testing 17

18. A/B testing 18

19. A/B testing 19

20. A/B testing 20

21. A/B testing 21
- Design:
Hypothesis If (…..) Then (……)
Based on [evidence] we believe that if we change
[your change] for [customer segment] it will help them
[impact].
Success Metrics Primary => Conversion
Secondary => Bounce Rate
We will know this is true if we see [your expected change]
in [primary metric]. This is good for our business because
an increase in [primary metric] is an increase in [business
KPI].
Launch Decision Variant A VS Variant B

22. A/B testing 22
- Setup:
1- Correct UX.
2- Correct Tracking.
3- Free of data quality
Issues.

23. A/B testing 23
- Run:
1- Reach minimum sample size.
Sample size calculator (https://cxl.com/ab-test-calculator/)
2- No mid-test changes to avoid attribution issues.

24. A/B testing 24
- Analyze:
1- Ship, iterate or abandon.
2- Statistical significance and power.
3- Confidence level.

25. A/B testing 25
- Learn:
1- Learn from both success and failure.
2- Uncover segment-specific insights and discoveries for future
iterations of ideas.

26. A/B testing 26

27. REAL-LIFE
EXAMPLES

28. A/B testing introduction 28

29. A/B testing introduction 29

30. A/B testing 30

31. A/B testing 31
Hypothesis: If we added
the sense of joining
people of joining others in
achieving success would
increase conversion
Success Metric:
Conversion from different
touch points.
Highest banner was
converting 15% from its
clicks, the banner
converted 55% from its
clicks
Variant A
Base

32. TIPS AND TRICKS

33. A/B testing introduction 33

34. A/B testing introduction 34

35. A/B testing introduction 35

36. A/B testing introduction 36

37. A/B testing introduction 37

38. Presentation title 38

39. Presentation title 39

40. Presentation title 40

41. Presentation title 41

42. Presentation title 42

43. Presentation title 43

44. SOURCES
A/B testing 44
• https://www.slideshare.net/productschool/ab-testing-101-by-bookingcom-product-manager-
238685991
• https://www.slideshare.net/productschool/how-to-run-ab-tests-successfully-by-vice-media-product-
manager?from_search=3
• https://www.slideshare.net/productschool/how-to-use-ab-testing-to-create-growth-by-crometrics-pm
• https://www.youtube.com/watch?v=RRsN7VLot10
• https://www.youtube.com/watch?v=di2e1QpKYUY
• https://www.youtube.com/watch?v=NsiDwGntnpA
• http://iterative.club/

45. THANK YOU
Ahmed Khaled
ahmed.khalid201030@gmail.co
m
akhaled@agel.io