A/B testing and optimization techniques for improving user experience
•
5 likes•1,066 views
This document discusses A/B testing and how companies like Netflix, Airbnb, and Globo use it. A/B testing involves experimenting with changes by exposing different groups of users to variations and analyzing metrics to see what works best. It's important to have a hypothesis, trust dashboards and data scientists for analysis, and continue tests long enough to account for seasonal behaviors. Multivariate testing (MVT) can test multiple variables at once but has higher statistical requirements. Multi-armed bandit algorithms can help optimize experiments by minimizing exposure to weaker options over time through exploration vs exploitation of results.
Recommended
More Related Content
What's hot
Similar to A/B testing and optimization techniques for improving user experience
Similar to A/B testing and optimization techniques for improving user experience (20)
More from Renan Moreira de Oliveira
More from Renan Moreira de Oliveira (20)
Recently uploaded
Recently uploaded (20)
A/B testing and optimization techniques for improving user experience
- 2. A GUY WHO LIKES DATA ON GLOBO.COM @renan_oliveira
- 3. What is A/B?
- 5. Experimentwith a group of users who will notbe impactedby change and other groups is impacted withit
- 8. Who use?
- 10. Netflix
- 11. When you have highly talented individuals who have lots of great ideas, it’s important to have a framework where any new idea can be developed and tested, and data, notopinion, is used to make decisions. https://medium.com/netflix-techblog/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation-and-data-5b0ae9295bdf
- 14. AirBnb
- 15. The outsideworld often has a much larger effectonmetrics than product changes do. Users can behave very differently depending on the day of week, the time of year, the weather, or whether they learned about the website through an online ad or found the site organically. Controlled experimentsisolatetheimpact of the product change while controlling for the aforementioned external factors. https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7
- 17. globo.com
- 18. We use it to validate our products both in the UX area and in the APIs. One of the steps we are seeking and increasingly integrate our A/B solution with the platforms developed by other teams.
- 19. A/B vs
- 20. First step to make your A/B
- 24. Aquestion! After observing a behavior create a question to try to guess the answer with the experiment.
- 25. When I get 99% confidence can I stop my experiment?
- 26. NO! This test says that the difference between the alternative and the control is real. If you do not include the time axis you lose the seasonal behavior.
- 27. How long do I leave my test running?
- 28. IDON’TKNOW! Each product has its own user seasonal behavior. You'll know it better than I do.
- 29. How do I analyze this?
- 30. Trust thedashboardsand consult a datascientist
- 32. It is a skill that will notbeableto escape, everyone should at some time know how to metrify their products in a correct way. Until there, look for specialists toguideyou.
- 33. More than one test on the same page?
- 35. The default answer is no.
- 36. No solution at all?
- 38. MVT - multivariate testing
- 40. For pages with a lotof audience all the experiments and alternatives of your product can be combined.
- 41. Think of settheory and the intersection of it, so we know what the bestcombination of experiments and alternatives for your product.
- 42. IsolatedMVT
- 44. If we build all the combination then we have one where the useronly sawthisexperiment.
- 45. AB vs MVT?
- 46. AB MVT High statistical confidence Does not depend on too much volume Non-visible noise may occur Best combination Population between experiments must be significant Intercessions are seen
- 49. Minimize the consumption of the worst
- 51. The best options may vary over time ARMsmostdeliveryperhour
- 55. Quem é Díaz-Canel, indicado para suceder Raúl Castro em Cuba? Indicado para suceder Raúl Castro nasceu após a Revolução Cubana https://g1.globo.com/mundo/noticia/quem-e-miguel-diaz-canel-o-discipulo-predileto-de-raul-castro-que-assume-o-poder-em-cuba.ghtml TITLE1 TITLE2 TITLE1 >15%
- 56. Viewability Use of an optimization algorithm (MAB) to set the best time to start advertising loading on the page and distribute users accordingly ALTERNATIVA A ALTERNATIVA B ALTERNATIVA C 200px 250px 300px