Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Wix Dev-Centric Culture And Continuous Delivery
Next
Download to read offline and view in fullscreen.

Share

The Art of A/B Testing

Download to read offline

How do you know what 60 millions users like? Wix.com is conducting hundreds of experiments per month on production to understand which features our users like and which hurt or improve our business. In this talk we’ll explain how the engineering team is supporting product managers in making the right decisions and getting our product road map on the right path. We will also present some of the open source tools we developed that help us experimenting our products on humans.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

The Art of A/B Testing

  1. 1. The Art of A/B Testing Experimenting on Humans Aviran Mordo Head of Engineering @aviranm www.linkedin.com/in/aviran www.aviransplace.com
  2. 2. Wix In Numbers Over 100M users + 1.5M new users/month Static storage is >5Pb of data 3 data centers + 3 clouds (Google, Amazon,Azure) 5B HTTP requests/day 1500 people work atWix, of which ~ 600 in Engineering
  3. 3. BasicA/B testing Experiment driven development PETRI –Wix’s 3rd generation open source experiment system Challenges and best practices Complexities and effect on product Agenda
  4. 4. 16:15 A/B Test
  5. 5. Home page results (How many registered)
  6. 6. This is the Wix editor
  7. 7. Gallery manager What can we improve?
  8. 8. Is this better?
  9. 9. Product ExperimentsToggles & Reporting Infrastructure
  10. 10. How do you know what is running?
  11. 11. If I “know” it is better, do I really need to test it? Why so many?
  12. 12. Sign-up Choose Template Edit site Publish Premium The theory
  13. 13. EVERY new feature is A/B tested We open the new feature to a % of users  Measure success  If it is better, we keep it  If worse, we check why and improve If flawed, the impact is just for % of our users Conclusion
  14. 14. New code can have bugs Conversion can drop Usage can drop Unexpected cross test dependencies Sh*t happens (Test could fail)
  15. 15. Language GEO Browser User-agent OS Minimize affected users (in case of failure) Gradual exposure (percentage of…) Company employees User roles Any other criteria you have (extendable) All users
  16. 16. First time visitors = Never visited wix.com New registered users = Untainted users Existing registered users = Already familiar with the service Not all users are equal
  17. 17. Start new experiment (limited population)
  18. 18. Calling Laboratory is Easy
  19. 19. Adding a mobile view
  20. 20. First trial failed Performance had to be improved
  21. 21. Halting the test results in loss of data. What can we do about it?
  22. 22. Solution – Pause the experiment! • Maintain NEW experience for already exposed users • No additional users will be exposed to the NEW feature
  23. 23. PETRI’s pause implementation Use cookies to persist assignment If user changes browser assignment is unknown Server side persistence solves this You pay in performance & scalability
  24. 24. Decision (What to do with the data) Keep feature Drop feature Improve code & resume experiment Keep backwards compatibility for exposed users forever? Migrate users to another equivalent feature Drop it all together (users lose data/work)
  25. 25. Numbers look good but sample size is small We need more data! Expand Reaching statistical significance 25% 50% 75% 100% 75% 50% 25% 0%Control Group (A) Test Group (B)
  26. 26. Keep user experience consistent Control Group (A) Test Group (B)
  27. 27. Signed-in user  Test group is determined by the user ID  Guarantee toss consistency across browsers Anonymous user (Home page)  Test group is randomly determined  Cannot guarantee consistent experience cross browsers 11% ofWix users use more than one desktop browser Keeping consistent UX
  28. 28. Always exclude robots Don’t let Google index a losing page Don’t let bots affect statistics
  29. 29. # of active experiment Possible # of states 10 1024 20 1,048,576 30 1,073,741,824 Possible states >= 2^(# experiments) Wix has ~1000 active experiments ~1.071509e+301
  30. 30. Supporting 2^N different users is challenging How do you know which experiment causes errors? Managing an ever changing production env.
  31. 31. Near real time user BI tools Override options (URL parameters, cookies, headers…) Specialized tools
  32. 32. Integrated into the product
  33. 33. Share document with other users
  34. 34. Document owner is part of a test that enables a new video component
  35. 35. What will the other user experience when editing a shared document ? Owner Friend
  36. 36. Assignment may be different than owner’s Owner (B) Friend (A)
  37. 37. Enable features by existing content  What will happened when you remove a component Enable features by document owner’s assignment  The friend now expects to find the new feature on his own docs Exclude experimental features from shared documents  You are not really testing the entire system Possible solutions
  38. 38. Petri is more than just an A/B test framework Feature toggle A/B Test Personalization Internal testing Continuous deployment Jira integration Experiments Dynamic configuration QA Automated testing
  39. 39. Petri is an open source project https://github.com/wix/petri
  40. 40. Q&A https://github.com/wix/petri http://goo.gl/dqyely Aviran Mordo Head of Engineering @aviranm www.linkedin.com/in/aviran www.aviransplace.com
  41. 41. Credits http://upload.wikimedia.org/wikipedia/commons/b/b2/Fiber_optics_testing.jpg http://goo.gl/nEiepT https://www.flickr.com/photos/ilo_oli/2421536836 https://www.flickr.com/photos/dexxus/5791228117 http://goo.gl/SdeJ0o https://www.flickr.com/photos/112923805@N05/15005456062 https://www.flickr.com/photos/wiertz/8537791164 https://www.flickr.com/photos/laenulfean/5943132296 https://www.flickr.com/photos/torek/3470257377 https://www.flickr.com/photos/i5design/5393934753 https://www.flickr.com/photos/argonavigo/5320119828
  42. 42. Modeled experiment lifecycle Open source (developed usingTDD from day 1) Running at scale on production No deployment necessary Both back-end and front-end experiment Flexible architecture Why Petri
  43. 43. PERTI Server Your app Laboratory DB Logs
  • jimmiethesun

    Sep. 1, 2015
  • bunkertor

    Jul. 19, 2015

How do you know what 60 millions users like? Wix.com is conducting hundreds of experiments per month on production to understand which features our users like and which hurt or improve our business. In this talk we’ll explain how the engineering team is supporting product managers in making the right decisions and getting our product road map on the right path. We will also present some of the open source tools we developed that help us experimenting our products on humans.

Views

Total views

1,146

On Slideshare

0

From embeds

0

Number of embeds

30

Actions

Downloads

12

Shares

0

Comments

0

Likes

2

×