Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Democratizing Online Controlled Experiments at

At we have been conducting evidenced based product development using online experiments for more than ten years. Our methods and infrastructure were designed from their inception to reflect culture, that is, with democratization and decentralization of experimentation and decision making in mind.

In this talk, based on this paper with the same title, we explain how our approach has allowed such a large organization as to truly and successfully democratize experimentation.

  • Be the first to comment

Democratizing Online Controlled Experiments at

  1. 1. Democratizing online controlled experiments at Lukas Vermeer | Senior Product Owner for Experimentation
  2. 2. TL;DR. Central repository of successes and failures Descriptions of all experiment iterations and of the final decision are available for all experimenters. Genericity and extensibility Experimental design is abstracted away. Reports are automated and product agnostic. Data which can be trusted We monitor the validity of the data by computing common metrics in two separate date pipelines. Loose coupling Business logic and experiment infrastructure are purposefully kept decoupled. Building safeguards We encourage sound methodology and provide data quality checks, but no rules or automation.
  3. 3. Focus.
  4. 4. This is a story about culture
  5. 5. This is a story about people
  6. 6. 1. Ask PR department for “pretty photos of employees”. 2. Filter out non-experiment roles (dba, support, management, etc). 3. Select six at random from the remainder. 4. Email them and ask “how do you use experiments in your job”. 5. Copy paste replies onto slides. 6. Highlight key phrases. 7. Sprinkle in some context. Method. Sample Survey Results
  7. 7. Lukas Vermeer. Senior Product Owner “I help hundreds of people run experiments to make better.”
  8. 8. Founded in 1996 1.600.000+ properties 1.500.000+ room nights per day 15.500+ employees 1.000+ concurrent tests Everything is a test Booking.
  9. 9. Empower people to experience the world.
  10. 10. Clyde Li. Client Side Developer “Data-driven is quite common these days in tech industry, however, empowering everyone to make data-driven decisions independently is quite unique in”
  11. 11. Nekeia Boone. Senior UX Copywriter “I can come up with an idea over breakfast, bike to the office and have it live well before lunch. I’ve never worked anywhere else that gives me this level of ownership and creative freedom to validate my ideas.”
  12. 12. Heloisa Biagi. Client Side Developer “I think it's great that the company encourages everyone to hack and test ideas among the users. No higher value opinions, no centralization of decision making, everyone is free to have their own ideas and at the end of the day, users are the ones who decide what's best for them.”
  13. 13. You cannot have a democracy without empowering the people
  14. 14. Hadeer Younis. Full-Stack Software Developer “Experimentation is a great way to figure out if something as small as a copy change or something as big as a whole book process flow will help users, but you can’t depend on it as your only source of product validation as this will greatly hinder the product development.”
  15. 15. “The plural of anecdote is not data” - Lots of misinformed speakers at conferences I’ve attended
  16. 16. “The plural of anecdote is not data” - Raymond Wolfinger (1969)
  17. 17. GreyBlue OrangeWhite 80%60% 70%50% Data is just data. To make good decisions, we need good evidence.
  18. 18. A B Website Optimisation. Let’s change the button from yellow to blue and see if it increases the magic number. Buy now! Buy now!
  19. 19. A B Hypothesis Testing. We observed in user research that some people have difficulty finding the “buy now” button. We suspect this is caused by the low contrast between the font and the background. To solve this user issue, we will change the button from yellow to blue. If this solution works, we expect to see more users hover and click, and eventually purchase. Buy now! Buy now!
  20. 20. Evidence-based customer-centric product development
  21. 21. Finn Hansen. Product Owner “We use experimentation to help us validate hypotheses with the goal of addressing well defined user problems. It's all about learning as fast as possible in the most rigorous way possible.”
  22. 22. Based on [prior] we believe [condition] for [users] will encourage them to [behavior] We will know this when we see [effects] happen to [metrics] This will be good for customers, partners and our business because [motivation]. Hypothesis template. Theory Validation Objective
  23. 23. Diogo Antunes. Principal Developer & Fellow “We work in a global scale. By validating my changes through experimentation I continuously get challenged on my assumptions and the way I look into the experience of our customers.”
  24. 24. 9/10 tests fail. “90% of product decisions have an inconclusive or negative effect on a product’s primary metric.” - VWO (2016) 90% 10%
  25. 25. Disclaimer: not what the actual experiment looked like.
  26. 26. Take the biggest small step so you can challenge your riskiest assumptions quickly
  27. 27. The order of items can be improved. The order of items matters to users. The feature can be improved. The feature matters to users. The page can be improved. What is on the page matters to users. Assumptions. Order Feature Page
  28. 28. Evidence-based customer-centric product development
  29. 29. This is a story about people
  30. 30. Empower people to experience the world.
  31. 31. Thank you!