Successfully reported this slideshow.
Your SlideShare is downloading. ×

CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 40 Ad

CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments

Download to read offline

(video summary at: https://conversionhotel.com/session/keynote-2019-run-better-experiments-srm-checks/)

Lukas combines industry experience in online experimentation and data science with an academic background in computing science and machine learning. When I first met him, he was already a smart guy, but still working at Oracle. A year later he moved on to Booking.com (almost 7 years ago already – he has seen a large bunch of new employees coming in :-)) to finally become their director of experimentation.

Lukas is responsible for the internal tooling and training that helps product development improve the customer experience in measurable steps through thousands of experiments. Of course he presented about this around the world, but never at Texel. Finally it was time to ask him – not to speak about democratizing experimentation or his storytelling on statistics. You have probably seen that before. We asked him because you, as CH2019 attendees, asked us to go advanced on experimentation and he is one of the only ones in Europe that has (co-)written in depth scientific papers on experimentation.

And that is exactly what Lukas is going to do: go in depth on his recent Sample Ratio Mismatch paper.

Enjoy the ride,

Ton Wesseling

Founder & host of The Conference formerly known as Conversion Hotel

(video summary at: https://conversionhotel.com/session/keynote-2019-run-better-experiments-srm-checks/)

Lukas combines industry experience in online experimentation and data science with an academic background in computing science and machine learning. When I first met him, he was already a smart guy, but still working at Oracle. A year later he moved on to Booking.com (almost 7 years ago already – he has seen a large bunch of new employees coming in :-)) to finally become their director of experimentation.

Lukas is responsible for the internal tooling and training that helps product development improve the customer experience in measurable steps through thousands of experiments. Of course he presented about this around the world, but never at Texel. Finally it was time to ask him – not to speak about democratizing experimentation or his storytelling on statistics. You have probably seen that before. We asked him because you, as CH2019 attendees, asked us to go advanced on experimentation and he is one of the only ones in Europe that has (co-)written in depth scientific papers on experimentation.

And that is exactly what Lukas is going to do: go in depth on his recent Sample Ratio Mismatch paper.

Enjoy the ride,

Ton Wesseling

Founder & host of The Conference formerly known as Conversion Hotel

Advertisement
Advertisement

More Related Content

Similar to CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments (20)

More from Webanalisten .nl (20)

Advertisement

Recently uploaded (20)

CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments

  1. 1. Lukas Vermeer Director of Experimentation @ Booking.com Experimentation Island 2019 One neat trick to run better experiments
  2. 2. Sample Ratio Mismatch 1. Why should we care? 2. How will we know? 3. Who suffers from SRM? 4. What can cause SRM? 5. What can we do?
  3. 3. Why should we care? The pitfalls of experimenting on the web
  4. 4. 12% Lower average weight for participants who described “how applying versus not applying eyeliner could make them feel differently.” Scientists in Chicago have discovered a revolutionary new way to lose weight. Read More >
  5. 5. How will we know? Detecting sample ratio mismatch
  6. 6. Sample Ratio Mismatch A mismatch between the expected sample ratio, and the observed sample ratio.
  7. 7. 2 Number of commercially available experimentation platforms (that I am aware of) that are actively checking for—and warning users about—SRM.
  8. 8. Who suffers from SRM? Prevalence of SRM in practice
  9. 9. 6%-10% Percentage of experiments affected by SRM in companies leading in experimentation which have been actively detecting and addressing SRM issues for the past few years. (Likely worse for companies which haven’t been checking at all.)
  10. 10. What can cause SRM? A taxonomy and rules of thumb for practitioners
  11. 11. What can we do? One neat trick to run better experiments
  12. 12. One neat trick to run better experiments 1. Check for SRM 2. ???????? 3. More reliable results 4. ???????? 5. Profit!
  13. 13. Rules of thumb 1. Examine scorecards 2. Examine user segments 3. Examine time segments 4. Analyse performance metrics 5. Analyse engagement metrics 6. Count frequency of SRMs 7. Examine AA experiment 8. Examine severity 9. Examine downstream 10. Examine across pipelines
  14. 14. Github Project github.com/lukasvermeer/srm Taxonomy Paper dl.acm.org/citation.cfm?id=329 2500.3330722
  15. 15. Lukas Vermeer Director of Experimentation @ Booking.com One neat trick to run better experiments
  16. 16. Bonus slide for when someone inevitably asks “but what about Bayesian!?” 🙄

×