Recommended
More Related Content
Similar to Causal inference-for-profit | Dan McKinley | DN18
Similar to Causal inference-for-profit | Dan McKinley | DN18 (20)
More from DataconomyGmbH
More from DataconomyGmbH (15)
Recently uploaded
Recently uploaded (20)
Causal inference-for-profit | Dan McKinley | DN18
- 1. CAUSAL INFERENCE for FUN and PROFIT @mcfunley Data Natives Berlin 2018
- 2. CAUSAL INFERENCE for FUN and PROFIT @mcfunley Data Natives Berlin 2018
- 5. xkcd.com/208 please give him all your money Trying science:
- 7. xkcd.com/208 please give him all your money Trying science: Kind of complicated, tbh
- 8. xkcd.com/208 please give him all your money Trying science: Kind of complicated, tbh I have now done this twice
- 10. Experiments?
- 11. user
- 12. A B C user
- 15. A B C
- 18. Sources of chaos: 1. It is tricky to do any of this correctly, turns out
- 19. Sources of chaos: 1. It is tricky to do any of this correctly, turns out 2. You are introducing the possibility that you will unlaunch things
- 20. The idea had been spoken. And the words wouldn’t go back after they’d been uttered aloud.
- 21. Things to worry about
- 22. Correctness
- 23. Effectiveness
- 24. Adoption
- 25. Use a vendor
- 27. Confidence, if constrained and directed tastefully, is not a bad thing.
- 29. Things that are hard: (non-exhaustive)
- 30. Counting things
- 31. Counting things (that arrive quickly or in large numbers or both)
- 32. Bucketing
- 33. The Stats Model
- 34. The Stats Model
- 35. IMPORTANT CAVEAT javascript testing is bad
- 36. IMPORTANT CAVEAT vendors are independent humans with motives
- 37. cool-testing-platform.io Advanced settings you probably don’t need to worry about lol 90% Significance level It’s cool trust me This is fineCancel
- 38. VENDOR MOTIVES: align with high experiment volume
- 39. Pavel Sokov
- 40. Ease into it
- 41. Gopal Vijayaraghavan One theory on experiment volume
- 42. Gopal Vijayaraghavan Individually small chances of finding a real effect
- 43. The “loads of experiments” theory: Knowing where to run experiments is a problem.
- 44. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem.
- 45. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem. Running a lot of experiments is a way to find such a place.
- 46. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem. Running a lot of experiments is a way to find such a place. Running a lot of experiments is a good way to find such a place.
- 47. if bucket(user) and eligible(user): return render(‘new.tpl’) return render(‘old.tpl’) derp The biggest problem is subtle mistakes
- 50. STATS: oh wow much worse
- 51. EXperIMENTpOWER ODDS OF ANARCHY ARE UNCORRELATED TO
- 52. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem. Running a lot of experiments is a way to find such a place. Running a lot of experiments is a good way to find such a place.
- 58. Gopal Vijayaraghavan Ok, once you know what you’re doing
- 59. Changing process
- 61. class CogWidget { // stuff } $w = new CogWidget();
- 62. class CogWidget { // stuff } class CogWidget2 { // stuff // new stuff } $w = (bucket($user) == ‘control’) ? new CogWidget() : new CogWidget2();
- 63. “ ~ The problem with code re-use is that it gets in the way of changing your mind later on. tef https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to
- 64. class CogWidget { // stuff } class CogWidget2 { // stuff // new stuff } $w = new CogWidget();
- 66. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 S M T W R F S meetings
- 67. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 S M T W R F S
- 68. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 S M T W R F S meetings
- 69. if bucket(user): return render(‘new.tpl’) return render(‘old.tpl’)
- 70. if has_no_edge_cases(user): if bucket(user): return render(‘new.tpl’) return render(‘old.tpl’)
- 71. IT’S EASY just change basically everything about development and add an entirely new technical discipline
- 72. OK NOT REALLY maybe start by building a single team with expertise
- 75. Concluding
- 76. Midway upon the journey of our life I found myself within a forest dark For the straightforward pathway had been lost
- 77. Correctness
- 78. YES NO A cross-disciplinary base of expertise for executing the full lifecycle of product experiments that can be built upon moving forward. A declarative YAML framework feeding realtime metrics into a Kafka cluster for self-adjusting multi-arm bandit testing.
- 79. Effectiveness
- 80. Adoption
- 81. “ ~ Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better… Douglas Bowman http://stopdesign.com/archive/2009/03/20/goodbye-google.html
- 82. CREDIBILITY Comes from success as well as avoiding egregious wastes of time
- 84. Humility