8 Rollouts a day

1. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices 8 Rollouts a day Kristian Köhntopp

2. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Most interesting!

3. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices This is what we do

4. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices This is what we do

5. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices This is what we do • Most hotels worldwide are not in a chain. • Many hotels communicate by fax.  • Guests need help. • Self-Service okay, Personal if needed.  • 110+ countries, 40+ languages.

6. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Problem statement

7. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Most likely your problem is different Problem statement

8. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices • Rapid growth - constantly: • Last years maximum  reached early in the new year • Year after year Problem statement

9. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices • The growth pattern is predictible and stable.   • Due to growth, good practice is constantly obsolete. • New bottlenecks appear all the time,  the larger part of the year we operate in 'blue water'. Problem statement

10. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Problem statement The Mission Statement:  Create the required organisational personal technical structure to match the expected size in time.

11. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Problem statement There are no constants.  Embrace Change.

12. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices About booking • 2006: • Under 40 persons in IT. • They know the code,   because they wrote it.  And the CPAN modules it uses. • Conditional code in production.  Collection of statistics. • No Ops team.

13. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices About booking • Long startup time for new devs. • Unstable, slow rollouts. • Code hard to test, too many variants. • Code hard to test, changing too fast. • Drowning in Ops work, too many variants.

14. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Decision • Velocity. Scalability. Performance.  • Prove business impact.  • Scale the method.   Create more great old ones.

19. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity

20. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity • Validate requirements:  Do they actually help improving the biz?

21. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity HIghest Paid Persons Opinion Experiments kill HIPPOs

22. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity

23. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity • Data proves: • HIPPO ideas are 0% better than improvements by random strangers.

24. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity • Data proves: • HIPPO ideas are 0% better than improvements by random strangers. • Conclusion: • The only reliable way to improve is to run many experiments.

25. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity • Data proves: • HIPPO ideas are 0% better than improvements by random strangers. • Conclusion: • The only reliable way to improve is to run many experiments. • Velocity: Roll out code as fast as possible, • and prove it with real users.

26. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/

27. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/ Experiments

30. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Experiments • »Experiment« Framework • Conditional Code, • measure impact on conversion, performance, etc.

31. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Velocity • Focus is on capturing quantitative (business) performance data. • and a low Time To Rollout.

33. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Proven requirements Experimental Implementation

34. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Proven requirements • Successful Experiment: • Full On, even if it is painful!  It is earning money! • Optimize that code.  • Trailing performance team sees only business-positive code.

35. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Proven requirements • Better insight than tests:  The best simulation of a real user is…  • There is a term for this:  Testing In Production http://www.thetestingplanet.com/2011/11/the-future-of-software-testing-part-one-testing-in-production/ http://www.setheliot.com/blog/wp-content/uploads/2012/03/STP2012_503_A_to_Z_Testing_in_Production-v2.0.pdf

36. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Testing in production • Experiment framework • decoupling deployment/activation • ramped activation • exposure control

37. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Testing in production • Load/Capacity test in production • determine true capacity,   true failure mode • use a probe to determine latency • concentrate real user load on few frontends, observe probe latency

38. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Testing in production • Telemetry • modern browsers can report page load times back • add additional system data • Data mining • real time monitors • ofﬂine data mining

40. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Scale the method Wait. What method?

41. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces

42. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces Risk compensation is an effect whereby individual people may tend to adjust their behaviour in response to perceived changes in risk. … Another way of stating this is that individuals will behave less cautiously in situations where they feel "safer" or more protected. http://en.wikipedia.org/wiki/Risk_compensation

43. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces Education Visibility Empowerment

44. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces

45. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces Education:You should know your trade. We're painting you blue.

46. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces Visibility: Effects of changes should be obvious. Linear, continuous systems.

47. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Shared Spaces Empowerment: Take away all mechanism that makes decisions  =  Give back responsibility

48. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Booking Rule #1 If you break it, will you even notice?

49. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Booking Rule #2 If you break it, can you ﬁx it?

50. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Craftsmanship The ability to debug a user story front to back, from Biz to deployment.

51. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Culture of Failure A good fail is fast and early. There is a lesson to take away from a good fail. Bad fails are extended suffering and pain.

52. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices So what exactly? • Hire experienced people • Real-world goal vs. SE academics  code earning money vs. beautiful code • with a proven track record in OSS • working in a cooperative environment,  self-organizing, self-driven personality

53. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices So what exactly? • We are not building on a green lawn in the middle of nowhere

54. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices So what exactly?

55. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices So what exactly? • We are not building on a green lawn in the middle of nowhere, • but we are also very aware of the fact that code has a ﬁnite lifetime. • In fact, more than 90% of our code changes are thrown away.

56. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Components • Experimentation • Testing in Production • Shared Spaces • Education,Visibility, Empowerment • A culture of failure

57. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV So how does this relate to DevOps?

58. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV

59. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Kill all sysadmins! • BOFH is a poisonous term 

60. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Kill all sysadmins! • BOFH is a poisonous term  • Infrastructure Developer • automate infrastructure tasks 

61. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Kill all sysadmins! • BOFH is a poisonous term  • Infrastructure Developer • automate infrastructure tasks  • Inhouse Operations Expert • support developers before and after failure

63. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Infrastructure Developer:

64. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Infrastructure Developer: • Make sure the same features work at a 10x scale.

65. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Infrastructure Developer: • Make sure the same features work at a 10x scale. • Make sure things work with zero manual interactions.

66. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Infrastructure Developer: • Make sure the same features work at a 10x scale. • Make sure things work with zero manual interactions. • Mindset differs from feature dev a lot.

68. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Methods differ

69. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Methods differ • Scrum-ish vs. Kanban-ish vs. Fireﬁghting

70. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Methods differ • Scrum-ish vs. Kanban-ish vs. Firefighting • All have prio lists • Feature dev: release oriented, timeboxed • Infrastructure dev: mission oriented,   # of things in flight • Translator,Agent: look at top10 things in flames right now, pick any

72. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Inhouse Operations Expert • Feature dev works on feature front to back, biz to deploy

73. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Inhouse Operations Expert • Feature dev works on feature front to back, biz to deploy • Infrastructure expert offer local special knowledge, possible solutions and their 'prices' • Enable dev to make informed decision

74. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Inhouse Operations Expert • Feature dev works on feature front to back, biz to deploy • Infrastructure expert offer local special knowledge, possible solutions and their 'prices' • Enable dev to make informed decision • Dev then still may choose to break things

76. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Example: • DBAs are not 'owning' their databases • Dev can get advice on schema design, query help, etc at any timπe • Dev can also elect to go forward against this advice   

77. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations POV • Example: • DBAs are not 'owning' their databases • Dev can get advice on schema design, query help, etc at any timπe • Dev can also elect to go forward against this advice    • The experts do not have control

78. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Lock in • We are in fact locked in… • Code changes must be small to debug,  so we MUST roll out often. • "Book": No rollout for x months,  took weeks to reintegrate.

79. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Lock in • Dependence on functional and fast monitoring: • Eventlogging breaks:   We cannot roll out. • Monitors all over the place. • Obsessed with data-based reasoning.

80. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Lock in • Individual changes must be small and rolled out fast: • We cannot use GIT branches, they are tools to accumulate large diffs. • Uniﬁed codebase everywhere. • We need to roll out systems even if they do not change.

81. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Lock in • We are load testing weekly: • Codebase changes quickly. • A lot of really bad code is put life in experiments, has capacity impact.  • Load testing? • In production, playing w/ LB weights,   of course.

82. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices It works for us "Have you broken the site already?"  "No."  "What am I paying you for?"  -- CEO to new dev over lunch

83. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Craftsmanship • Hire experts: "Fachwissen"  They know all possible design choices and the price tags for each of them. • Teach history: "Sachwissen"  The choices we made.

84. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Craftsmanship • C-Level endorsement. • Teach business to devs, dev to business people. 

85. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Craftsmanship • No cop out:  You can get counselling,  but the choice to roll out is yours. • You wrote it.You are the only expert on that code we have.Your call. • Avoiding risk compensation.

86. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices End-to-end responsibility

87. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices End-to-end responsibility • DevOps is magic:

88. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices End-to-end responsibility • DevOps is magic: If you treat adult experts  as responsible adult experts,    they will handle themselves as responsible adult experts.

89. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices No biz vs. dev • We are doing many small projects. • We are creating tight feedback loops. • We are working interdisciplinary.  Business people sit with devs,   as needed.  Operations people sit with devs,   as needed.

90. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations as a service

91. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations as a service • Primary goal:Velocity. • Everything else we do as needed.

92. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations as a service • Primary goal:Velocity. • Everything else we do as needed. • We are deploying imperfect, badly performing or otherwise broken code, crap schema designs, or break internal rules.    Tough luck. Deal.

93. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Operations as a service • Primary goal:Velocity. • Everything else we do as needed. • We are deploying imperfect, badly performing or otherwise broken code, crap schema designs, or break internal rules.    Tough luck. Deal. • Some people can't.They quit.

94. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Should you do it? • Should you do it? • Website.We control the rollout. • High growth environment. • 8 months/year in blue water. • »Slack is good.«  Baby fat eaten by growth rapidly anyway.

95. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Hell yes • Treat your responsible adults as responsible adults, and they will handle themselves as responsible adults. • Kill HIPPOs with data. • Measure. Real world data. For that, you need to be live.

96. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices Live by the rules If you break it, will you even notice? If you break it, can you ﬁx it?

97. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices http://booking.com/jobs Join us!

98. Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices http://booking.com/jobs Join us!