Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pat Hermens - From 100 to 1,000+ deployments a day - Codemotion Amsterdam 2019

18 views

Published on

Coolblue is a proud Dutch company, with a large internal development department; one that truly takes CI/CD to heart. Empowerment through automation is at the heart of these development teams, and with more than 1000 deployments a day, we think it's working out quite well. In this session, Pat Hermens (a Development Managers) will step you through what enables us to move so quickly, which tools we use, and most importantly, the mindset that is required to enable development teams to deliver at such a rapid pace.


Published in: Technology
  • Be the first to comment

  • Be the first to like this

Pat Hermens - From 100 to 1,000+ deployments a day - Codemotion Amsterdam 2019

  1. 1. From 100 to 1000+ deployments a day Pat Hermens Amsterdam | April 2-3, 2019
  2. 2. From 100 to 1,000+ deployments What does it take to get to the next level?
  3. 3. Pat Hermens Development Manager Coding for 15+ years Father & husband Rotterdam, Netherlands @phermens pat@coolblue.nl
  4. 4. Why 1,000+ deployments/day?
  5. 5. 6 @phermens ¯_(ツ)_/¯
  6. 6. 7 @phermens “It depends”
  7. 7. 100 / 1,000 / 10,000; so what?
  8. 8. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf
  9. 9. … Apparently we are too much trained to disregard differences in scale, to treat them as “gradual differences that are not essential.” We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true! — Edsger Dijkstra, 1969
  10. 10. Prerequisites...
  11. 11. 23 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  12. 12. 24 @phermens Developing ☑ Coding guidelines ☑ Code reviews ⬜ Code analysis ☑ Short-lived branches ⬜ Feature toggles Monitoring ☑ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ☑ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code The “faster to master” checklist
  13. 13. 25 @phermens Developing ☑ Coding guidelines ☑ Code reviews ⬜ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ⬜ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ⬜ Infrastructure as Code The “faster to master” checklist
  14. 14. 26 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
  15. 15. Story time!
  16. 16. Responsibility Autonomy FailureOwnership
  17. 17. Responsibility Autonomy FailureOwnership
  18. 18. Responsibility
  19. 19. 32 @phermens Hosting & Deployment “DevOps”
  20. 20. 33 @phermens Hosting & Deployment “DevOps” Team A Team D Team B Team C
  21. 21. 34 @phermens Hosting & Deployment “DevOps” Team A Team D Team F Team B Team C Team E
  22. 22. 35 @phermens Hosting & Deployment “DevOps” Team A Team D Team F Team B Team C Team E
  23. 23. 36 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
  24. 24. 37 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
  25. 25. 38 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
  26. 26. 39 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
  27. 27. 40 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
  28. 28. 41 @phermens Team H Team A Team C Team G Team E Team J Team F Team B Team I Team C Team D Team L Deploy- ment
  29. 29. 44 @phermens
  30. 30. Responsibility Autonomy FailureOwnership
  31. 31. Responsibility Autonomy FailureOwnership
  32. 32. Autonomy
  33. 33. Autonomy
  34. 34. 49 @phermens Serilog + Splunk + DataDog
  35. 35. Autonomy
  36. 36. 58 @phermens
  37. 37. 59 @phermens
  38. 38. 60 @phermens
  39. 39. 61 @phermens $ $ $ $ $
  40. 40. 62 @phermens $ $ $ $ $
  41. 41. 63 @phermens
  42. 42. 64 @phermens UDP
  43. 43. 65 @phermens UDP
  44. 44. 66 @phermens UDP
  45. 45. 67 @phermens UDP
  46. 46. 68 @phermens UDP
  47. 47. 69 @phermens
  48. 48. 70 @phermens TCP
  49. 49. 71 @phermens TCP
  50. 50. 72 @phermens
  51. 51. 73 @phermens
  52. 52. 74 @phermens
  53. 53. 75 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
  54. 54. 76 @phermens
  55. 55. 77 @phermens AWS Lambda stats for Vanessa Optima Prime (as of 2019-04-01) Last 30 days Last 7 days
  56. 56. 78 @phermens AWS Lambda stats for Vanessa Optima Prime (as of 2019-04-01) Last 30 days Last 7 days
  57. 57. 79 @phermens
  58. 58. 80 @phermens
  59. 59. Responsibility Autonomy FailureOwnership
  60. 60. Responsibility Autonomy FailureOwnership
  61. 61. 83 @phermens Ownership
  62. 62. 84 @phermens Build System
  63. 63. 85 @phermens
  64. 64. 86 @phermens
  65. 65. 87 @phermens Development team tooling choices
  66. 66. 88 @phermens Infrastructure team tooling choices
  67. 67. 89 @phermens
  68. 68. 90 @phermens What’s next?
  69. 69. 91 @phermens What’s next?
  70. 70. 92 @phermens ¯_(ツ)_/¯
  71. 71. 93 @phermens
  72. 72. Responsibility Autonomy FailureOwnership
  73. 73. Responsibility Autonomy FailureOwnership
  74. 74. 96 @phermens Fail early, fail often, but always fail forward.
  75. 75. 97 @phermens Failure
  76. 76. 106 @phermens [REDACTED] [REDACTED]
  77. 77. NOTE: 20 cubic centimeters
  78. 78. NOTE: Now 50cc!
  79. 79. NOTE: 49 cubic centimeters
  80. 80. NOTE: More than 49cc’s!
  81. 81. NOTE: Now 45cc!
  82. 82. 123 @phermens
  83. 83. 124 @phermens
  84. 84. 126 @phermens #war-room
  85. 85. Responsibility Autonomy FailureOwnership
  86. 86. Responsibility Autonomy FailureOwnership
  87. 87. 130 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  88. 88. 131 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces (or contracts?) ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  89. 89. Questions?
  90. 90. Checklists!
  91. 91. 134 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  92. 92. 135 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  93. 93. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf
  94. 94. 137 @phermens Thanks! @phermens hermens.com.au pat@coolblue.nl careersatcoolblue.com

×