Dev opstalks 2018 releasing the monolith on a daily basis
•
0 likes•303 views
Continuous Delivery is a solved problem. Really? A lot of organisations still struggle for many reasons and Confluence was one of them. What if the world your engineering team lives in is not perfect. This talk is all about making continues deployment happen in a real and not so perfect engineering organisation. This talk is all about the options you have a to achieve a goal without trying to solve all the problems at once. The talk will start with setting the scene of Confluence’s build and deployment process. There will be some interesting insights in the sheer size of builds, tests and the complexities we have around deployments. Once the stage is set for Confluence’s environment, the next step is to explore the problems that are preventing the team to move faster both culturally and technically. To name a few: Build structure, build times, flakey tests, lack of automation, wrong culture, dealing with release blockers, test automation, post-deployment verification, monitoring and operating a production system. With the problems defined, we will go into them in more detail and also learn about the ways the Confluence team resolved them. To wrap it all up we go through the current situation and explain how far we have come with the strategies taken and implemented by the team. As a bonus, a sneak peak of the road ahead for Confluence's Continuous Deployment and Site Reliability Engineering will be shared with the audience.
Recommended
Recommended
More Related Content
Similar to Dev opstalks 2018 releasing the monolith on a daily basis
Similar to Dev opstalks 2018 releasing the monolith on a daily basis (20)
More from Vincent Kok
More from Vincent Kok (10)
Recently uploaded
Recently uploaded (20)
Dev opstalks 2018 releasing the monolith on a daily basis
- 1. VINCENT KOK | ENGINEERING MANAGER, TRELLO | @VINCENTKOK Releasing the Monolith On a daily basis
- 2. @vincentkok About me Trello Engineering Manager on the Trello team Dutch You’ve probably heard that already ;)
- 6. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Cool stories
- 7. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Got issues? https://flic.kr/p/fSopv4
- 8. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Journey https://flic.kr/p/8WQutt
- 9. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Monolith https://flic.kr/p/nA4S49
- 10. FOCUS
- 11. FOCUS
- 13. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Confluence pipeline https://flic.kr/p/jus4Bt
- 15. 81,000 CI jobs
- 18. NaN flakes
- 19. 3 size of the Pipeline team
- 20. 72 steps to release Confluence :(
- 21. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Culture https://flic.kr/p/DsetEn
- 23. Experience The way you’ve been shaped Beliefs Based on your experience Actions Based on your beliefs Results The outcome feeding your experience The Oz Principle
- 24. Early results Theory in real life MomentumClear goal
- 25. Good intentions don’t work ASANKA JAYASURIYA, HEAD OF ENGINEERING STRIDE
- 26. GreenDex
- 27. GreenDex
- 28. Adoption Drive adoption by i.e. training people Mechanisms Audit Does it move the needle? Process What process can improve the situation?
- 29. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Microservice envy https://flic.kr/p/b7ak8e
- 31. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Monolith https://flic.kr/p/nA4S49
- 32. < 1 hour
- 33. < 1 day
- 35. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace #1: Complexity https://flic.kr/p/cFM3cd
- 36. A TYPICAL PIPELINE ReleaseInner Outer Publish Deploy
- 37. SOME FLAWS ReleaseInner Outer Publish Deploy Staging deploy Manual release branch
- 38. Hard to understand . Impact of complexity Inconsistent feedback Manual tasks
- 39. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Fix the model https://flic.kr/p/8ykpkW
- 40. FORK & JOIN Release Publish Deploy Inner Outer Fork Join
- 41. FORK & JOIN Release Publish Deploy Inner Outer Fork Join
- 42. FORK & JOIN IMPLEMENTATION
- 44. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace #2: Ownership https://flic.kr/p/bfRuoX
- 45. Getting the model wrong
- 46. Getting the model wrong
- 47. Broken windows Impact No improvementsFire fighting
- 48. Feedback loop Even more impact MTTRDev speed
- 49. OWNERSHIP Sustainable environment Pipeline team - BuildOps SREEngineering teams
- 50. WHERE DOES YOUR TEAM SIT? Busy work Keeping the lights on Improving team life
- 51. Enforce, don’t fix SUPPORTING TEAMS
- 52. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Sustainable system https://flic.kr/p/awVLLs
- 53. Early results Visibility Create awareness across the team Measure Gather data to allow data driven decisions Move the needle Don’t boil the ocean, focus on a set of smaller results
- 58. SHARED OWNERSHIP
- 59. PROVIDE INSIGHT
- 60. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace #3: Testing https://flic.kr/p/cBbxuQ
- 61. Unit Service UI
- 62. Unit Service
- 63. Balance
- 64. Balance
- 66. The impact of flakes Build times Restarts cause build times to increase drastically Culture The culture of green goes out of the window Broken windows It starts with one. Soon, there will be a lot
- 67. Retries as a painkiller Automatic retries relieve some pain. This is an early result, so see it as a work-around.
- 69. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Fix the root cause https://flic.kr/p/J44Nm
- 70. Test strategy Flake policy Iterate
- 72. Detect Raise ticket Ignore Fix or remove Flake policy
- 73. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Deleting tests, seriously? https://flic.kr/p/8BYXV4
- 74. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace #4: Releases https://flic.kr/p/91MFkh
- 75. RELEASE CYCLE
- 76. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Congestion https://flic.kr/p/bRgBZP
- 77. Ends at master Big bang Releases on hold Dog-fooding becomes hard Impact
- 78. For once a free lunch Fork and Join solved most of our problems.
- 79. 12 FACTOR CONFLUENCE Health check 200 app is alive. 500 app is unhealthy, destroy the node Stateless* Run as many nodes as you need Expose a port Only access to the service
- 80. DECLARATIVE DEPLOYMENT name: Confluence description: Confluence Cloud links: binary: type: docker name: docker.atlassian.io/confluence tag: latest healthcheck: uri: /wiki/internal/healthcheck deepcheck: uri: /wiki/internal/deepcheck semanticCheck: dockerImage: +
- 81. RESOURCES resources: - type: sqs name: default attributes: MaxReceiveCount: 20 VisibilityTimeout: 60 scaling: instance: m3.xlarge min: 7
- 82. Reduce build times Fix dog-fooding and the feedback loop Iterate
- 83. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Dog-fooding https://flic.kr/p/dxpLTm
- 84. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Feedback https://flic.kr/p/dxaAdk
- 85. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Soak time https://flic.kr/p/fmfG2q
- 86. Merge often Feature flags One processDecouple code and features
- 87. Things to take home
- 88. Release cadence Ownership Test automation Complexity Identify your problem areas
- 89. Early results Focus on momentum MomentumClear goal
- 90. Good intentions don’t work. BUILD MECHANISMS
- 91. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace Sustainable system https://flic.kr/p/awVLLs
- 93. VINCENT KOK | ENGINEERING MANAGER, TRELLO | @VINCENTKOK Thanks!