A look at the past, present and future of continuous integration and delivery at the biggest Dutch online retail platform.

1. CI/CD @ bol.com
Taking control from developers and giving it back again
1

2. •Maarten Dirkse
• CI/CD engineer
• @ bol.com for 4.5 years (started as a Java developer)
• Member of the DPI (read: Tools) team
• @mdirkse on twitter
•bol.com
• Largest online retail platform in the Netherlands and (the Flemish part of) Belgium
• 50+ scrum teams that work on 200+ applications and services
• Platform team of ~50 people that supports them
2
Who am I and what is bol.com?

3. 1. Bol.com build fact(oids)
Stats, numbers, etc.
2. A long time ago, in a build toolchain far far away…
The old toolchain
3. That time we automated all the things
Adopting JobDSL and taking control of our builds
4. Mayfly buzz
Creating a CI/CD platform based on our own vision
5. Unkept gates
The future and how we plan to definitively get out of every team’s way
6. Q&A
The part where we all stare at each other in awkward silence ‘cause no one wants to go first
3
What I’ll be discussing

4. Bol.com build fact(oids)
4
Source
• 200+ services & apps
• ~ 3000 repositories in Gitlab
and Stash (yes… and)
• Java, Go, Groovy, PLSQL
Build
• 140 Jenkins slaves (VM’s)
• 4000+ Jenkins jobs
Deploy
• 2500+ VM’s
• ~ 1000 deploy jobs in
Rundeck
• 100’s of lines of Puppet
code to manage deploys

5. •SVN!
•Jenkins
• Configuration wild west: every job is different!
•DeployIT (now known as XLDeploy)
• Deploys done by “deployment engineers”
5
The old ball and (tool-)chain

6. •Teams responsible for their own configuration of jobs
• No standardization
• No quality control
•Tools team was responsible for Jenkins
• And thus, by extension, everyone’s problems (scalability issue)
•Deployment process was opaque to developers
• Code gets “thrown over the wall”, ops engineers deploy and run it
•Automated tests checks were done twice daily against test environment
• Completely divorced from the actual deploys of services
Problems
6
The old ball and (tool-)chain

7. • Jenkins JobDSL plugin
• Wrote our own config format that is used, in combination with the JobDSL, to generate all our
Jenkins jobs
• Result: teams control config, tools team controls output to Jenkins
• We can limit variation between jobs
• We can easily generate helper jobs (like check-jobs that follow immediately on the deploy)
• We can easily re-generate our entire Jenkins config
• Have a central place to implement best practices across all jobs
• But….
• One, central config repo
• Any individual config could still potentially screw up the Jenkins server, so…
• We employ a merge request model where the Tools team checks every config change (gatekeeping)
•On the ops side: deployment moved from DeployIT to Rundeck
Standardize!
7
Enter the JobDSL

8. •We had/have a shared testing environment
• Break it, and everyone’s work is on hold, while the team that has to fix it sweats
•We needed isolation, to provide
• Insulation from failure
• Stable master
• Breaking dependencies on other teams
• A stable environment to do automated tests checks
•We developed Mayfly
8
Mayfly

9. • Separate environment per feature
• Ability to mock out dependencies
• Ability to define your own build pipeline (including running checks)
• Block a merge to master before certain (team-defined) quality conditions are met, like:
• Basic build
• Automated tests
• “4-eyes” principle
• Successful deploy to the automated environment
• Custom, team-defined rule
• (Almost) complete team control over the CI part of CI/CD
• Everything is based on containers
• But... deploy still out of the hands of teams
What does it provide?
9
Mayfly

10. 10

11. Unkept gates
What does the future hold for CI/CD?
11
Autonomy Automation
Build “consulting”

12. •Autonomy
teams should be able to build and deploy without blocking on dependencies on
anyone (API’s & self-service)
• But teams are responsible! No taxation without representation autonomy without
responsibility.
•Automation
Reduce the amount of manual action to a complete minimum
•Build consulting
“Hi, we’re the tools team. We’ve developed some wheels which work pretty
well and which we think you’ll like. But if not, feel free to invent your own.”
• (the same would hold for a “test tooling team”)
12
Unkept gates

13. The vision and approach of the tools
team must be validated by uptake.
Developers should want to use your
stuff, not be forced to adopt it.
Your tools must be better and more
appealing than something that can be
put together in some team’s spare
time.
13
Unkept gates
The Discerning User

14. 14
Workflow of the future

15. 15
http://banen.bol.com

16. Thanks!
till next bol.com