Configuration As Code - Adoption of the Job DSL Plugin at Netflix
•
35 likes•68,445 views
The Jenkins Job DSL plugin allows programmers to express job configurations as code. Learn about the benefits, from the obvious (store your configurations in the SCM of your choice) to the not-so-obvious (focus on intent, instead of succumbing to the distraction of multiple, complex job configuration options). We will share our experience adopting the plugin over the past year to create and maintain more complex job pipelines at Netflix.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (10)
Similar to Configuration As Code - Adoption of the Job DSL Plugin at Netflix
Similar to Configuration As Code - Adoption of the Job DSL Plugin at Netflix (20)
Recently uploaded
Recently uploaded (20)
Configuration As Code - Adoption of the Job DSL Plugin at Netflix
- 1. Configuration as Code Adoption of the Job DSL Plugin at Netflix Justin Ryan <jryan@netflix.com> @quidryan http://www.slideshare.net/quidryan
- 2. Who Are We • Engineering Tools • Justin Ryan / jryan@netflix.com / @quidryan • Curt Patrick / cpatrick@netflix.com • Freedom & Responsibility
- 3. Situation • No single flow that builds everything • Every team has their own jobs • Each branch needs multiple jobs
- 4. Problem • Lots of copying of jobs • Subtle fields hidden in Advanced button • No ability to mimic another team • Change in UI can be slow
- 5. Alternatives • Template Project Plugin from Cloudbees • Job Generator Plugin • Literate Plugin • etc...
- 6. Infrastructure As Code “The end goal of infrastructure as code is to perform as many infrastructure tasks as possible programmatically. Key word is automation." (http://www.somic.org/2012/09/28/concise-introduction-to-infrastructure-as-code/)
- 7. Configuration As Code • Netflix - We understand the value of scaling and automation • Jobs should be responsible citizens • Team can own their scripts
- 8. Job DSL Plugin • Create a Free-form project • Add a “Process Job DSL” build step • Enter Groovy script using a Jenkins DSL • Run job
- 9. Step 2. Add Build Step
- 10. Step 3. Specify Script
- 11. Step 4. Run
- 12. Job DSL Language • Groovy • One additional method, “job” • Documentation in Github Wiki • https://github.com/jenkinsci/job-dsl-plugin/wiki • Job DSL Commands • Job Reference Page
- 13. Single Job from Github job { name "RxJava-Test” scm { github(“git@github.com:Netflix/RxJava.git”) } steps { gradle("test") } }
- 14. Leveraging Groovy import groovy.json.JsonSlurper def project = 'quidryan/aws-sdk-test' def github = 'https://api.github.com' def api = new URL("${github}/repos/${project}/branches") def branches = new JsonSlurper().parse(api.newReader()) branches.each { def branchName = it.name job { name "${project}-${branchName}".replaceAll('/','-') scm { git("git://github.com/${project}.git", branchName) } steps { maven("test -Dproject.name=${project}/${branchName}") } } }
- 15. Disadvantages • Learning Curve • Can get complicated • Not all plugins exist • Possible problems with plugin version skew
- 16. The Project • 408 installs at last check • Active Google Group, with 78 members • Regular Google Hangout • Constant stream of contributors • Core contributors - myself, Daniel Spilker, Stefan Wolf, Andrew Harmel-Law
- 17. Use Case - Opower “Grab json blobs, which define pipeline, from an internal Rails app then loop through with Job DSL. I now have only one job to set up manually. So, in theory, I set up one job and I end up with 600 jobs. Throw each set into a folder and you've got yourself a red/green dashboard.” James Levinson <james.levinson@opower.com>
- 19. Use Case - Chicago Trading “I use the Job DSL to create gerrit build and release build jobs, parameterized on the git repository. Then I use the Build Flow plugin to orchestrate those jobs” Alan Beale <alan.beale@chicagotrading.com>
- 20. Netflix Conventions • Utilities methods to create common jobs • Groovy Category to enhance “job” object • Published as a jar
- 21. Merge Jobs import netflix.Merge Merge.nfMergeJobs( [jobFactory: this, securityGroup: ‘nbs’, base: '//depot/IT_Engineering/NBS/nbsrules', projectPrefix: 'BILLING-rules'] ) Merge.nfMergeJobs( [jobFactory: this, securityGroup: ‘nbs’, base: '//depot/IT_Engineering/NBS/Phoenix', projectPrefix: 'BILLING-phoenix'] )
- 22. Merge Jobs • BILLING-rules-dev-force-to-test • BILLING-rules-test-force-to-prod • BILLING-rules-prod-merge-to-test • BILLING-rules-test-force-to-dev • BILLING-phoenix-dev-force-to-test • BILLING-phoenix-test-force-to-prod • BILLING-phoenix-prod-merge-to-test • BILLING-phoenix-test-force-to-dev
- 23. Jobs as Objects import netflix.* def defaults = [...] def jobs = [‘dev’,’test’,’prod’].collectMany {[ CBFQuickBuild(defaults + [branch:branch]), CBFQualityBuild(defaults + [branch:branch]), nfBakeJob(defaults + [branch:branch]) ]} jobs.each { it.publishers { extendedEmail('jryan@netflix.com', 'Oops') { trigger(triggerName: 'StillUnstable', subject: 'Subject', recipientList:'RecipientList', sendToDevelopers: true, sendToRequester: true, includeCulprits: true, sendToRecipientList: false) } }
- 24. Extra features • Environment variables • Queue jobs to run • Read files workspace • Configure blocks • @Grab • @Category
- 25. @Category use(netflix.Conventions) { def myjob = job { name ‘BuildFromP4’ } myjob.addPerforce("//depot/webapps/astrid/...") }
- 26. Good Practice • Respect “source of truth” • All scripts go in version control • Use configure block if needed • Trigger “seed” job on SCM changes
- 27. Future of project • Plugins provide DSL methods • Produce DSL from real job • Support for utility classes • Produce jobs from build, e.g. Gradle or Maven
- 28. Jenkins User Conference Palo Alto , Oct 23 2013 Thank You To Our Sponsors Platinum Gold Silver #jenkinsconf
- 29. Thank You We’re hiring. Justin Ryan <jryan@netflix.com> @quidryan http://www.slideshare.net/quidryan
Editor's Notes
- Send feedback to @quidryan, presentation will be up later on slideshare, if you miss some link.
- I’d like to start by thanking my co-worker Curt Patrick. While I mostly work on writing the Job DSL Plugin, he has the spent the most time using it and integrating it into teams. Engineering Tools Is part of a larger organization who makes developing for the cloud palatable. Tasked with: Building and Deploying to the cloud Try to: Keep simple, simple. Keep hard, possible. Coupled with Freedom & Responsibility See the Netflix culture slide deck. Or listen to episode 28 of theshipshow.com’s podcast to hear two members of the team explain how F&R work in the context of our team. In general, we can’t necessarily tell people what to do
- We use a Service Orientated Architecture, with lots of little services. There is no one monolithic app to zip together. Each team has their own release schedule, some are short, some are long. Meaning there is not a set of jobs that all pushes go through. The jobs used between teams are “similar", and varying across branches and potentially environments. Think of a matrix of flows, measuring hundreds of projects by three branches. We have 3200 active jobs on our primary master.
- The problem then becomes the complexity. We provide some canonical jobs, and teams will copy among themselves. But in the end these are copies, and we don’t have a way to keep them up to date or effectively share them. We have a lot of plugins on our master, which makes the rendering of the configure screen slow.
- Caveat: Our problems are not necessarily your problems. We are not the teams making jobs for people, but we’re the ones who try to curate and streamline the usage of Jenkins. This gives us a unique prospective. So if I say something disparaging, it might just be because our situation is different than yours. It’s not giving anything away to say that we ended up using Job DSL. But it’s worth pointing out the alternatives at this stage. Since it was what we had to think about without this plugin. I’ll be the first to admit that I’m lazy when evaluating plugins, unless they have lots of screenshots. Template Project Plugin - UI Based, which is hard to scale out to many dozens of teams which all have their own preferences for jobs. The variations we have don’t map to a system being driven by parameters. Their docs state that it’s best when “only differ by a few attributes”, that’s not us. Job Generator Plugin - Good integration aligning input/outputs of jobs. Great for standardized flows, but that’s not us. Literate Plugin - Very new. Solves one of our use-cases which is to tie job creation to the source code. But we’re learned that the minutia of each plugin extremely subtle and important to get right. The idea of checking in specific settings in a subdirectory isn’t very scalable either across projects. Speaking of templates, we found that we don’t use them.
- Recognizing that the problem is about complexity, we can look at other areas which have had to tackle complexity recently. There have been plenty of attempts to wrap complex systems in pretty UIs. UML for coding. There are plenty of “Enterprise” solutions for configuration management, but it’s Chef/Puppet that’s winning that war. In Operations, their world is being up-ended by using Chef/Puppet. It’s making the problem of configuring hundreds if not thousands of systems manageable. Our problem is the same.
- Proper automation has to deal with all the quirks and customizations that real world requires. And to tackle those customizations we write code. Code is code. Build is Code (Ant/Gradle). Infrastructure is Code (Chef). The only missing piece is our build jobs, why are they special? In this model, Jobs are just artifacts of the DSL scripts. We expect any piece of our infrastructure to be a responsible citizen. They should be in SCM, they should get code reviews, re-usability through higher-level methods. Teams write and own the scripts. What they write is isolated from the work of others. There are no global constructs like templates. Meaning that changes are managed by them on their schedule, i.e. they don’t have the take a template change in the middle of a prod push. I’m sold!
- This is accomplished using the Job DSL Plugin. Let me explain the how it’s used. Create a “Seed” job, which will create the other jobs. Groovy runtime’s been instrumented to expose certain domain specific language features while maintaining the language. The specific build step will execute the script and collect a list of jobs to maintain. The general model is you create the ideal situation and the plugin take care of create/update/delete.
- We have 7 SCMs, 6 triggers, 15 build steps, 28 publishers, ~20 wrappers.
- It can be difficult to get started. I find that once there’s a “seed” job to modify, it’s easy to tweak it.Ideally, you could make the job the way you like it, then translate it back to dsl code, but that’s a future feature.Mental mind-shift, given that SEED jobs are jobs themselves.Plugin itself is dependent on contributors to fill in unimplemented plugins and we have a problem with version skew.
- Friendly mailing list and a monthly Google Hangout to let people ask questions. Constant stream of contributors adding miscellaneous plugins as they find a need. I’m always amazed how many plugins are out there. A few contributors stand out as contributing to the core
- sic.
- sic. Showing that the job dsl is good at creating jobs, while other plugins are better at executing them.
- Back to Netflix. We want to be a launching off point for teams, where EngTools can add value and not be gating them. We do that producing a .jar with utility methods and enhancements. We end up with our own language above the job dsl. Users are able to use the DSL directly or leverage what we offer on top of it. Focus on making most complicated problems easier (e.g. nftaskbuilder), while also addressing common problems (branching and merging).
- There’s some internals showing up here, around jobFactory, just ignore them for now. These each create 4 merge jobs, 2 to clobber up from dev to test, test to prod. And 2 merge jobs to merge back from prod to test and test to dev. Making this code even cleaner is left up to the reader.
- Creating 8 standardized jobs, which if changed by EngTools would get updated. We can abstract out the appropriate methods for our use-case.
- Some caveats, we’re using Groovy.Using groovy variables like properties, or variables defined earlier. Groovy strings, aka GStrings, have token replacement.
- addPerforce is a Netflix specific function that suites our needs and environment.
- E.g. Create a builds for every public Netflix project on GitHub. The Github API is the source of truth. Or Store model of your builds somewhere, like a web-app, so that users have a nice UI.Put scripts in VCS, we put them in a shared repo, to encourage teams learning from each other.
- Leave here, just Google “job dsl”, it’ll come right up. This was maybe Job DSL 101, for 102 tune into the Google Hangout to answer all new questions. As you might have guessed, we’re not just your standard Jenkins shop. We’re always looking out for someone who deeply understands Jenkins and can contribute back to the community, as part of their job.