Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 1 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 2 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 3 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 4 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 5 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 6 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 7 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 8 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 9 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 10 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 11 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 12 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 13 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 14 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 15 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 16 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 17 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 18 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 19 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 20 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 21 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 22 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 23 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 24 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 25 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 26 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 27 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 28 Configuration As Code - Adoption of the Job DSL Plugin at Netflix Slide 29
Upcoming SlideShare
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)
Next
Download to read offline and view in fullscreen.

34 Likes

Share

Download to read offline

Configuration As Code - Adoption of the Job DSL Plugin at Netflix

Download to read offline

The Jenkins Job DSL plugin allows programmers to express job configurations as code. Learn about the benefits, from the obvious (store your configurations in the SCM of your choice) to the not-so-obvious (focus on intent, instead of succumbing to the distraction of multiple, complex job configuration options). We will share our experience adopting the plugin over the past year to create and maintain more complex job pipelines at Netflix.

Related Books

Free with a 30 day trial from Scribd

See all

Configuration As Code - Adoption of the Job DSL Plugin at Netflix

  1. 1. Configuration as Code Adoption of the Job DSL Plugin at Netflix Justin Ryan <jryan@netflix.com> @quidryan http://www.slideshare.net/quidryan
  2. 2. Who Are We • Engineering Tools • Justin Ryan / jryan@netflix.com / @quidryan • Curt Patrick / cpatrick@netflix.com • Freedom & Responsibility
  3. 3. Situation • No single flow that builds everything • Every team has their own jobs • Each branch needs multiple jobs
  4. 4. Problem • Lots of copying of jobs • Subtle fields hidden in Advanced button • No ability to mimic another team • Change in UI can be slow
  5. 5. Alternatives • Template Project Plugin from Cloudbees • Job Generator Plugin • Literate Plugin • etc...
  6. 6. Infrastructure As Code “The end goal of infrastructure as code is to perform as many infrastructure tasks as possible programmatically. Key word is automation." (http://www.somic.org/2012/09/28/concise-introduction-to-infrastructure-as-code/)
  7. 7. Configuration As Code • Netflix - We understand the value of scaling and automation • Jobs should be responsible citizens • Team can own their scripts
  8. 8. Job DSL Plugin • Create a Free-form project • Add a “Process Job DSL” build step • Enter Groovy script using a Jenkins DSL • Run job
  9. 9. Step 2. Add Build Step
  10. 10. Step 3. Specify Script
  11. 11. Step 4. Run
  12. 12. Job DSL Language • Groovy • One additional method, “job” • Documentation in Github Wiki • https://github.com/jenkinsci/job-dsl-plugin/wiki • Job DSL Commands • Job Reference Page
  13. 13. Single Job from Github job { name "RxJava-Test”  scm {   github(“git@github.com:Netflix/RxJava.git”)  }  steps {   gradle("test")  } }
  14. 14. Leveraging Groovy import groovy.json.JsonSlurper def project = 'quidryan/aws-sdk-test' def github = 'https://api.github.com' def api = new URL("${github}/repos/${project}/branches") def branches = new JsonSlurper().parse(api.newReader()) branches.each {    def branchName = it.name    job {        name "${project}-${branchName}".replaceAll('/','-')        scm {            git("git://github.com/${project}.git", branchName)        }        steps {            maven("test -Dproject.name=${project}/${branchName}")        }    } }
  15. 15. Disadvantages • Learning Curve • Can get complicated • Not all plugins exist • Possible problems with plugin version skew
  16. 16. The Project • 408 installs at last check • Active Google Group, with 78 members • Regular Google Hangout • Constant stream of contributors • Core contributors - myself, Daniel Spilker, Stefan Wolf, Andrew Harmel-Law
  17. 17. Use Case - Opower “Grab json blobs, which define pipeline, from an internal Rails app then loop through with Job DSL. I now have only one job to set up manually. So, in theory, I set up one job and I end up with 600 jobs. Throw each set into a folder and you've got yourself a red/green dashboard.” James Levinson <james.levinson@opower.com>
  18. 18. Use Case - Chicago Trading “I use the Job DSL to create gerrit build and release build jobs, parameterized on the  git repository. Then I use the Build Flow plugin to orchestrate those jobs” Alan Beale <alan.beale@chicagotrading.com>
  19. 19. Netflix Conventions • Utilities methods to create common jobs • Groovy Category to enhance “job” object • Published as a jar
  20. 20. Merge Jobs import netflix.Merge Merge.nfMergeJobs( [jobFactory: this, securityGroup: ‘nbs’,    base: '//depot/IT_Engineering/NBS/nbsrules',    projectPrefix: 'BILLING-rules'] ) Merge.nfMergeJobs( [jobFactory: this, securityGroup: ‘nbs’,    base: '//depot/IT_Engineering/NBS/Phoenix',    projectPrefix: 'BILLING-phoenix'] )
  21. 21. Merge Jobs • BILLING-rules-dev-force-to-test • BILLING-rules-test-force-to-prod • BILLING-rules-prod-merge-to-test • BILLING-rules-test-force-to-dev • BILLING-phoenix-dev-force-to-test • BILLING-phoenix-test-force-to-prod • BILLING-phoenix-prod-merge-to-test • BILLING-phoenix-test-force-to-dev
  22. 22. Jobs as Objects import netflix.* def defaults = [...] def jobs = [‘dev’,’test’,’prod’].collectMany {[ CBFQuickBuild(defaults + [branch:branch]), CBFQualityBuild(defaults + [branch:branch]), nfBakeJob(defaults + [branch:branch]) ]} jobs.each {    it.publishers {        extendedEmail('jryan@netflix.com', 'Oops') {            trigger(triggerName: 'StillUnstable', subject: 'Subject', recipientList:'RecipientList',                 sendToDevelopers: true, sendToRequester: true, includeCulprits: true, sendToRecipientList: false)    } }
  23. 23. Extra features • Environment variables • Queue jobs to run • Read files workspace • Configure blocks • @Grab • @Category
  24. 24. @Category use(netflix.Conventions) { def myjob = job { name ‘BuildFromP4’ }  myjob.addPerforce("//depot/webapps/astrid/...") }
  25. 25. Good Practice • Respect “source of truth” • All scripts go in version control • Use configure block if needed • Trigger “seed” job on SCM changes
  26. 26. Future of project • Plugins provide DSL methods • Produce DSL from real job • Support for utility classes • Produce jobs from build, e.g. Gradle or Maven
  27. 27. Jenkins User Conference Palo Alto , Oct 23 2013 Thank You To Our Sponsors Platinum Gold Silver #jenkinsconf
  28. 28. Thank You We’re hiring. Justin Ryan <jryan@netflix.com> @quidryan http://www.slideshare.net/quidryan
  • MaximIvanitskiy

    Jan. 31, 2018
  • danielgalassi

    Jan. 26, 2017
  • angaur

    Oct. 11, 2016
  • yilingredlee

    Aug. 31, 2016
  • TannerWatson

    Jun. 11, 2016
  • biztalk72

    May. 10, 2016
  • rinostilo

    Mar. 10, 2016
  • PhillipShannon

    Feb. 12, 2016
  • MichaelNeufeld

    Nov. 17, 2015
  • lepffm

    Nov. 4, 2015
  • justinschmidt80

    Sep. 19, 2015
  • ricardogarfe

    Sep. 1, 2015
  • MichaelBaran1

    Aug. 21, 2015
  • nmunson

    Aug. 7, 2015
  • bunkertor

    Jul. 14, 2015
  • larrycai

    Jul. 14, 2015
  • alexis.tejeda

    Mar. 2, 2015
  • SylvainLeroy

    Feb. 17, 2015
  • jacobdanner

    Oct. 27, 2014
  • chelusly

    Oct. 6, 2014

The Jenkins Job DSL plugin allows programmers to express job configurations as code. Learn about the benefits, from the obvious (store your configurations in the SCM of your choice) to the not-so-obvious (focus on intent, instead of succumbing to the distraction of multiple, complex job configuration options). We will share our experience adopting the plugin over the past year to create and maintain more complex job pipelines at Netflix.

Views

Total views

68,134

On Slideshare

0

From embeds

0

Number of embeds

48,728

Actions

Downloads

130

Shares

0

Comments

0

Likes

34

×