Managing Jenkins with Jenkins (Jenkins User Conference Palo Alto, 2013)

  • 2,737 views
Uploaded on

How we use system Groovy scripts to automate housekeeping and monitoring of our Jenkins instances.

How we use system Groovy scripts to automate housekeeping and monitoring of our Jenkins instances.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Thanks!
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
2,737
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
58
Comments
1
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Managing Jenkins with Jenkins Tweet @garethbowles with feedback! Monday, October 28, 13 Hi, I’m Gareth from Netflix. It’s great to be back for my third JUC ! I’m going to talk about how we use Jenkins’ System Groovy Scripts feature to automate the management of our Jenkins infrastructure. I hope you’ll get inspired by some of these ideas to try them out yourselves. As this is a short talk there may not be much time for questions, but find me afterwards or tweet me if you want to follow up.
  • 2. @ • • • • • • 3200 jobs 3TB build data 6 masters ~100 slaves / ~250 executors 15% build failure rate All 700 engineers can set up / run jobs Tweet @garethbowles with feedback! Monday, October 28, 13 We have a pretty big Jenkins setup, it’s the centrepiece of the pipeline that builds our streaming service. Our main master has 3200 builds, 3TB of build data data We have a growing number of masters as we split jobs between different masters for speed and reliability Housekeeping and monitoring has to be automated due to this scale. We need to make sure our slaves are online, disk space for builds is kept under control, builds run on the right slaves, and be able to update the configuration of many jobs or slaves at a time. And because all our engineers have job configuration access, we need to make sure that some basic conventions are followed and nothing gets too out of control.
  • 3. How can we automate all the Jenkins things ? Tweet @garethbowles with feedback! Monday, October 28, 13 Jenkins has quite a few options for automation.
  • 4. How can we automate all the Jenkins things ? • Remote API - nice, but limited update functions Tweet @garethbowles with feedback! Monday, October 28, 13 Jenkins has quite a few options for automation.
  • 5. How can we automate all the Jenkins things ? • • Remote API - nice, but limited update functions Jenkins CLI - ditto Tweet @garethbowles with feedback! Monday, October 28, 13 Jenkins has quite a few options for automation.
  • 6. How can we automate all the Jenkins things ? • • • Remote API - nice, but limited update functions Jenkins CLI - ditto Custom plugins - powerful but heavyweight Tweet @garethbowles with feedback! Monday, October 28, 13 Jenkins has quite a few options for automation.
  • 7. A Better Option Tweet @garethbowles with feedback! Monday, October 28, 13 Here’s a more lightweight but still very powerful way to automate Jenkins tasks.
  • 8. System Groovy Scripts Tweet @garethbowles with feedback! Monday, October 28, 13 Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain. You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example). Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
  • 9. System Groovy Scripts • Full access to the Jenkins master’s JVM Tweet @garethbowles with feedback! Monday, October 28, 13 Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain. You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example). Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
  • 10. System Groovy Scripts • • Full access to the Jenkins master’s JVM Much quicker than writing a plugin Tweet @garethbowles with feedback! Monday, October 28, 13 Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain. You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example). Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
  • 11. System Groovy Scripts • • • Full access to the Jenkins master’s JVM Much quicker than writing a plugin Make use of Jenkins' job scheduling, logging, notification and data visualization   Tweet @garethbowles with feedback! Monday, October 28, 13 Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain. You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example). Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
  • 12. System Groovy Scripts • • • Full access to the Jenkins master’s JVM • Needs the Groovy plugin: https://wiki.jenkinsci.org/display/JENKINS/Groovy+plugin Much quicker than writing a plugin Make use of Jenkins' job scheduling, logging, notification and data visualization   Tweet @garethbowles with feedback! Monday, October 28, 13 Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to write and maintain. You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing data (with the plot plugin for example). Many of our needs aren’t generic enough to make it worth writing a plugin, anyway. But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it back easily if something goes wrong.
  • 13. Examples Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 14. • Examples Disable old or broken jobs Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 15. • • Examples Disable old or broken jobs Relabel a set of slaves Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 16. • • • Examples Disable old or broken jobs Relabel a set of slaves Monitor slave uptime Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 17. • • • • Examples Disable old or broken jobs Relabel a set of slaves Monitor slave uptime Disconnect bad slaves Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 18. • • • • • Examples Disable old or broken jobs Relabel a set of slaves Monitor slave uptime Disconnect bad slaves Set number of builds to keep Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 19. • • • • • • Examples Disable old or broken jobs Relabel a set of slaves Monitor slave uptime Disconnect bad slaves Set number of builds to keep Check build labels Tweet @garethbowles with feedback! Monday, October 28, 13 Here are a few examples of the work we do with system Groovy scripts. If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs. We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them. We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the slaves (a reconnect usually works, otherwise we leave the slave offline for analysis). We monitor custom build slaves and send notifications to the owners if they get disconnected. We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk space fast). We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave for example.
  • 20. How do I figure out what to write ? Tweet @garethbowles with feedback! Monday, October 28, 13 Figuring out where to start writing a script can be tricky, but there are lots of resources available.
  • 21. Resources Tweet @garethbowles with feedback! Monday, October 28, 13 There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples. You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing ! The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate. We wrote a simple object inspector method that shows you what type of object you’re working with.
  • 22. • Resources Sample scripts - Scriptler plugin (https:// wiki.jenkins-ci.org/display/JENKINS/Scriptler +Plugin) or GitHub repo (https://github.com/ jenkinsci/jenkins-scripts) Tweet @garethbowles with feedback! Monday, October 28, 13 There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples. You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing ! The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate. We wrote a simple object inspector method that shows you what type of object you’re working with.
  • 23. Resources • Sample scripts - Scriptler plugin (https:// wiki.jenkins-ci.org/display/JENKINS/Scriptler +Plugin) or GitHub repo (https://github.com/ jenkinsci/jenkins-scripts) • Script console in Jenkins - be careful ! Tweet @garethbowles with feedback! Monday, October 28, 13 There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples. You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing ! The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate. We wrote a simple object inspector method that shows you what type of object you’re working with.
  • 24. Resources • Sample scripts - Scriptler plugin (https:// wiki.jenkins-ci.org/display/JENKINS/Scriptler +Plugin) or GitHub repo (https://github.com/ jenkinsci/jenkins-scripts) • • Script console in Jenkins - be careful ! API JavaDoc: http://javadoc.jenkins-ci.org/ overview-summary.html Tweet @garethbowles with feedback! Monday, October 28, 13 There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples. You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing ! The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate. We wrote a simple object inspector method that shows you what type of object you’re working with.
  • 25. Resources • Sample scripts - Scriptler plugin (https:// wiki.jenkins-ci.org/display/JENKINS/Scriptler +Plugin) or GitHub repo (https://github.com/ jenkinsci/jenkins-scripts) • • Script console in Jenkins - be careful ! • Object inspector method API JavaDoc: http://javadoc.jenkins-ci.org/ overview-summary.html Tweet @garethbowles with feedback! Monday, October 28, 13 There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples. You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing ! The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate. We wrote a simple object inspector method that shows you what type of object you’re working with.
  • 26. Resources • Sample scripts - Scriptler plugin (https:// wiki.jenkins-ci.org/display/JENKINS/Scriptler +Plugin) or GitHub repo (https://github.com/ jenkinsci/jenkins-scripts) • • Script console in Jenkins - be careful ! • Object inspector method API JavaDoc: http://javadoc.jenkins-ci.org/ overview-summary.html Tweet @garethbowles with feedback! Monday, October 28, 13 There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples. You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing ! The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate. We wrote a simple object inspector method that shows you what type of object you’re working with.
  • 27. Object Inspector Method Tweet @garethbowles with feedback! Monday, October 28, 13 Here’s the handy object inspector method that display an object’s type, properties and methods. Once yuo know the type you can look in the JavaDoc or search existing code to find out how to do what you want.
  • 28. Be careful about testing ! Tweet @garethbowles with feedback! Monday, October 28, 13 System Groovy scripts can do some serious damage ! You often want to loop over every job or build or slave, so be careful to test your code. I learned the hard way when I accidentally kicked off every job on our main Jenkins master. We’ve made a few other mistakes too.
  • 29. Tweet @garethbowles with feedback! Monday, October 28, 13 Here’s another one where we accidentally disabled all the jobs in the system and then had to figure out how to identify the jobs that should really be disabled. We keep this one around as a cautionary tale.
  • 30. Developing scripts Tweet @garethbowles with feedback! Monday, October 28, 13 Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves. For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets ! Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
  • 31. Developing scripts • Use a local Jenkins instance (java -jar jenkins.war) Tweet @garethbowles with feedback! Monday, October 28, 13 Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves. For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets ! Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
  • 32. Developing scripts • Use a local Jenkins instance (java -jar jenkins.war) • Use the script console or inline code in the job config. while developing Tweet @garethbowles with feedback! Monday, October 28, 13 Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves. For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets ! Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
  • 33. Developing scripts • Use a local Jenkins instance (java -jar jenkins.war) • Use the script console or inline code in the job config. while developing • Code editor or IDE for longer scripts Tweet @garethbowles with feedback! Monday, October 28, 13 Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves. For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets ! Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
  • 34. Developing scripts • Use a local Jenkins instance (java -jar jenkins.war) • Use the script console or inline code in the job config. while developing • • Code editor or IDE for longer scripts Check the script into SCM when it’s ready Tweet @garethbowles with feedback! Monday, October 28, 13 Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves. For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code completion - way fewer missing brackets ! Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a backup of the code outside of Jenkins.
  • 35. Awesome ! Where can I get this stuff ? • • https://github.com/jenkinsci/jenkins-scripts http://scriptlerweb.appspot.com/catalog/list Tweet @garethbowles with feedback! Monday, October 28, 13 Here’s where you can get lots of System Groovy script goodness. I just added a bunch of our scripts that I mentioned in this talk.
  • 36. Jenkins User Conference Palo Alto , Oct 23 2013 Thank You To Our Sponsors Platinum Gold Silver Tweet @garethbowles with feedback! Monday, October 28, 13 #jenkinsconf
  • 37. Thank You ! Email: gbowles@{gmail,netflix}.com Twitter: @garethbowles Linkedin: www.linkedin.com/in/garethbowles Want to do Jenkins development for your day job ? We’re hiring ! http:// jobs.netflix.com Tweet @garethbowles with feedback! Monday, October 28, 13 Here’s how to get in touch. We’re hiring for lots of positions, but in particular we’re looking for someone to own Jenkins at Netflix - all the way from keeping the infrastructure running to writing custom plugins and fixing the Jenkins core. If you want to take Jenkins in new directions as your full time job - with a heavy dose of cloud thrown in - come and find a Netflixer. thanks again