Jenkins talk at Silicon valley DevOps meetup


Published on

Kohsuke Kawaguchi's Jenkins talk at 2011 June Silicon Valley DevOps meetup.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Jenkins talk at Silicon valley DevOps meetup

  1. 1. Jenkins<br />
  2. 2. What’s Jenkins?<br />©2010 CloudBees, Inc. All Rights Reserved<br />
  3. 3. What’s Jenkins?<br />Jenkins is a “continuous integration” server<br />It looks for changes and builds your project<br />It publishes the result<br />It keeps track of how your project is used elsewhere<br />It brings transparency to projects<br />3<br />
  4. 4. What’s Jenkins?<br />Community-driven open-source project<br /><br />Used to be called Hudson<br />Get going with “java -jar jenkins.war”<br />Extensible architecture<br />Lots of community developed plugins<br />Public plugins as well as in-house plugins<br />Very active<br />7 years old, ~400 releases to date<br />300+ plugins, 250+ committers<br />4<br />
  5. 5. Used Everywhere<br />5<br />5<br />
  6. 6. What does Jenkins do?<br />Notice changes and checking out<br />Polling, post-commit hooks<br />ClearCase, Perforce, Git, Mercurial, …<br />Runs builds and tests<br />Lots of build/reporting tools integrations<br />6<br />
  7. 7. What does Jenkins do?<br />Whatever you can script can be run in Jenkins<br />It needs to be non-interactive<br />Very flexible and can work with any build/test tools/frameworks<br />But it is not by itself a build/test framework<br />7<br />
  8. 8. Non-Java Suport<br />Jenkins is used to build non-Java projects<br />Ruby, Python, PHP, C++, .NET, …<br />Rapidly expanding as many of those languages do not have viable CI server<br />Various tool/report integrations done by community<br />8<br />
  9. 9. Demo<br />©2010 CloudBees, Inc. All Rights Reserved<br />
  10. 10. Why do I care?<br />Life before Jenkins<br />RE nightly build breaks. It blocks SQE<br />Dev rebuilds before commit to avoid this<br />People waste time<br />Life after Jenkins<br />Dev just commits it right away<br />Dev moves on to work on another thing<br />Jenkins will tell you in 5 mins if a build broke<br />10<br />
  11. 11. Why do I care?<br />Life before Jenkins<br />Dev makes a change Monday AM<br />SQE nightly finds a bug Monday night<br />Dev fixes it Tuesday<br />Lot of wasted time<br />Life after Jenkins<br />Dev makes a change Monday AM<br />Jenkins finds a regression 30 mins later<br />Dev can fix it before lunch<br />11<br />
  12. 12. Why do I care?<br />Life before Jenkins<br />SQE test runs every night, results sent out in e-mail<br />After the enthusiasm of the 1st week is gone, nobody looks at them anymore<br />Regressions go unnoticed until it’s too late<br />Life after Jenkins<br />Tests run Jenkins after every commit<br />E-mail sent out only when tests start failing<br />So it manages to keep people’s attention<br />12<br />
  13. 13. Got the idea?<br />Automation<br />Reduce turn-around time<br />Make things transparent<br />Remove people from the loop<br />Save people’s time<br />Push jobs to servers, keep workstations idle for you<br />13<br />
  14. 14. Advanced Integration Techniques<br />©2010 CloudBees, Inc. All Rights Reserved<br />
  15. 15. Doing Distributed Builds<br />15<br />Photo by skreuzer<br />
  16. 16. Why?<br />You need to use multiple computers because…<br />You need different environments<br />You need isolation<br />You have non-trivial work loads<br />There’s only so much you can do with 1 computer<br />
  17. 17. Lots of computers = tedious<br />Pains grow as your build farm grows<br />Software installation problem<br />Node failure problem<br />Remote maintenance problem<br />…<br />What does Jenkins do to help you with these?<br />
  18. 18. Installing new slaves<br />For first 20 or so slaves, we did it manually<br />Insert CD, click, type, click, type, click, …<br />But that doesn’t scale<br />Then we automated<br />Available as “Jenkins PXE Plugin”<br />
  19. 19. Automated System Installations<br /><ul><li>Jenkins + PXEplugin
  20. 20. ISO images of OS</li></ul>Your corporate IT guy & his DHCP server<br />Slaves<br />Power on, hit F12<br />PC boots from network (PXE)<br />
  21. 21. Automated System Installations<br /><ul><li>Jenkins+ PXEplugin
  22. 22. ISO images of OS</li></ul>Your corporate IT guy & his DHCP server<br />Slaves<br />Power on, hit F12<br />PC boots from network (PXE)<br />Choose OS from menu<br />Installs non-interactively<br />
  23. 23. Automated System Installations<br />Supports OpenSolaris, Ubuntu, CentOS, Fedora<br />Trivial with most Linux<br />Cooperate with Windows, too<br />Quite useful outside Jenkins, too<br />No more broken CD drives<br />No more CD-Rs<br />
  24. 24. Master<br />Serves HTTP requests<br />Stores all important info<br />Slaves<br />170KB single jar<br />Assumed to be unreliable<br />Scale to at least 100<br />Distributed builds with Jenkins<br />
  25. 25. Building on slaves<br />Slave agent is a little Java program<br />Runs locally on the machine that it controls<br />Access files, fork processes, etc., on behalf of master<br />Communication with master<br />Needs a bi-di byte stream<br />But very flexible in how this gets set up<br />No shared file system, no network topology constraints, etc<br />
  26. 26. How master and slaves start talking<br />For Unix slaves, via SSH<br />Only need SSHD on slaves<br />We even install Java for you<br />We just need a host name<br />
  27. 27. How master and slaves start talking<br />For Windows, DCOM<br />We just need admin user name and password<br />No manual intervention<br />Works even from Unix masters<br />
  28. 28. How master and slaves start talking<br />Via Java Web Start<br />When master cannot see slaves<br />A separate socket connection is made<br />
  29. 29. Automating JNLP launch<br />Once started, can be installed as Windows service<br />
  30. 30. Automating JNLP launch<br />Emulate the JNLP client headlessly<br />$ java -jar slave.jar -jnlpUrl URL<br />
  31. 31. Automated Tool Installations<br />JDK from<br />It automatically chooses the right bundle<br />
  32. 32. Automated Tool Installations<br />Ant and Maven from Apache<br />General purpose implementations<br />Grab an archive from some URL and extract<br />Run arbitrary shell script<br />Extensible<br />
  33. 33. Labels<br />Tags that group slaves together<br />Name: alpha<br />Label: windows 32bit<br />Name: bravo<br />Label: linux 32bit<br />Name: bravo<br />Label: linux 64bit<br />
  34. 34. Labels<br />Name: alpha<br />Label: windows 32bit<br />Name: bravo<br />Label: linux 32bit<br />This job needs to run on “linux”<br />Name: bravo<br />Label: linux 64bit<br />
  35. 35. Labels<br />Name: alpha<br />Label: windows 32bit<br />Name: bravo<br />Label: linux 32bit<br />This job needs to run on “linux && 32bit”<br />Name: bravo<br />Label: linux 64bit<br />
  36. 36. More Label Benefits<br />Labels can be anything<br />Geographic locations<br />Availability of proprietary software installed<br />Better resource utilization<br />Jenkins have maximum freedom<br />Resilient to node addition/removal<br />Treat servers like livestocks, not like pets<br />
  37. 37. Making builds sticky<br />We want jobs to be mostly on the same slave<br />Faster check out<br />Consistent results<br />Minimizes disk consumption<br />But do it softly<br />Jenkins uses consistent hash to achieve this<br />More schedule controls become possible:<br />Use faster machines more frequently<br />Slowly ramp up newly installed slaves<br />*<br />
  38. 38. Forecasting failures<br />Jenkins monitors key health metrics of slaves<br />Low disk space, insufficient swap<br />Clock out of sync<br />Extensible<br />Slaves put offline automatically<br />Catch problems before it breaks builds<br />36<br />
  39. 39. Clean up mess after builds<br />Kill run-away processes<br />Daemons, background processes left by your build<br />Works on Windows, Linux, Mac, and Solaris<br />
  40. 40. Cloud support<br />Slaves can come and go<br />Current/future workload and queue are all monitored continuously<br />We can let Jenkins provision slaves on cloud on-the-fly<br />
  41. 41. Jenkins EC2 plugin<br />Automatically provision slaves on EC2 on demand<br />Pick the right AMI depending on demand<br />Start slave agent<br />Shut down unused instances<br />*<br />
  42. 42. Integration with VMWare<br />Similar mechanism exists with VMWare<br />Create a pool of virtual machines upfront<br />Jenkins powers them up/down based on load<br />Optionally revert to snapshot at the end<br />Achieves benefits similar to EC2 without going public<br />Nectar<br />Feature<br />
  43. 43. Matrix Project<br />Photo:<br />41<br />
  44. 44. Motivation<br />You often do the same thing with slight variations<br />Compile C++ code for different platforms<br />Test with different browsers<br />Or more generally, think of it as<br />for (x in [a,b,c]) {<br /> for (y in [d,e,f]) {<br /> …<br />doSomethingWith(x,y,z,..);<br /> …<br /> }<br />}<br />
  45. 45. Model<br />Define axes<br />One axis ≈ one for loop<br />Choose from pre-defined types of Axis<br />Slave axis: pick slaves by their names or their labels<br />e.g., linux, solaris, and windows<br />Generic axis: arbitrary values exposed as environment variables<br />
  46. 46. Matrix Project Gimmicks<br />Filtering<br />Otherwise combinations increase exponentially<br />Not all combinations make sense<br />Use boolean expression to trim down the size<br />Or tell Jenkins to cut the workload to N%<br />Jenkins will thin out the combinations by itself<br />(label==“windows”).implies(browser==“iexplore”) &&(label==“mac”).implies(browser==“safari”)<br />
  47. 47. Demo<br />
  48. 48. Build Promotion<br />
  49. 49. Progressive Exposure<br />AKA Build pipeline<br />Once built, bits go through progressively more expensive “certification”<br />Think of tests, deployment to QA, staging, etc.<br />Some builds fail in the middle, others go through<br />
  50. 50. Then it gets more complex<br />Each stage can be several parallel jobs<br />Needs to be retryable<br />You can’t afford to execute every pipeline until it fails<br />
  51. 51. Hence “build promotion”<br />Configuration<br />“Foo gets promoted if FooTest passes”<br />“When Foo gets promoted, do X, Y, and Z”<br />Sequence of events<br />Foo #N successfully completes<br />FooTest #M executes on Foo #N<br />FooTest #M successfully completes<br />Foo #N gets promoted<br />X, Y, and Z happens on Foo #N<br />
  52. 52. More on build promotion<br />Promotion condition can be more complex<br />If FooTest, BarTest, and ZotTest all passes<br />If PHB manually approves<br />If upstream project gets promoted<br />Extensible<br />Promotion actions can be anything<br />Copy files, execute scripts, trigger jobs, …<br />
  53. 53. More on build promotion<br />Multiple different promotion levels<br />Foo #5<br />FooIntTest #9<br />FooIntTest #10<br />FooTest #10<br />BarTest #16<br />Deploy #19<br />
  54. 54. Promotion benefits<br />No rebuilding of binaries<br />Asynchronous and naturally throttled<br />Audit Trail<br />
  55. 55. Demo<br />