HudsonSF JUG<br />Kohsuke Kawaguchi<br />Sun Microsystems, Inc.<br />
Rise of Continuous Integration<br />Offload from people, push to computers<br />$<br />computers<br />people<br />time<br ...
Spend more CPU power to help you<br />First on your laptops and workstations<br />IDEs are at the forefront<br />And then ...
Hudson<br />Open-source CI server<br />Emphasis on ease of installation and use<br />“java -jar hudson.war” execution<br /...
Adoption in all kinds of businesses<br />5<br />
Support Subscriptions<br />Community releases every week<br />Over 340 releases to date<br />Rapid new features & bug fixe...
Basic Feature Set<br />Hudson notices that there is a change<br />Check out the source code<br />CVS, Subversion, ClearCas...
Demo<br />
Why do I care?<br />Life before Hudson<br />Dev forgets to commit a file<br />RE nightly build breaks. It blocks SQE<br />...
Why do I care?<br />Life before Hudson<br />Dev makes a change Monday AM<br />SQE nightly finds a bug Monday night<br />De...
Why do I care?<br />Life before Hudson<br />SQE test runs every night, results sent out in e-mail<br />After the enthusias...
Matrix projects<br />Run the same thing on different environments<br />such as different JDKs, different databases, and di...
FindBugs integration<br />13<br />
Tracking changes<br />14<br />
… and more<br />Browse workspaces<br />Build time trend report<br />15<br />
Inter-Project Support<br />You can host many projects on 1 Hudson<br />And when you do, 1+1 becomes more than 2<br />You c...
Test/Build Separation<br />Tests often dominate the build time<br />Multiple tests are run in parallel<br />Improves turn-...
Test/Build Separation Technique<br />Copy bits from the build to test via wget by permalinks<br />Ant has the equivalent &...
Build Promotion<br />CI produces a lot of “successful” builds<br />Often overwhelming to downstream consumers<br />Run tes...
Got the idea?<br />Automation<br />Reduce turn-around time<br />Make things transparent<br />Remove people from the loop<b...
Doing Distributed Builds<br />21<br />
Why?<br />There’s only so much 1 computer can do<br />But lots of PCs = tedious<br />What does Hudson do to help you?<br /...
Installing new slaves<br />For first 20 or so slaves, we did it manually<br />Insert CD, click, type, click, type, click, ...
Automated System Installations<br /><ul><li>Hudson + PXEplugin
ISO images of OS</li></ul>Slaves<br />Power on, hit F12<br />PC boots from network (PXE)<br />24<br />
Automated System Installations<br /><ul><li>Hudson + PXEplugin
ISO images of OS</li></ul>Your corporate IT guy & his DHCP server<br />Slaves<br />Power on, hit F12<br />PC boots from ne...
Distributed builds with Hudson<br />Master<br />Serves HTTP requests<br />Stores all important info<br />Slaves<br />170KB...
How master and slaves start talking<br />For Unix slaves, via SSH<br />Only need SSHD and JRE on slaves<br />We just need ...
How master and slaves start talking<br />For Windows, DCOM<br />We just need admin user name and password<br />No manual i...
How master and slaves start talking<br />Via Java Web Start<br />When master cannot see slaves<br />A separate socket conn...
Automating JNLP launch<br />Once started, can be installed as Windows service<br />30<br />
Automating JNLP launch<br />Emulate the JNLP client headless<br />$ java -jar slave.jar -jnlpUrl URL<br />31<br />
Automated Tool Installation - JDK<br />JDK from http://java.sun.com/<br />Hudson automatically chooses the right bundle<br...
Automated Tool Installation - Apache<br />Ant and Maven from Apache<br />33<br />
Heterogeneous Cluster Challenge<br />Your builds/tests need to run in specific environment<br />Dependency on individual n...
Labels to rescue<br />Label is a group of slaves<br />Tie jobs to labels<br />jobs<br />slaves<br />WombatWindows test<br ...
Concurrent Builds<br />36<br />Hudson #10<br />Hudson #11<br />Hudson #12<br />Hudson #13<br />Hudson #14<br />Hudson #15<...
Making builds sticky<br />We want jobs to be mostly on the same slave<br />Faster check out<br />Consistent results<br />M...
Forecasting failures<br />Hudson monitors key health metrics of slaves<br />Low disk space, insufficient swap<br />Clock o...
Clean up mess after builds<br />Kill run-away processes<br />Daemons, background processes left by your build<br />Works o...
Load Statistics Monitoring<br />40<br />
When it’s time to add more slaves<br />41<br />
Hudson EC2 plugin<br />Automatically provisions slaves on EC2 on demand<br />Based on the current load<br />Picks the righ...
Putting it all together<br />capacity<br /># of executors<br />usage<br />queue  length<br />time<br />43<br />
Hudson “Appliance” on EC2<br />Run the master in the cloud too, if you like<br />Hudson on stock OpenSolaris AMI<br />Data...
And a lot more I couldn’t show<br />IDE plugins<br />iPhone/Android apps<br />REST API and CLI<br />Deployment Automation<...
Conclusions<br />CI is here to stay<br />Using lots of PCs is a key<br />Hudson is very easy to get started<br />Once star...
Upcoming SlideShare
Loading in...5
×

Learn About Continuous Integration With Hudson Directly From the Source

8,127

Published on

San Francisco Java User Group presents Kohsuke Kawaguchi from Sun who introduces us to Hudson, an open-source continuous integration (CI) system, which improves the productivity of a development team by automating various things.

Hosted By: SUPINFO International University
Sponsored By: TEKsystems, Sun, O'Reilly, JetBrains, and Marakana
Organized By: Marakana
Video By: Max Walker

Video available at http://marakana.com/forums/java/general/79.html

Published in: Education
1 Comment
13 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,127
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
898
Comments
1
Likes
13
Embeds 0
No embeds

No notes for slide
  • Plugins --- up 80 from last year in JavaOne, contributers – up 70 from last year
  • Demo cheat sheet:Ant project build. Build unit test. Polling.Maven project. Auto configuration. Javadoc.Plugin manager, update center
  • We’ve seen that how Hudson implements different launch protocol for Windows and slaves, but that’s not the only place where we take heterogeneous environments seriously.
  • Transcript of "Learn About Continuous Integration With Hudson Directly From the Source"

    1. 1. HudsonSF JUG<br />Kohsuke Kawaguchi<br />Sun Microsystems, Inc.<br />
    2. 2. Rise of Continuous Integration<br />Offload from people, push to computers<br />$<br />computers<br />people<br />time<br />2<br />
    3. 3. Spend more CPU power to help you<br />First on your laptops and workstations<br />IDEs are at the forefront<br />And then to the servers<br />a.k.a. “Continuous Integration”<br />More frequent build/test executions<br />Static code analysis tools<br />And more to come<br />3<br />
    4. 4. Hudson<br />Open-source CI server<br />Emphasis on ease of installation and use<br />“java -jar hudson.war” execution<br />Or your OS-specific package<br />Configure everything from browsers<br />Extensibility<br />230+ community-developed public plugins<br />By 230+ contributors<br />Estimated 13,000 installations<br />4<br />https://hudson-ci.org/<br />
    5. 5. Adoption in all kinds of businesses<br />5<br />
    6. 6. Support Subscriptions<br />Community releases every week<br />Over 340 releases to date<br />Rapid new features & bug fixes delivery<br />“Sun Continuous Integration Server”<br />For enterprise that needs stability<br />Bug fix only patch releases for 18 months<br />6<br />
    7. 7. Basic Feature Set<br />Hudson notices that there is a change<br />Check out the source code<br />CVS, Subversion, ClearCase, Mercurial, Accurev, Perforce, Git, StarTeam, …<br />Do builds<br />Ant, Maven, shell script, NAnt, …<br />Record and publish results<br />Close the feedback loop<br />RSS feeds<br />E-mail/IM notification<br />Tray application to notify developers<br />Tool integration<br />Findbugs, cobertura, …<br />7<br />
    8. 8. Demo<br />
    9. 9. Why do I care?<br />Life before Hudson<br />Dev forgets to commit a file<br />RE nightly build breaks. It blocks SQE<br />People waste time<br />Life after Hudson<br />Hudson will tell you in 5 mins if a build broke<br />No one else notices that you forgot to commit a file<br />9<br />
    10. 10. Why do I care?<br />Life before Hudson<br />Dev makes a change Monday AM<br />SQE nightly finds a bug Monday night<br />Dev fixes it Tuesday<br />Lot of wasted time<br />Life after Hudson<br />Dev makes a change Monday AM<br />Hudson finds a regression 30 mins later<br />Dev can fix it before lunch<br />10<br />
    11. 11. Why do I care?<br />Life before Hudson<br />SQE test runs every night, results sent out in e-mail<br />After the enthusiasm of the 1st week is gone, nobody looks at them anymore<br />Regressions go unnoticed until it’s too late<br />Life after Hudson<br />Tests run Hudson after every commit<br />E-mail sent out only when tests start failing<br />So it manages to keep people’s attention<br />11<br />
    12. 12. Matrix projects<br />Run the same thing on different environments<br />such as different JDKs, different databases, and different OSes<br />Aggregate the results<br />12<br />
    13. 13. FindBugs integration<br />13<br />
    14. 14. Tracking changes<br />14<br />
    15. 15. … and more<br />Browse workspaces<br />Build time trend report<br />15<br />
    16. 16. Inter-Project Support<br />You can host many projects on 1 Hudson<br />And when you do, 1+1 becomes more than 2<br />You can…<br />Execute tests separately and correlate them with builds<br />Mark builds as “promoted” based on test outcome<br />Track which build of libraries are used where<br />And more<br />16<br />
    17. 17. Test/Build Separation<br />Tests often dominate the build time<br />Multiple tests are run in parallel<br />Improves turn-around time<br />time<br />JAX-WS #10<br />JAX-WS #11<br />JAX-WS #12<br />…<br />Unit test with GF<br />Unit test with Tomcat<br />SQE test<br />Test w/o container<br />3OS x 2JDK = 6 tests<br />17<br />
    18. 18. Test/Build Separation Technique<br />Copy bits from the build to test via wget by permalinks<br />Ant has the equivalent &lt;get&gt; task<br />“Fingerprinting” to keep track of where binaries are used<br />$ wget –O foo.zip<br /> http://server/hudson/job/foo/lastSuccessfulBuild/ artifact/foo.zip<br />18<br />
    19. 19. Build Promotion<br />CI produces a lot of “successful” builds<br />Often overwhelming to downstream consumers<br />Run tests as fast as you can<br />If a build passes tests, promote it<br />19<br />
    20. 20. Got the idea?<br />Automation<br />Reduce turn-around time<br />Make things transparent<br />Remove people from the loop<br />Save people’s time<br />Push jobs to servers, keep workstations idle for you<br />20<br />
    21. 21. Doing Distributed Builds<br />21<br />
    22. 22. Why?<br />There’s only so much 1 computer can do<br />But lots of PCs = tedious<br />What does Hudson do to help you?<br />22<br />
    23. 23. Installing new slaves<br />For first 20 or so slaves, we did it manually<br />Insert CD, click, type, click, type, click, …<br />But that doesn’t scale<br />Then we automated<br />Available as “Hudson PXE Plugin”<br />23<br />
    24. 24. Automated System Installations<br /><ul><li>Hudson + PXEplugin
    25. 25. ISO images of OS</li></ul>Slaves<br />Power on, hit F12<br />PC boots from network (PXE)<br />24<br />
    26. 26. Automated System Installations<br /><ul><li>Hudson + PXEplugin
    27. 27. ISO images of OS</li></ul>Your corporate IT guy & his DHCP server<br />Slaves<br />Power on, hit F12<br />PC boots from network (PXE)<br />Choose OS from menu<br />Installs non-interactively<br />25<br />
    28. 28. Distributed builds with Hudson<br />Master<br />Serves HTTP requests<br />Stores all important info<br />Slaves<br />170KB single JAR<br />Assumed to be unreliable<br />Scale to at least 100<br />Link<br />Single bi-di byte stream<br />No other requirements<br />26<br />
    29. 29. How master and slaves start talking<br />For Unix slaves, via SSH<br />Only need SSHD and JRE on slaves<br />We just need a host name<br />27<br />
    30. 30. How master and slaves start talking<br />For Windows, DCOM<br />We just need admin user name and password<br />No manual intervention<br />Works even from Unix masters<br />28<br />
    31. 31. How master and slaves start talking<br />Via Java Web Start<br />When master cannot see slaves<br />A separate socket connection is made<br />29<br />
    32. 32. Automating JNLP launch<br />Once started, can be installed as Windows service<br />30<br />
    33. 33. Automating JNLP launch<br />Emulate the JNLP client headless<br />$ java -jar slave.jar -jnlpUrl URL<br />31<br />
    34. 34. Automated Tool Installation - JDK<br />JDK from http://java.sun.com/<br />Hudson automatically chooses the right bundle<br />Always up to date with new releases<br />32<br />
    35. 35. Automated Tool Installation - Apache<br />Ant and Maven from Apache<br />33<br />
    36. 36. Heterogeneous Cluster Challenge<br />Your builds/tests need to run in specific environment<br />Dependency on individual nodes hurts utilization<br />jobs<br />slaves<br />WombatWindows test<br />Windows #1<br />GlassFishWindows test<br />Windows #2<br />Hudson Windows test<br />Solaris#1<br />Hudson Solaris test<br />34<br />
    37. 37. Labels to rescue<br />Label is a group of slaves<br />Tie jobs to labels<br />jobs<br />slaves<br />WombatWindows test<br />Windows #1<br />Windows<br />GlassFishWindows test<br />Windows #2<br />Hudson Windows test<br />Solaris#1<br />Solaris<br />Hudson Solaris test<br />Windows #3<br />35<br />
    38. 38. Concurrent Builds<br />36<br />Hudson #10<br />Hudson #11<br />Hudson #12<br />Hudson #13<br />Hudson #14<br />Hudson #15<br />time<br />
    39. 39. Making builds sticky<br />We want jobs to be mostly on the same slave<br />Faster check out<br />Consistent results<br />Minimizes disk consumption<br />But does it softly<br />Hudson uses consistent hash* to achieve this<br />More schedule controls become possible:<br />Use faster machines more frequently<br />Slowly ramp up newly installed slaves<br />* http://en.wikipedia.org/wiki/Consistent_hashing<br />37<br />
    40. 40. Forecasting failures<br />Hudson monitors key health metrics of slaves<br />Low disk space, insufficient swap<br />Clock out of sync<br />Extensible<br />Slaves go offline automatically<br />38<br />
    41. 41. Clean up mess after builds<br />Kill run-away processes<br />Daemons, background processes left by your build<br />Works on Windows, Linux, Mac, and Solaris<br />39<br />
    42. 42. Load Statistics Monitoring<br />40<br />
    43. 43. When it’s time to add more slaves<br />41<br />
    44. 44. Hudson EC2 plugin<br />Automatically provisions slaves on EC2 on demand<br />Based on the current load<br />Picks the right AMI depending on demand<br />Starts slave agent<br />Shuts down unused instances<br />42<br />
    45. 45. Putting it all together<br />capacity<br /># of executors<br />usage<br />queue length<br />time<br />43<br />
    46. 46. Hudson “Appliance” on EC2<br />Run the master in the cloud too, if you like<br />Hudson on stock OpenSolaris AMI<br />Data stored persistently in Elastic Block Storage<br />Dynamically expandable thanks to ZFS<br />Online, too<br />Packaged as a wizard<br />44<br />
    47. 47. And a lot more I couldn’t show<br />IDE plugins<br />iPhone/Android apps<br />REST API and CLI<br />Deployment Automation<br />Hadoop, Selenium<br />…<br />45<br />
    48. 48. Conclusions<br />CI is here to stay<br />Using lots of PCs is a key<br />Hudson is very easy to get started<br />Once started, there are a lot you can do<br />46<br />
    49. 49. Q&A<br />Kohsuke Kawaguchi<br />Sun Microsystems, Inc.<br />
    50. 50. Compartmentalization<br />full Metro build #10<br />full Metro build #11<br />JAXB #10<br />JAXB #11<br />JAXB #12<br />Faster feedback<br />Good engineering practice anyway<br />Isolation of problems<br />JAX-WS #20<br />JAX-WS #21<br />JAX-WS #22<br />time<br />48<br />
    51. 51. Automated System Installations<br />Supports OpenSolaris, Ubuntu, CentOS, Fedora<br />Trivial with most Linux<br />Cooperate with Windows, too<br />Quite useful outside Hudson, too<br />No more broken CD drives<br />No more CD-Rs<br />49<br />

    ×