Learn About Continuous Integration With Hudson Directly From the Source

HudsonSF JUG
Kohsuke Kawaguchi
Sun Microsystems, Inc.

Rise of Continuous Integration
Offload from people, push to computers
$
computers
people
time
2

Spend more CPU power to help you
First on your laptops and workstations
IDEs are at the forefront
And then to the servers
a.k.a. “Continuous Integration”
More frequent build/test executions
Static code analysis tools
And more to come
3

Hudson
Open-source CI server
Emphasis on ease of installation and use
“java -jar hudson.war” execution
Or your OS-specific package
Configure everything from browsers
Extensibility
230+ community-developed public plugins
By 230+ contributors
Estimated 13,000 installations
4
https://hudson-ci.org/

Adoption in all kinds of businesses
5

Support Subscriptions
Community releases every week
Over 340 releases to date
Rapid new features & bug fixes delivery
“Sun Continuous Integration Server”
For enterprise that needs stability
Bug fix only patch releases for 18 months
6

Basic Feature Set
Hudson notices that there is a change
Check out the source code
CVS, Subversion, ClearCase, Mercurial, Accurev, Perforce, Git, StarTeam, …
Do builds
Ant, Maven, shell script, NAnt, …
Record and publish results
Close the feedback loop
RSS feeds
E-mail/IM notification
Tray application to notify developers
Tool integration
Findbugs, cobertura, …
7

Why do I care?
Life before Hudson
Dev forgets to commit a file
RE nightly build breaks. It blocks SQE
People waste time
Life after Hudson
Hudson will tell you in 5 mins if a build broke
No one else notices that you forgot to commit a file
9

Why do I care?
Life before Hudson
Dev makes a change Monday AM
SQE nightly finds a bug Monday night
Dev fixes it Tuesday
Lot of wasted time
Life after Hudson
Dev makes a change Monday AM
Hudson finds a regression 30 mins later
Dev can fix it before lunch
10

Why do I care?
Life before Hudson
SQE test runs every night, results sent out in e-mail
After the enthusiasm of the 1st week is gone, nobody looks at them anymore
Regressions go unnoticed until it’s too late
Life after Hudson
Tests run Hudson after every commit
E-mail sent out only when tests start failing
So it manages to keep people’s attention
11

Matrix projects
Run the same thing on different environments
such as different JDKs, different databases, and different OSes
Aggregate the results
12

FindBugs integration
13

Tracking changes
14

… and more
Browse workspaces
Build time trend report
15

Inter-Project Support
You can host many projects on 1 Hudson
And when you do, 1+1 becomes more than 2
You can…
Execute tests separately and correlate them with builds
Mark builds as “promoted” based on test outcome
Track which build of libraries are used where
And more
16

Test/Build Separation
Tests often dominate the build time
Multiple tests are run in parallel
Improves turn-around time
time
JAX-WS #10
JAX-WS #11
JAX-WS #12
…
Unit test with GF
Unit test with Tomcat
SQE test
Test w/o container
3OS x 2JDK = 6 tests
17

Test/Build Separation Technique
Copy bits from the build to test via wget by permalinks
Ant has the equivalent <get> task
“Fingerprinting” to keep track of where binaries are used
$ wget –O foo.zip
http://server/hudson/job/foo/lastSuccessfulBuild/ artifact/foo.zip
18

Build Promotion
CI produces a lot of “successful” builds
Often overwhelming to downstream consumers
Run tests as fast as you can
If a build passes tests, promote it
19

Got the idea?
Automation
Reduce turn-around time
Make things transparent
Remove people from the loop
Save people’s time
Push jobs to servers, keep workstations idle for you
20

Doing Distributed Builds
21

Why?
There’s only so much 1 computer can do
But lots of PCs = tedious
What does Hudson do to help you?
22

Installing new slaves
For first 20 or so slaves, we did it manually
Insert CD, click, type, click, type, click, …
But that doesn’t scale
Then we automated
Available as “Hudson PXE Plugin”
23

Automated System Installations
Hudson + PXEplugin
ISO images of OS
Slaves
Power on, hit F12
PC boots from network (PXE)
24

Hudson + PXEplugin
ISO images of OS
Your corporate IT guy & his DHCP server
Slaves
Power on, hit F12
PC boots from network (PXE)
Choose OS from menu
Installs non-interactively
25

Distributed builds with Hudson
Master
Serves HTTP requests
Stores all important info
Slaves
170KB single JAR
Assumed to be unreliable
Scale to at least 100
Link
Single bi-di byte stream
No other requirements
26

How master and slaves start talking
For Unix slaves, via SSH
Only need SSHD and JRE on slaves
We just need a host name
27

For Windows, DCOM
We just need admin user name and password
No manual intervention
Works even from Unix masters
28

Via Java Web Start
When master cannot see slaves
A separate socket connection is made
29

Automating JNLP launch
Once started, can be installed as Windows service
30

Automating JNLP launch
Emulate the JNLP client headless
$ java -jar slave.jar -jnlpUrl URL
31

Automated Tool Installation - JDK
JDK from http://java.sun.com/
Hudson automatically chooses the right bundle
Always up to date with new releases
32

Automated Tool Installation - Apache
Ant and Maven from Apache
33

Heterogeneous Cluster Challenge
Your builds/tests need to run in specific environment
Dependency on individual nodes hurts utilization
jobs
slaves
WombatWindows test
Windows #1
GlassFishWindows test
Windows #2
Hudson Windows test
Solaris#1
Hudson Solaris test
34

Labels to rescue
Label is a group of slaves
Tie jobs to labels
jobs
slaves
WombatWindows test
Windows #1
Windows
GlassFishWindows test
Windows #2
Hudson Windows test
Solaris#1
Solaris
Hudson Solaris test
Windows #3
35

Concurrent Builds
36
Hudson #10
Hudson #11
Hudson #12
Hudson #13
Hudson #14
Hudson #15
time

Making builds sticky
We want jobs to be mostly on the same slave
Faster check out
Consistent results
Minimizes disk consumption
But does it softly
Hudson uses consistent hash* to achieve this
More schedule controls become possible:
Use faster machines more frequently
Slowly ramp up newly installed slaves
* http://en.wikipedia.org/wiki/Consistent_hashing
37

Forecasting failures
Hudson monitors key health metrics of slaves
Low disk space, insufficient swap
Clock out of sync
Extensible
Slaves go offline automatically
38

Clean up mess after builds
Kill run-away processes
Daemons, background processes left by your build
Works on Windows, Linux, Mac, and Solaris
39

Load Statistics Monitoring
40

When it’s time to add more slaves
41

Hudson EC2 plugin
Automatically provisions slaves on EC2 on demand
Based on the current load
Picks the right AMI depending on demand
Starts slave agent
Shuts down unused instances
42

Putting it all together
capacity
# of executors
usage
queue length
time
43

Hudson “Appliance” on EC2
Run the master in the cloud too, if you like
Hudson on stock OpenSolaris AMI
Data stored persistently in Elastic Block Storage
Dynamically expandable thanks to ZFS
Online, too
Packaged as a wizard
44

And a lot more I couldn’t show
IDE plugins
iPhone/Android apps
REST API and CLI
Deployment Automation
Hadoop, Selenium
…
45

Conclusions
CI is here to stay
Using lots of PCs is a key
Hudson is very easy to get started
Once started, there are a lot you can do
46

Q&A
Kohsuke Kawaguchi
Sun Microsystems, Inc.

Compartmentalization
full Metro build #10
full Metro build #11
JAXB #10
JAXB #11
JAXB #12
Faster feedback
Good engineering practice anyway
Isolation of problems
JAX-WS #20
JAX-WS #21
JAX-WS #22
time
48

Supports OpenSolaris, Ubuntu, CentOS, Fedora
Trivial with most Linux
Cooperate with Windows, too
Quite useful outside Hudson, too
No more broken CD drives
No more CD-Rs
49

http://jptotoolnx11	837
http://blog.hudson-ci.org	209
http://jenkins-ci.org	141
http://static.slidesharecdn.com	37
http://jptotoolnx11.corpusers.net	37
http://www.slideshare.net	35
http://vincent-grayzone.blogspot.com	28
http://calebesantos.wordpress.com	16
http://www.hudson-labs.org	10
http://marakana.com	7
http://android-ci.sonyericsson.net	7
http://android-ci.jpto.sonyericsson.net	5
http://hudson-labs.org	4
http://www.techgig.com	3
http://translate.googleusercontent.com	2
http://webcache.googleusercontent.com	1
http://www.hanrss.com	1
http://10.152.9.104	1
https://jenkins-ci.org	1

Learn About Continuous Integration With Hudson Directly From the Source

by Marakana Inc. on Feb 19, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

19 Embeds 1,382

Statistics

Learn About Continuous Integration With Hudson Directly From the Source — Presentation Transcript