• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Learn About Continuous Integration With Hudson Directly From the Source
 

Learn About Continuous Integration With Hudson Directly From the Source

on

  • 9,585 views

San Francisco Java User Group presents Kohsuke Kawaguchi from Sun who introduces us to Hudson, an open-source continuous integration (CI) system, which improves the productivity of a development team ...

San Francisco Java User Group presents Kohsuke Kawaguchi from Sun who introduces us to Hudson, an open-source continuous integration (CI) system, which improves the productivity of a development team by automating various things.

Hosted By: SUPINFO International University
Sponsored By: TEKsystems, Sun, O'Reilly, JetBrains, and Marakana
Organized By: Marakana
Video By: Max Walker

Video available at http://marakana.com/forums/java/general/79.html

Statistics

Views

Total Views
9,585
Views on SlideShare
7,029
Embed Views
2,556

Actions

Likes
13
Downloads
775
Comments
1

20 Embeds 2,556

http://jptotoolnx11 1752
http://jenkins-ci.org 364
http://blog.hudson-ci.org 209
http://static.slidesharecdn.com 37
http://jptotoolnx11.corpusers.net 37
http://www.slideshare.net 35
http://vincent-grayzone.blogspot.com 28
http://marakana.com 22
http://calebesantos.wordpress.com 22
https://jenkins-ci.org 12
http://www.hudson-labs.org 10
http://android-ci.sonyericsson.net 7
http://android-ci.jpto.sonyericsson.net 5
http://hudson-labs.org 4
http://www.techgig.com 3
http://10.152.9.104 3
http://translate.googleusercontent.com 2
https://thenewcircle.com 2
http://www.hanrss.com 1
http://webcache.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Good one..!
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Plugins --- up 80 from last year in JavaOne, contributers – up 70 from last year
  • Demo cheat sheet:Ant project build. Build unit test. Polling.Maven project. Auto configuration. Javadoc.Plugin manager, update center
  • We’ve seen that how Hudson implements different launch protocol for Windows and slaves, but that’s not the only place where we take heterogeneous environments seriously.

Learn About Continuous Integration With Hudson Directly From the Source Learn About Continuous Integration With Hudson Directly From the Source Presentation Transcript

  • HudsonSF JUG
    Kohsuke Kawaguchi
    Sun Microsystems, Inc.
  • Rise of Continuous Integration
    Offload from people, push to computers
    $
    computers
    people
    time
    2
  • Spend more CPU power to help you
    First on your laptops and workstations
    IDEs are at the forefront
    And then to the servers
    a.k.a. “Continuous Integration”
    More frequent build/test executions
    Static code analysis tools
    And more to come
    3
  • Hudson
    Open-source CI server
    Emphasis on ease of installation and use
    “java -jar hudson.war” execution
    Or your OS-specific package
    Configure everything from browsers
    Extensibility
    230+ community-developed public plugins
    By 230+ contributors
    Estimated 13,000 installations
    4
    https://hudson-ci.org/
  • Adoption in all kinds of businesses
    5
  • Support Subscriptions
    Community releases every week
    Over 340 releases to date
    Rapid new features & bug fixes delivery
    “Sun Continuous Integration Server”
    For enterprise that needs stability
    Bug fix only patch releases for 18 months
    6
  • Basic Feature Set
    Hudson notices that there is a change
    Check out the source code
    CVS, Subversion, ClearCase, Mercurial, Accurev, Perforce, Git, StarTeam, …
    Do builds
    Ant, Maven, shell script, NAnt, …
    Record and publish results
    Close the feedback loop
    RSS feeds
    E-mail/IM notification
    Tray application to notify developers
    Tool integration
    Findbugs, cobertura, …
    7
  • Demo
  • Why do I care?
    Life before Hudson
    Dev forgets to commit a file
    RE nightly build breaks. It blocks SQE
    People waste time
    Life after Hudson
    Hudson will tell you in 5 mins if a build broke
    No one else notices that you forgot to commit a file
    9
  • Why do I care?
    Life before Hudson
    Dev makes a change Monday AM
    SQE nightly finds a bug Monday night
    Dev fixes it Tuesday
    Lot of wasted time
    Life after Hudson
    Dev makes a change Monday AM
    Hudson finds a regression 30 mins later
    Dev can fix it before lunch
    10
  • Why do I care?
    Life before Hudson
    SQE test runs every night, results sent out in e-mail
    After the enthusiasm of the 1st week is gone, nobody looks at them anymore
    Regressions go unnoticed until it’s too late
    Life after Hudson
    Tests run Hudson after every commit
    E-mail sent out only when tests start failing
    So it manages to keep people’s attention
    11
  • Matrix projects
    Run the same thing on different environments
    such as different JDKs, different databases, and different OSes
    Aggregate the results
    12
  • FindBugs integration
    13
  • Tracking changes
    14
  • … and more
    Browse workspaces
    Build time trend report
    15
  • Inter-Project Support
    You can host many projects on 1 Hudson
    And when you do, 1+1 becomes more than 2
    You can…
    Execute tests separately and correlate them with builds
    Mark builds as “promoted” based on test outcome
    Track which build of libraries are used where
    And more
    16
  • Test/Build Separation
    Tests often dominate the build time
    Multiple tests are run in parallel
    Improves turn-around time
    time
    JAX-WS #10
    JAX-WS #11
    JAX-WS #12

    Unit test with GF
    Unit test with Tomcat
    SQE test
    Test w/o container
    3OS x 2JDK = 6 tests
    17
  • Test/Build Separation Technique
    Copy bits from the build to test via wget by permalinks
    Ant has the equivalent <get> task
    “Fingerprinting” to keep track of where binaries are used
    $ wget –O foo.zip
    http://server/hudson/job/foo/lastSuccessfulBuild/ artifact/foo.zip
    18
  • Build Promotion
    CI produces a lot of “successful” builds
    Often overwhelming to downstream consumers
    Run tests as fast as you can
    If a build passes tests, promote it
    19
  • Got the idea?
    Automation
    Reduce turn-around time
    Make things transparent
    Remove people from the loop
    Save people’s time
    Push jobs to servers, keep workstations idle for you
    20
  • Doing Distributed Builds
    21
  • Why?
    There’s only so much 1 computer can do
    But lots of PCs = tedious
    What does Hudson do to help you?
    22
  • Installing new slaves
    For first 20 or so slaves, we did it manually
    Insert CD, click, type, click, type, click, …
    But that doesn’t scale
    Then we automated
    Available as “Hudson PXE Plugin”
    23
  • Automated System Installations
    • Hudson + PXEplugin
    • ISO images of OS
    Slaves
    Power on, hit F12
    PC boots from network (PXE)
    24
  • Automated System Installations
    • Hudson + PXEplugin
    • ISO images of OS
    Your corporate IT guy & his DHCP server
    Slaves
    Power on, hit F12
    PC boots from network (PXE)
    Choose OS from menu
    Installs non-interactively
    25
  • Distributed builds with Hudson
    Master
    Serves HTTP requests
    Stores all important info
    Slaves
    170KB single JAR
    Assumed to be unreliable
    Scale to at least 100
    Link
    Single bi-di byte stream
    No other requirements
    26
  • How master and slaves start talking
    For Unix slaves, via SSH
    Only need SSHD and JRE on slaves
    We just need a host name
    27
  • How master and slaves start talking
    For Windows, DCOM
    We just need admin user name and password
    No manual intervention
    Works even from Unix masters
    28
  • How master and slaves start talking
    Via Java Web Start
    When master cannot see slaves
    A separate socket connection is made
    29
  • Automating JNLP launch
    Once started, can be installed as Windows service
    30
  • Automating JNLP launch
    Emulate the JNLP client headless
    $ java -jar slave.jar -jnlpUrl URL
    31
  • Automated Tool Installation - JDK
    JDK from http://java.sun.com/
    Hudson automatically chooses the right bundle
    Always up to date with new releases
    32
  • Automated Tool Installation - Apache
    Ant and Maven from Apache
    33
  • Heterogeneous Cluster Challenge
    Your builds/tests need to run in specific environment
    Dependency on individual nodes hurts utilization
    jobs
    slaves
    WombatWindows test
    Windows #1
    GlassFishWindows test
    Windows #2
    Hudson Windows test
    Solaris#1
    Hudson Solaris test
    34
  • Labels to rescue
    Label is a group of slaves
    Tie jobs to labels
    jobs
    slaves
    WombatWindows test
    Windows #1
    Windows
    GlassFishWindows test
    Windows #2
    Hudson Windows test
    Solaris#1
    Solaris
    Hudson Solaris test
    Windows #3
    35
  • Concurrent Builds
    36
    Hudson #10
    Hudson #11
    Hudson #12
    Hudson #13
    Hudson #14
    Hudson #15
    time
  • Making builds sticky
    We want jobs to be mostly on the same slave
    Faster check out
    Consistent results
    Minimizes disk consumption
    But does it softly
    Hudson uses consistent hash* to achieve this
    More schedule controls become possible:
    Use faster machines more frequently
    Slowly ramp up newly installed slaves
    * http://en.wikipedia.org/wiki/Consistent_hashing
    37
  • Forecasting failures
    Hudson monitors key health metrics of slaves
    Low disk space, insufficient swap
    Clock out of sync
    Extensible
    Slaves go offline automatically
    38
  • Clean up mess after builds
    Kill run-away processes
    Daemons, background processes left by your build
    Works on Windows, Linux, Mac, and Solaris
    39
  • Load Statistics Monitoring
    40
  • When it’s time to add more slaves
    41
  • Hudson EC2 plugin
    Automatically provisions slaves on EC2 on demand
    Based on the current load
    Picks the right AMI depending on demand
    Starts slave agent
    Shuts down unused instances
    42
  • Putting it all together
    capacity
    # of executors
    usage
    queue length
    time
    43
  • Hudson “Appliance” on EC2
    Run the master in the cloud too, if you like
    Hudson on stock OpenSolaris AMI
    Data stored persistently in Elastic Block Storage
    Dynamically expandable thanks to ZFS
    Online, too
    Packaged as a wizard
    44
  • And a lot more I couldn’t show
    IDE plugins
    iPhone/Android apps
    REST API and CLI
    Deployment Automation
    Hadoop, Selenium

    45
  • Conclusions
    CI is here to stay
    Using lots of PCs is a key
    Hudson is very easy to get started
    Once started, there are a lot you can do
    46
  • Q&A
    Kohsuke Kawaguchi
    Sun Microsystems, Inc.
  • Compartmentalization
    full Metro build #10
    full Metro build #11
    JAXB #10
    JAXB #11
    JAXB #12
    Faster feedback
    Good engineering practice anyway
    Isolation of problems
    JAX-WS #20
    JAX-WS #21
    JAX-WS #22
    time
    48
  • Automated System Installations
    Supports OpenSolaris, Ubuntu, CentOS, Fedora
    Trivial with most Linux
    Cooperate with Windows, too
    Quite useful outside Hudson, too
    No more broken CD drives
    No more CD-Rs
    49