Setting up continuous integration for a single project can be a pretty daunting task. Doing that for hundreds of projects becomes a challenge of a different magnitude. Not only are their capacity problems, but some tests are destructive to the testing environment, some have esoteric environment demands. See how this is solved in the real world using Jenkins, jclouds, CloudStack to build an on-demand build infrastructure.
About David Nalley
David Nalley is the Vice President, Infrastructure at the Apache Software Foundation and a CloudStack PMC member.
2. #whoami
• Recovering sysadmin
• Former server hugger, now cloud
addict
• Contributor to a few open source
projects:
– Fedora, Zenoss, CloudStack, jclouds
• VP, Infrastructure for the Apache
Software Foundation
• Employed by Citrix in the Open
Source Business Office
Jenkins + jclouds + Centos + CloudStack
5. Background: Apache CloudStack
Formerly from Cloud.com
Compute- focused IaaS management solution.
Started in 2008
Now a top level project at the ASF
Written in Java
ASLv2
Jenkins + jclouds + Centos + CloudStack
6. The Problem
Lots of different testing needs – multiple
platforms, multiple versions, multiple
versions of software.
Jenkins + jclouds + Centos + CloudStack
7. The Problem – at scale
We have nearly 200 software projects each with
their own set of needs, distinct platforms.
Some want CentOS 5, 6, or 7; others want OSX, Win
2012, Ubuntu, FreeBSD, or Solaris.
Demand varies widely – projects getting closer to
release test more; some days we had hundreds of
jobs in the queue, other days we had 20 or 30
Jenkins + jclouds + Centos + CloudStack
8. The transition for monolith to scalable.
Jenkins + jclouds + Centos + CloudStack
9. Where we were
Dedicated Jenkins master
Esoteric things had VMs
~30 dedicated machines, strewn across
multiple datacenters.
Jenkins + jclouds + Centos + CloudStack
10. How we started our move
• Defining what a build slave needs
• Make sure that Jenkins is working
efficiently
(Misconfigured jobs are a huge drain)
• Adopted a LTS version of Jenkins
Jenkins + jclouds + Centos + CloudStack
11. Building images
Packer – building images
Start with JeOS – declare everything else in
puppet, run puppet as a job.
Create a jenkins job for this – everytime a
job changes – rebuild, reupload.
Jenkins + jclouds + Centos + CloudStack
12. How you plan to use Cloud
Server huggers wanted to use fixed VMs
(deploy fixed machines)
Jenkins + jclouds + Centos + CloudStack
13. How you want to use cloud
Fresh, dynamically provisioned VM for each
build
Jenkins + jclouds + Centos + CloudStack
15. The right answer
(for us)
Spin up build slaves in response to demand.
Leave the machine up for 30 minutes after
demand subsides to keep from flapping.
Jenkins + jclouds + Centos + CloudStack
16. Make it all work
Jenkins + jclouds + Centos + CloudStack
23. Get Involved
Web: http://cloudstack.apache.org/
Mailing Lists: cloudstack.apache.org/mailing-lists.html
IRC: irc.freenode.net: 6667 #cloudstack
Twitter: @cloudstack
LinkedIn: www.linkedin.com/groups/CloudStack-Users-Group-3144859
If it didn’t happen on the mailing list, it didn’t happen.
Jenkins + jclouds + Centos + CloudStack
Editor's Notes
Do you know what continuous integration is?
CI is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.
So Jenkins is merely a testing tool for use in CI.
CloudStack maintains its own CI server
The ASF has a top-20 in the world, Jenkins instance.
Originally written by Adrian Cole – now a top level project at the ASF
Java cloud bindings regardless of what your underlying cloud actually is
Do you know what continuous integration is?
CI is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.
So Jenkins is merely a testing tool for use in CI.
And that’s just CloudStack
And that’s just CloudStack
We started this with Ansible – and for many of our build slaves still is the default.
The rest of our infrastructure is Puppet – so much of this is moving.
Developers – committers on Jenkins as a matter of fact – had us running bleeding edge, unreleased, with custom patches.
The first struggle I faced was explaining how IP addresses were dynamic, and that they couldn’t really reserve a block of them.
A perfectly clean workspace – nothing muddled – and perfect isolation – only a single job running at a time.
Downside is that spin up time can be excessive. Depending on the cloud provider you might have several minutes waiting – for the ASF – we discovered that using our own custom images made spin up much slower – a problem our public cloud vendor since solved.
Don’t do what we did and install a personal snapshot.
That said – a number of folks have been discussing creating provider-specific plugins.
