1.
Next Step in Automation:
Elastic Build Environment
Kohsuke Kawaguchi / CloudBees, Inc.
kk@kohsuke.org / @kohsukekawa
Jesse Glick / CloudBees, Inc.
jglick@cloudbees.com / @tyvole
©2013 CloudBees, Inc. All Rights Reserved 1

2.
Have You Met Jenkins? http://jenkins-ci.org/
©2013 CloudBees, Inc. All Rights Reserved 2

3.
©2013 CloudBees, Inc. All Rights Reserved 3

4.
©2013 CloudBees, Inc. All Rights Reserved 4

5.
©2013 CloudBees, Inc. All Rights Reserved 5

6.
My Jenkins around 2006
©2013 CloudBees, Inc. All Rights Reserved 6

7.
©2013 CloudBees, Inc. All Rights Reserved 7

8.
©2013 CloudBees, Inc. All Rights Reserved 8

9.
©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/lgl bRyirgnhetss/ 9R1e25s7e6rv8e83d/ 9

10.
If only we had more computers…
• Just building & testing them all…
• Running tests more frequently
• Testing individual commits
©2013 CloudBees, Inc. All Rights Reserved 10

11.
©2013 CloudBees, Inc. All Rights Reserved 11
http://www.flickr.com/photos/drocpsu/8546730021/

12.
Just enough computers
just in time
©2013 CloudBees, Inc. All Rights Reserved 12
Elasticity

13.
My Jenkins around 2007
©2013 CloudBees, Inc. All Rights Reserved 13

14.
©2013 CloudBees, Inc. All Rights Reserved 14

15.
©2013 CloudBees, Inc. All Rights Reserved 15

16.
©2013 CloudBees, Inc. All Rights Reserved 16

17.
©2013 CloudBees, Inc. All Rights Reserved 17
http://www.flickr.com/photos/drocpsu/8546730021/

18.
©2013 CloudBees, Inc. All Rights Reserved 18
18

19.
Just enough computers
of the right kind
just in time
©2013 CloudBees, Inc. All Rights Reserved 19
Elasticity!

20.
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 20

21.
Correct answer
• Test assumes a fixture running on port 8080
– Doesn’t check if it’s already being used
• If another test runs at the same time…?
©2013 CloudBees, Inc. All Rights Reserved 21

22.
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 22

23.
Correct answer
• Because of “pkill -f -9 tomcat” cleanup
©2013 CloudBees, Inc. All Rights Reserved 23

24.
©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/ljlu Rmiigllha/ts8 6R6e76s4e8rv7e97d/ 24

25.
Isolation
• At odds with large multi-core systems
• x86 virtual machines
• User isolation
• Kernel containers
©2013 CloudBees, Inc. All Rights Reserved 25

26.
©2013 CloudBees, Inc. All Rights Reserved 26

27.
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 27

28.
Correct answer
• Same Maven ID, two different jars
• Different projects designate different ones
• Local cache gets cleaned up periodically
• Whichever first runs after cache cleanup
“wins”
©2013 CloudBees, Inc. All Rights Reserved 28

29.
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 29

30.
Correct answer
• Test script leaves background daemon
process behind
• Over time it’ll slowly choke slaves
©2013 CloudBees, Inc. All Rights Reserved 30

31.
©2013 CloudBees, Inc. All Rights Reserved 31

32.
Throw away & create new
©2013 CloudBees, Inc. All Rights Reserved 32
Elasticity!

33.
©2013 CloudBees, Inc. All Rights Reserved 33

34.
©2013 CloudBees, Inc. All Rights Reserved 34
Ladder to Cloud
Single
Multiple
Elastic

35.
Solid OSS Elasticity Plugins
• EC2 plugin
• Jclouds plugin
– OpenStack, CloudStack
• Launch and tear down slaves on demand
©2013 CloudBees, Inc. All Rights Reserved 35

36.
VMWare auto-scaling plugin
• Snapshot
• Power on-off management
• Hypervisor-aware scheduling
• Folder based pooling
• VMWare tools integration
• One-time use support
©2013 CloudBees, Inc. All Rights Reserved 36

37.
Host that runs Docker
©2013 CloudBees, Inc. All Rights Reserved 37
Docker plugin

38.
CloudBees DEV@cloud
©2013 CloudBees, Inc. All Rights Reserved 38
Mansion
Slave Slave
Slave Slave

39.
Linux Container = zero cost virtualization
©2013 CloudBees, Inc. All Rights Reserved 39
Maven
Git
Ant
Mercurial
Gradle
Subversion
Linux Kernel
Hardware

40.
©2013 CloudBees, Inc. All Rights Reserved 40
For OS X
Maven
Git
XCode
Git
XCode
Subversion
OS X OS X OS X
QEMU QEMU QEMU
Linux Kernel
Apple Hardware

41.
Kernel Same-page Merging
OS X OS X OS X
©2013 CloudBees, Inc. All Rights Reserved 41

42.
©2013 CloudBees, Inc. All Rights Reserved 42

43.
©2013 CloudBees, Inc. All Rights Reserved 43
Mansion
Slave
Slave
Slave
Workspace 1
Workspace 2
Workspace 3
Workspace 4

44.
Slave
Slave
Workspace’
©2013 CloudBees, Inc. All Rights Reserved 44
Workspace
Workspace’’

45.
Slave
Slave
~/.m2/repository
©2013 CloudBees, Inc. All Rights Reserved 45
~/.m2/repository
~/.m2/repository

46.
©2013 CloudBees, Inc. All Rights Reserved 46
Parallel Testing
Test Group #1 Test Group #2 Test Group #3

47.
©2013 CloudBees, Inc. All Rights Reserved 47
Parallel Testing
foo #10 Test Group #1
foo #11 Test Group #2
foo #12 Test Goup #3

48.
©2013 CloudBees, Inc. All Rights Reserved 48
Validated Merge
upstream
repo
gate
repo

49.
©2013 CloudBees, Inc. All Rights Reserved 49

50.
Workflow System
• Alternative to “freestyle” projects
• Scripted control flow
• Resumable execution across restarts
• All-in-one build/test/deploy pipelines
• Under active development
©2013 CloudBees, Inc. All Rights Reserved 50

51.
Workflow with Elastic Slaves
• One-line provisioning from cloud
• Language-level parallelism
• Run commands, archive files, test results
• Now integrates with parallel test plugin
©2013 CloudBees, Inc. All Rights Reserved 51

52.
©2013 CloudBees, Inc. All Rights Reserved 52

53.
Conclusion: Elasticity Benefits
• Just-in-time capacity
• Diversity without overhead
• Isolation
• Productivity gain
– parallel testing
– validated merge
– workflow
©2013 CloudBees, Inc. All Rights Reserved 53

54.
©2013 CloudBees, Inc. All Rights Reserved 54