Next Step in Automation:
Elastic Build Environment
Kohsuke Kawaguchi / CloudBees, Inc.
kk@kohsuke.org / @kohsukekawa

©201...
Have You Met Jenkins?

http://jenkins-ci.org/

©2013 CloudBees, Inc. All Rights Reserved

2
©2013 CloudBees, Inc. All Rights Reserved

3
©2013 CloudBees, Inc. All Rights Reserved

4
©2013 CloudBees, Inc. All Rights Reserved

5
My Jenkins around 2006

©2013 CloudBees, Inc. All Rights Reserved

6
©2013 CloudBees, Inc. All Rights Reserved

7
©2013 CloudBees, Inc. All Rights Reserved

8
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/gbyrnes/912576883/

9
If only we had more computers…
• Just building & testing them all…
• Running tests more frequently
• Testing individual co...
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/drocpsu/8546730021/

11
Elasticity

Just enough computers
just in time

©2013 CloudBees, Inc. All Rights Reserved

12
My Jenkins around 2007

©2013 CloudBees, Inc. All Rights Reserved

13
©2013 CloudBees, Inc. All Rights Reserved

14
©2013 CloudBees, Inc. All Rights Reserved

15
©2013 CloudBees, Inc. All Rights Reserved

16
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/drocpsu/8546730021/

17
©2013 CloudBees, Inc. All Rights Reserved

18

18
Elasticity!

Just enough computers
of the right kind
just in time

©2013 CloudBees, Inc. All Rights Reserved

19
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

20
Correct answer
• Test assumes a fixture running on port 8080
– Doesn’t check if it’s already being used

• If another test...
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

22
Correct answer
• Because of “pkill -f -9 tomcat” cleanup

©2013 CloudBees, Inc. All Rights Reserved

23
http://www.flickr.com/photos/jumilla/8667648797/
©2013 CloudBees, Inc. All Rights Reserved

24
Isolation
• At odds with large multi-core systems
• x86 virtual machines
• User isolation
• Kernel containers

©2013 Cloud...
©2013 CloudBees, Inc. All Rights Reserved

26
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

27
Correct answer
• Same Maven ID, two different jars

• Different projects designate different ones
• Local cache gets clean...
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

29
Correct answer
• Test script leaves background daemon
process behind
• Over time it’ll slowly choke slaves

©2013 CloudBee...
©2013 CloudBees, Inc. All Rights Reserved

31
Elasticity!

Throw away & create new

©2013 CloudBees, Inc. All Rights Reserved

32
©2013 CloudBees, Inc. All Rights Reserved

33
Ladder to Cloud

Elastic
Multiple

Single

©2013 CloudBees, Inc. All Rights Reserved

34
Parallel Testing

Test Group #1

Test Group #2

©2013 CloudBees, Inc. All Rights Reserved

Test Group #3

35
Parallel Testing

foo #10

Test Group #1

foo #11

Test Group #2

foo #12

Test Goup #3

©2013 CloudBees, Inc. All Rights ...
Validated Merge: Jenkins Enterprise by
CloudBees

upstream
repo
gate
repo

©2013 CloudBees, Inc. All Rights Reserved

37
Recap: Elasticity Benefits
•
•
•
•

Just-in-time capacity
Diversity without overhead
Isolation
Productivity gain
– paralle...
Solid OSS Elasticity Plugins
• EC2 plugin
• Jclouds plugin
– OpenStack, CloudStack

• Launch and tear down slaves on deman...
©2013 CloudBees, Inc. All Rights Reserved

mage © http://sfcitizen.com/blog/wp-content/uploads/2011/11/6302790910_c4eb8658...
Distributed builds with Jenkins
• Master
– Serves HTTP requests
– Stores all important info

• Slaves

slave1
slave8

– 28...
Lots More Way To Deploy Slaves
• Good old shell scripting

$ java -jar slave.jar -jnlpUrl URL

©2013 CloudBees, Inc. All R...
How Flexible Does It Get?
• Slave agent is a little Java program
– Runs locally on the machine that it controls
– Access f...
Static Virtual Machines As Build Slaves
• Allocate lots of identical VMs
– Better consistency, lower overhead

• Curve up ...
Labels
Name: alpha
Label: windows 32bit

• Refer to slaves
but not by name

Name: bravo
Label: linux 32bit

Name: charlie
...
Labels
Name: alpha
Label: windows 32bit

Name: bravo
Label: linux 32bit

This job needs to run
on “linux”

Name: charlie
L...
Labels
Name: alpha
Label: windows 32bit

Name: bravo
Label: linux 32bit

This job needs to run
on “linux && 32bit”

Name: ...
More Label Benefits
• Labels can be anything
– Geographic locations
– Availability of proprietary software installed

• Be...
Load Statistics Monitoring

©2013 CloudBees, Inc. All Rights Reserved

49

49
“Cloud” plugin

©2013 CloudBees, Inc. All Rights Reserved

50
EC2 Plugin
• Label → Amazon Machine Image (AMI)
• Integrates with various AWS features
– Spot Instance
– Identity and Acce...
jclouds plugin

Jenkins

jclouds

cloud

• CloudStack, OpenStack, and many more
• “One-time use” support

©2013 CloudBees,...
Then Adjust Capacity based on it

©2013 CloudBees, Inc. All Rights Reserved

53
Jenkins Enterprise by CloudBees
• VMWare auto-scaling plugin
– Snapshot
– Power on-off management
– Hypervisor-aware sched...
CloudBees DEV@cloud
Mansion

Slave

Slave

Slave

Slave

©2013 CloudBees, Inc. All Rights Reserved

55
Linux Container = zero cost virtualization

Subversion

Gradle

Mercurial

Ant

Git

Maven

Linux Kernel
Hardware

©2013 C...
For OS X
Subversion

XCode

Git

XCode

Git

Maven

OS X

OS X

OS X

QEMU

QEMU

QEMU

Linux Kernel
Apple Hardware

©2013...
Kernel Same-page Merging

OS X

OS X

©2013 CloudBees, Inc. All Rights Reserved

OS X

58
©2013 CloudBees, Inc. All Rights Reserved

59
Mansion

Workspace 1

Slave

Workspace 2
Slave
Workspace 3
Slave
Workspace 4

©2013 CloudBees, Inc. All Rights Reserved

6...
Workspace’

Slave

Workspace

Workspace’’

Slave

©2013 CloudBees, Inc. All Rights Reserved

61
~/.m2/repository

Slave

~/.m2/repository

~/.m2/repository

Slave

©2013 CloudBees, Inc. All Rights Reserved

62
この苦しみから解脱する方法はないかと
• お釈迦様の絵
• 徐々に蓄積してきたノウハウ・ツールを大公
開

©2013 CloudBees, Inc. All Rights Reserved

63
©2013 CloudBees, Inc. All Rights Reserved

64
Testing things in pieces

©2013 CloudBees, Inc. All Rights Reserved

65
Testing the whole is hard
•
•
•
•
•

Database
Load balancer
Messaging layer
Identity service
…

©2013 CloudBees, Inc. All ...
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/46471983@N00/339605220/

67
What you really want = real deal!
Test and production environment should be
identical
… which means automating:
– Database...
The Challenge

Infrastructure as Code

©2013 CloudBees, Inc. All Rights Reserved

69
The Real Challenge

Elasticity

©2013 CloudBees, Inc. All Rights Reserved

70
The same evolution ladder

Elastic
Multiple

Single

©2013 CloudBees, Inc. All Rights Reserved

71
©2013 CloudBees, Inc. All Rights Reserved

72
Upcoming SlideShare
Loading in …5
×

Elastic build environment

671 views

Published on

Published in: Technology, Sports
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
671
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • JavaOSS
  • More than 30% uses no slaves at all or just one
  • 50+ slaves. There’s a divide here.
  • Growing “Cloud divide”
  • Because if you are doing it right, just building and testing will require a dozen or computers.
  • As I get used to controlling a handful of computers, I started thinking what more we can do.If you don’t think more computers are helpful, you are doing it wrong / Can’t be said about people.
  • Don’t build up capacity that’s enough on a few days a year but go idle most other time.
  • One of the reasons I needed so many computers is because I needed all the different environments / some combinations were very rare and old, keeping them pristine was hard.
  • Needing to have diversity in the environment adds to the capacity planning problem.
  • But you don’t want to make everything too slow by over-subscribing. I’ve seen hypervisors used to run many virtual machines.
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • So the lesson and the best practice = isolate builds and tests / treat them like untrusted code
  • Various techniques has been deployed successfully today
  • but as I found out the hard way, this isn’t enough to solve this problem
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • Turns out isolation in the time dimension is just as important / somewhat like a human body --- if you live long enough, things tend to break down / beyond certain point it becomes unsalvageable, as Windows users know all too well!
  • Turns out elasticity solves this problem, too, by allowing you to simply throw away and create new instances in the same predictable state /
  • Episode from scalability summit / everyone explains their monitoring system
  • Either this slide or more details Jenkins.
  • Another common mode of deployment is…Even if it’s static…
  • If you are willing to invest on creating a great slave virtualization environment, you can.
  • HS: if somebody misses the CoW concept, he’d be lost for the next two slides
  • Milestone in build environment elasticity / you’ve reached a new level of mental peace, enlightenment / all is well, let’s pack up and head home, right?
  • But this story doesn’t end there. The case for elasticity applies equally well to tests and test environments. As that’s really the heart of continuous integration. The hard problem.
  • A traditional attack vector to the testing is to test individual piece one at a time, then hope it still works when put together.We do this in Jenkins & CB a lot. Runs fast, anywhere, great!
  • Especially in a connected world
  • In Jenkins & CloudBees I do both all the times / Jetty / access token.And sometimes it’s a major accomplishment just to do it. Subversion server / OpenID service.But sometimes you just can’t do it. LDAP server / Active Directory
  • single “runtime environment” definition and just multiple copies of it.
  • Take load balancer as an example. Having chef configure haproxy is now a well understood problem.
  • Provision and dispose through API / creating the box that chef/puppet runs in.Usefulness of such elasticity is not just about elasticity for running tests. For development, for review, too.
  • Nowadays, that’s what I think of CloudBees as. Elasticity build environmenttaken to its logical conclusion demands elastic platform as a service.
  • Need to split this into multiple pages.
  • Elastic build environment

    1. 1. Next Step in Automation: Elastic Build Environment Kohsuke Kawaguchi / CloudBees, Inc. kk@kohsuke.org / @kohsukekawa ©2013 CloudBees, Inc. All Rights Reserved 1
    2. 2. Have You Met Jenkins? http://jenkins-ci.org/ ©2013 CloudBees, Inc. All Rights Reserved 2
    3. 3. ©2013 CloudBees, Inc. All Rights Reserved 3
    4. 4. ©2013 CloudBees, Inc. All Rights Reserved 4
    5. 5. ©2013 CloudBees, Inc. All Rights Reserved 5
    6. 6. My Jenkins around 2006 ©2013 CloudBees, Inc. All Rights Reserved 6
    7. 7. ©2013 CloudBees, Inc. All Rights Reserved 7
    8. 8. ©2013 CloudBees, Inc. All Rights Reserved 8
    9. 9. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/gbyrnes/912576883/ 9
    10. 10. If only we had more computers… • Just building & testing them all… • Running tests more frequently • Testing individual commits ©2013 CloudBees, Inc. All Rights Reserved 10
    11. 11. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/drocpsu/8546730021/ 11
    12. 12. Elasticity Just enough computers just in time ©2013 CloudBees, Inc. All Rights Reserved 12
    13. 13. My Jenkins around 2007 ©2013 CloudBees, Inc. All Rights Reserved 13
    14. 14. ©2013 CloudBees, Inc. All Rights Reserved 14
    15. 15. ©2013 CloudBees, Inc. All Rights Reserved 15
    16. 16. ©2013 CloudBees, Inc. All Rights Reserved 16
    17. 17. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/drocpsu/8546730021/ 17
    18. 18. ©2013 CloudBees, Inc. All Rights Reserved 18 18
    19. 19. Elasticity! Just enough computers of the right kind just in time ©2013 CloudBees, Inc. All Rights Reserved 19
    20. 20. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/82219206@N00/7003641975/ 20
    21. 21. Correct answer • Test assumes a fixture running on port 8080 – Doesn’t check if it’s already being used • If another test runs at the same time…? ©2013 CloudBees, Inc. All Rights Reserved 21
    22. 22. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/82219206@N00/7003641975/ 22
    23. 23. Correct answer • Because of “pkill -f -9 tomcat” cleanup ©2013 CloudBees, Inc. All Rights Reserved 23
    24. 24. http://www.flickr.com/photos/jumilla/8667648797/ ©2013 CloudBees, Inc. All Rights Reserved 24
    25. 25. Isolation • At odds with large multi-core systems • x86 virtual machines • User isolation • Kernel containers ©2013 CloudBees, Inc. All Rights Reserved 25
    26. 26. ©2013 CloudBees, Inc. All Rights Reserved 26
    27. 27. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/82219206@N00/7003641975/ 27
    28. 28. Correct answer • Same Maven ID, two different jars • Different projects designate different ones • Local cache gets cleaned up periodically • Whichever first runs after cache cleanup “wins” ©2013 CloudBees, Inc. All Rights Reserved 28
    29. 29. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/82219206@N00/7003641975/ 29
    30. 30. Correct answer • Test script leaves background daemon process behind • Over time it’ll slowly choke slaves ©2013 CloudBees, Inc. All Rights Reserved 30
    31. 31. ©2013 CloudBees, Inc. All Rights Reserved 31
    32. 32. Elasticity! Throw away & create new ©2013 CloudBees, Inc. All Rights Reserved 32
    33. 33. ©2013 CloudBees, Inc. All Rights Reserved 33
    34. 34. Ladder to Cloud Elastic Multiple Single ©2013 CloudBees, Inc. All Rights Reserved 34
    35. 35. Parallel Testing Test Group #1 Test Group #2 ©2013 CloudBees, Inc. All Rights Reserved Test Group #3 35
    36. 36. Parallel Testing foo #10 Test Group #1 foo #11 Test Group #2 foo #12 Test Goup #3 ©2013 CloudBees, Inc. All Rights Reserved 36
    37. 37. Validated Merge: Jenkins Enterprise by CloudBees upstream repo gate repo ©2013 CloudBees, Inc. All Rights Reserved 37
    38. 38. Recap: Elasticity Benefits • • • • Just-in-time capacity Diversity without overhead Isolation Productivity gain – parallel testing – validated merge – keep failed tests on the side ©2013 CloudBees, Inc. All Rights Reserved 38
    39. 39. Solid OSS Elasticity Plugins • EC2 plugin • Jclouds plugin – OpenStack, CloudStack • Launch and tear down slaves on demand ©2013 CloudBees, Inc. All Rights Reserved 39
    40. 40. ©2013 CloudBees, Inc. All Rights Reserved mage © http://sfcitizen.com/blog/wp-content/uploads/2011/11/6302790910_c4eb865892_o-copy.jpg 40
    41. 41. Distributed builds with Jenkins • Master – Serves HTTP requests – Stores all important info • Slaves slave1 slave8 – 280KB single jar – Assumed to be unreliable – Scale to 100s slave7 slave2 Master slave6 slave3 slave4 slave5 ©2013 CloudBees, Inc. All Rights Reserved 41
    42. 42. Lots More Way To Deploy Slaves • Good old shell scripting $ java -jar slave.jar -jnlpUrl URL ©2013 CloudBees, Inc. All Rights Reserved 42
    43. 43. How Flexible Does It Get? • Slave agent is a little Java program – Runs locally on the machine that it controls – Access files, fork processes, etc., on behalf of master • Communication with master – Just needs a bi-di byte stream • No shared file system, no network topology constraints, etc ©2013 CloudBees, Inc. All Rights Reserved 43
    44. 44. Static Virtual Machines As Build Slaves • Allocate lots of identical VMs – Better consistency, lower overhead • Curve up 1 big machine to small VMs • Throw away > maintain ©2013 CloudBees, Inc. All Rights Reserved 44
    45. 45. Labels Name: alpha Label: windows 32bit • Refer to slaves but not by name Name: bravo Label: linux 32bit Name: charlie Label: linux 64bit ©2013 CloudBees, Inc. All Rights Reserved 45
    46. 46. Labels Name: alpha Label: windows 32bit Name: bravo Label: linux 32bit This job needs to run on “linux” Name: charlie Label: linux 64bit ©2013 CloudBees, Inc. All Rights Reserved 46
    47. 47. Labels Name: alpha Label: windows 32bit Name: bravo Label: linux 32bit This job needs to run on “linux && 32bit” Name: charlie Label: linux 64bit ©2013 CloudBees, Inc. All Rights Reserved 47
    48. 48. More Label Benefits • Labels can be anything – Geographic locations – Availability of proprietary software installed • Better resource utilization – Jenkins have maximum freedom • Resilient to node addition/removal – Treat servers like livestocks, not like pets ©2013 CloudBees, Inc. All Rights Reserved 48
    49. 49. Load Statistics Monitoring ©2013 CloudBees, Inc. All Rights Reserved 49 49
    50. 50. “Cloud” plugin ©2013 CloudBees, Inc. All Rights Reserved 50
    51. 51. EC2 Plugin • Label → Amazon Machine Image (AMI) • Integrates with various AWS features – Spot Instance – Identity and Access Management (IAM) – Virtual Private Cloud (VPC) ©2013 CloudBees, Inc. All Rights Reserved 51
    52. 52. jclouds plugin Jenkins jclouds cloud • CloudStack, OpenStack, and many more • “One-time use” support ©2013 CloudBees, Inc. All Rights Reserved 52
    53. 53. Then Adjust Capacity based on it ©2013 CloudBees, Inc. All Rights Reserved 53
    54. 54. Jenkins Enterprise by CloudBees • VMWare auto-scaling plugin – Snapshot – Power on-off management – Hypervisor-aware scheduling – Folder based pooling – VMWare tools integration – One-time use support ©2013 CloudBees, Inc. All Rights Reserved 54
    55. 55. CloudBees DEV@cloud Mansion Slave Slave Slave Slave ©2013 CloudBees, Inc. All Rights Reserved 55
    56. 56. Linux Container = zero cost virtualization Subversion Gradle Mercurial Ant Git Maven Linux Kernel Hardware ©2013 CloudBees, Inc. All Rights Reserved 56
    57. 57. For OS X Subversion XCode Git XCode Git Maven OS X OS X OS X QEMU QEMU QEMU Linux Kernel Apple Hardware ©2013 CloudBees, Inc. All Rights Reserved 57
    58. 58. Kernel Same-page Merging OS X OS X ©2013 CloudBees, Inc. All Rights Reserved OS X 58
    59. 59. ©2013 CloudBees, Inc. All Rights Reserved 59
    60. 60. Mansion Workspace 1 Slave Workspace 2 Slave Workspace 3 Slave Workspace 4 ©2013 CloudBees, Inc. All Rights Reserved 60
    61. 61. Workspace’ Slave Workspace Workspace’’ Slave ©2013 CloudBees, Inc. All Rights Reserved 61
    62. 62. ~/.m2/repository Slave ~/.m2/repository ~/.m2/repository Slave ©2013 CloudBees, Inc. All Rights Reserved 62
    63. 63. この苦しみから解脱する方法はないかと • お釈迦様の絵 • 徐々に蓄積してきたノウハウ・ツールを大公 開 ©2013 CloudBees, Inc. All Rights Reserved 63
    64. 64. ©2013 CloudBees, Inc. All Rights Reserved 64
    65. 65. Testing things in pieces ©2013 CloudBees, Inc. All Rights Reserved 65
    66. 66. Testing the whole is hard • • • • • Database Load balancer Messaging layer Identity service … ©2013 CloudBees, Inc. All Rights Reserved 66
    67. 67. ©2013 CloudBees, Inc. All Rights Reserved http://www.flickr.com/photos/46471983@N00/339605220/ 67
    68. 68. What you really want = real deal! Test and production environment should be identical … which means automating: – Database – Load balancer – Messaging layer – Identity service –… ©2013 CloudBees, Inc. All Rights Reserved 68
    69. 69. The Challenge Infrastructure as Code ©2013 CloudBees, Inc. All Rights Reserved 69
    70. 70. The Real Challenge Elasticity ©2013 CloudBees, Inc. All Rights Reserved 70
    71. 71. The same evolution ladder Elastic Multiple Single ©2013 CloudBees, Inc. All Rights Reserved 71
    72. 72. ©2013 CloudBees, Inc. All Rights Reserved 72

    ×