Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Next Step in Automation: Elastic Build Environment Kohsuke Kawaguchi / CloudBees, Inc. kk@kohsuke.org / @kohsukekawa J...
Have You Met Jenkins? http://jenkins-ci.org/ ©2013 CloudBees, Inc. All Rights Reserved 2
©2013 CloudBees, Inc. All Rights Reserved 3
©2013 CloudBees, Inc. All Rights Reserved 4
©2013 CloudBees, Inc. All Rights Reserved 5
My Jenkins around 2006 ©2013 CloudBees, Inc. All Rights Reserved 6
©2013 CloudBees, Inc. All Rights Reserved 7
©2013 CloudBees, Inc. All Rights Reserved 8
©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/lgl bRyirgnhetss/ 9R1e25s7e6rv8e83d/ 9
If only we had more computers… • Just building & testing them all… • Running tests more frequently • Testing individual...
©2013 CloudBees, Inc. All Rights Reserved 11 http://www.flickr.com/photos/drocpsu/8546730021/
Just enough computers just in time ©2013 CloudBees, Inc. All Rights Reserved 12 Elasticity
My Jenkins around 2007 ©2013 CloudBees, Inc. All Rights Reserved 13
©2013 CloudBees, Inc. All Rights Reserved 14
©2013 CloudBees, Inc. All Rights Reserved 15
©2013 CloudBees, Inc. All Rights Reserved 16
©2013 CloudBees, Inc. All Rights Reserved 17 http://www.flickr.com/photos/drocpsu/8546730021/
©2013 CloudBees, Inc. All Rights Reserved 18 18
Just enough computers of the right kind just in time ©2013 CloudBees, Inc. All Rights Reserved 19 Elasticity!
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 20
Correct answer • Test assumes a fixture running on port 8080 – Doesn’t check if it’s already being used • If another te...
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 22
Correct answer • Because of “pkill -f -9 tomcat” cleanup ©2013 CloudBees, Inc. All Rights Reserved 23
©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/ljlu Rmiigllha/ts8 6R6e76s4e8rv7e97d/ 24
Isolation • At odds with large multi-core systems • x86 virtual machines • User isolation • Kernel containers ©2013 C...
©2013 CloudBees, Inc. All Rights Reserved 26
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 27
Correct answer • Same Maven ID, two different jars • Different projects designate different ones • Local cache gets cle...
©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 29
Correct answer • Test script leaves background daemon process behind • Over time it’ll slowly choke slaves ©2013 Cloud...
©2013 CloudBees, Inc. All Rights Reserved 31
Throw away & create new ©2013 CloudBees, Inc. All Rights Reserved 32 Elasticity!
©2013 CloudBees, Inc. All Rights Reserved 33
©2013 CloudBees, Inc. All Rights Reserved 34 Ladder to Cloud Single Multiple Elastic
Solid OSS Elasticity Plugins • EC2 plugin • Jclouds plugin – OpenStack, CloudStack • Launch and tear down slaves on de...
VMWare auto-scaling plugin • Snapshot • Power on-off management • Hypervisor-aware scheduling • Folder based pooling ...
Host that runs Docker ©2013 CloudBees, Inc. All Rights Reserved 37 Docker plugin
CloudBees DEV@cloud ©2013 CloudBees, Inc. All Rights Reserved 38 Mansion Slave Slave Slave Slave
Linux Container = zero cost virtualization ©2013 CloudBees, Inc. All Rights Reserved 39 Maven Git Ant Mercurial Grad...
©2013 CloudBees, Inc. All Rights Reserved 40 For OS X Maven Git XCode Git XCode Subversion OS X OS X OS X QEMU QE...
Kernel Same-page Merging OS X OS X OS X ©2013 CloudBees, Inc. All Rights Reserved 41
©2013 CloudBees, Inc. All Rights Reserved 42
©2013 CloudBees, Inc. All Rights Reserved 43 Mansion Slave Slave Slave Workspace 1 Workspace 2 Workspace 3 Workspa...
Slave Slave Workspace’ ©2013 CloudBees, Inc. All Rights Reserved 44 Workspace Workspace’’
Slave Slave ~/.m2/repository ©2013 CloudBees, Inc. All Rights Reserved 45 ~/.m2/repository ~/.m2/repository
©2013 CloudBees, Inc. All Rights Reserved 46 Parallel Testing Test Group #1 Test Group #2 Test Group #3
©2013 CloudBees, Inc. All Rights Reserved 47 Parallel Testing foo #10 Test Group #1 foo #11 Test Group #2 foo #12 Test...
©2013 CloudBees, Inc. All Rights Reserved 48 Validated Merge upstream repo gate repo
©2013 CloudBees, Inc. All Rights Reserved 49
Workflow System • Alternative to “freestyle” projects • Scripted control flow • Resumable execution across restarts • ...
Workflow with Elastic Slaves • One-line provisioning from cloud • Language-level parallelism • Run commands, archive fi...
©2013 CloudBees, Inc. All Rights Reserved 52
Conclusion: Elasticity Benefits • Just-in-time capacity • Diversity without overhead • Isolation • Productivity gain ...
©2013 CloudBees, Inc. All Rights Reserved 54
Upcoming SlideShare
Loading in …5
×

JavaOne 2014: Next Step in Automation: Elastic Build Environment

1,632 views

Published on

This is the JavaOne 2014 session by Jesse Glick & myself that makes a case for elastic build environment (such as EC2, OpenStack, etc) for Jenkins

Published in: Software
no profile picture user

  • Be the first to comment

  • Be the first to like this

JavaOne 2014: Next Step in Automation: Elastic Build Environment

  1. 1. Next Step in Automation: Elastic Build Environment Kohsuke Kawaguchi / CloudBees, Inc. kk@kohsuke.org / @kohsukekawa Jesse Glick / CloudBees, Inc. jglick@cloudbees.com / @tyvole ©2013 CloudBees, Inc. All Rights Reserved 1
  2. 2. Have You Met Jenkins? http://jenkins-ci.org/ ©2013 CloudBees, Inc. All Rights Reserved 2
  3. 3. ©2013 CloudBees, Inc. All Rights Reserved 3
  4. 4. ©2013 CloudBees, Inc. All Rights Reserved 4
  5. 5. ©2013 CloudBees, Inc. All Rights Reserved 5
  6. 6. My Jenkins around 2006 ©2013 CloudBees, Inc. All Rights Reserved 6
  7. 7. ©2013 CloudBees, Inc. All Rights Reserved 7
  8. 8. ©2013 CloudBees, Inc. All Rights Reserved 8
  9. 9. ©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/lgl bRyirgnhetss/ 9R1e25s7e6rv8e83d/ 9
  10. 10. If only we had more computers… • Just building & testing them all… • Running tests more frequently • Testing individual commits ©2013 CloudBees, Inc. All Rights Reserved 10
  11. 11. ©2013 CloudBees, Inc. All Rights Reserved 11 http://www.flickr.com/photos/drocpsu/8546730021/
  12. 12. Just enough computers just in time ©2013 CloudBees, Inc. All Rights Reserved 12 Elasticity
  13. 13. My Jenkins around 2007 ©2013 CloudBees, Inc. All Rights Reserved 13
  14. 14. ©2013 CloudBees, Inc. All Rights Reserved 14
  15. 15. ©2013 CloudBees, Inc. All Rights Reserved 15
  16. 16. ©2013 CloudBees, Inc. All Rights Reserved 16
  17. 17. ©2013 CloudBees, Inc. All Rights Reserved 17 http://www.flickr.com/photos/drocpsu/8546730021/
  18. 18. ©2013 CloudBees, Inc. All Rights Reserved 18 18
  19. 19. Just enough computers of the right kind just in time ©2013 CloudBees, Inc. All Rights Reserved 19 Elasticity!
  20. 20. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 20
  21. 21. Correct answer • Test assumes a fixture running on port 8080 – Doesn’t check if it’s already being used • If another test runs at the same time…? ©2013 CloudBees, Inc. All Rights Reserved 21
  22. 22. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 22
  23. 23. Correct answer • Because of “pkill -f -9 tomcat” cleanup ©2013 CloudBees, Inc. All Rights Reserved 23
  24. 24. ©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/ljlu Rmiigllha/ts8 6R6e76s4e8rv7e97d/ 24
  25. 25. Isolation • At odds with large multi-core systems • x86 virtual machines • User isolation • Kernel containers ©2013 CloudBees, Inc. All Rights Reserved 25
  26. 26. ©2013 CloudBees, Inc. All Rights Reserved 26
  27. 27. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 27
  28. 28. Correct answer • Same Maven ID, two different jars • Different projects designate different ones • Local cache gets cleaned up periodically • Whichever first runs after cache cleanup “wins” ©2013 CloudBees, Inc. All Rights Reserved 28
  29. 29. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 29
  30. 30. Correct answer • Test script leaves background daemon process behind • Over time it’ll slowly choke slaves ©2013 CloudBees, Inc. All Rights Reserved 30
  31. 31. ©2013 CloudBees, Inc. All Rights Reserved 31
  32. 32. Throw away & create new ©2013 CloudBees, Inc. All Rights Reserved 32 Elasticity!
  33. 33. ©2013 CloudBees, Inc. All Rights Reserved 33
  34. 34. ©2013 CloudBees, Inc. All Rights Reserved 34 Ladder to Cloud Single Multiple Elastic
  35. 35. Solid OSS Elasticity Plugins • EC2 plugin • Jclouds plugin – OpenStack, CloudStack • Launch and tear down slaves on demand ©2013 CloudBees, Inc. All Rights Reserved 35
  36. 36. VMWare auto-scaling plugin • Snapshot • Power on-off management • Hypervisor-aware scheduling • Folder based pooling • VMWare tools integration • One-time use support ©2013 CloudBees, Inc. All Rights Reserved 36
  37. 37. Host that runs Docker ©2013 CloudBees, Inc. All Rights Reserved 37 Docker plugin
  38. 38. CloudBees DEV@cloud ©2013 CloudBees, Inc. All Rights Reserved 38 Mansion Slave Slave Slave Slave
  39. 39. Linux Container = zero cost virtualization ©2013 CloudBees, Inc. All Rights Reserved 39 Maven Git Ant Mercurial Gradle Subversion Linux Kernel Hardware
  40. 40. ©2013 CloudBees, Inc. All Rights Reserved 40 For OS X Maven Git XCode Git XCode Subversion OS X OS X OS X QEMU QEMU QEMU Linux Kernel Apple Hardware
  41. 41. Kernel Same-page Merging OS X OS X OS X ©2013 CloudBees, Inc. All Rights Reserved 41
  42. 42. ©2013 CloudBees, Inc. All Rights Reserved 42
  43. 43. ©2013 CloudBees, Inc. All Rights Reserved 43 Mansion Slave Slave Slave Workspace 1 Workspace 2 Workspace 3 Workspace 4
  44. 44. Slave Slave Workspace’ ©2013 CloudBees, Inc. All Rights Reserved 44 Workspace Workspace’’
  45. 45. Slave Slave ~/.m2/repository ©2013 CloudBees, Inc. All Rights Reserved 45 ~/.m2/repository ~/.m2/repository
  46. 46. ©2013 CloudBees, Inc. All Rights Reserved 46 Parallel Testing Test Group #1 Test Group #2 Test Group #3
  47. 47. ©2013 CloudBees, Inc. All Rights Reserved 47 Parallel Testing foo #10 Test Group #1 foo #11 Test Group #2 foo #12 Test Goup #3
  48. 48. ©2013 CloudBees, Inc. All Rights Reserved 48 Validated Merge upstream repo gate repo
  49. 49. ©2013 CloudBees, Inc. All Rights Reserved 49
  50. 50. Workflow System • Alternative to “freestyle” projects • Scripted control flow • Resumable execution across restarts • All-in-one build/test/deploy pipelines • Under active development ©2013 CloudBees, Inc. All Rights Reserved 50
  51. 51. Workflow with Elastic Slaves • One-line provisioning from cloud • Language-level parallelism • Run commands, archive files, test results • Now integrates with parallel test plugin ©2013 CloudBees, Inc. All Rights Reserved 51
  52. 52. ©2013 CloudBees, Inc. All Rights Reserved 52
  53. 53. Conclusion: Elasticity Benefits • Just-in-time capacity • Diversity without overhead • Isolation • Productivity gain – parallel testing – validated merge – workflow ©2013 CloudBees, Inc. All Rights Reserved 53
  54. 54. ©2013 CloudBees, Inc. All Rights Reserved 54

×