Successfully reported this slideshow.

JavaOne 2014: Next Step in Automation: Elastic Build Environment

1

Share

Loading in …3
×
1 of 54
1 of 54

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

JavaOne 2014: Next Step in Automation: Elastic Build Environment

  1. 1. Next Step in Automation: Elastic Build Environment Kohsuke Kawaguchi / CloudBees, Inc. kk@kohsuke.org / @kohsukekawa Jesse Glick / CloudBees, Inc. jglick@cloudbees.com / @tyvole ©2013 CloudBees, Inc. All Rights Reserved 1
  2. 2. Have You Met Jenkins? http://jenkins-ci.org/ ©2013 CloudBees, Inc. All Rights Reserved 2
  3. 3. ©2013 CloudBees, Inc. All Rights Reserved 3
  4. 4. ©2013 CloudBees, Inc. All Rights Reserved 4
  5. 5. ©2013 CloudBees, Inc. All Rights Reserved 5
  6. 6. My Jenkins around 2006 ©2013 CloudBees, Inc. All Rights Reserved 6
  7. 7. ©2013 CloudBees, Inc. All Rights Reserved 7
  8. 8. ©2013 CloudBees, Inc. All Rights Reserved 8
  9. 9. ©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/lgl bRyirgnhetss/ 9R1e25s7e6rv8e83d/ 9
  10. 10. If only we had more computers… • Just building & testing them all… • Running tests more frequently • Testing individual commits ©2013 CloudBees, Inc. All Rights Reserved 10
  11. 11. ©2013 CloudBees, Inc. All Rights Reserved 11 http://www.flickr.com/photos/drocpsu/8546730021/
  12. 12. Just enough computers just in time ©2013 CloudBees, Inc. All Rights Reserved 12 Elasticity
  13. 13. My Jenkins around 2007 ©2013 CloudBees, Inc. All Rights Reserved 13
  14. 14. ©2013 CloudBees, Inc. All Rights Reserved 14
  15. 15. ©2013 CloudBees, Inc. All Rights Reserved 15
  16. 16. ©2013 CloudBees, Inc. All Rights Reserved 16
  17. 17. ©2013 CloudBees, Inc. All Rights Reserved 17 http://www.flickr.com/photos/drocpsu/8546730021/
  18. 18. ©2013 CloudBees, Inc. All Rights Reserved 18 18
  19. 19. Just enough computers of the right kind just in time ©2013 CloudBees, Inc. All Rights Reserved 19 Elasticity!
  20. 20. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 20
  21. 21. Correct answer • Test assumes a fixture running on port 8080 – Doesn’t check if it’s already being used • If another test runs at the same time…? ©2013 CloudBees, Inc. All Rights Reserved 21
  22. 22. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 22
  23. 23. Correct answer • Because of “pkill -f -9 tomcat” cleanup ©2013 CloudBees, Inc. All Rights Reserved 23
  24. 24. ©2013 CloudBees, http://www.flickr.com/p Ihnoct.o As/ljlu Rmiigllha/ts8 6R6e76s4e8rv7e97d/ 24
  25. 25. Isolation • At odds with large multi-core systems • x86 virtual machines • User isolation • Kernel containers ©2013 CloudBees, Inc. All Rights Reserved 25
  26. 26. ©2013 CloudBees, Inc. All Rights Reserved 26
  27. 27. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 27
  28. 28. Correct answer • Same Maven ID, two different jars • Different projects designate different ones • Local cache gets cleaned up periodically • Whichever first runs after cache cleanup “wins” ©2013 CloudBees, Inc. All Rights Reserved 28
  29. 29. ©2013 CloudBees, http://www.flickr.com/p hInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 29
  30. 30. Correct answer • Test script leaves background daemon process behind • Over time it’ll slowly choke slaves ©2013 CloudBees, Inc. All Rights Reserved 30
  31. 31. ©2013 CloudBees, Inc. All Rights Reserved 31
  32. 32. Throw away & create new ©2013 CloudBees, Inc. All Rights Reserved 32 Elasticity!
  33. 33. ©2013 CloudBees, Inc. All Rights Reserved 33
  34. 34. ©2013 CloudBees, Inc. All Rights Reserved 34 Ladder to Cloud Single Multiple Elastic
  35. 35. Solid OSS Elasticity Plugins • EC2 plugin • Jclouds plugin – OpenStack, CloudStack • Launch and tear down slaves on demand ©2013 CloudBees, Inc. All Rights Reserved 35
  36. 36. VMWare auto-scaling plugin • Snapshot • Power on-off management • Hypervisor-aware scheduling • Folder based pooling • VMWare tools integration • One-time use support ©2013 CloudBees, Inc. All Rights Reserved 36
  37. 37. Host that runs Docker ©2013 CloudBees, Inc. All Rights Reserved 37 Docker plugin
  38. 38. CloudBees DEV@cloud ©2013 CloudBees, Inc. All Rights Reserved 38 Mansion Slave Slave Slave Slave
  39. 39. Linux Container = zero cost virtualization ©2013 CloudBees, Inc. All Rights Reserved 39 Maven Git Ant Mercurial Gradle Subversion Linux Kernel Hardware
  40. 40. ©2013 CloudBees, Inc. All Rights Reserved 40 For OS X Maven Git XCode Git XCode Subversion OS X OS X OS X QEMU QEMU QEMU Linux Kernel Apple Hardware
  41. 41. Kernel Same-page Merging OS X OS X OS X ©2013 CloudBees, Inc. All Rights Reserved 41
  42. 42. ©2013 CloudBees, Inc. All Rights Reserved 42
  43. 43. ©2013 CloudBees, Inc. All Rights Reserved 43 Mansion Slave Slave Slave Workspace 1 Workspace 2 Workspace 3 Workspace 4
  44. 44. Slave Slave Workspace’ ©2013 CloudBees, Inc. All Rights Reserved 44 Workspace Workspace’’
  45. 45. Slave Slave ~/.m2/repository ©2013 CloudBees, Inc. All Rights Reserved 45 ~/.m2/repository ~/.m2/repository
  46. 46. ©2013 CloudBees, Inc. All Rights Reserved 46 Parallel Testing Test Group #1 Test Group #2 Test Group #3
  47. 47. ©2013 CloudBees, Inc. All Rights Reserved 47 Parallel Testing foo #10 Test Group #1 foo #11 Test Group #2 foo #12 Test Goup #3
  48. 48. ©2013 CloudBees, Inc. All Rights Reserved 48 Validated Merge upstream repo gate repo
  49. 49. ©2013 CloudBees, Inc. All Rights Reserved 49
  50. 50. Workflow System • Alternative to “freestyle” projects • Scripted control flow • Resumable execution across restarts • All-in-one build/test/deploy pipelines • Under active development ©2013 CloudBees, Inc. All Rights Reserved 50
  51. 51. Workflow with Elastic Slaves • One-line provisioning from cloud • Language-level parallelism • Run commands, archive files, test results • Now integrates with parallel test plugin ©2013 CloudBees, Inc. All Rights Reserved 51
  52. 52. ©2013 CloudBees, Inc. All Rights Reserved 52
  53. 53. Conclusion: Elasticity Benefits • Just-in-time capacity • Diversity without overhead • Isolation • Productivity gain – parallel testing – validated merge – workflow ©2013 CloudBees, Inc. All Rights Reserved 53
  54. 54. ©2013 CloudBees, Inc. All Rights Reserved 54

Editor's Notes

  • Java
    OSS

  • More than 30% uses no slaves at all or just one
  • 50+ slaves. There’s a divide here.
  • Growing “Cloud divide”
  • Because if you are doing it right, just building and testing will require a dozen or computers.
  • As I get used to controlling a handful of computers, I started thinking what more we can do.

    If you don’t think more computers are helpful, you are doing it wrong / Can’t be said about people.
  • Don’t build up capacity that’s enough on a few days a year but go idle most other time.
  • One of the reasons I needed so many computers is because I needed all the different environments / some combinations were very rare and old, keeping them pristine was hard.
  • Needing to have diversity in the environment adds to the capacity planning problem.
  • But you don’t want to make everything too slow by over-subscribing. I’ve seen hypervisors used to run many virtual machines.
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • So the lesson and the best practice = isolate builds and tests / treat them like untrusted code
  • Various techniques has been deployed successfully today
  • but as I found out the hard way, this isn’t enough to solve this problem
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • Hey Kohsuke, my builds are failing. Can you take a look?
  • Turns out isolation in the time dimension is just as important / somewhat like a human body --- if you live long enough, things tend to break down / beyond certain point it becomes unsalvageable, as Windows users know all too well!
  • Turns out elasticity solves this problem, too, by allowing you to simply throw away and create new instances in the same predictable state /
  • Episode from scalability summit / everyone explains their monitoring system
  • Either this slide or more details Jenkins.
  • If you are willing to invest on creating a great slave virtualization environment, you can.
  • HS: if somebody misses the CoW concept, he’d be lost for the next two slides
  • brand-new job type / based on feedback from many / scalability summit
  • ×