Downloaded 12 times
Uploaded byKohsuke Kawaguchi
JavaOne 2014: Next Step in Automation: Elastic Build Environment
The document discusses advancements in automation using an elastic build environment, particularly focusing on the capabilities of Jenkins and CloudBees. It highlights the benefits of elasticity in testing and deployment, including just-in-time capacity, isolation, and productivity gains through parallel testing and workflows. Various plugins and tools for managing resources like EC2, Docker, and VMware are also mentioned to enhance automation processes.
More Related Content
Similar to JavaOne 2014: Next Step in Automation: Elastic Build Environment
More from Kohsuke Kawaguchi
JavaOne 2014: Next Step in Automation: Elastic Build Environment
- 1. Next Step inAutomation: Elastic Build Environment Kohsuke Kawaguchi / CloudBees, Inc. kk@kohsuke.org / @kohsukekawa Jesse Glick / CloudBees, Inc. jglick@cloudbees.com / @tyvole ©2013 CloudBees, Inc. All Rights Reserved 1
- 2. Have You MetJenkins? http://jenkins-ci.org/ ©2013 CloudBees, Inc. All Rights Reserved 2
- 3. ©2013 CloudBees, Inc.All Rights Reserved 3
- 4. ©2013 CloudBees, Inc.All Rights Reserved 4
- 5. ©2013 CloudBees, Inc.All Rights Reserved 5
- 6. My Jenkins around2006 ©2013 CloudBees, Inc. All Rights Reserved 6
- 7. ©2013 CloudBees, Inc.All Rights Reserved 7
- 8. ©2013 CloudBees, Inc.All Rights Reserved 8
- 9. ©2013 CloudBees, http://www.flickr.com/pIhnoct.o As/lgl bRyirgnhetss/ 9R1e25s7e6rv8e83d/ 9
- 10. If only wehad more computers… • Just building & testing them all… • Running tests more frequently • Testing individual commits ©2013 CloudBees, Inc. All Rights Reserved 10
- 11. ©2013 CloudBees, Inc.All Rights Reserved 11 http://www.flickr.com/photos/drocpsu/8546730021/
- 12. Just enough computers just in time ©2013 CloudBees, Inc. All Rights Reserved 12 Elasticity
- 13. My Jenkins around2007 ©2013 CloudBees, Inc. All Rights Reserved 13
- 14. ©2013 CloudBees, Inc.All Rights Reserved 14
- 15. ©2013 CloudBees, Inc.All Rights Reserved 15
- 16. ©2013 CloudBees, Inc.All Rights Reserved 16
- 17. ©2013 CloudBees, Inc.All Rights Reserved 17 http://www.flickr.com/photos/drocpsu/8546730021/
- 18. ©2013 CloudBees, Inc.All Rights Reserved 18 18
- 19. Just enough computers of the right kind just in time ©2013 CloudBees, Inc. All Rights Reserved 19 Elasticity!
- 20. ©2013 CloudBees, http://www.flickr.com/phInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 20
- 21. Correct answer •Test assumes a fixture running on port 8080 – Doesn’t check if it’s already being used • If another test runs at the same time…? ©2013 CloudBees, Inc. All Rights Reserved 21
- 22. ©2013 CloudBees, http://www.flickr.com/phInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 22
- 23. Correct answer •Because of “pkill -f -9 tomcat” cleanup ©2013 CloudBees, Inc. All Rights Reserved 23
- 24. ©2013 CloudBees, http://www.flickr.com/pIhnoct.o As/ljlu Rmiigllha/ts8 6R6e76s4e8rv7e97d/ 24
- 25. Isolation • Atodds with large multi-core systems • x86 virtual machines • User isolation • Kernel containers ©2013 CloudBees, Inc. All Rights Reserved 25
- 26. ©2013 CloudBees, Inc.All Rights Reserved 26
- 27. ©2013 CloudBees, http://www.flickr.com/phInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 27
- 28. Correct answer •Same Maven ID, two different jars • Different projects designate different ones • Local cache gets cleaned up periodically • Whichever first runs after cache cleanup “wins” ©2013 CloudBees, Inc. All Rights Reserved 28
- 29. ©2013 CloudBees, http://www.flickr.com/phInotco. sA/8ll2 R21ig9h20ts6 @ReNs0e0r/v7e00d3641975/ 29
- 30. Correct answer •Test script leaves background daemon process behind • Over time it’ll slowly choke slaves ©2013 CloudBees, Inc. All Rights Reserved 30
- 31. ©2013 CloudBees, Inc.All Rights Reserved 31
- 32. Throw away &create new ©2013 CloudBees, Inc. All Rights Reserved 32 Elasticity!
- 33. ©2013 CloudBees, Inc.All Rights Reserved 33
- 34. ©2013 CloudBees, Inc.All Rights Reserved 34 Ladder to Cloud Single Multiple Elastic
- 35. Solid OSS ElasticityPlugins • EC2 plugin • Jclouds plugin – OpenStack, CloudStack • Launch and tear down slaves on demand ©2013 CloudBees, Inc. All Rights Reserved 35
- 36. VMWare auto-scaling plugin • Snapshot • Power on-off management • Hypervisor-aware scheduling • Folder based pooling • VMWare tools integration • One-time use support ©2013 CloudBees, Inc. All Rights Reserved 36
- 37. Host that runsDocker ©2013 CloudBees, Inc. All Rights Reserved 37 Docker plugin
- 38. CloudBees DEV@cloud ©2013CloudBees, Inc. All Rights Reserved 38 Mansion Slave Slave Slave Slave
- 39. Linux Container =zero cost virtualization ©2013 CloudBees, Inc. All Rights Reserved 39 Maven Git Ant Mercurial Gradle Subversion Linux Kernel Hardware
- 40. ©2013 CloudBees, Inc.All Rights Reserved 40 For OS X Maven Git XCode Git XCode Subversion OS X OS X OS X QEMU QEMU QEMU Linux Kernel Apple Hardware
- 41. Kernel Same-page Merging OS X OS X OS X ©2013 CloudBees, Inc. All Rights Reserved 41
- 42. ©2013 CloudBees, Inc.All Rights Reserved 42
- 43. ©2013 CloudBees, Inc.All Rights Reserved 43 Mansion Slave Slave Slave Workspace 1 Workspace 2 Workspace 3 Workspace 4
- 44. Slave Slave Workspace’ ©2013 CloudBees, Inc. All Rights Reserved 44 Workspace Workspace’’
- 45. Slave Slave ~/.m2/repository ©2013 CloudBees, Inc. All Rights Reserved 45 ~/.m2/repository ~/.m2/repository
- 46. ©2013 CloudBees, Inc.All Rights Reserved 46 Parallel Testing Test Group #1 Test Group #2 Test Group #3
- 47. ©2013 CloudBees, Inc.All Rights Reserved 47 Parallel Testing foo #10 Test Group #1 foo #11 Test Group #2 foo #12 Test Goup #3
- 48. ©2013 CloudBees, Inc.All Rights Reserved 48 Validated Merge upstream repo gate repo
- 49. ©2013 CloudBees, Inc.All Rights Reserved 49
- 50. Workflow System •Alternative to “freestyle” projects • Scripted control flow • Resumable execution across restarts • All-in-one build/test/deploy pipelines • Under active development ©2013 CloudBees, Inc. All Rights Reserved 50
- 51. Workflow with ElasticSlaves • One-line provisioning from cloud • Language-level parallelism • Run commands, archive files, test results • Now integrates with parallel test plugin ©2013 CloudBees, Inc. All Rights Reserved 51
- 52. ©2013 CloudBees, Inc.All Rights Reserved 52
- 53. Conclusion: Elasticity Benefits • Just-in-time capacity • Diversity without overhead • Isolation • Productivity gain – parallel testing – validated merge – workflow ©2013 CloudBees, Inc. All Rights Reserved 53
- 54. ©2013 CloudBees, Inc.All Rights Reserved 54
Editor's Notes
- #3 Java OSS
- #4 More than 30% uses no slaves at all or just one
- #5 50+ slaves. There’s a divide here.
- #6 Growing “Cloud divide”
- #10 Because if you are doing it right, just building and testing will require a dozen or computers.
- #11 As I get used to controlling a handful of computers, I started thinking what more we can do. If you don’t think more computers are helpful, you are doing it wrong / Can’t be said about people.
- #12 Don’t build up capacity that’s enough on a few days a year but go idle most other time.
- #17 One of the reasons I needed so many computers is because I needed all the different environments / some combinations were very rare and old, keeping them pristine was hard.
- #18 Needing to have diversity in the environment adds to the capacity planning problem.
- #19 But you don’t want to make everything too slow by over-subscribing. I’ve seen hypervisors used to run many virtual machines.
- #21 Hey Kohsuke, my builds are failing. Can you take a look?
- #23 Hey Kohsuke, my builds are failing. Can you take a look?
- #25 So the lesson and the best practice = isolate builds and tests / treat them like untrusted code
- #26 Various techniques has been deployed successfully today
- #27 but as I found out the hard way, this isn’t enough to solve this problem
- #28 Hey Kohsuke, my builds are failing. Can you take a look?
- #30 Hey Kohsuke, my builds are failing. Can you take a look?
- #32 Turns out isolation in the time dimension is just as important / somewhat like a human body --- if you live long enough, things tend to break down / beyond certain point it becomes unsalvageable, as Windows users know all too well!
- #33 Turns out elasticity solves this problem, too, by allowing you to simply throw away and create new instances in the same predictable state /
- #34 Episode from scalability summit / everyone explains their monitoring system
- #36 Either this slide or more details Jenkins.
- #39 If you are willing to invest on creating a great slave virtualization environment, you can.
- #45 HS: if somebody misses the CoW concept, he’d be lost for the next two slides
- #50 brand-new job type / based on feedback from many / scalability summit