Successful PaaS and CI in the Cloud - EclipseCon 2012
•
8 likes•1,589 views
Session at AgileALM/EclipseCon 2012 presented by Steve Harris of CloudBees
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Successful PaaS and CI in the Cloud - EclipseCon 2012
Similar to Successful PaaS and CI in the Cloud - EclipseCon 2012 (20)
More from CloudBees
More from CloudBees (20)
Recently uploaded
Recently uploaded (20)
Successful PaaS and CI in the Cloud - EclipseCon 2012
- 1. Successful PaaS and CI in the Cloud Steven G. Harris steven.g.harris@cloudbees.com @stevengharris AgileALM/EclipseCon 2012 ©2012 CloudBees, Inc. All Rights Reserved
- 2. Platform as a Service ©2012 CloudBees, Inc. All Rights Reserved
- 3. As-a-Service Examples Today SaaS "Cloud computing is on- demand access to virtualized IT resources that are housed outside of your own data center, shared by others, simple to use, paid for via PaaS subscription, and accessed over the Web.“ - John Foley, Information Week IaaS ©2012 CloudBees, Inc. 3 All Rights Reserved
- 4. Traditional Software Stack Validate Setup Update Monitor Patch ©2012 CloudBees, Inc. 4 All Rights Reserved
- 5. Cloud Computing: How to do it? Who does what? Validate Setup Cloud Provider? Update Monitor Patch ©2012 CloudBees, Inc. 5 All Rights Reserved
- 6. PaaS in Summary • Applications and data are at the center of your world! – Forget about servers, VMs, load-balancers, etc. • Cloud concepts are applied to applications as first class citizens – On-demand, pay-as-you-go, elasticity, etc. – No need to handle updates, patches, scalability, failover, etc. Delivered as a Service, not as Packaged Software! ©2012 CloudBees, Inc. 6 All Rights Reserved
- 7. External Overview of CloudBees PaaS cloudbees.com Partner Test Ecosystem Forge Repositories Git Grand Session Central Clustering Web Jenkins SVN Code Stage Console Master SDK mvn Router CloudBees Build API Application Jenkins Executor MySQL Developer and Development Runtime End User Operations Services Services Interaction Interaction ©2012 CloudBees, Inc. 7 All Rights Reserved
- 8. What’s So Hard About Building PaaS, Anyway? • Technology investment to operate at scale • Intersection of app development/deployment and complex infrastructure • Cultural shift can be more challenging than technology • Efficient use of infrastructure resource pools ©2012 CloudBees, Inc. 8 All Rights Reserved
- 9. Internal Overview of CloudBees PaaS cloudbees.com Partner Test Ecosystem Forge Repositories Git Grand Session Central Clustering Web Jenkins SVN Code Stage Console Master SDK mvn Router CloudBees Build API Application Jenkins Executor MySQL Identity Provisioning AS Agent DB Agent … Scaling Monitoring AnyCloud Message Bus Alerting Auditing Shared Services Agents ©2012 CloudBees, Inc. 9 All Rights Reserved
- 10. Example of What PaaS and Cloud Enable ©2012 CloudBees, Inc. All Rights Reserved
- 11. Easier Deployment Model Load balancer App v7 Building Cloud Tools for NetFlix: http://www.slideshare.net/joesondow/building-cloudtoolsfornetflix-9419504 ©2012 CloudBees, Inc. 11 All Rights Reserved
- 12. Easier Deployment Model Load balancer App v7 App v8 ©2012 CloudBees, Inc. 12 All Rights Reserved
- 13. Easier Deployment Model Load balancer App v7 App v8 ©2012 CloudBees, Inc. 13 All Rights Reserved
- 14. Easier Deployment Model Load balancer App v7 App v8 ©2012 CloudBees, Inc. 14 All Rights Reserved
- 15. Easier Deployment Model Load balancer App v7 App v8 ©2012 CloudBees, Inc. 15 All Rights Reserved
- 16. Easier Deployment Model Load balancer App v7 App v8.1 ©2012 CloudBees, Inc. 16 All Rights Reserved
- 17. Easier Deployment Model Load balancer App v8.1 ©2012 CloudBees, Inc. 17 All Rights Reserved
- 18. Easier Deployment Model Load balancer • Doubled resources – briefly & cheaply! App v8.1 • Reduced risk dramatically ©2012 CloudBees, Inc. 18 All Rights Reserved
- 19. CI and the Cloud ©2012 CloudBees, Inc. All Rights Reserved
- 20. Development Tradeoffs for the Cloud Ecosystem Elasticity Managed Services Scale Latency Turnaround Local speed Local control Your environment Your IDE Your debugger ©2012 CloudBees, Inc. 20 20 All Rights Reserved
- 21. Use the Right Tool for the Job Low Resource High Resource Machines High IT Integration Cloud Testing Advantages Unit X-Platform Testing Functional Testing Performance Testing Testing Stress Debugging Testing Regression Cutting Testing Acceptance Code! Testing Local Code People Low IT Advantages Quality High Touch Low Touch ©2012 CloudBees, Inc. 21 21 All Rights Reserved
- 22. CI Resources Over a Few Days What you would need What you have (and pay for) What you consume ©2012 CloudBees, Inc. 22 All Rights Reserved
- 23. CI Resources Over a (Traditional) Product Lifecycle What you would need What you consume Project Team Public start working Team Hollidays Release Bug working ! fix Maintenance ©2012 CloudBees, Inc. 23 All Rights Reserved
- 24. Your CI feels like… but looks like… ©2011 CloudBees, Inc. All Rights Reserved 24
- 25. Cloud & PaaS – An Abundance of Compute Resource • How do you get it when you need it? • Once you have it, how do you harness it? • Once you’ve harnessed it, how do you keep it control and running at peak efficiency? What you want What you want to avoid ©2012 CloudBees, Inc. 25 All Rights Reserved
- 26. Jenkins Has Been Doing Distributed Builds for 5 Yrs • Easily able to control and manage 100+ computers from a single place • Plugin mechanism simplifies doing more ©2012 CloudBees, Inc. 26 All Rights Reserved
- 27. Validated Merge of Commits Taking advantage of an abundance of computing: #1 ©2012 CloudBees, Inc. ©2010 CloudBees, Inc. All Rights Reserved All Rights Reserved
- 28. Is CI Really Helping You as Much as It Could? • Does your CI server shift work from laptops to servers? – You need to commit to have Jenkins test it – But if your commit is bad, it blocks others – You end up testing locally before committing – … #fail ©2012 CloudBees, Inc. 28 All Rights Reserved
- 29. Mathematics of Large Projects Every developer makes mistakes once in a while ↓ The more developers you add, the less stable the repository gets 100% 80% 60% 40% 20% 0% 0 5 10 15 20 ©2012 CloudBees, Inc. 29 All Rights Reserved
- 30. Solution: Validated Merge • Dev commits to branches • Jenkins tests branches • If good, Jenkins merges to the trunk Alice Bob Master ©2012 CloudBees, Inc. 30 All Rights Reserved
- 31. It’s Not as Bad as It Might Look • More realistic commit graph is like this – Especially if devs remain close to the tip – Your coding is slower than “slow tests” ©2012 CloudBees, Inc. 31 All Rights Reserved
- 32. Advantages • Your mistake doesn’t impact others – Commit without worrying • Tests run on servers – Large environment-dependent tests are no longer a problem • Tests run asynchronously – Commit, then move on – You don’t wait for tests to complete • Works with other plugins… – Subversion Merge Plugin – Git Plugin – Gerrit Plugin Ref: http://blog.cloudbees.com/2012/03/dont-phunk-with-my-stable-branch.html ©2012 CloudBees, Inc. 32 All Rights Reserved
- 33. Hierarchical Validated Merges • NetBeans team works like this master repo team team team repo repo repo ©2012 CloudBees, Inc. 33 All Rights Reserved
- 34. Automated Deployment Taking advantage of an abundance of computing: #2 ©2012 CloudBees, Inc. ©2010 CloudBees, Inc. All Rights Reserved All Rights Reserved
- 35. 35
- 36. But You Need To Test Before Deploying • It compiled ≠ it’s ready to ship • Hence pipeline – Progressively run more expensive tests – A failure, and it’s out – Avoids wasting computer cycles and improves feedback speed Unit Build Int. Test Staging UAT Test ©2012 CloudBees, Inc. 36 All Rights Reserved
- 37. Build Pipeline: #1 Build Integration • Waterfall model Code • Connect jobs via upstream/ Quality downstream • Copy binaries from upstream Deploy ©2012 CloudBees, Inc. 37 All Rights Reserved
- 38. Same Model from Different Angle Time Build Build Build Build Build Build Integration Integration Integration Deploy Deploy ©2012 CloudBees, Inc. 38 All Rights Reserved
- 39. Build Pipeline Plugin ©2012 CloudBees, Inc. 39 All Rights Reserved
- 40. Promoted Builds Plugin • Think about state flow, not process flow – What conditions trigger a transition? Good build QC Pass Stability • Compiled OK • Coverage > 60% • Ran > 3d in • Unit test passed • Int. tests passed UAT env ©2012 CloudBees, Inc. 40 All Rights Reserved
- 41. Combining Pipeline and Promotions Developer Org Developer Tests Developer Job Functional Tests Security Tests X-Team Tests Access Tests QA Org Database Tests SQE Job Compliance Tests Stress Tests Production Org Backup Production Production Job Deploy Monitor ©2012 CloudBees, Inc. 41 41 All Rights Reserved
- 42. Advantages of Promotion • Asynchronous • Retryable • Flexibility in shape • More stable – Conditions can be tweaked – Human teams often interface by such states • Break free from triggers – More flexible conditions • More opportunity to take advantage of resources ©2012 CloudBees, Inc. 42 All Rights Reserved
- 43. Traceability Finding the needle in the compute haystack ©2012 CloudBees, Inc. ©2010 CloudBees, Inc. All Rights Reserved All Rights Reserved
- 44. 44
- 45. Machine-Assisted Traceability • Stitch together information in siloed systems – Dev: commit ID – QA: test execution log – Ops: deployment log, running version check ©2012 CloudBees, Inc. 45 All Rights Reserved
- 46. CI Server Is Big Brother • Source : commit ID = Binary : checksum – In Jenkins, we call it “fingerprints” • Liberally record fingerprints every time you “use” binaries – When you run tests – When you deploy – When you integrate with something else • Then cross check where they showed up ©2012 CloudBees, Inc. 46 All Rights Reserved
- 47. Traceability and Pipeline Time 1a5d a820 83ad 2f03 02d9 ecda Build Build Build Build Build Build 1a5d 83ad 2f03 Integration Integration Integration 1a5d 2f03 Deploy Deploy ©2012 CloudBees, Inc. 47 All Rights Reserved
- 48. Use Case ©2012 CloudBees, Inc. All Rights Reserved
- 49. Major Retailer (courtesy Marco Vermeulen) • Directive from top management to move to cloud • Assessment of various solutions – Committed to CI – Did not want to manage infrastructure, monitoring – Grails app – Wanted to use standard .war • Continuous delivery in Dev and Test environments • Release after each 2-week sprint • Source code on GitHub • Plugins… – Git, Cobertura, Codenarc, Violations, Grails, Pipeline, Twitter ©2012 CloudBees, Inc. 49 All Rights Reserved
- 50. Production Pipeline View • Exploits parallelism • Mirrored DEV and PRODUCTION pipeline • Manual trigger to push to production ©2012 CloudBees, Inc. 50 All Rights Reserved
- 51. Enforcing Your Team’s Values • Code coverage < 80 triggers FAILURE • Excessive Codenarc violations trigger FAILURE ©2012 CloudBees, Inc. 51 All Rights Reserved
- 52. Summary • PaaS is serving up platform capabilities to you without the headaches of install, update, patch, and ongoing infrastructure management • Removing roadblocks to dev/test/production resource availability changes the way we have to work • Extensible CI server like Jenkins provides hooks to harness the abundance of resources ©2012 CloudBees, Inc. 52 All Rights Reserved
- 53. Thank You! steven.g.harris@cloudbees.com @stevengharris ©2012 CloudBees, Inc. 53 All Rights Reserved
- 54. ©2012 CloudBees, Inc. All Rights Reserved
Editor's Notes
- Intro CloudBees, myself.Java PaaS and principal sponsors of the Jenkins open source project.Talk about how the combo of PaaS and CI can change the way you develop, deploy, and deliver applicationsReal world examples.
- Let’s start with PaaSWhat it is, what it does, a bit about what’s under the covers
- Canonical taxonomy of IaaS, PaaS, SaaS, with some familiar examplesAmazon, SalesforcePaaS follows Salesforce model, but for the platform you build to typically. No install, no maintenance. Examples.
- Compare PaaS to traditional stackYou control all these layers.You have to deal with all these layers.
- By adopting one of the *aaS models, you offload some parts of this stackYou also offload the maintenance and most ops aspects of that part of the stackDepends on your choice, tho, as you may just be taking on another piece of software that Is supposed to offload you, etc
- In short, this lets you focus on what you want to focus on – the applications and the data – and leave the rest of it to CloudBees and the PaaS.Just like you get on-demand, pay-as-you-go, elasticity of infrastructure resources using AWS, you get the same for applications in PaaSYou don’t deal with the maintenance and update issues for the stack below.And, this is a service, not another piece of packaged software you install.
- Switching from legos to more traditional powerpoint, then…Services and Partner Ecosystem.Again, no install/maintain/config of the partner services either.
- Side trip: Do you want to build this yourself?After all, you have an IT department, VM images, Chef/Puppet?Example: NetFlix.
- From a technology standpoint, it’s not just about racking and stacking VM images.In our case as an example, it’s about a set of shared services to address…And an architectural approach that gives flexibility in deployment models and IaaS providers.
- Aside from never having to argue with my IT department again, what can I get out of this.Well, there is time-to-market, better lower-case agility to go with your Agile practice.There is lower cost. But let me show you an operational practice that is nearly impossible to do without PaaS and the cloud.
- This is a cloud variation on something that probably takes weeks of time, extensive planning and meetings to do in most places. This is what they do at NetFlix.You have an existing version of your app running in production and you want to roll a new one out.
- Since you’re paying on-demand and you can provision new systems simply, quickly and reliablyYou set up a 100% duplicate environment as your original, with the same number of resources to handle the full load.Don’t cannibalize your existing pool, why should you?
- You roll over load to the new version until you can stop using the old app.
- When something goes wrong…
- You just shift the load back to the old system. It’s all still there and can do what it was doing the day before, because it hasn’t changed at all.
- You roll over load to the new version until you can stop using the old app.
- You roll over load to the new version until you can stop using the old app.
- You roll over load to the new version until you can stop using the old app.
- So now let’s switch gears away from PaaS itself and into CI and the cloud.
- To start with, I’d just make the observation that there are some tradeoffs of working in the cloud, particularly for development.You want all the benefits we just spoke about, but you also want the local speed of your hot new machine, local control, and so on.Furthermore, you have to make some tradeoff between cloud benefits and things like latency and turnaround.
- So here’s a way I like to think about it. Across the spectrum of activities during your development, QA, and overall release process,There are some things that play to the cloud advantages more than others. Human intensive things, like cutting code, debugging, and so on at least to me seem to be biased toward local systems. Whereas activities that can benefit from larger resources, like stress testing, etc, find a very natural home in the cloud.
- It turns out that CI is one of those things for a couple of reasons.First, activity is spiky. It’s tied to the time your team is awake, and when builds kick off, and when you do more compute intensive operations.Second, if you had all the resources you could possibly make use of, you’d be in good shape.But typically you don’t, so you have to make do with what you have. So you get a double-whammy of being resource constrained, but even with the resources you have, you can’t use them efficiently because of the spiky nature of CI activities.
- And if you look beyond a sort of normal day, the situation is just as bad.You end up with spiky load that is coupled to the stage of your release.
- In a lot of ways, you feel like you’re always fighting traffic, but the road is really quite empty.
- What happens is that PaaS is the solution for the problem of how to get these resources when you want them and only pay for the ones you use.But you still have some problems. How do you harness all that compute resource?And onec you’ve harnessed it, how to do you keep it in control and running at peak efficiency, without bad things happening?
- And it turns out that Jenkins, even if it went by another name in an earlier life, has been driving distributed builds for five years now.Jenkins can easily control MANY servers, and at the heart of its value is a community-driven set of plugins that make that simple.
- So let’s talk about some specific techniques to take advantage of this abundance of compute resource.The first is
- So there is a problem with CI that people discover, particularly in complex interrelated projects.You have to commit code to cause Jenkins to test things.But you don’t want to commit something that breaks things.So you end up doing more testing locally before committing.Seems like a problem.
- And the problem gets worse in large projects, where the mathematics work against you.
- The solution is to set up an arrangement where developers commit to branches, causing Jenkins to test the branch, and if it passes, to merge to the trunk.
- You might think this could be worse than dealing with the occasional bad commit.It turns our it’s not.Typical developers remain close to the tip, and that helps.And the reality is that your coding cycle is slower than the test suites you would use to gate merges.So it would more typically interleave merges.
- This approach has some real advantages.
- A team that uses this kind of approach is the NetBeans team. They use multiple team repositories they merge to using the team’s gating tests, those in turn kick off tests asynchornously, which if successful, merge the code to the master repo.
- Next, let’s look at ways to automate deployment.
- You have all these compute resources, you should take advantage of them. Why build something if you’re not going to run it?
- But, you need to test it before deploying it. Just because it compiles, it ain’t ready to ship.So, you should set up a build pipeline, typically of more and more expensive tests.When something fails at any stage of the pipeline, you want to throw out the checkin that caused the problem, and avoid going any further.You get feedback upstream more quickly, and you avoid wasting cycles.
- Typically, people just start with a waterfall type model.An upstream job triggers a downstream one if it’s successful, using the binaries from the upstream job.
- If you look at this on a time basis, it ends up looking more like this.Checkin triggers a build, that succeeds, that triggers a more complicated downstream test that triggers another one if it’s successful.And m
- It turns out there is a very nice plugin – the Build Pipeline plugin – that gives you a nice visual representation of your pipeline as each build is kicked off. So you can see very quickly when your pipeline is stalled and why. You can kick off retries manually.
- There is another way to build your pipeline beyond build triggers. In the case of promotions, you’re not thinking in terms of build process flow as much as state transition. So, you think about “When is my development-team bits ready to be picked up by the QA team” or “When is my product ready to be rolled out for stability testing”.
- So here is an example of combining pipeline and parallel processing with promotions.
- Promotions get you in a mode where code advances asynchronously, more on its own pace that makes sense to you and the teams you work with. They give you a lot more flexibility in the way you construct a pipeline and take advantage of the abundance of resources.
- Finally, let’s talk about traceability. When you have so many resources working on your behalf…
- How do you cope when bad things happen?
- You have a complicated pipeline that may be consumed by teams in different ways. As a developer, you commit code and create a binary artifact. During QA, you may create other artifacts, like logs, that are then consumed elsewhere or will contain some kind of incriminating evidence of a problem. And so on with other areas.
- What you can do is let Jenkins track these artifacts as they progress through your pipeline. In Jenkins, it’s call fingerprints. Basically, Jenkins computes a checksum for the artifacts and tracks them for every build. What you can do is just be very liberal in your use of this fingerprinting capability at every stage. It’s not like you have to fingerprint just a binary. You can do it with any artifact, like logs, at any stage of the lifecycle you use Jenkins to drive. Then when a problem arises, you can find out when the fingerprint showed up that caused the problem.
- Here’s what it looks like in our example from before. The fingerprint is associated with artifacts you identify at each step, so you can easily tell in this example that the 3rd build was used in the 2nd integration test. Jenkins has built-in capabilities to track fingerprints and make it easy to chase them down.
- Finally, I want to talk briefly about a customer of ours who is putting a lot of this into practice.
- This is a “quote” major retailer, building a cataloging application providing easier access to their many stores, both from desktops as well as mobile devices. They had a top-down directive to move things to the cloud. He surveyed a variety of solutions, coming in with his own prejudices. He and his team were completely committed to agile practice and CI. They didn’t want to deal with managing infrastructure, etc. They were building the app using Grails and wanted to use standard war file delivery. So they very smartly chose CloudBees. They use a continuous delivery approach to deliver the app into a Dev and Test environment. Then they “manually” push out a new version after every two week sprint.
- And this is what there build pipeline looks like. This happens to be their production pipeline, which is an exact mirror of their DEV pipeline, except that there is a manual trigger to deploy to production. Note they have exploited parallelism right off the bat.
- And here’s another interesting view into their project. They use Jenkins to, as Marco put it, keep the pressure up on things they value. So, they require tests to be checked in with all code, and they track coverage all the time. If coverage falls below some threshold, it triggers a failure. Similarly with some code quality metrics they measure with Codenarc.
- So, in summary…PaaS is serving up the old platform capabilities you used to have to install and maintain yourself as hosted and fully managed services.This ability to get resources delivered to you on-demand really changes the way you work and drives even more need for a proper CI and CD process.Jenkins itself provides the kind of extensibility and distributed build capability you need to harness the abundance of riches.