1st conference 2015 devops
•Download as PPTX, PDF•
2 likes•750 views
Although DevOps is primarily a mindset and cultural movement, it has concrete practices which are natural manifestations of this culture and mindset. In this sense, DevOps is very similar to Agile. And like Agile, it’s possible to cargo cult DevOps and focus on “how” to implement practices without paying sufficient attention to “why”... There be dragons here. However, the organisations that truly embrace the DevOps approach have publicised practices that are far less susceptible to cargo culting. This presentation examines several of these practices and asks the audience to consider what organisational attitudes must be in place to support these practices. This process of consideration should lead the audience to a deeper understanding of what the key values of DevOps are and how to approach moving an organisation in this direction.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to 1st conference 2015 devops
Similar to 1st conference 2015 devops (20)
More from Andy Marks
More from Andy Marks (17)
Recently uploaded
Recently uploaded (20)
1st conference 2015 devops
- 1. what if... what your devops practices say about organisational maturity
- 4. lies of omission! “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”to production HOW?!? “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
- 5. missing pieces of the puzzle agile how do we build “it”? lean*what “it” should we build? devops we’ve built “it”... now what?
- 6. devops -?
- 7. 2014 State of DevOps report 30x more frequent deployments 8000x faster lead times than peers
- 8. high functioning devops in the wild
- 9. low hanging fruit deployment focus @ scale
- 10. low hanging fruit what your practices (can) say about you
- 11. you build it, you fix it “we found that when you woke up developers at 2am, defects got fixed faster than ever” ???
- 12. you build it, you fix it “we found that when you woke up developers at 2am, defects got fixed faster than ever”
- 13. share ownership “Backend services typically have two so-called System Owners - one from Dev and one from Ops.” ???
- 14. share ownership Backend services typically have two so-called System Owners - one from Dev and one from Ops. ???
- 17. no more snowflakes “remove (login) access to all deployed systems” ???
- 18. no more snowflakes “remove (login) access to all deployed systems”
- 19. deployment focus what your practices (can) say about you
- 20. use short release cycles “... we have one hard and fast rule for new Engineers on their first day: deploy to production” ???
- 21. use short release cycles “... we have one hard and fast rule for new Engineers on their first day: deploy to production”
- 22. separate deployment from release “right now ... there is already the code [in production] for every major thing [we’re] going to launch in the next six months and beyond!” ???
- 23. separate deployment from release “right now ... there is already the code for every major thing [we’re] going to launch in the next six months and beyond!”
- 24. separate deployment from release “Feature toggles should be a first-class part of your stories. One team … builds feature bits into their stories such that the first task they perform when they play a story is to add the feature bit for that story.” ???
- 25. separate deployment from release “Feature toggles should be a first-class part of your stories. One team … builds feature bits into their stories such that the first task they perform when they play a story is to add the feature bit for that story.”
- 26. @ scale what your practices (can) say about you
- 27. embrace continuous deployment “In 2011, [we] were doing a maximum of 1,079 deployments” ???
- 28. embrace continuous deployment “In 2011, [we] were doing a maximum of 1,079 deployments PER HOUR.” ???
- 29. embrace continuous deployment “In 2011, [we] were doing a maximum of 1,079 deployments PER HOUR.”
- 30. integrate continuously... “... engineers have built a powerful continuous integration system running 60 million tests per day” ???
- 31. integrate continuously... “... engineers have built a powerful continuous integration system running 60 million tests per day”
- 32. “Over the last year Chaos Monkey has terminated over 65,000 instances running in our production and testing environments. Most of the time nobody notices” architect for network failure ???
- 33. “Over the last year Chaos Monkey has terminated over 65,000 instances running in our production and testing environments. Most of the time nobody notices” architect for network failure
- 34. Conclusion
- 35. Conclusion
- 37. REFERENCE S http://bit.ly/1C3S0DZ http://etsy.me/1FEyyex http://bit.ly/1v70LcY http://bit.ly/198ccb4 http://bit.ly/1hqFE9d 2014 State of DevOps report http://bit.ly/1Hte2id http://tcrn.ch/1AZIaMe http://nflx.it/1ia88X4 http://bit.ly/1AcOLUB http://bit.ly/1E2vztN
- 38. Andy Marks @andee_marks “we should optimise for the time it takes us to deliver valuable software to users”
Editor's Notes
- START TIMER - 40 MINUTES Good afternoon everyone! This talk is about practices… and DevOps. Agile has practices; TDD, pair programming, retrospectives, so too does DevOps. They’re not as well understood, nor are they as widely agreed upon, but when done well, they can be incredibly effective at giving your organisation a competitive advantage, especially as Agile becomes more and more of a commodity with each passing year. And I’ll also be introducing a number of organisations that do indeed implement some of these practices incredibly well. OK, let’s move on...
- So who am I? I’m Andy Marks and I work, along with Cam and Karl, and Kelsey who spoke earlier today, and Nick Thorpe who’ll be speaking in the next slot, at ThoughtWorks. We’re a global professional services company that works with organisations looking to build and grow digital businesses. And for us, Agile is just part of the way we work. I’ve been working in the DevOps community for the last couple of years; I created and ran the first DevOps Meetup group in Singapore during my time there in 2012 and took over running the Melbourne DevOps Meetup group from early 2013 to the end of last year. Introductions aside, let’s get started...
- Let’s start by killing some sacred cows, shall we? How about these dodgy looking characters here - one of whom looks suspiciously like our keynote speaker from earlier today!?! These people are, of course, the 12 founding fathers of Agile. Or, as I like to call them… Liars! And what lies did these good folk inflict upon us?
- Lies of omission actually. It was the stuff they didn’t tell us that I want to speak about today. Here’s what I mean: here are two of the 12 principles from the Agile Manifesto. Each of which is missing some very important information that, I suspect, has led many a team and organisation to do local optimisation of their software delivery processes: I’ll explain what I mean by this… Firstly, the principal about continuous delivery…What’s been omitted here? How about the little point about delivery being “to production”. In retrospect, I think that would have been a nice thing to have had in the original manifesto. Lord knows I’ve seen and been part of many teams that have produced high quality software very quickly… right up to the point where it needs to get released outside the development environment. Which leads us to the second principle that falls a little short of complete IMO; the one about short releases being ideal. And the missing piece of the puzzle here? HOW should we go about doing those frequent releases. It’s all well and good to tell us they’re good things to have, but there was precious little in the way of specifics around how this goal was to be achieved. Certainly, when I starting on my first Agile project back in 2002, with the Kent Beck White Book of Extreme Programming as my guiding light and mentor, nothing was mentioned in those hallowed pages about how to release quickly into production. Indeed, it’s only been in the last 4-5 years that we’ve had much guidance in this perilous activity; in 2011 Jez Humble and Dave Farley released the Continuous Delivery book, and the subsequent rise and rise again of the DevOps movement made this sort of stuff cool again. [Need to ref White Book]
- To be completely fair to the 12 authors of the manifesto, the work they did that weekend in 2001 has had enormous ramifications on our industry, almost all the better. I’m really being very very harsh by picking on them in these respects, but the key point I have is that blind faith and a Scrum Master certification isn’t going to get you where you need to be. Agile is necessary but not sufficient to really give a competitive difference to your organisation. And what do I say this? [*] Well, if you are to look at the main thing Agile helps us with, is to answer the question: “how do we build it?” [*] Which then begs the follow on question: what is this “it” to which we refer? And many organisations have been asking themselves this or similar questions over the years, even after significant investments in Agile. These investments have likely helped morale in and around the delivery teams, but not really make a substantial difference to their ability to get the right features in front of customers. [*] Thankfully, there’s been a surge of interest recently in answering this question by applying many of the techniques espoused in the Lean Startup and Lean Enterprise world. [*] And then the remaining question for lot of folks has been; OK we’ve identified an MVP which think might make a valuable opening hand for what “it” is, we’ve applied our Agile ninja skills to build it quickly and to suitable quality, how do we put ourselves in a position where we can reliably and with low risk get our MVP to market, especially under the assumption that we may need to iterate over this quickly once real customers get their hands on it. [*] And DevOps attempts to be this missing piece of the puzzle: a cultural approach to tearing down the traditional barriers between development and operations with the purpose of helping organisations get software to customers faster and with less risk.
- There’s lots of definitions of what DevOps is, but one of the ones I ran across recently that had me initially chuckling away and then sitting there thinking “you know, there’s a lot of truth in that” is this little gem from Jason Huggins on Twitter: [*] So the nugget of truth in this Tweet is that, in a very similar way to how Agile made developers also look ahead to where their work traditionally went (i.e., testing) and then build that skill into their toolbox, the DevOps philosophy has made Developers looks ahead again and the next stage which is traditionally the realm of Operations teams, and then bring a lot of those skills and priorities into their own way of thinking. Now this is a hopelessly Dev-centric perspective on things, because there’s the Ops side of this equation, where developers skills in automation have led to some great leaps forward in tooling as well.
- But beyond fodder for witty Tweets, the correct adoption of a DevOps culture within organisations does seem to have some positive impact on those organisations’ ability to deliver software to production, in line with those 2 principles from the Agile manifesto we were looking at earlier. I cannot honestly say I’ve read the 2014 State of DevOps report cover-to-cover, so I cannot vouch for the rigour of the method they’ve used in their research, but even if these numbers are gross approximations of the truth, I think they still make a compelling argument for at least considering what a DevOps approach might do for your particular organisation. Yes?
- I truly hope so, because that’s what I’m going to talk about now: organisations that are deeply invested in this way of thinking. I’m going to use a number of examples of companies ranging from medium sized up to the very large. Most will be US-based organisations, some Australian and I suspect you’ll know all of them, or certainly the vast majority.
- And I’m going to progressively introduce these examples in order from what I think might be the simplest ones to achieve, or at least require the smallest amount of elapsed time, through to the ones that work at relatively small scale but have been optimised for extremely large scenarios.
- Why were they woken up? Because they were carrying the pager which received alerts when the software was down. Now this first practice can be found in lots of places because it’s a fairly low hurdle for most teams looking to start to break down the barriers between Dev and Ops, it’s often known as “Devs carry the pager” and this particular quote is from the company formerly known as BrowserMob and now called Neustar.
- Now it might sound like passive aggressive behavior, but when you think about what’s driving this change, it’s wanting to build empathy within the development community for the people who traditionally have been charged with looking after their software in production. In essence, it’s a case of “make them feel the pain” in order to build a more operational mindset in the development teams, a mindset that appreciates what happens when flaky software goes pear-shaped in the wee small hours of the morning. This is a good case to start with because, of all my examples, it brings out this core value of encouraging empathy most strongly.
- Another way to bake the importance of a shared responsibility for software into your culture is to clearly establish a shared ownership model for your systems, as has been done by this organisation. Each owner will no doubt bring their own expertise and perspective to this role, but by making ownership bi-lateral, you can help ensure these perspectives have a voice at key stages during the system’s lifecycle --- “the dev system owner owns the code, design and architecture, while the ops system owner owns the service once life is blown into it when it is deployed and is serving traffic. However, these two areas have great overlap, and thus the two system owners have regular checkups to discuss scalability, changes is neighbouring backend topology, coming new products which will affect service behaviour, etc.”
- And this is an example from Spotify. If you’ve read much about modern organisational models for supporting agile ways of working, you’ve probably come across a few videos from Spotify evangelising their tribes, squads, chapters and guilds, but they’ve also been experimenting quite a lot with the integration of a DevOps mindset into their DNA as well.
- This next one is about going out of your way to celebrate your ability to deploy software. We know how complex this activity can be, but even if you’re very good at it, there’s nothing wrong with giving credit where credit is due. Surprisingly enough, this used to be the footer on this organization's’ homepage, but it isn’t there anymore, which is a shame. I’m a particular fan of celebrating the people side of this activity; it’s not just the number of deployments which are shown, but the actual people who contributed to them. And this information is usually readily available as well; commit history in your version control system will give you the people and the number of changes, and I suspect your Continuous Integration tooling can supply the rest.
- But whether you publish these metrics internally or externally, we know people change their behaviour based on the measures applied to them. With this in mind, I’m sure this page was a strong inducement for the team at Flickr to continue to deploy regularly.
- So whenever you see a movie involving the hacker archetype, there’s an unwritten law that states that there needs to be at least 1 scene where they’re lit only by the reflective glow of the green screen terminals they’re sitting in front of. These terminals spew forth a ceaseless stream of gibberish presumably understandable by said hacker and, depending on whether they’re Good Hacker or Bad Hacker, they then proceed to either save the world or attempt to destroy it by issuing a furious series of keystrokes. Apart from being complete bullshit, the practice of being able to login to your production servers and administer them in this fashion is quickly fading into obsolescence for many organisations investing in the sort of practices we’re looking at today. You want to treat your servers as completely replaceable commodity machines and the more people have access to them, the more chance there is of those people changing the systems and starting to turn them into special little snowflakes, each unique in their own undefinable and likely unreproducible way. This emphasis is often referred to as “treat your servers as cattle, not pets”. And many organisations that have strong DevOps cultures have also invested heavily into horizontal scaling of systems which means being able to aggregate information from many servers is crucial… in these sorts of environments, allowing individuals login access to production servers is counter to the “cattle, not sheep” philosophy I mentioned before and also grossly inefficient. And this example is from a local organisation that has recently made this discovery.
- There’s nothing quite like the inability to access a machine that might be in trouble to force your hand into building a highly mature tooling and processes around logging, monitoring and alerting.
- OK, enough of the people related items - let’s move onto more process-centric stuff.
- What better way to emphasis the values of your engineering organisation that to make a first day engineer do a production deployment? Ask yourselves what the mean time is for new engineers to deploy to production in your organisation? And beyond this, consider also all the pieces that need to be in place to allow this to occur! Think of the logistics required to enable this; presumably the engineer needs a computer with all the necessary software installed, they’ll need access to the version control system to get the source code, access to wherever the requirements are held, they’ll need to know how to build and test the software after they’ve made their change and how to do the deployment itself. For many organisations, there’s a good week of bootstrapping and training right there.
- The company in question here is Etsy, which is a digital commerce platform for handicrafts. Etsy also has a great reputation for it’s engineering practices. John Allspaw, one of the founding fathers of the modern DevOps movement, is a VP at Etsy, so no coincidence there.
- Now this is a relatively old quote (2011, I think) which perhaps makes it even more impressive given how much practices have matured since then. As the title suggests, the key concept here is separating the deployment of software into production from the act of releasing it into the hands or browsers of your customers. Often much of the stress around doing deployments comes as much from the assumption that the two things are irrevocably tied together and you cannot do the former without the latter. But if you can separate these two activities, then what does that give you? Firstly, it allows you to test newly deployed software in the only real test environment that counts: production. It also allows you to selectively release software into the market to perform any number of variations of A-B tests. At a certain scale, you can potentially use entire countries as a contained yet statistically significant test environment prior to releasing to the rest of the world. And the company to which this quote pertains does operate at that sort of scale.
- Facebook I believe uses New Zealand as the test bed for many of it’s new features; the timezone difference being one of main drivers. Next we’re going to talk more about “how” Facebook support this mode of working.
- This Facebook style of releasing is often called “dark launching” and the main tool employed at the development team level is this thing call a “feature toggle”, which is really just a fancy term for a good old fashioned “if” statement in your codebase that hides the functionality until a certain condition has been met, often a value in a configuration file somewhere that tells the system that the feature should be “toggled on”. Feature Toggles are not a tool you should use indiscriminately because of the added code and testing complexity they introduce, but when employed with discipline, they are the primary tool needed to separate deployment from release, as per the previous Facebook example.
- Flickr, Google and Etsy are also in the public domain as to their use of Toggles, but I like this quote the best.
- And our final step on the journey is with tools-related practices...
- This is one of those statistics that, whilst anonymous, narrows down the list of candidate companies considerably just based on the numbers used. Oh hang on, I left an important part of the original quote out… There!
- Let’s have a think about the commitment and efforts required to produce this sort of performance. Obviously this is not the same single system which is being updated 18 times per minute, but rather a large number of loosely coupled systems, each capable of being independently updated. In some cases, up to 10,000 hosts were being deployed to at a time. And to be honest, it’s highly unlikely that any of us will ever need to build and use a deployment capability this performant, but when you take all the effort and risk out of doing deployments, you don’t think twice about deploying tiny increments at a truly scary rate. This, of course, can only be from amazon.com.
- When you get to the point where it’s a completely reasonable and viable idea to spin off an entirely separate business based around virtualised infrastructure, you know you’re doing this stuff quite well.
- So Continuous Integration (or CI) is the pivotal practice from Agile that carries over to the DevOps world. The mindset and toolset required to frequently commit to a common place in your version control system and have automated tests run for each of those changes is what gives both Agile development teams the ability to fearlessly respond to change and keep code production ready AND provide the infrastructure for the pipeline to production. Granted, you need to do a little more than run up a single instance of Bamboo or Jenkins to get the sort of throughput seen here, but the underlying principal is the same, whether your working on a 10 developer team or have over 10000 developers in 40 offices contributing 20 to 60 code changes every minute, as is the case of...
- Google, at least in 2012.
- This quote is another dead giveaway for anyone who has spent any time in this space; indeed, Chaos Monkey is certainly one of the most well known software tools for driving home some honest truths about how operationally ready your software actually is. The basic concept is simple; the development teams need to build software to run in an environment where network failure is not just a possibility, it’s a statistical certainly because of the scale at which this organisation operates. Some organisations include DR simulations into their operations processes, this organisation runs software built to intentionally kill random server instances... and runs it in production! The end result: a hardened architecture that has been repeatedly proven to be resilient against unexpected outages, leaving this company operational and online on Amazon Web Services, when many others running on the same infrastructure have been hit by serious network outages.
- The company that created Chaos Monkey? - Netflix, who through their serious commitment to open sourcing much of the support tooling, is probably the most recognised company in the realm of development and operational expertise at web scale
- So by way of conclusion
- I’ve presented a large number of real examples of organisations that have optimised their development, deployment, release and operational mechanisms to a level that would have been almost unheard of just 10 years ago and is still beyond the vast majority of IT organisations worldwide. Hopefully these examples have got each of you thinking about areas in which you can improve your own organization's abilities in these areas, because here’s the really important part. Ready for it? It can be tempting to avoid comparing your own situation with these companies because they differ in some way to your own; possibly in size, given Facebook, Amazon and Netflix were included. But you do so at your peril; yes, you may never need the capacity they currently have, but they would have been unable to grow to their current size without the efforts they’ve expended in these areas. They all got to a point in their own lifecycle where they recognised the need to build maturity around these aspects of their software delivery or they wouldn’t be able to operate at the scale they wanted to. And I think this is where the respective lessons should be for each of us; it’s probably fair to say an inability to deploy and release suitably hardened software in a regular, low risk way is preventing your current company from moving forward. This barrier might be seen by the slow pace at which your organisation can release software, or it might be by the collective holding of breath that occurs everytime a release goes out the door, or it might be something as simple as the dreaded pager being passed around the operations team like a loaded gun during a game of russian roulette. And you might well be in this situation even after a signficant investment into agile techniques. So however slick your development teams are at churning through stories on the wall, unless their definition of “Done” means released in Production, then you’re just locally optimising. Unreleased software is waste - we know this from the writings of Mary and Tom Poppendisk in the Lean space. And DevOps brings a philosophy and associated practices with it that can really help to tackle that waste, as these companies have all shown.
- Show Continuous Delivery and Lean Enterprise books Spruik DevOps and Infracoders meetups
- If you want to hear a bit more about how more local companies are embracing the DevOps way, come along to either the DevOps Melbourne of Infrastructure Coders meetup groups.