Download to read offline
Uploaded byPete Cheslock
DevOpsDays PGH: How to Fail With One Weird Trick
1. The document is a series of tweets by Pete Cheslock about DevOps failures and successes. 2. Cheslock discusses how organizational structure, priorities, and culture can enable or prevent effective DevOps practices. 3. He emphasizes building strong, collaborative teams over job titles and encourages learning from failures as an organization rather than assigning blame.
More Related Content
Similar to DevOpsDays PGH: How to Fail With One Weird Trick
DevOpsDays PGH: How to Fail With One Weird Trick
- 1. SCRUM MASTERS HATEHIM!!!! How To Fail With One Weird Trick @petecheslock
- 2. Pete Cheslock WasAsked to Run the DevOps Team But You Won’t Believe What Happens Next @petecheslock
- 3.
- 4.
- 5.
- 6. @petecheslock Who is thisfailure? ❖ Pete Cheslock ❖ Currently: Funemployed ❖ Previously: ❖ Systems, Automation, RelEng for Dyn ❖ Tech Ops for Sonian
- 7.
- 8.
- 9. @petecheslock –Thomas Edison “I havenot failed 700 times. I have not failed once. I have succeeded in proving that those 700 ways will not work. When I have eliminated the ways that will not work, I will find the way that will work.”
- 10.
- 11.
- 12. @petecheslock My One “DevOps”Slide I Blame DevOps on Agile
- 13.
- 14.
- 15. @petecheslock CHIEF DEVOPS OFFICERSHATE HIM!!!!! Weird Trick #1 Build a DevOps team to work along side your Dev and Ops team.
- 16.
- 17.
- 18. @petecheslock “DevOps is acultural and professional movement. Period. That’s it… There’s no technology, you can’t patent ‘devops’.” –Adam Jacobs, Velocity 2010 https://www.youtube.com/watch?v=Fx8OBeNmaWw
- 19.
- 20. @petecheslock What does your organizationrepresent? What are your “real” values, vs the actual values of your org Do you incentivize collaboration or politics?
- 21.
- 22. @petecheslock “Devops means givinga shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world.” –John Vincent (@lusis) http://blog.lusis.org/blog/2013/06/04/devops-the-title-match/
- 23. @petecheslock Ops Ops Sr. Ops Dir VP OpsVP Eng Dev Dev Sr. Dev Dir C-Level
- 24. @petecheslock –Dr. W. EdwardsDemming “The worker is not the problem. The problem is at the top! Management!”
- 25. @petecheslock You Get theDevOps You Deserve
- 26.
- 27.
- 28. @petecheslock Build strong teamsfirst. Assign them problems later
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37. @petecheslock Ops Ops Sr. Ops Dir VP OpsVP Eng Dev Dev Sr. Dev Dir C-Level
- 38. @petecheslock Ops Ops Sr. Ops Dir VP OpsVP Eng Dev Dev Sr. Dev Dir C-Level 1. 2. 3. Priorities Priorities 1. 2. 3.
- 39.
- 40. @petecheslock “Organizations which designsystems ... are constrained to produce designs which are copies of the communication structures of these organizations” –Melvin Conway http://en.wikipedia.org/wiki/Conway's_law
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47. @petecheslock Deploy Openstack a betterway to manag Deploy Openstack
- 48.
- 49.
- 50. @petecheslock GROWTH HACKERS HATETHEM!!!!! Bonus Weird Trick Buy My DevOps First taste is free
- 51.
- 52.
- 53. @petecheslock Would you ratherbe Systems Engineer IV or Senior DevOps Engineer? Answer: They both kinda suck.
- 54. @petecheslock “I’m a systemadmin, I don’t write code” –Soon to be Unemployable SysAdmin
- 55.
- 56.
- 57. @petecheslock A failure ofDevOps is EVERYONE’s failure
- 58. @petecheslock “It is notnecessary to change. Survival is not mandatory.” –Dr. W. Edwards Deming “There is no substitute for knowledge.”
- 59.
- 60. @petecheslock Reading List • AdamJacobs Velocity 2010 talk - https://www.youtube.com/watch?v=Fx8OBeNmaWw • John Vincent - Devops, the Title Match - http://blog.lusis.org/blog/2013/06/04/devops-the-title-match/ • Chris Fry on the Power of Small Teams - http://firstround.com/article/Twitter-Engineering-SVP-Chris-Fry-on-the-Power-of- Stable-Teams • Conway’s Law - http://en.wikipedia.org/wiki/Conway's_law • Patty McCord - https://www.youtube.com/watch?v=o3e1lnixKBM
Editor's Notes
- #2 How’s everyone doing. having a good time so far today? Who here is at their first devopsdays? So - my name is Pete, and I’m here to teach you how to fail at devops with one weird trick. It’s so easy. I’ve been working on this talk for quite a while, and lots of titles come up. So this is the original title. - But the actual title is more like…
- #3 this. I saw lots of talks that talked about how to do devops, how to buy devops, how to outsource devops. But there was not talk that talked about actual concrete things that didn’t work and did work. Only a few got that specific. So that’s what I hope to do here today.
- #4 You may have seen these ads on the internet. They are everywhere. And they make some pretty bold claims about how simple tricks can solve complex problems. Similar maybe to Agile, Scrum and DevOps. What are some of the things they did that caused DevOps to fail. How did they react to those failures. What did they do to improve things.
- #5 I’m going to show you one weird trick you can use to fail at DevOps. I’m going to share with you some of my real life experiences at real companies. Many of these stories I was directly involved with. Some of these decisions I even made myself, and each one of these failures helped me learn. Failure is a good thing. We should embrace it, maybe give it a nice hug. So hopefully you can learn from my failure and not have to experience the same.
- #6 But wait! - I keep saying there is one weird trick - actually there are lots of weird tricks you can you to start your failure today. But unfortunately I only have 30 mins, soooooo. Wether you are trying to transform your organization by creating a new silo, trying to use politics to improve the communication in your organizations, or building a tool to solve your people problems, I have probably already done that and failed epically.
- #7 But before we start - I want to share a little bit about my background and myself. Again I’m Pete Cheslock - Currently Funemployed and looking for my next big challenge. Most recently Built out the Automation and Release Engineering teams for Dyn. Might remember them from back in the day, DynDNS updater client. Now a global internet performance company. Before that - Sonian where I built out and lead the Technical Operations team. Anyone here heard of/use Sensu? That came out of my team at Sonian and we open sourced it almost 3 years ago. Lots of consulting prior - seeing all manner of companies and their struggles.
- #8 So that is my background, but there is really only one thing that you need to know about me. And it’s this. If you haven’t figured it out yet… Cause in the grand scheme of things… that’s really all that matters. Well, that and Hugs. Now if we meet for the first time, i’ll probably shake your hand because i will try to respect boundaries. But I’m an equal opportunity hugger.
- #9 Ok - I lied, three things. I love animated gifs. So even if this talk sucks - there will be some cool animated gifs for you to enjoy.
- #10 When people talk about failure this is often the quote by Thomas Edison that gets brought up. He said.
- #11 What a pompous asshole. We are still in the early stages of DevOps. It’s only been about 5 years or so, and there is still so much confusion about what DevOps is. But, many people, you, me, all of us, are trying to help companies understand and improve their organizations. We don’t yet have all the answers, no one does. We are all still figuring it out as we go. And there are no single answers. There is no silver bullet.
- #12 We are all still trying to learn as we go, but by sharing our stories at events like these we can learn some of those 700 ways w/o going thru them ourselves. We just need to remember one important thing. And that is not to fail the same way every time, but that we should be failing in new and novel ways. Learn from those failures. Share those failures.
- #13 So - I really didn’t want to do a “What is DevOps” slide, but what But later i will give you my belief on devops, only because it frames the talk. But - why devops.. why is this important now. I was talking with a friend of mine, and he just blew my mind saying this. His belief is that the push to agile development practices of the early 2000s’ are to blame for the necessity of DevOps. In the past you had one giant poorly functioning organization. A slow to react ops team was not holding back a slow to deploy dev team.
- #14 But then you had agile increase the speed of delivery, which the Operations team was not ready for that transformation. Before the bottleneck was Engineering, it was far before Ops got involved. Agile and Scrum does not work for Operations teams. Operations is far too interruptive in nature. So - we need to change, ALL OF US. I don’t care if you are in Ops or Dev or QA or Finance or Business. We need to change how we do business. We need to change how we work with other people on the team.
- #15 I feel like many of us have DevOps at different places in the Hype Cycle. For many, getting recently involved and learning more - will find themselves on their way up the peak. For those of us that have been involved for many years, many of us are disillusioned with Devops. What i’m most looking forward to is the path to enlightenment. I hope we move towards a place where honestly, devops doesn’t exist anymore, I would love to see DevOps die. And someday that will happen. Because this will all become standard practice.
- #16 So - what’s the weird trick. How can you fail really well. Maybe your Chief DevOps Officer comes to you, says, we need to build out a DevOps team. They should sit right where that wall of confusion is. So everything that Dev is throwing over the wall to ops.… Nah… just toss that over to the DevOps team. Done. And Done. This obviously sounds insane. But this is what’s happening more and more everyday.
- #17 In the past I ran a devops team - even called myself the director of DevOps. This was about 2010 at the time. We had always called our Ops team “DevOps”. One of the Senior members said to me at one point, “Well, I write code and manage systems, I do both Dev and Ops….DevOps”. Made sense to me at the time. This is often what I hear about now a days when it comes to DevOps teams.
- #18 Having been out of front line operations for a few years while I was consulting, I was shocked how deep my team was at programming, in some cases they were delivering applications of higher quality than our actual engineering team. But now many years later, I still stand by the Adam Jacobs version of “DevOps”,
- #19 <quote> People want to talk about how you need to have tools in order to do DevOps. You don’t have a technology problem - you have a people problem. Not to says that tools isn’t important, they are extremely important. But if you deploy tools while ignoring your people problem - you are going to have a bad time.
- #20 Every tool in the world will not help you if your engineering team pushed untested code to production without talking to anyone, or if your operations team says “no” so often that the engineering team just stops asking them for help. You just can not solve social or communication issues with any process and technology. There is no amount of Agile or Scrum in the world that will help you fix your people problem.
- #27 How many people’s organization resemble this? Who’s ever worked at a highly political organization? Remember that slide a few back? How the communication goes up and back down again thru what is basically the worst game of telephone ever? I make the assertion that if your org looks like this, you will never be able to do “DevOps”. Because people will be too busy weaponizing data, and using it, as Deming said, using it to “become a selfish, independent, profit centre.”
- #28 So - how should we build an org? One of the most successful and scalable organizations of all time was the Roman Army. Their units were 8 men who shared a tent, a millstone, a mule and cooking pot. Why 8? Because that was how many fit in a tent. The Romans dominated 500 years of history and created a distributed system that controlled the western world. There is a fantastic article by Chris Fry who is a SVP of Twitter Engineering. - the link is there (and don’t worry i’ll share these slides on twitter later.
- #29 Chris talks a few things regarding scalable teams in his blog post. I want to focus on one thing. And by the way…strong doesn’t mean senior. We need DevOps so badly because these antiquated business structures from 30, 40, 50 years ago have created a world where there is little to no incentive for teams to work together. Where in fact often times, teams are incentivized to work against each other. Because if another team screws up, maybe I can get their budget. Maybe I can grow my ivory tower even higher.
- #30 About the time you hit 40 people, is when things start to break down. You want to spend your time building strong teams of people. Stop hiring silos. You don’t “need a web person” - you need someone who’s willing to learn. Knowledge can be fast to acquire by those people that are willing. If learning in a core initiative for your organization (pro tip, it is). Then you should incentivise that.
- #31 When you are a starting out - everyone does a bit of everything. So you tend to staff out those projects for short periods of time. It’s a lot of highly interruptive work you are doing, and it costs you in the long run. As you grow a couple things will become clear, you have too few people, working on too many things.
- #32 And if you are the person leading that team that is assigning all the work, you now have become the newest bottleneck in your organization. And now you are moving so fast you can’t stop and fix all that technical debt you are incurring. Just like financial debt, technical debt is evil and will eventually catch up you. You’ll reach a point where you can no longer innovate, because you need to pay down on that interest.
- #33 Your only focus should be to create small teams of dedicated people. There is no replacement for having a small group of people that work well together. You can then take these teams and then apply them to problems. If you are a manager, that should be your ONLY goal.
- #34 So why don’t people do this? Because it requires one thing that most people that run organizations don’t have. Anyone want to guess? You think the Roman army asked that squad of 8 people to track their time on the mission?
- #35 Did the Roman army have the squad sit down to plan every step of an invasion piece by piece and estimate the time for each piece? Let’s say they did do that… What happens when the solders try to go over the wall, and it turns out there in a moat in front? Do they go back to the scrum master…. i mean squad leader to find out what to do.
- #36 Oh sorry, we’re blocked on that moat over there. Better get the bridge builders over here. Meanwhile the squad leader is freaking out because their estimates are blown. THE BURNDOWN CHART IS RUNIED I TELL YOU!!!! RUINED!!!!
- #37 You’re basically hiring skilled adults into a high functioning team. Then you’re taking someone who has never done software engineering, and having them run the team. But don’t worry, they are certified… They went thru a class and everything at the strip mall down the street next to that sushi place. You are removing the most important skill you can ever get out of your team. And that is critical thought. How does your team react to uncertainty and disorder? Do they solve it themselves, or do they throw their hands up and say they are blocked?
- #38 Top down management does not work. I see this time and time again, as companies grow by stating flexible and nimble, then they focus on people, versus teams, and then bring in people and process that mimics the same slow, lumbering organizations they originally started to disrupt. So then what happens… they become those same large bureaucratic organizations they tried to replace. And another company comes along that can move faster. So the cycle continues… So - does anyone see a problem with this org structure?
- #39 What happens when those two teams create their own list of priorities? Are they the same? Or are they different. Can you go to a person on your ops team, and a person on your dev team and ask what the most important project is right now? Do you get the same answer from both?
- #40 So what happens? You have two teams, with their own list of priorities and you end up with two sides of a bridge that don’t connect. How many people know what Conway’s Law is?
- #51 Because no talk is complete until you’ve listed to me rant about marketing, recruiting or titles.
- #52 DevOps Automation Service. This is one of many companies that is like this. And honestly, I’m really beyond caring if you want to sell your devops tool, because I think the market is smarter than you. Look - there is nothing in this list that solves your problems. Many of these things actually will give you more problems than they will solve. Everyone wants to sell you devops, but it’s not something you can buy.
- #53 Devops Engineers, Devops Architects. Directors of Devops, VP’s of Devops. When will it end???
- #54 Look - I get it. The tech world moved forward much faster than our ability to come up with job titles that don’t suck. Tools such as EC2, Puppet, Chef, etc. essentially made programming much more accessible to a group of people that normally would have avoided it. So now people that didn’t want anything to do with “programming” now need to do it as a part of their day to day job, because seriously, have you even tried to manage more than like 10 systems on EC2 without config management??? So let’s be honest - both of those titles sucks.
- #55 So… If you want to use DevOps as a qualifier for what is essentially an advanced generalist… fine. Because as bad as it is… devops has come to mean things like, Continuous Integration/Deployment, Cloud, Configuration Management, programming. Sometimes its used as “doing both dev and ops work” - and trust me - you should definitely say the hell away from those roles.
- #56 And look - you want to be a devops or hire devops or whatever. I ain’t even mad. Do what you have to do. It’s your company, it’s your title. Own it.
- #57 If someone can be a Digital Prophet, then then you can be a DevOp. It’s probably fine.
- #58 So - let’s sum this all up. And this is really i think the most important thing. A failure of DevOps is everyone’s failure. EVERYONE’s. If you want to undertake a DevOps transformation, and you don’t have the buy in of everyone involved (which is normally all the way up to your CEO). Then you are going to fail. As I meet with companies looking for someone like me to come in and help them grow and scale their organizations I will always try to investigate how willing they are to change. Because if they are not willing to change, then they are doomed to fail.
- #59 So I want to sum this all up with some Deming quotes, as one does. Let’s continue these discussions during the openspaces after lunch. Let’s talk about what you like about your organization. What you don’t like. What you think are the barriers that are holding you back. You either want to improve your organization. Or you are losing to someone who is.