Puppet Labs EMC DevOps Day NYC Aug-2015
•Download as PPTX, PDF•
1 like•1,058 views
Puppet Labs - Alanna Brown - How to Build High-Performing IT Teams. State of the DevOps Report 2015
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Puppet Labs EMC DevOps Day NYC Aug-2015
Similar to Puppet Labs EMC DevOps Day NYC Aug-2015 (20)
More from Bob Sokol
More from Bob Sokol (12)
Recently uploaded
Recently uploaded (20)
Puppet Labs EMC DevOps Day NYC Aug-2015
- 1. How to Build High-Performing IT Teams Presented by Alanna Brown
- 2. Agenda • Build the case • Identify the desired state • Align incentives • Align your teams • Implement key technical practices
- 5. People don’t buy what you do, they buy why you do it. Simon Sinek http://bit.ly/sinektedtalk
- 7. High-performing IT orgs are more agile 30x More frequent deployments 200x Faster lead times than their peers Source: Puppet Labs 2015 State of DevOps Report
- 8. High-performing IT orgs are more reliable 60x Change success rate 168x Faster mean time to recover (MTTR) Source: Puppet Labs 2015 State of DevOps Report
- 9. High-performing IT orgs are winning 1.5x More likely to exceed profitability, market share & productivity goals 50% Higher market capitalization growth over 3 years.* Source: Puppet Labs 2015 State of DevOps Report
- 10. Learning is not compulsory, but neither is survival. Edward W. Deming http://bit.ly/deming14pts
- 12. Organization Low trust culture High trust culture Siloed teams Cross-functional teams Lack of alignment Aligned around business goals
- 13. Processes Lots of manual work Mostly automated work Long cycle times Short cycle times Poor visibility Fast feedback & insight
- 14. People High burnout High job satisfaction Stagnant Able to grow Checking boxes Creative innovation
- 17. Align Incentives
- 18. “Trying to effect process, people, technology and cultural changes across the entire application portfolio, in a globally dispersed team and with a lot of associated technical debt, is an epic challenge.” Jonathan Fletcher Enterprise Architect and Lead for Technology, Platform and DevOps at Hiscox http://bit.ly/devopshiscox
- 19. Hiscox: Results • Reduced cost per release on one application by 97% • Reduced time per release by 89% • Reduced staff required to release by 75% • Automated testing reduced multiple man days of effort down to an overnight hands-free process
- 20. Conflicting Incentives Business Delivering value to customers Dev teams Delivering new features Ops teams Ensuring stability of systems Quality teams Ensuring quality of software releases
- 21. Everyone is responsible for quality and we’re all trying to deliver the best solution for our customers. Reena Mathew, Principle Architect Quality Engineering, Salesforce http://bit.ly/sfdevops
- 24. We can’t do DevOps because our application is ________________________.
- 25. Architectural Characteristics Significant affect on IT performance • Does not require integrated testing environment • Can deploy/release app independently of other apps/services it depends on • Custom software with microservices architecture No affect on IT performance • Packaged commercial software / COTS. • Systems of record / systems of engagement • New, not-yet-deployed systems. • Software w/ embedded component that runs on manufactured hardware device / user-installed component that runs on user’s machine
- 26. Align Your Teams
- 27. Typical Enterprise Org Structure IT Operations NOC Commercial Banking Business Units Credit Cards Mortgages Investment Banking Systems Engineers Network Engineers Storage Admins DBAs InfosecDev teams reside in BU
- 28. Pattern 1: Smooth Operations Dev Ops Recommended Reading: http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to- flourish/
- 29. Pattern 2: Cross-functional team Characteristics • Consists of devs, testers, ops, product owner, etc. • Focused on delivering a single application • Self-sufficient • Optimized for throughput
- 30. Pattern 3: DevOps Team Dev Ops Dev Ops Characteristics • Consists ideally of devs with systems experience, or sysadmins with programming experience • Focused on automating pain points • Responsible for building a platform that allows devs to self-service • Provides a toolchain to enable devs to build, test and deploy their systems • Coaches other teams
- 31. Roles & Responsibilities Roles Responsibilities “The Business” Understand market trends and identify customer needs IT Manager Build trust with counterparts on other teams; create culture of learning and continuous improvement; delegate authority; remove roadblocks Dev Manager Build trust with Ops counterpart; bring Ops into the planning process early Systems Engineer Automate the things that are painful; help devs get feedback QE Provide input into scale and performance; provide feedback on staging environments Devs Plan for deployment as you’re planning new features; get feedback from ops and work with them on deployment process
- 34. Infrastructure as Code Infrastructure as Code Version Control Peer Review Continuous Delivery Collaboration Iteration Fast Feedback Visibility
- 35. Version Control Source: Puppet Labs 2014 State of DevOps Report
- 36. Peer-Reviewed Change Process • Code can be contributed by anyone • Code changes can be reviewed by anyone • Code can be worked on as a team
- 37. Continuous Delivery • Code is repeatable • Code is sharable • Code is promotable • Code is testable
- 38. Measuring Results Throughput • Deployment frequency • Change lead time (from dev’s laptop to production) • Cycle time Stability / Reliability • Change fail rate • Mean time to recover • Availability / downtime
- 39. Deployment Pain Tomorrow, ask your team two questions: • How painful are your deployments? • What’s causing the pain?
- 40. Diversify Your Team Teams with more women have higher: • Financial performance • Stock market performance • Hedge fund returns • Collective intelligence
- 41. Q&A
- 42. Resources • The 2015 State of DevOps Report is here! puppetlabs.com/2015- devops-report • The Phoenix Project by Gene Kim • Continuous Delivery by Jez Humble • PuppetConf 2015: http://2015.puppetconf.com/ • DevOps Enterprise Summit: http://devopsenterprise.io/
Editor's Notes
- - Hi, I’m Alanna Brown. - I spearheaded Puppet Labs’ annual State of DevOps Survey and Report and I’ve spent the past 4 years studying DevOps and high-performing IT teams. - How many of you have taken our survey or read the report? - If you’re like most people, you’re struggling with how to “operationalize DevOps” and implement it in a pragmatic and sustainable way. - Based on data and real world examples from customers, I’m going to share with you the keys to building a high-performing IT team. - Today, I’m going to help you cut through the noise so you can start your DevOps initiative on the right foot.
- Here’s how I’m going to do it: Today, I’m going to demystify high-performance and show you 5 steps to kickstarting your DevOps initiative, supported by findings from our 2015 State of DevOps Report.
- The first and most critical step in kickstarting your DevOps initiative is building the case and creating consensus. It sounds really simple, but most people get this critical step wrong.
- You can’t just say “We’re going to do DevOps now!” and expect everyone to drop what they’re doing and jump on board. How many of you have ever had a higher up tell you you had to implement the latest buzzword without fully understanding what it would entail? How many of you have been the ones telling your staff to implement the latest buzzword?
- There’s a great TED talk by Simon Sinek, who wrote a book called Start with Why. The basic premise is this: People don’t buy what you do, they buy why you do it. You probably already know why DevOps matters, but that doesn’t mean the rest of your team, your peers, or the people you report to get it. Just as an experiment, go ask ten people “What problem are we trying to solve?” and see what answers you get. I bet they’ll be different depending on who you talk to. Are there any managers or team leads in the room? Well, listen up: you play a critical role here. One of the most unexpected findings from the 2015 State of DevOps Report was that when the DevOps initiative was driven purely from the top-down, OR purely from a grassroots level, we saw a negative effect on organizational performance. That’s why all of you managers and team leads are so important: you’re the ones who connect the strategic goals of the business with on-the-ground execution. In the next couple of slides, I’m going to give you the tools to make a really strong business case for DevOps in your organization.
- One of the challenges every organization faces is the need to move faster without compromising the reliability, security and stability of their systems. These two seemingly opposing forces — agility and reliability — are possible to achieve. And it’s not just the unicorns who are achieving these results, it’s the horses, too.
- Over the past 4 years, our data has consistently shown that: High performing IT organizations are more agile. They deploy code 30 times more frequently. And they deploy 200 times faster than their peers.
- We also know that high-performing IT orgs are more reliable. We were shocked when we compared our high performing group this year to the high performers last year. We found that while they were deploying at the same speed and frequency, their overall reliability was substantially better than the 2014 respondents. Our high performers this year have 60 times fewer failures and they’re able to restore service 168 times faster. This suggests that organizations are shifting quality to the left. If you think of the software delivery process as a manufacturing assembly line, on the far left you have the developer’s laptop where the code originates and on the far right you have your production environment where that code eventually ends up. The fact that stability is increasing suggests that orgs are improving the system as a whole and building quality into the entire software development pipeline.
- - We also know that strong IT performance is a competitive advantage. - IT is not just a cost center — recent outages at the Wall Street Journal, New York Stock Exchange and United Airlines — illustrate the point that every company is a technology company and IT is critical to helping businesses achieve their goals. - Companies with high performing IT orgs are 1.5x more likely to exceed profitability, market share & productivity goals than their peers. - Preliminary financial analysis from our 2014 report also hints that it can even increase the total value of publicly traded companies as measured by market capitalization.
- - I think this quote by Edward Deming sums it up best. Deming is considered by many to be the godfather of DevOps. I’ve included the link to Deming’s 14 points which has influenced some of the best thinking in DevOps today. The point of DevOps isn’t to automate all the things — although automation should be one of the outcomes; it’s to create a culture of continuous improvement, because only organizations that can adapt and learn will ultimately survive and win.
- After you’ve gotten everyone on the same page about the problems you’re trying to solve as an organization, you need to get everyone aligned on what the desired outcomes are so you have a clear understanding of what success looks like.
- - DevOps requires some big changes across the organization: changes that impact culture, processes and people. - At the organizational level, the desired outcomes are to create a culture of high trust, with high-performing teams that are all aligned around a common business goal.
- - To do that, a lot of processes have to change as you go from a very manual and siloed way of working to adopting automation and learning how to collaborate. - These process changes lead to shorter cycle times, higher productivity and greater visibility.
- - Let’s not forget the human cost of all of this. - Burnout is an important issue in IT and one that we researched for this year’s report. The conditions that lead to burnout — unsustainable workloads, lack of control, and low-trust, pathological cultures — can be prevented. - Not surprisingly, the data shows that where burnout is high, organizational culture is poor, deployment pain is high and organizational performance is low. - How many of you are struggling to hire tech talent? Did you know that it can cost upwards of $20k to hire one engineer? - What if instead of spending that money on hiring new engineers every time one gets fed up and quits, you spent that money on changing the work conditions that lead to burnout and stagnation in the first place? - In the next two slides, I’m going to show you what that investment looks like.
- Here’s the good news: these desired outcomes are achievable. In the 2015 State of DevOps report, we found a direct link between DevOps practices, IT performance and organizational performance. The diagram you see here represents a statistical model that shows how the practices that make up continuous delivery — deployment automation, automated testing, continuous integration and use of version control for all production artifacts — those practices predict deployment pain, IT performance and change fail rate. And in turn, IT performance predicts organizational performance.
- One of the new constructs we added this year was around lean management practices. Lean management practices include limiting work in process, use of visual displays to monitor quality and productivity, and proactive application and infrastructure monitoring. This diagram shows how lean management practices, predict organizational culture, IT performance and burnout. While culture and IT performance both predict organizational performance. I love this finding because it proves that we don’t have reinvent the wheel. Lean manufacturing has evolved over 100 years and there’s so much we can learn from these battle-tested and codified principles and practices. When we apply these principles and practices to software delivery, we get the same results: higher quality products, faster cycle times, lower costs and less waste.
- Doing DevOps doesn’t mean you’ll all be sitting around a campfire singing Kumbaya. A DevOps transformation is hard work and there will be plenty of challenges along the way.
- One of our customers, Jonathan Fletcher from Hiscox, lead his organization through a DevOps transformation where he faced the epic challenge of driving change across a large, siloed organization. You can read Jonathan’s amazing story on our blog.
- - The results speak volumes. Hiscox was able to reduce the cost per release on one application by 97 percent and by automating testing, they reduced multiple days of manual effort to an overnight hands-free process. - Their transformation didn’t happen overnight. It took a lot of evangelizing, experimenting and restructuring.
- - I want to talk about the biggest challenge Hiscox faced and the one you’re going to face as you embark on your journey. - When I talk to customers about their challenges with adopting DevOps, they often cite resistance to change as their number one barrier. - But I’d argue that the root cause isn’t resistance to change, but rather the way people are incentivized to do their work. - The business is incentivized to deliver value to customers so it can generate more revenue - Dev teams are incentivized to deliver new features faster - Ops teams are incentivized to keep the systems running at all costs Quality teams are responsible for the quality of the software. Those incentives are all different and therefore lead to very different behaviors.
- One of our long-time customers, Reena Mathews, principal architect in Quality Engineering at Salesforce said this: “Everyone is responsible for quality and we’re all trying to deliver the best solution for our customers.” For Salesforce, every single participant in the software delivery process is incentivized to improve the service their teams deliver to their customers, and again the results speak for themselves.
- Salesforce has been been ranked the most innovative company by Forbes Magazine for a record of four consecutive years. They also rank No. 7 on Fortune’s 100 Best Places to Work list.
- Salesforce and Hiscox are great examples of companies that are aligned around the goal of delivering value to customers. In order to achieve alignment, you have to be structured correctly in the first place, which we’ll talk about in a minute.
- But first, I want to talk about another common objection you’ll hear. Who here has tried to make the case for DevOps and heard this: “We can’t do DevOps because our application is <fill in the blank>.
- We found in our 2015 report that it doesn’t matter if your apps are greenfield, brownfield or legacy — as long as they are architected for testability and deployability, high performance is possible. We found that high performing respondents shared some common architectural characteristics. They: - Do most of their testing without requiring an integrated environment - Can deploy their applications independently of other dependent applications/services - And many were using a microservices architecture - Just to illustrate this point, one of our customers has an app running on a mainframe with a million line code base, and they’re doing automated testing.
- One of the questions we get asked a lot is “What is the ideal team structure for DevOps?” And I think the answer depends on a few variables, like: - how flexible is your current org structure - what skillsets do you currently have on your team - what’s the relationship between teams and team leaders
- - How many of you work in an org structure that looks something like this? - This is pretty typical of large enterprise organizations and what this results in is multiple silos of highly specialized individuals, which then leads to multiple hand-offs to get changes into production. - I was talking to the head of infrastructure at a large retailer and he said it took 15 teams to deploy one service. - If you count each hand-off as a conservative three days to account for lead times, work in progress, configuration errors, etc. that would be 45 days to deploy, and the majority of that time is waste. Silos exist for a reason, and they’re not going away anytime soon, so how do we work around them? I’m going to show you the three common patterns we see that work well for DevOps.
- The first pattern is one in which Dev and ops remain distinct teams but work closely together to deploy applications. The goal is not to have devs do operations and ops do development, it’s for each team to have an awareness and understanding of how their work impacts each other. This pattern works really well for smaller organizations, but it’s extremely important in this structure to have a common toolchain between dev and ops to facilitate smoother collaboration.
- A lot of our large enterprise customers, like Salesforce and Hiscox, have had success implementing a cross-functional model. Cross-functional teams consist of representatives from all disciplines responsible for developing and deploying a service — business analysts, product owners, developers, quality engineers, ops, security, etc. These teams are fully empowered and self-sufficient — they write it, test it, and deploy it. These teams are more efficient because there are less hand-offs and more opportunities for knowledge sharing.
- - We’re seeing the rise of dedicated DevOps teams: In our 2014 study, 16% of respondents belonged to a DevOps Department. In 2015, that grew to 19%. Dedicated DevOps teams are often made up of experienced operations people with a mix of skills including using version control, writing infrastructure as code, and continuous delivery. They often exist to address a specific pain point such as deployment automation, but often grow to become more than that. One of the pitfalls of a dedicated DevOps team is that it can become just another silo. In order to prevent that from happening, it’s critical that these teams evolve to providing shared services for the rest of the organization and also provide coaching, training and support to evangelize DevOps best practices. Best for: orgs where deployments are extremely painful, need quick results, have the necessary skills in-house, and can’t get appropriate buy-in. What we see in more mature DevOps organizations is a hybrid approach. There’s typically a small team responsible for providing self-service platforms, and also several product-focused teams that own the end-to-end delivery of a product or service.
- Another question we get asked a lot is what are the roles and responsibilities of team members in a DevOps implementation? If you’re an IT manager or a team lead, I highly recommend checking out the 2015 State of DevOps Report, because there’s a whole section for how IT managers can help their teams win. The main thing if you’re a manager, either on the dev or ops side, is to remove roadblocks for your team by building trust with your counterparts on other teams so that everyone can work more effectively together. Earlier, I talked about how lean management practices contribute to IT performance. If you want to increase the performance of your team, the answer isn’t to pile on more work, but to actually limit work in process so people can complete work. Additionally, providing visible metrics that show flow and quality are great ways to keep everyone on track. If you’re a practitioner, you’re not off the hook. You also have to learn how to collaborate with your counterparts on other teams as well because trust is one of the major factors in both IT and organizational performance. If you’re in ops, most of your initial work will be automating away the pain points so you can get to a place where you’re no longer constantly fighting fires. Focus on the quick wins to start and share your your successes. Are there any QA or QE folks in the room? Quality folks have a lot to teach us about building quality in everything we do. And devs, please talk to ops early and often as you’re planning deployments.
- Some people get really grumpy when you start talking about “DevOps tools” — you’ve all heard the argument, “DevOps is about culture, not tools!” I agree with that statement, but I also believe that tools can drive behavioral change, and ultimately our behaviors become part of the culture. How many of you at some point moved away from email to a ticketing system? Did your behavior change as a result? Did it force you to think about ownership and priority and how to provide the right level of info to actually get your ticket processed? Or for those of you working in agile organizations - you know Kanban itself doesn’t make you agile, but it does help you rethink the flow of work. And we know from our research that these tools, and the practices they enable, are highly correlated with IT performance...
- This is just the short list of some of the more common tools in a DevOps toolchain which consists of a distributed version control system, configuration management, continuous integration, deployment and monitoring tools. Here are some considerations as you’re building your toolchain: Standardize on the toolchain, and make sure that each tool you choose is compatible with the rest of the toolchain. Where possible, try to use the same tools your devs are using. Are they already using GitHub and Jenkins? Then those are probably the best choices for your team if you want to promote better cross-collaboration. Set up lunch and learns with them, leverage their expertise, this will help build trust between the teams. If you're starting from scratch, Start on the left with version control and configuration management and work your way towards the right. Figure out what the smallest thing you can do is, and then build on it, whether that means simply getting all of your scripts under version control, or writing a manifest for a basic OS configuration. 1-800-Flowers, another one of our big customers started by standardizing all of their configurations for their PCI requirements. That’s no small feat, they were able to start small by standardizing configurations for each necessary component, such as NTP, firewall, user permissions, etc. until they had fully configured and automated their entire compliance process, replacing dozens of custom scripts.
- Infrastructure as code is necessary for many DevOps practices, but it’s not entirely sufficient on it’s own. When you combine it with version control, you get this force multiplier effect. This is where ops teams can learn a lot from devs. With infrastructure as code, you can apply proven, agile development best practices to infrastructure code. Things like code review, automated testing, and continuous delivery are now possible for system configuration code, making your teams more agile and your systems more reliable. Together, these practices enable fast feedback loops, the ability to iterate, greater collaboration between dev and ops teams because they’re using the same tools and similar processes, and also greater visibility into infrastructure changes. These are the things we see as key to becoming an agile, lean organization that continuously learns and improves.
- We found that use of version control is highly correlated with all of our IT performance metrics: deployment frequency, lead time for changes and mean time to recover. With your infrastructure code in a version control system, you can quickly produce environments for testing and troubleshooting which increases deployment frequency and reduces your lead time for changes. It also allows you to quickly identify cause of failure and roll back to the last known good state, which significantly reduces your mean time to recover.
- Peer review is one of the biggest benefits of a distributed version control system and this is why. The data tells us that the traditional way of controlling is highly ineffective. When external approval is required to deploy to production, IT performance decreases, and contrary to popular belief, change approval boards have very little impact on stability. But when the technical team holds itself accountable for the quality of its code through peer review, we saw that performance actually increases. In a peer review process, code can be contributed by anyone, devs, ops, quality engineers, infosec, etc. It can can be reviewed by anyone and you can work on it as a team.
- One of the awesome benefits of having your infrastructure as code is that you can do continuous delivery to make sure each change to your infrastructure is fully tested before it hits production. This is important because it a) forces your team to work incrementally, in small batches, making it easier to fix issues as they occur. It also enables fast feedback loops, which is a common theme across the devops toolchain. Tools like Puppet, have these continuous delivery workflows built into them so you can easily manage and promote your code.
- As you start your DevOps initiative, you also want to think about how you measure success. There are a lot of different metrics, but we suggest you start with with some simple metrics for throughput and reliability.
- My favorite new construct from this year’s report hands down is deployment pain. We found that deployment pain is highly correlated with IT performance. The more painful code deployments are, the poorer the organization's IT performance, overall performance and culture. The reason I love this construct so much is because it gives us a really quick temperature gauge to assess IT performance. Tomorrow, I want everyone to ask their teams this question: “How painful are your deployments?” and then follow up with “What are the things causing the pain?” Then actively work to prioritize and fix those pain points.
- One of the things I’m most proud about is our section on #WomenInTech There’s a whole body of research that shows that teams with more women members have higher collective intelligence and achieve better business outcomes. Despite that, women are leaving STEM fields at alarming rates. In our survey, only 5% of survey respondents were women More than half of the survey respondents work on teams that are less than 10% female A third of the survey respondents said they worked on teams with no women. We can do better and its up to all of us to make IT better for everyone.