Where is the test in DevOps?
•Download as PPTX, PDF•
1 like•141 views
This presenation was given at an ANZTB SigIST in Wellington in November 2015. It is an overview of testing and the changed role that testing has in it.
Recommended
Recommended
More Related Content
Similar to Where is the test in DevOps?
Similar to Where is the test in DevOps? (20)
More from Matt Mansell
Recently uploaded
Recently uploaded (20)
Where is the test in DevOps?
- 1. Where is the Test in DevOps? Catch the wave or be swamped by the wave
- 2. Waiting for the wave Origins Lean Principles The Three Ways of DevOps Defining DevOps Beyond the Surf Line Test is Everywhere The Old Value Stream The DevOps Value Stream The wave builds Everything Changes What To Do? Surf or Be Swamped (c) Copyright Matt Mansell and IntegrationQA: Permission to share 13/06/2017
- 3. Origins DevOps Toyota Kata Agile Lean (c) Copyright Matt Mansell and IntegrationQA: Permission to share 13/06/2017
- 4. Lean Principles Identify customers and specify value Identify and map the value stream Create flow by eliminating waste Respond to customer pull Pursue perfection 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share
- 5. Defining DevOps “DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information- technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.” - https://en.wikipedia.org/wiki/DevOps 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share
- 6. The Three Ways 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share The Principles of flow The Principles of Feedback The Principles of Continuous Learning and Experimentation Dev Ops Dev Ops Dev Ops
- 7. 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share Test is the water that enables DevOps to grow
- 8. Test is Everywhere 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share Test Test Service System Unit Integration Build Acceptance Test
- 9. The Traditional Value Stream 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share Code • 50% Complete • 25% Accurate Build • 60% Complete • 45% Accurate System Test • 70% Complete • 50% Accurate Acceptance Test • 75% Complete • 65% accurate Deploy • 75% Complete • 70% Accurate Support • 75% Complete • 60% Accurate
- 10. The DevOps Value Stream 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share Code • 50% Complete • 100% Accurate Build • 60% Complete • 100% Accurate System Test • 70% Complete • 100% Accurate Acceptance Test • 75% Complete • 100% accurate Deploy • 75% Complete • 100% Accurate Support • 75% Complete • 100% Accurate
- 11. Everything Changes – Surf or get Swamped 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share
- 12. What does this mean for testing? Get Close to the User and / Or Get Technical 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share
- 13. Fin – Any Questions? 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share
- 14. Interested in talking more? Connect on LinkedIn @mjmansell Matt.Mansell@integrationQA.com 13/06/2017(c) Copyright Matt Mansell and IntegrationQA: Permission to share
Editor's Notes
- Hi, I’m Matt Mansell, a Senior Consultant with IntegrationQA. We specialise in automating software delivery. I’m a management consultant. I work with CEOs, CIOs and other leaders to help them understand how they can transform their organisations to deliver higher quality solutions more quickly. My contact details are on the last slide, I’m happy to talk afterwards. In the interests of time it would be good if we can save questions until the end.
- If you characterised IT approaches maturity as human age I reckon DevOps is in its mid to late teens. It is pulling away from its parents and looking to craft its own identity. And it is doing this reasonably successfully. In the DevOps handbook, the authors describe the parents of DevOps being the Lean movement, the Agile manifesto, the Agile Infrastructure and Velocity movement, the Continuous Delivery movement and the Toyota Kata. I could easily spend my whole time talking about just one of these, if you want to know more about them you can easily find out more online. I want to talk briefly about two now and then spend a little more time on one in a moment. I’ve heard some people say that if you are doing DevOps you don’t need Agile. I think there is an element of truth in this, but I also think it’s largely false. I think a part of the problem here lies in the confusing of Agile practices, say something like SCRUM, with implementing and operating out of the Agile values as expressed in the Agile manifesto. You can be Agile without doing Agile. Obviously if you can implement the practices you’ll be more successful, but it doesn’t require the practices to be successful. I once had a litigious, contract bashing supplier try to tell me that his company was being Agile because they timeboxed the deliveries. Despite the rhetoric and the implementation of a practice popular in Agile, they weren’t Agile at all. DevOps is like this. You could implement DevOps in a waterfall of V-model type delivery world and still get benefit out of it. It would drive you towards Agile behaviours but you don’t have to be ‘doing’ Agile to do DevOps. Yes, it’s better with it, but it’s not a requirement. If you haven’t heard of it, the Toyota Kata is a continuous learning paradigm that a guy call Mike Rother elaborated from his studies of Toyota’s manufacturing. If you google “Improvement Kata” and “Mike Rother” you’ll find his website where all of the material is available for free. It’s a way of implementing daily continuous learning in teams and organisations. Not just any kind of learning, but learning that is guiding by an overarching vision. I like the knuckles. I sometimes feel like DevOps is a bit like a punch in the face for traditional ways of doing things.
- I’ve seen a few different versions of lean principles, but all of them have these 5 at their core. First you should know who your customers are and what they value. Second you should understand the system that delivers value to your customers. In lean language this is the value stream. Third you try to create flow through the value stream by eliminating waste. The writers of the DevOps handbook also talked about this as eliminating hardship, as sometimes the language of eliminating waste can be seen as dehumanising. You want to optimise your system to respond to customer pull, not push things at your customer. Finally you want to pursue perfection; you want to continuously learn. The reason I highlight these principles is that they underlie all the movements I talked about before, obviously lean, but also Agile, continuous delivery and the improvement kata. And DevOps is an explicit implementation of these principles in the software delivery value stream.
- So, I’ve talked a lot about DevOps, but what is it? The authors of the DevOps Handbook don’t really offer a specific definition, and I think that’s wise. DevOps is, in my view, primarily a cultural movement that seeks to build great cross functional teams that are focussed on implementing these lean values throughout their work. For the sake of providing a definition here is the one from Wikipedia” “DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.” The things that irks me, as a test professional, is that there is not really much mention of testing in this. It’s not like it’s DevTestOps. Which I get, because that is clumsy. I guess we are just another part of the “other IT professionals” part of the definition. At least it mentions testing in the second part. So how does testing fit into DevOps. Before going there we need to understand the three ways of DevOps.
- The Principles of Flow accelerate the delivery of work from Development to Operations to our customer. The Principles of Feedback enable us to create ever safer systems of work The Principles of Continual Learning and Experimentation foster a high trust culture and a scientific approach to organisational improvement risk taking as a part of our daily work. The three ways are the embodiment of the lean values in a way of work that drives efficiency, quality and continuous improvement. So back to the question, “where is the test in DevOps?” Well that depends what you mean by test? Without getting too philosophical on you, can you tell me what testing is? What is the value of testing? Or, to put it more bluntly, why should anyone pay for testing? My view is that testing is the feedback and learning mechanism in the SDLC. I used to work as a teacher. One of the things I found was most valuable to student’s learning was when I could give them immediate feedback on what they were doing. This is called Just in Time Learning. My daughter wants to be physicist, but she acknowledges that maths is hard for her. She struggles to learn it. She has had many teachers say to her: “just apply the rules, it’ll make sense one day”. I see what they are getting at. Through rote repetition of the rules she will come to understand them. Unfortunately, she doesn’t work that way, she is too much like her old man. I need to know why before I can implement how. She’s the same. It wasn’t until she found a teacher that said: “Zoe, I understand you work differently. If you want you can come and see me at lunch time or after school, and we can work through problems and I can explain the why as you go.” Now she is doing much better. Not only does she need feedback, she needs the right feedback, at the right time. So where is the test in DevOps?
- To step away from the surf metaphor for a second. I think test is the water that enables DevOps to grow If you think back to the three ways, who does testing fit it? Well, testing done well, throughout the lifecycle, reduces rework. If the developer builds it and tests it, effectively getting it right first time, then they don’t have to rework it later. And so on throughout the lifecycle. So testing enables flow. If the developer makes a mistake and the tester finds it quickly they can provide feedback to the developer, ideally just in time. So the developer and the tester can both remedy the problem and then learn how not to make it in the future. And if there are independent testers they can act as a mechanism of continual learning for the team. I’ve said before, if you talk to an architect about what they are interested in they talk about architecture. BA’s talk about requirements, Developers talk about code and design. The best testers, in my view, talk about architecture, requirements, design, code and testing. Test has a unique view of the lifecycle that can be profoundly useful in a DevOps world. In 2013 I went to a presentation by Michael Polino the global head of QA for EBAY and they had restructured all of their teams so that test was at the centre.
- So where is testing in DevOps? Test as I have defined it is everywhere. That’s the good news.
- Here is a typical ‘traditional’ value stream. I’ve made this up based on my experience. It goes from Development to Support. A developer builds something, in my experience their unit testing is ad hoc at best, often there is none at all. They then try to build it, and get some errors which they fix. This might mean they complete some features, increasing the completeness of the solution. Then system test gets a hold of it. They often have a set timeframe that is immutable and they strive to get everything done that they can. Smart testers will adopt what I call the CYA risk based approach. They’ll agree a set of risks and then use that to force other people to make priority decisions about what gets tested and what doesn’t. In this process, they might be able to convince the project that some undeveloped features get added. Once their time is up they hand it over to Acceptance test quietly praying that they found all the major defects so that they can say their CYA risk based testing worked. Then the acceptance testers get it. They freak out because ‘nothing works’. If they are lucky they get extra time to test, and they might get a few more features added, usually by playing politics. Once the project budget runs out, or an arbitrary date is met, they stop and the product is deployed. Often into a complex, spaghetti junction production environment and onto an Operations team who has no idea what it is or what it is doing here. They will often stall and try to use change and release processes to hold up deployment and fix what they perceive (often rightly) as quality problems that will make their lives hell. In the end, they get over ridden and have to support an application that barely works. You’ll see here I have completeness degrading. That usually because the much vaunted ‘phase two’ project that will solve all the problems gets binned as too expensive, then the application is too unstable to upgrade things like patches and OS’s or do version upgrades and so it functionally degrades over time. Despite all the advances in IT delivery, PRINCE 2, ITIL, etc. I know of dozens of projects in various stages like this right now. And I’m willing to bet there are hundreds in Wellington. Even in supposedly agile organisations. And test in this space; as an engine of feedback and learning. How much will a developer learn if their defect is found in Acceptance test, maybe 3 or 6 months, or a year or years after they wrote the code? Not a lot.
- The key difference here is that in DevOps quality really is everyone’s responsibility. In good DevOps there is a ruthless effort to eliminate hardship and waste. Redoing something you’ve already done qualifies as both hardship and waste. Not getting feedback in time to really learn and change because of it is also hardship and waste. In DevOps everyone tests as soon as they do the work. And from the test point of view, repeating the same boring test manually over and over again qualifies as hardship and waste. It is the number one thing I hate about testing. Having to do the same test every release. So ground up automation is a key part of this lifecycle. From the moment a developer touches a keyboard they are writing automated tests. System and acceptance testers might manually explore a new feature, but as soon as they have they automate as many of those tests as possible. And so on at every step.
- So I would argue that test is at the heart of good DevOps. As I’ve said, test is the water that helps the plant grow. But it’s not testing as we know it. Who here has done testing on an Agile project? Okay, who has done testing on an Agile project where the developers have not done automated unit testing? Okay, who has ended up working on an Agile project where there were separate test, or regression sprints? I once heard Agile sold as the way for testers to be engaged in the whole lifecycle, a vital part of the team. But I have seen many Agile teams that silo testing into its own sprint. They certainly don’t deliver on their promise. But I would argue that you simply cannot do DevOps without testing, and testing mercilessly, at every single step of the process. And they you rinse and repeat and do it again. This diagram is a much better depiction of the lifecycle. But we are no in Kansas anymore Toto. For your average tester who has worked in more traditional projects and organisations, or in low functioning agile teams, everything changes. Who here does boundary value analysis in their testing? Why? That’s a unit test. It’s the developers job to test that. I once saw a tester want to run 9 tests for every date field in an application. They were all variations on boundary value testing. There were 11 such fields in the application. 99 tests. And they were in the regression suite, so 99 tests run manually every time there was a new release. This is hardship and waste. All were served by the same piece of code. Wouldn’t it have been more efficient for that tester to tell the developer; “can you please automate these 9 tests?” I see a lot of distrust of developers among testers, and not without reason. I was recently looking to hire graduate Comp Sci students, I asked them what they learnt about testing in their degrees. The most was, “Oh unit testing was mentioned a few times”. They are simply not trained to know what to do. The good ones pick it up, but then it’s often beaten out of them in the rush to deliver. It becomes Test’s problem. In DevOps you should be required to be much more deeply involved. You are likely to have to work right across the team to help everyone test their ideas, whether in code or not. The days of waiting until the application is ‘complete’ and then proceeding to try and manually unit test it through the GUI are long gone. In fact, in some organisations, manual testing is largely a thing of the past. Apparently, Amazon does 26 production releases an hour. There’s not a lot of time for manual testing in that. Rather the developer hits the button to commit the code and it automagically goes through all the quality gates and is deployed to production. So, testing is deeply involved in DevOps, despite the title (as are architecture, business analyst, and “other IT professionals”) but what they do, how they do it and why they do it all changes. So, what does this mean for you? Well, there’s no point having your head in the sand about it. DevOps is coming here. Westpac are on a transformation journey to shift to it. The Department of Internal Affairs, Statistics New Zealand, the Ministry of Justice and even the Ministry of Social Development are trying to shift in that direction. The reality is that it is coming, driven largely by increasing business demands for faster and higher quality IT change. If you are planning for a long career in testing, I’d argue that you need to seriously think about how you might need to change to continue in it.
- I’m going to come right out and say it. In a DevOps world, the need for test specialists to perform manual functional testing through the GUI of an application is almost non – existent. If you are trying to deliver value to your customer as quickly and well as possible then manual functional testing through the GUI is going look like waste in your value stream. So what can you do? The short answer is get technical or get close to the users or both. Dion, one of my colleagues, had a question when I told him what I was talking about. He said: “Where do you see business process testing in a DevOps world? That still needs to be done manually doesn’t it?” My response was: “Well yes and no.” If you are delivering a product straight to the public then you should enable rigorous gathering of metrics about user behaviour in production. Also speak to engaged users on forums or whatever you have. What you want to know is how people actually use the system. In this sense testing some abstract business process is next to useless compared to seeing the business process users actually use. If you are working on a system that has specific compliance requirements, say an evidence management system, then getting your business experts to use to the system and see how they interact with it, especially with how you’ve encoded that business process into the application is very effective. If you are releasing to an internal audience, you might want to get test consultants to work with the business to identify key subject matter experts who will run through the business processes to see if they work from a user point of view. This is effectively getting into the realm of user experience and usability testing. I set up a test programme for a client recently and we didn’t want to call it acceptance testing as that term is laden with too many different meanings to too many different people. Instead we ran Business Functional Feedback sessions with users. In our BFF sessions, we asked the users to perform key work tasks and recorded their feedback, observations, concerns, queries, areas they got stuck. In QA there are two kinds of assurance, validation and verification. Validation is: “have we built the system right?”. Verification is: “have we build the right system?” Validation is a testers domain and in DevOps we should mercilessly seek to automate it to reduce waste and hardship and to increase flow. Verification is the domain of the users. I know of many organisations that have independent test teams that claim to run acceptance testing. I ran a team like this for 6 years. I now see that they couldn’t truly do acceptance testing because they didn’t truly understand the business. Some had been brought from the business, but their knowledge of the business units waned over time. I see the testers role in this situation is to manage the acceptance testing doing all the logistics and management so that the business users don’t have too. So that’s one avenue. Get close to the users. Read about user experience, understand the domain, be their IT advocate. The other response is to get close to the developers. Get more technical. Learn a programming language so that you can write awesome test code. Understand continuous integration. Learn about test driven development and how you can partner with developers to help them get it right first time. Learn how to test systems without using the UI. Learn how to test API’s and service layers. Learn about service orient architecture and how to test in that space. Learn how to automate these tests. Learn about to check your tests in as code along side the application code. Don’t learn an automation tool. Learn how to be a test automation expert. At the same time develop our manual testing skills so that you can do the little bit you might need to do as well as possible. Read James Whitaker’s books on Software Attacks. And / or get close to the operations team. Understand environments and infrastructure. Learn about software defined networks, they are the next big game changer, potentially incredibly useful for testing. Learn how to automate environment provision. Learn about service virtualisation to dis-integrate tightly coupled systems and systems of systems. Learn about cloud services like Amazon Web Services. Learn how to use the tools that will validate that the right environment has been built. DevOps isn’t going away. I don’t know if government departments like DIA or Statistics will be wholly successful in implementing it, as it required culture change far beyond the business. But they are trying. Government are the most conservative in this space. They are the slow movers, the laggards. And yet they are changing. The pressure to change in organisations like this is palpable. So if you think you can escape these changes, I think you’re out of luck. Unless you’re planning to retire or change career in the next 5 to 10 years. And here’s the thing with change. We can fight it and almost certainly lose out. We can ignore it and be overtaken by it. Or we can see it as an opportunity and exploit it. The wave is building, it’s time to surf or be swamped. Any questions?