DevOps Test Engineering: Putting the ‘Continuous’ in Testing, an ITSM Academy Webinar
•Download as PPTX, PDF•
1 like•844 views
Presenter: Anne Hungate President, Daring Systems You’ve heard about Continuous Integration and Continuous Delivery but what’s common as code makes its way through those processes? Testing. With DevOps Testing (also known as Continuous Testing), testing tasks are engineered to be continuously completed end-to-end across the entire development to deployment pipeline. Developers, QA analysts, security professionals, IT Operations analysts…everyone is a tester in a DevOps environment. Join us to learn more about DevOps Testing and the emerging role of DevOps Test Engineer.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to DevOps Test Engineering: Putting the ‘Continuous’ in Testing, an ITSM Academy Webinar
Similar to DevOps Test Engineering: Putting the ‘Continuous’ in Testing, an ITSM Academy Webinar (20)
More from ITSM Academy, Inc.
More from ITSM Academy, Inc. (20)
Recently uploaded
Recently uploaded (20)
DevOps Test Engineering: Putting the ‘Continuous’ in Testing, an ITSM Academy Webinar
- 1. DevOps Test Engineering: Putting the Continuous in Testing ANNE HUNGATE DARING SYSTEMS @ANNEHUNGATE MAY 18, 2017
- 2. © 2017 Daring Systems Continuous Testing … Central to DevOps . DevOps Continuou s Release & Deploy Continuous Testing Continuous Monitoring Collaborative Customer Fdbk & Optimize Continuous Business Planning Collaborative Development https://devops.com/state-devops-adoption-trends-2017/ Image adapted from the state of devops Web site Continuous Release & Deployment @AnneHungate 2
- 3. © 2017 Daring Systems Key Take Aways 1. Understanding of DevOps Testing aka Continuous Testing 2. Appreciation of DevOps Test Engineer Role 3. Insight to adoption challenges with strategies to overcome 4. Personal call to action to prepare for Continuous Testing 3
- 4. © 2017 Daring Systems Key Take Aways 1. Understanding of DevOps Testing aka Continuous Testing 2. Appreciation of DevOps Test Engineer Role 3. Insight to adoption challenges with strategies to overcome 4. Personal call to action to prepare for Continuous Testing 4
- 5. © 2017 Daring Systems Definition of Continuous Testing Source - https://en.wikipedia.org/wiki/Continuous_testing Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.[1][2] 5
- 6. © 2017 Daring Systems Definition of Continuous Testing Four key elements … in order of importance 1. At every stage of the pipeline 2. Emphasizing business value 3. Providing feedback 4. Automated 6
- 7. © 2017 Daring Systems Definition of Continuous Testing Testing as a Phase … is too late Time Resources Initiate / ideate Requirements Design Development Testing Install Run / Maintain At this point, you’ve spent the resources 7
- 8. © 2017 Daring Systems Definition of Continuous Testing Problems become more expensive to find & fix IBM System Science Institute Relative Cost of Fixing Defects Source: ASTQB ( American Software Testing Qualifications Board) 8
- 9. © 2017 Daring Systems Definition of Continuous Testing At Every Stage … applies to waterfall Time Resources Initiate / ideate Requirements Design Development Testing Install Run / Maintain Test Here Test Here Test Here Test Here Test Here Test Here Test Here too 9
- 10. © 2017 Daring Systems 5/14/2017 Devops-toolchain(5).svg By Kharnagy (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons Test Here Test Here Test Here Test Here Test Here Test Here Test Here Image adapted from the Wikipedia definition of DevOps 10 Definition of Continuous Testing At Every Stage … in the Pipeline supports DevOps
- 11. © 2017 Daring Systems Definition of Continuous Testing Business Value Relevance Tests are organized by what matters most Tests are executed in a timely manner – get the hard ones done first Answer the so-what If this doesn’t work, how is the business affected If this doesn’t work, how will our customers respond 11
- 12. © 2017 Daring Systems Definition of Continuous Testing Feedback Work incrementally Test components at each stage and in small chunks Don’t test the full workflow until all the parts work Perform “assessments” At each stage - develop artifacts Perform an evaluation of the artifact – can it move forward to the next stage or not 12
- 13. © 2017 Daring Systems Definition of Continuous Testing Automated Integrated into DevOps Master Framework ◦ Includes: ◦ Production Equivalent Test Environment ◦ Environment Orchestration and Release ◦ Data Provisioning ◦ Scenario Generation ◦ Automated Execution ◦ Automated Analysis & Reporting ◦ Leverages: ◦ Containers ◦ Microservices ◦ Elements of development environment 13 • DevOps Test Framework is not separate from the Development environment • The test tools must work with the development tools • Need to get the first three dimension of continuous test right BEFORE automating • Automation is NOT a silver bullet • Automating the wrong stuff, the wrong way at the wrong time creates technical debt
- 14. © 2017 Daring Systems Key Take Aways 1. Understanding of DevOps Testing aka Continuous Testing 2. Appreciation of DevOps Test Engineer Role 3. Insight to adoption challenges with strategies to overcome 4. Personal call to action to prepare for Continuous Testing 14
- 15. © 2017 Daring Systems Role of a DevOps Test Engineer DevOps Institute and experts in the field advocate for a cross-functional team that develops automation. In fact, letting the team self-organize is highly encouraged – utilizing a checklist to ensure relevant tasks are covered. You may have heard of ‘full stack’ engineers – that means these engineers can use the full stack of tools and techniques used to develop software 15
- 16. © 2017 Daring Systems Role of a DevOps Test Engineer Creating a culture that allows for DevOps Testing is critical - Developers are automation developers / testers - QA resources are automation developers / testers - BA’s are automation developers / testers Friction can result is LEADERSHIP does not enable the DevOps Test culture Skills and knowledge for DevOps Test culture require training and opportunity to practice 16
- 17. © 2017 Daring Systems Key Take Aways 1. Understanding of DevOps Testing aka Continuous Testing 2. Appreciation of DevOps Test Engineer Role 3. Insight to adoption challenges with strategies to overcome 4. Personal call to action to prepare for Continuous Testing 17
- 18. © 2017 Daring Systems Adoption Challenges / Strategies Common excuses for not adopting continuous testing There’s a lack of understanding about what continuous testing is The org planning processes doesn’t include continuous testing We don’t have the skills necessary We don’t know what tools we need We don’t have time to add continuous testing practices into our work and still hit our commitments People upstream don’t want to include continuous testing tasks We are not a DevOps shop Our vendor partners don’t do it, so we can’t 18
- 19. © 2017 Daring Systems Adoption Challenges / Strategies Common excuses for not getting healthy I don’t know where to start I don’t have time to plan and cook every meal at home Produce is expensive I don’t know what broccoli sprouts are and why I need flaxseed My family doesn’t want to eat that way I don’t have time to go to the gym The gym intimidates me My friends will make fun of me for being on a health kick 19
- 20. © 2017 Daring Systems Adoption Challenges / Strategies Be realistic about what you can do Stop worrying about what you cannot control 20
- 21. © 2017 Daring Systems Adoption Challenges / Strategies Be realistic about what you can do Stop worrying about what you cannot control 21 People Process Technology In Your Control • We don't have the skills necessary • There's a lack of understanding about what continuous testing is • Org planning processes don't include continuous testing • We don't know what tools we need Not In Your Control • Our vendor partners don't do it • People upstream don't want to include testing tasks • We don't have time to add CT • We are not a DevOps shop
- 22. © 2017 Daring Systems Adoption Challenges / Strategies Enlist the help of others 22
- 23. © 2017 Daring Systems Adoption Challenges / Strategies Enlist the help of others 23 People Process Technology In Your Control • We don't have the skills necessary • There's a lack of understanding about what continuous testing is • Org planning processes don't include continuous testing • We don't know what tools we need HR business partner L&D organization Vendor partners DevOps Coach Process Consultant DevOps Coach Architecture & Dev
- 24. © 2017 Daring Systems Adoption Challenges / Strategies Know your numbers 24
- 25. © 2017 Daring Systems Adoption Challenges / Strategies Know your numbers 25 Production Project - Are defect backlogs trending the right direction - Are our regression suites as effective and efficient as we'd like - Do our test methods and suites align with our production priorities - Is there a correlation between system test and post-implementation issues - When do we forecast the release will certify for deployment - How far ahead / behind schedule - Are we finding the right type of defects in System Test - complex defects - What do our test defects say about delivery methods Test Inventory Test Tools - Are the test suites current - Do we have coverage for all our applications - Are we keeping track of what we have - Is there redundancy - Do we have a deliberate set of tools or are we just using legacy tools - Are teams using the same tool differently - Do we have duplicate tools doing the same function - What is our tool utilization
- 26. © 2017 Daring Systems Adoption Challenges / Strategies Start small and use what you’ve got 26
- 27. © 2017 Daring Systems Adoption Challenges / Strategies Start small and use what you’ve got 27 Start small - Don’t try to solve for the entire organization on the first try - Use Lean methods to build, measure, learn - Find a willing partner - Have a baseline - Compare / contrast before and after - Add one test process improvement before adding tons - Streamline what you already have Use what you’ve got - CALMS - Don’t rush to the Automation - Old, enterprise tools have been updated to support processes, so don’t throw them out yet - Process first, then people, then tools… what you have may be fine
- 28. © 2017 Daring Systems Adoption Challenges / Strategies Focus on business value / outcomes 28 Awake for Drive Home Sleeping Well Laugh Daily
- 29. © 2017 Daring Systems Adoption Challenges / Strategies Focus on business value / outcomes 29 Grow Business Enhance Brand Reduce Risk Lower Cost Be in Compliance
- 30. © 2017 Daring Systems Key Take Aways 1. Understanding of DevOps Testing aka Continuous Testing 2. Appreciation of DevOps Test Engineer Role 3. Insight to adoption challenges with strategies to overcome 4. Personal call to action to prepare for Continuous Testing 30
- 31. © 2017 Daring Systems Make it personal More Polling! 1. Is everyone in your organization on board with DevOps 2. Does your boss support continuous testing 3. Are stakeholders looking for shift left testing or do they just want you to automate testing as a phase? 4. Do you have a cohesive tool set right now? 31
- 32. © 2017 Daring Systems Make it personal Join the community and the movement Educate yourself Surround yourself with people who are going to change 32
- 33. © 2017 Daring Systems Make it Personal Who told you to wait You’re not going to learn any younger 33
- 34. © 2017 Daring Systems Key Take Aways 1. Understanding of DevOps Testing aka Continuous Testing 2. Appreciation of DevOps Test Engineer Role 3. Insight to adoption challenges with strategies to overcome 4. Personal call to action to prepare for Continuous Testing 34
Editor's Notes
- My name is Anne Hungate, I am the president and principal consultant for Daring Systems, a firm that helps organizations migrate to DevOps – from understanding their readiness to advising on the specifics of continuous testing. In addition, we coach and advise leaders who are driving the transformation – since DevOps begins with Culture change. Today, we’re going to introduce the concepts of continuous testing – also know as DevOps Testing. For those of you familiar with DevOps and the State of DevOps report – until now, emphasis has been on three C’s—Continuous Integration, Continuous Deployment, and Continuous Delivery. Those three C’s are not possible without Continuous Testing… the silent backbone of DevOps. Today’s talk is going to take a personal turn because this understanding an embracing of continuous testing has been a nine year journey for me. My hope is that I can share the lessons I learned on my journey to accelerate yours. I also discovered a parallel journey that many of us have taken which should help bring some light and understanding to the imperative of continuous testing.
- When we’re through with today’s Webinar, you will understand what continuous testing is, you will know the common reasons why organizations struggle with adoption, you will have strategies for accelerating the rollout in your organization, and you will have a personal call to action to drive up your own adoption of continuous testing. In making this a four bullet talk, I broke one of my fundamental rules which is only have three bullets in any talk, more than that and people won’t remember it – but I added the personal call to action because I want everyone to leave knowing what is in their control to change today.
- I am making the assumption that continuous testing and DevOps are terms that are gaining traction in your organization – we don’t need to take a poll on that because the State of DevOps report tells us that as of 2017, the interest and volume of DevOps searches has grown exponentially. Also, almost 85% of organizations are implementing some DevOps practices – so if you’re not working on DevOps now, you will be soon.
- The first time I saw this graph, I was a freshman in college. I was taking an introductory Decision Sciences class (now called analytics) and the professor laughed because this common sense was overlooked by US auto manufacturers for years and now Toyota and Honda were taking over – I was a freshman in college a LONG time ago. I knew this simply from my house painting days. In college, I paid my tuition by working as an exterior house painter – I learned many things and can build an entire project management course from the experience – but the most important was to get the job right before you start working. We did a walkthrough with the homeowner before starting to understand specific challenges and the desired impact she wanted. We understood special conditions such as big trees, toddlers, or moss on a roof. We checked the paint color for accuracy – we looked at the weather forecast (Cleveland, Ohio). We made sure our equipment was up to snuff, then most importantly, we prepped the house for paint. The goal was to bring the house in on time and budget but at a level of quality that we wouldn’t be getting a call back. Why are the same principles overlooked when building software. What are some of the tests you do in the early phases? Another name for this is Shift Left. Left means, make the change earlier, before you spend your resources.
- Start with static testing. When we say static, we refer to tests that don’t involve executing code. For those of you trained in the V model – these are our verification tests. Static tests are tests that don’t involve executing the code – they are tests of the requirements, the idea, the design, the environment. Static tests can include checklists of the things you consider before getting into a development phase. I LOVE static testing of ideas and requirements. Before launching into building software, we verify the existence of a business opportunity or problem. We answer the question, why. In 2010, my BA and QA teams came together and implemented cross-domain, static reviews of requirements. The result was adding two weeks onto our requirements phase – but cutting four out of integrating testing. Furthermore, we implemented a function that we had tried and failed launch multiple times in the past. We found 864 requirements defects. I will never forget that number – because we learned that coding and testing weren’t the source of our greatest pain.
- Continuous Testing when you’re in a DevOps environment is even more powerful than a waterfall environment. Your verification level test activities include assembling the packages that orchestrate a production equivalent test environment – so that tests are generating useful results for assessment. Those of you who take the DevOps Test Engineering course will learn about the different test strategies employed – important to note is that you don’t limit yourself to one strategy – there are strategies to ensure the environment is ready, strategies to validate the interdependencies of code, strategies to leverage virtualization and tools that can accelerate your automation. I am very excited about the orchestration automation available today – we can orchestrate environments, orchestrate all your test scenarios – and orchestrate the data. These are not only time saving activities – but improve the accuracy. In DevOps Test Engineering, we call this early stage testing pre-flight testing. Once we start developing code, we can do both static and dynamic testing. Static analysis of code for quality and accuracy is easy. Here I go dating myself again, but back in the day, when I was a programming supervisor, I would print out my teams’ code on green bar paper, grab a red pen, and find improvements in their code while I watched Quantum Leap. Scott Backula was very young then. The role of programming supervisor has been automated away with static code analyzers. Teams that employ test driven development have automated unit test cases developed at the same time as design – so unit test is merely an execution of an already automated test. The Wikipedia definition stresses that continuous testing means tests are automated – I have stretched that a bit with my examples – but you’ll understand why when we get to the next section.
- Automated testing is a no brainer. When we are testing linear logic, nothing is better and more complete than automated testing. When we are testing data, services, and processes with no UI, automated testing is the best. We rely on manual testing when testing non-linear activity and generating unpredictable behavior – these are the exception conditions, not the bulk of testing. One last note on What continuous test is before we get into the hard stuff… continuous test means treating all your test assets with the same standards, care, and measurement that you treat your code with. We need standards, inventories, tools, and aging. Unused or irrelevant test scenarios are technical debt – just the same as software build on non-standard versions or straying from the enterprise reference architecture. During the DTE class, we’ll go through guiding principles for establishing and enforcing those standards. We’ll discuss the checklists, pitfalls, and the requirements for well-managed test automation.
- These are real challenges – these are pains we feel like we’re knocking our head against a wall sometimes. I have experienced all of these in my journey. I have examples of happily sharing information that showed we were off track – and being excited because we could adjust, to find my peers embarrassed that I called them out. I have had partners ask me why testing costs so much and asking me to personally sign up for making the change – still don’t know what that meant. I combed through articles, notes from books I have read, cool Web sites… but the aha moment that brought this Webinar together for me was realizing that Continuous Testing is a good habit, a life habit that doesn’t merely keep you out of trouble, it creates opportunity and capacity to deliver more to customers. You don’t have to be fully automated to get substantial benefit. Small changes in the right place can have HUGE impact. Getting continuous testing is like getting healthy.
- This is not a choice – it is what we need to do to build value for our customers, to reduce risk, and to lower cost. We also don’t need to be ironmen to live healthy, productive lives. Your organization does not need to be Google to build great software and secure systems.
- Step 1 – Put it in perspective Getting healthy, we get overwhelmed, intimidated, don’t know where to start. The answer is just start, one step at a time. If we dissect the common reasons we don’t do continuous testing, we can eliminate half of them immediately from our worries – because they are outside of our control. When we’re trying to get healthy, we don’t demand that our local grocery story get rid of Cap’n Crunch, we don’t demand that McDonald’s stop making delicious fries, and we don’t take health advice from unhealthy people. When we’re trying to get healthy, we can control our diet, our movement, our attitude, how we spend our time, and with whom we associate…
- When we’re trying to move to continuous testing… We can control the skills we have (and our teams), we can learn new processes, we can change the way we plan work, and we can learn tools. Unless you’ve got a C in your title, you cannot change other companies, departments, existing commitments, or overall enterprise methods. Don’t try, and don’t spin another cycle there.
- When I started my journey to health, I had an infant, a toddler, and a husband with his own business. My job was to hold down a job and keep us all together. We had a house payment, two car payments, and a huge daycare bill… it was all resting on me, and then I got sick. It changed everything…and nothing. My perspective changed – the healthy choices were always there. I asked for help. A friend encouraged me to find a nutrionist… who had the time… not me, but I could join the weight loss program offered through work. I could educate myself on what healthy eating meant and month over month make little changes that would become habits. I didn’t walk into the gym, see a bunch of people working out and feel comfortable. I asked for help… I had a trainer teach me how to use the machines and craft a few workouts for me. I could not afford an extended, personal trainer so I worked with her on which group fitness classes mattered. These experts help me make small changes that added up.
- You don’t have the skills for continuous testing… ask for help. Get a coach who knows what they are and bring in your HR business partner. Start building models that facilitate your education and the education of your team. These don’t have to be high investment plans, but they need to be specific and high impact. I strongly encourage you to not wing it. I made that mistake when I first tried implementing continuous testing. I thought I knew methodology and managing people, I could do this by myself – but my perception was clouded by norms I had accepted and my time was filled with administrative and organizational burdens. It took one of my direct reports telling me – get a coach, Anne for me to realize I was trying to do all of this myself.
- When you’re getting healthy, you have numbers that guide you – weight, BMI, sleep, cholesteral, glucose, etc. Some of these numbers you can get yourself, some you need help with. Some you grab daily, some you pull annually… but you have numbers.
- When you’re adopting continuous testing, know your numbers. I was at a local tech event a couple weeks ago. I met a consultant for the first time, we exchanged pleasantries and he said, “so what percentage of testing should be automated?” Then he looks at me with a knowing smile. I said, percent automated and percent complete are two numbers that don’t tell me anything as a leader. Automating scripts that are duplicate or wasteful doesn’t bring value to a company. Being 80% complete, but you saved the hardest 20% for the end gives a misrepresentation of your progress. Work with your coach to figure out what numbers make the most sense for you… you probably have some great baseline data in test and defect repositories. You have some great insight and data points from other teams. Once you know the relevant numbers for you now, pull your baseline. Just like a health journey, the numbers you need to know and what constitutes ‘good’ will change over time. You can cross a threshold and some become moot – while others become important.
- In my health journey, I have added devices and gadgets to help gather sophisticated numbers. But, I started with my scale, my observations about sleep, the size of my clothes, and my energy level… The GPS, heart rate monitor, and home blood pressure cuff came much later. There is a ton of noise and confusion right now with which tools to use – open source, full stack-proprietary, a hybrid… use what you have!!! When you are employing continuous testing you will want the right tools – but you have the right tools right now. I didn’t invest in the fancy blender until I went completely vegan and decided to eat smoothies multiple times a week (and make faux creamy soups). When getting healthy, are the big changes coming from intense cross-fit sessions … no, they’re coming from simple diet choices at home. Opting for the apple instead of the candy bar. Eating the salad first and saying no to dessert. Not fancy or worth posting on FB, but they’re the changes that make the lasting impact.
- Lots of organizations start with the tools. DON’T. If you invest thousands in the ultimate kayak and realize you prefer SUP, you’ve thrown away money. If you get a top of the line road bike but find running is your thing, you’ll need different equipment. Your coach can help you use the tools you have right now – looking at it objectively is powerful. Work with the tools you have. Many of you are probably familiar with the acronym CALMS for DevOps. That means Culture, Automation, Lean, Measurement, and Sharing. That means tools are 1/5 of the equation. Back to Pareto – why spend 80% of your time on something that only matters 20%. When you are introducing healthy lifestyle changes, you do one or two at a time – and if you’re doing more than one it is because they are small or complementary. Pick one project with one product and a supporting partner. Pick one step and add it. Use the lean method of build, measure, learn… get the supporting partner to be your advocate and champion – if he talks up what you’ve done it will be more powerful than you advocating for your own team.
- When making life changes to be healthier, I focused on what mattered – being there for my peeps. Could I stay awake while driving home from daycare, did the kids have clean diapers… maybe those are extreme – but those are the outcomes that mattered, not my VO2 max or how much weight I could squat… was I keeping it together for what mattered.
- When you’re instituting continuous testing, do it because it affects the performance of your company, not because it is cool or trendy. When you’re correcting a process or driving a dialog, do it because you can grow your business, enhance your brand, differentiate from a competitor, lower your cost, or reduce risk. So often, in technology, we love what we do so much we gold-plate (using a PMI term). We do more than we need do. Let’s look at the bell curve again – where do you think you’ll have the most impact? Right – by automating end to end functional test or left where you’re auto checking requirements, and automatically provisioning environments? Shift left…
- Any of these can inhibit the progress of your team – but they should not stop YOU!
- You know about CT. You’re on-board. You can control your actions. Be a learner. Start reading a lot Go to local tech events Find a favorite blog or podcast to follow Write your own blog Stretch yourself Incremental improvements become habits Join the larger community Attend DevOps events Contribute to events – speak, teach, tell your story Make connections with people with similar stories and help each other Seek certificaitons to make sure you’re up to Say goodbye to relationships that don’t support this journey or derail you Have a purpose – and don’t give up Remember why you’re doing this – shifting left and adding continuous testing is the right way to build software. If you’re in the software game, don’t intentionally build software the wrong way This journey is not direct and it is not easy, but it is the right journey This isn’t a fad or a choice Continuous testing is an if, it is a when – put down the Twinkie and get in the game