Downloaded 55 times
Uploaded byPerfecto by Perforce
Continuous Testing: The Path Forward
The document discusses continuous testing as a crucial component in the software delivery pipeline, emphasizing rapid feedback on business risks associated with software releases. It outlines five core practices that differentiate successful DevOps leaders from laggards, including the importance of end-to-end test automation and integrating testers into delivery teams. Key recommendations include prioritizing business risk in metrics and ruthlessly automating testing processes.
More Related Content
Similar to Continuous Testing: The Path Forward
Continuous Testing: The Path Forward
- 1. 1 Continuous Testing ThePath Forward Wayne Ariola & Eran Kinsbruner
- 2. Eran Kinsbruner• Lead SoftwareEvangelist at Perfecto • Blogger and Speaker • http://continuoustesting.blog • https://www.infoworld.com/author/Eran-Kinsbruner/ • 18+ Years in Development & Testing • Author of: The Digital Quality Handbook Continuous Testing for DevOps Professionals @ek121268 https://www.linkedin.com/in/eran-kinsbruner-4b47a81/ erank@perfecto.io
- 3. Wayne Ariola • Authorof Continuous Testing for IT Leaders • 6 Patents in Software Testing • Continuous Testing Evangelist @wayneariola https://www.linkedin.com/in/wayneariola w.ariola@tricentis.com
- 4. 4 AGENDA • Digital TransformationImplications • Introduction to Continuous Testing (CT) • Good vs. Bad CT Practices • 5 Core CT Recommended Practices • Short Demo – Tricentis Tosca & Perfecto Continuous Quality Lab • Q & A
- 5. 5 Digital Transformation isa Boardroom Topic
- 6.
- 7.
- 8. 8 What is ContinuousTesting? Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release as rapidly as possible. • Right feedback, right time, for the right stakeholder • Can happen at any point in the service delivery lifecycle
- 9. 9 Testing in Transition DecentralizedCentralized SystemArchitectures Dev. Team Structure Test Team Structure
- 10. 10 Testing Percent ofIT Application Budget 23% 26% 35% 34% 37% 39% 40% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 2013 2014 2015 2016 2017 2018 (est.) 2019 (est.) TESTING AS A PERCENTAGE OF IT APPLICATION BUDGET Source: 2017, World Quality Report.
- 11. 11 J-Curve of Transformation Source:Accelerate State of DevOps Report 2018
- 12. 12 Forrester Research innovate.tricentis.com/forrester-devops The DefinitiveSoftware Quality Metrics for Agile + DevOps Five core "Continuous Testing" practices separate successful DevOps leaders from laggards Business risk must be accurately and continuously managed. Leaders ruthlessly drive end-to-end test automation Top 20 “Definitive Quality Metrics” measured by the leaders
- 14. 14 5 Core ContinuousTesting Practices Most significant differentiators between leaders and laggards? 1) They implement continuous testing to meet the demands of release frequency 2) They automate end-to-end functional testing 3) They shift testing to earlier in the development life cycle 4) They include testers as part of their integrated delivery teams 5) They allocate proper testing budgets and focus on upgrading their testing skills
- 15. 15 Advanced Testing Practices Both advancedtechniques and the array of advanced techniques in use differentiated Leaders vs. Laggards
- 16. Not only theuse of certain testing practices but also the “automation” of those practices differentiated leaders from laggards Automation
- 17. Source: A commissionedstudy conducted by Forrester Consulting on behalf of Tricentis, March 2018 “Automation” is considered critical as a business differentiator versus a priority or important. Automation is Critical DevOps Leaders truly consider automation a critical business differentiator
- 18. 18 Reasons for “Unstable”Test Automation 80% of issues have a pattern52% success rate 10% of devices, causing 80% of lab issues Lab 25% Orchestr ation 25% Scripts & FW 50% FAILURE REASON Objects Codding Time Other Scripts & FW issues Device in use No Device Orchestration issues Networking Stability Lock Other Lab issues What’s wrong With my Scripts What’s wrong With my Lab What’s wrong With my Executions
- 19. We are the Champions! TheC-Level is very confident in the organizations ability to transform. v v v
- 20. …We are the Champions…We TheC-Level is very confident in the organizations ability to really do anything… CxO significantly overestimate their organization’s maturity
- 21. Key Recommendations Make businessrisk the driver for your metrics program Ruthlessly automate your testing and QA processes, end to end Prioritize test execution predicated around business risk
- 22.
- 23.
Editor's Notes
- #2 Questions 1) How can I get the survey
- #5 -Resources for testing should be easy for any dev, tester, etc. to create -Needs to be integrated into the pipeline -Needs to provide feedback (i.e. visibility, reporting) fast - testing quickly without feedback is worthless
- #6 Why is DT a boardroom issue? Because DT is essential for growth and innovation IDC estimates that 40 percent of all technology spending will go toward digital transformations, with enterprises spending in excess of $2 trillion by 2019. 62% plan to increase their investment in digital transformation over the next 12-18 months (deloitte) Companies that reach a higher level of digital maturity are 26% more profitable, grow 9% faster, and achieve 12% higher market valuations than their industry peers. (MIT, Accenture
- #8 risk associated with software failure at all time high and SW = Bus -SW testing is generally in the wrong spot – Dev/Release – release and business – verifying fucnitonaly versus mitigating business risk -SW testing entering a new era with DevOps Shift left? (Dev) Why because is dev was bad at testing Shift right? (Ops) Because of speed and demand for nwe2 software we need governance Analogy with stop light Volume and velocity – more / faster with more your need automation and with faster you need rules As a CIO how do you need the quality process to change? What do you wan tot ask of developers and testers. At this time in the evolution of the digital economy we need to see the role of quality shift from a tactical deliverable….to quality framework. Users/Customers have more choices than ever before. Risk is real. Business failure is real. Requires a transformation of the IT and Business Culture to Agile … the key ingredient if testing. Without it you run the risk of failure which impacts Trust, …
- #9 -Resources for testing should be easy for any dev, tester, etc. to create -Needs to be integrated into the pipeline -Needs to provide feedback (i.e. visibility, reporting) fast - testing quickly without feedback is worthless
- #12 Automation is the lever that development teams pull to go faster. However, if not properly managed, automation can jeopardize quality and increase risk. As seen in the “J-curve of Transformation” , automation increases test requirements, which are dealt with manually. A mountain of technical debt blocks progress and add additional risk that needs to be mitigated before productivity increases. To ensure that quality issues and unacceptable risk do not negate the velocity benefits of automation, firms must ensure they are tracking the right metrics to measure quality throughout development- testing-deployment. In order to meet their goal to deliver exceptional end-user experience, enterprises need to answer this question before deploying any software release candidate into production – “Does the release candidate has acceptable amount to risk?” You can only do this when you are measuring the right quality metrics that matters.
- #22 Forrester’s in-depth survey of global enterprises using Agile and/or DevOps about the software quality metrics they track in software development yielded several important recommendations. Make business risk the driver for your metrics program. Most firms acknowledge gaps in their ability to manage business risk with the right metrics but are nevertheless optimistic on risk coverage. First, you must clearly define risk. Second, you must continuously measure your exposure versus that risk. You should be able to answer this question before deploying any software release candidate into production – “Does the release candidate has acceptable amount to risk?” Ruthlessly automate your testing and QA processes, end to end. If your goal is to deliver high-quality software faster, then you need to automate your software development pipeline. Automating end-to-end testing is a key step in implementing continuous testing and should be a top priority for Agile+DevOps leaders today. Prioritize test execution predicated around business risk. Understanding which tests deliver the greatest risk coverage is a significant advantage when speed matters. This will allow you to maximize business coverage by executing minimal number of text cases to support frequent software delivery. Finally, Enterprises should realistically assess and up-level your continuous testing practices for Agile+DevOps. For example, the DevOps assessment by DORA, based on the state of DevOps research data, can help organizations benchmark their current IT performance against the industry, quantify the state of existing DevOps capabilities and show which capabilities will provide the greatest impact if developed. They can then measure and track these software quality metrics to up-level and improve their continuous testing practices.
- #23 Forrester’s in-depth survey of global enterprises using Agile and/or DevOps about the software quality metrics they track in software development yielded several important recommendations. Make business risk the driver for your metrics program. Most firms acknowledge gaps in their ability to manage business risk with the right metrics but are nevertheless optimistic on risk coverage. First, you must clearly define risk. Second, you must continuously measure your exposure versus that risk. You should be able to answer this question before deploying any software release candidate into production – “Does the release candidate has acceptable amount to risk?” Ruthlessly automate your testing and QA processes, end to end. If your goal is to deliver high-quality software faster, then you need to automate your software development pipeline. Automating end-to-end testing is a key step in implementing continuous testing and should be a top priority for Agile+DevOps leaders today. Prioritize test execution predicated around business risk. Understanding which tests deliver the greatest risk coverage is a significant advantage when speed matters. This will allow you to maximize business coverage by executing minimal number of text cases to support frequent software delivery. Finally, Enterprises should realistically assess and up-level your continuous testing practices for Agile+DevOps. For example, the DevOps assessment by DORA, based on the state of DevOps research data, can help organizations benchmark their current IT performance against the industry, quantify the state of existing DevOps capabilities and show which capabilities will provide the greatest impact if developed. They can then measure and track these software quality metrics to up-level and improve their continuous testing practices.
- #24 Forrester’s in-depth survey of global enterprises using Agile and/or DevOps about the software quality metrics they track in software development yielded several important recommendations. Make business risk the driver for your metrics program. Most firms acknowledge gaps in their ability to manage business risk with the right metrics but are nevertheless optimistic on risk coverage. First, you must clearly define risk. Second, you must continuously measure your exposure versus that risk. You should be able to answer this question before deploying any software release candidate into production – “Does the release candidate has acceptable amount to risk?” Ruthlessly automate your testing and QA processes, end to end. If your goal is to deliver high-quality software faster, then you need to automate your software development pipeline. Automating end-to-end testing is a key step in implementing continuous testing and should be a top priority for Agile+DevOps leaders today. Prioritize test execution predicated around business risk. Understanding which tests deliver the greatest risk coverage is a significant advantage when speed matters. This will allow you to maximize business coverage by executing minimal number of text cases to support frequent software delivery. Finally, Enterprises should realistically assess and up-level your continuous testing practices for Agile+DevOps. For example, the DevOps assessment by DORA, based on the state of DevOps research data, can help organizations benchmark their current IT performance against the industry, quantify the state of existing DevOps capabilities and show which capabilities will provide the greatest impact if developed. They can then measure and track these software quality metrics to up-level and improve their continuous testing practices.