Downloaded 87 times
Uploaded byCA Technologies
Drive Continuous Delivery With Continuous Testing
The document discusses the importance of continuous testing in achieving effective continuous delivery of software, emphasizing that many organizations struggle with quality due to outdated testing practices. It stresses the need for modernization of testing tools and a cultural shift towards integrating testing early in the software development lifecycle. Continuous testing is portrayed as a critical enabler for businesses to enhance speed, quality, and overall efficiency in their development processes.
More Related Content
Similar to Drive Continuous Delivery With Continuous Testing
Drive Continuous Delivery With Continuous Testing
- 1. 1 Keith Puzey CA Technologies DriveContinuous Delivery With Continuous Testing Titus Fortner Sauce Labs August 2017
- 2. © 2017 CA.Confidential. All rights reserved. 2 Continuous Testing How do you build better apps, faster?
- 3. © 2017 CA.Confidential. All rights reserved. Why Many Companies Think They’re Agile… They moved some Dev projects from waterfall to agile They’re having daily standups They have a scrum master Product owner is part of the team They are all talking and walking agile… And are talking about Continuous Delivery BUT…
- 4. © 2017 CA.Confidential. All rights reserved. QA is STILL a Bottleneck… Even in DevOps Shops http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf …of delays were occurring at the Test/QA stage of the cycle. “Where are the main hold-ups in the software production process?” 32% 16% 22% 63% 21% 23% 30% 63%A 2017 survey of self- proclaimed DevOps practitioners found that … 4
- 5. © 2017 CA.Confidential. All rights reserved. Why Quality is So Hard to Achieve? Clear evidence that you are unlikely to enjoy the full benefits of continuous delivery unless you modernize your testing tools and practices. 75% highlight continuous testing as critical or important 76% see Continuous Testing as critical or highly desirable to ‘shift-left’ testing activity from downstream back into development Traditional testing tools are often too complex and expensive to use Adoption of continuous testing is much easier with the right tools Continuous testing requires a major shift in mindset and culture agree or strongly agree 63% agree or strongly agree 79% agree or strongly agree 72% “Continuous Testing as a Digital Business Enabler”, 2017 “ 5
- 6. © 2017 CA.Confidential. All rights reserved. “Continuous Testing as a Digital Business Enabler”, 2017 Tangible Evidence of the incremental value continuous testing represents for “Leaders” more likely to have succeeded in left-shifting testing activity to requirements phase more confident in quality of output more confident in speed of delivery more likely to experience rapid revenue growth more likely to deliver 10x faster iterations more likely to reduce defects by 50%+ 2.5x 1.5x 1.9x 2.6x 2.4x 3.9x SPEED QUALITY ROI 6
- 7. © 2017 CA.Confidential. All rights reserved. What is Continuous Testing? First need to understand what it’s not! It’s not… Test Automation It’s not… Continuous Integration It’s not… Unit Testing It’s not… done only in the Test environment It’s not… a QA-led activity 7
- 8. © 2017 CA.Confidential. All rights reserved. What is Continuous Testing? The practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected
- 9. © 2017 CA.Confidential. All rights reserved. Why are Traditional Testing Activities a Bottleneck? Enable Interfaces (real or virtual) Create Detailed Test Cases Execute Test Cases Identify Interfaces (internal and 3P) Automate Tests (usually Regression) Create Defects Fix Defects Define an Idea or Customer Needs Write Requirements Create Requirements Traceability Matrix Read Requirements Manual Processes Find/Create Test Data Testing hasn’t changed in the past 30 years Identify Test Data Requirements
- 10. © 2017 CA.Confidential. All rights reserved. But Continuous Testing Changes the Game… Enable Interfaces (real or virtual) Create Detailed Test Cases Execute Test Cases Identify Interfaces (internal and 3P) Automate Tests (usually Regression) Create Defects Fix Defects Define an Idea or Customer Needs MODEL Requirements Create Requirements Traceability Matrix Read Requirements Find/Create Test Data Identify Test Data Requirements NOW ALL AUTOMATED
- 11. © 2017 CA.Confidential. All rights reserved. End-to-End Integrated Continuous Testing from CA Automatically generate test automation scripts from requirements 1 2 Simulate test environment 3 Access test data on- demand & ensure PII compliance 4 Start multi-layer / backend “requests & response” testing prior to UI 5 Democratize performance testing Integrate open source testing tools6 7 Ensure comprehensive, cloud-based API testing 8 Built-in automated application security testing 9 Promote artifacts when tests pass by orchestrating & automating the pipeline 10 Harness application insight across the SDLC to improve user experience Continuous Delivery Director
- 12. © 2017 CA.Confidential. All rights reserved. WHAT IS SAUCE LABS? The world’s largest cloud-based platform for running automated tests • Co-founded by Jason Huggins, the creator of Selenium; Selenium is the de facto standard for front-end test automation (Gartner) • One platform that spans desktop, simulators, emulators and real devices. • Run tests across over 900 browser/OS/platform combinations & 1,000’s of real mobile devices • Enterprise-grade security trusted by thousands of customers • Optimized for Continuous Testing and CI/CD
- 13. © 2017 CA.Confidential. All rights reserved. ACCELERATE YOUR PIPELINE Faster Releases = Competitive Advantage • Test on every build; integrate with your CI Server* and pipeline orchestration • Run tests in parallel to speed releases • Test more scenarios per unit of time • Catch bugs earlier in the development cycle & fix faster A C H I E V E C O N T I N U O U S D E L I V E R Y W I T H A U T O M AT E D T E S T I N G *Sauce supports: Jenkins, Bamboo, Travis, Circle CI, MSFT VSTS, Serial Testing (9 minutes) Chrome 1 Chrome 2 Chrome 3 FF 2 FF 2 FF 3 IE 1 IE 2 IE 3 Parallel Testing (1 minute) Chrome 1 Chrome 2 Chrome 3 FF 1 FF 2 FF 3 IE 1 IE 2 IE 3
- 14. © 2017 CA.Confidential. All rights reserved. Quality Code = Quality Experience • Catch more bugs with increased coverage • Higher quality keeps users from abandoning sites and apps • Increased coverage broadens your addressable market • Expand testing to GA / Beta & Dev releases • Videos, screenshots and logs help speed bug resolution IMPROVE QUALITY THROUGH COVERAGE E X PA N D Y O U R T E S T C O V E R A G E W I T H O U T M O R E O V E R H E A D Increased Quality
- 15. © 2017 CA.Confidential. All rights reserved. IMPROVE EFFICIENCY Less Time Testing = More Time Innovating • Enable team to focus on high value activities instead of infrastructure • Eliminate time spent on supporting & maintaining an internal test grid • Improve productivity of Devs & QE • Increase confidence in test results Fewer false failures Video debugging R E D U C E C O S T S A N D S P E E D T I M E T O VA L U E Infrastructure Costs False Failure Ongoing Maintenance
- 16. © 2017 CA.Confidential. All rights reserved. 16 DEMO
- 17. © 2017 CA.Confidential. All rights reserved. 17 Q&A
- 18. © 2017 CA.Confidential. All rights reserved. 18 Thank You! Come see us: Sauce Labs Booth #205 CA Booth #302 CI to CD in the Cloud: Orchestrate All the Things Wed 8, 2017 12:30 - 12:45pm
Editor's Notes
- #2 Throughout the global economy, across all industries, companies are re-inventing themselves―to become better at sensing the next big thing their customers need, and finding ways to deliver it to them faster than the competition. At CA, we’re fueling a big part of this revolution where ideas are the new currency. Our passion is helping you to champion your best ideas and navigate the obstacles across the application lifecycle to break through and produce real measurable outcomes. What’s slowing you down? What are your bottlenecks preventing you from getting your product, your software, out the door with exceptional quality? What’s keeping you from getting to market as fast as possible? Have you already predefined what “as fast as possible” is with how things have been that you can’t see what it could be? [next]
- #3 It’s done through Continuous Testing. Well define it later on but to serve as a compass to this discussion: We represent the Continuous Delivery division of CA We define Continuous Delivery as the ability to release _ any application at any time to any environment with zero touches But the caveat to that is, you can’t achieve Continuous Delivery, without Continuous Testing. Testing has to be a focus, just like everything else, because it’s become one of the most glaring bottlenecks in the SDLC. It’s hindering DevOps and threatening Agile. We have customers all the time tell us how agile they are. [next]
- #5 http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf The majority of delays were still occurring at the Test/QA stage of the SDLC Therefore, even when DevOps exists, companies are still experiencing a trade-off between quality and speed 29% of our DevOps practitioners told us that quality was being sacrificed to at least a degree, which means they were looking for speed. 43% are choosing to focus on quality as an area for improvement. Were they going too fast? [next]
- #6 https://www.ca.com/us/collateral/industry-analyst-report/continuous-testing-lessons-from-leading-practitioners.html CA commissioned Freeform Dynamics to conduct market research surrounding Continuous Testing. The came back with a great quote saying there’s, “Clear evidence…” They found that: The need for adoption exists 75% Organizations recognize the critical need to shift testing left with continuous testing practices. 72% say it requires a mind shift. They need champions and evangelists to convince the powers that be. And the need for technology exists 63% say the tools they use and know of wont get them there 79% say that if the right tools existed, continuous testing could be achievable [next]
- #7 Of the respondents that were surveyed, they weren’t oblivious to continuous testing. Everybody knows and sees testing as a critical piece. However, within the sample there was also evidence of those who were ahead of the pack. This survey also concluded, industry “Leaders” who are already focused on continuous testing, are more successful shifting their testing activities further to the left of the SDLC. Are two times more confident in both the speed and quality of deliverables to meet demands are 1.5 times more likely to deliver 10x faster iterations [next] Are 2.6 times more likely to reduce defects by 50% or more [next] And 3.9 times more likely to experience rapid revenue growth against their peers. There’s obviously more to their stories and this wouldn’t all be credited continuous testing, but it reaffirms, right along with other research that’s out there, that: being more agile, cultivating a Devops culture, pressing on towards Continuous delivery, keeping speed and quality at the forefront, you WILL become a leader and increase revenue. [next]
- #8 So before you start to define Continuous Testing yourself, this is what Continuous Testing is NOT. [next]
- #9 We define continuous testing as… “” This picture helps to illustrate our point. Quality is not an owned discipline of QA department. Just like how evolution of TQM (Total Quality Management) and Lean Manufacturing put quality checks between each stage of the production line in the 80’s and 90’s, Quality is EVERYONE’s responsibility and it should be imbedded at each stage of the SDLC. [next]
- #10 Once an idea is approved and requirements are written: Requirements are read An RTM, or requirements traceability matrix is created Test cases are created and tied to the requirements in the RTM Test data requirements are defined Internal and external interfaces necessary for the requirement to be fulfilled are identified Automation tests are created from the requirements and test cases Test data has to be found or created before tests can be run Internal and external interfaces needed for tests to actually run are made available or virtually enabled Finally… the test is executed, any defects found are logged and developers start the code fixing cycle Testing activities are still a bottleneck because the process hasn’t changed in 30 years! There’s been plenty of advancements in surrounding departments and disciplines like development, business analysis and operations. Everything you see on the blue line in the middle is the problem. What’s interesting, what REALLY matters is to provide quick feedback to the developer. So the intention is to get to the Execute Test Cases as fast as possible. But with the current process you have to go through ALL the activities in blue first, so you can get to where you really care – being able to give a developer a thumbs up or down. Depending on how agile your organization is (remember the agile adoption phases), you may still be working on the workload of the previous “sprint” when this new code hits QA. So, you’re already late, the developer has moved on and now has to go back and address defects. But arguably, everything in blue, these steps can’t just go away. And you’re right. [next]
- #11 Let’s just automate them instead. If we model our requirements well, if we are intentional in our design, and have the right tools in place, ..then we can shift all these efforts to the design phase and capture everything we need in the requirements. Outside CA, the other solutions on the market don’t address these manual aspects, their focus is only in making the pain less painful by increasing efficiency of those intermediate manual steps, as opposed to removing them from the equation altogether. [next]
- #12 When we look at the end-to-end we’re bringing quality to the attention of everyone. Continuous Testing how it’s accomplished. [next] The way we do this at CA the following… 1) Starting with Intake – aka requirements – Agile Requirements Designer (ARD) provides the capability to align business requirements and remove ambiguity by creating models that allow for the automatic creation of test cases and test case automation from the defined models. In addition, this capability allows for optimal and efficient test case coverage and can identify 100% coverage of the defined model. The model based approach also allows you to deal with change impact – if a requirement changes, ARD can model down and enable adaptive test case design and automation. ARD models requirements directly from user stories from ALM, JIRA and Agile Central (and several other sources). 2) Once we have defined our test cases and user stories, Service Virtualisation (SV) helps accelerate the delivery of quality code by virtualized services that can be shared across the enterprise to simulate those unavailable systems, both internal and external and remove constraints from Dev and Test. SV is available as Code so can be integrated directly into the Development IDE, thus accelerating adoption. SV also provides performance testing capability against these virtualized services. 3) Following on from removing constraints with system access, next we may need access to Test Data. Test Data Manager (TDM) provides automatic generation of synthetic test data, on demand - available across the enterprise and in a compliant manner to assist with regulations such as GDPR. TDM also supports traditional data masking and sub-setting but builds upon this with its Profiling capability to ensure you also understand your potential exposure to PII data, as well as providing the capability for data portability across the enterprise with its vTDM (virtual data) capability – effectively removing test data constraints across the organization in a compliant manner. 4) Once we have our Test cases modelled and aligned with our user stories and test systems available, along with the right test data, CA AppTest enables us to accelerate testing outside of the UI by allowing you to test against APIs – following workflows and scenarios built, full API testing can occur much earlier in the Dev lifecycle, even if the dependent system is not yet available through integration with CA SV. 5) CA Blazemeter provides SaaS infrastructure capabilities to move Performance and Load testing much earlier in the lifecycle. CA Blazemeter provides a Domain Specific Language so that Performance Testing can even occur at the Unit Testing phase within the developer IDE. CA Blazemeter is also built upon open source technologies that protect your existing investment with support of JMeter, Selenium, Gatling, Taurus – and other common opens source load testing frameworks. CA Blazemeter also provides SaaS based functional API testing capability. The choice for CA AppTest or CA Blazemeter will in part be driven by your direction with regards to SaaS adoption in your organisation. 6) As above 7) As above 8) In addition to being able to shift Performance Testing left within the SDLC, we provide capabilities with respect to Application Security with CA Veracode – direct integration into the IDE with the “Greenlight” capability – security unit testing - that not only identifies risk in near real time whilst coding but also identifies good behaviours that helps improve the capabilities of the team. Coupled with Static and Dynamic code analysis to identify flaws and vulnerabilities, CA Veracode is also a SaaS based capability - that has so far scanned more than 3 trillion lines of code for our customers. 9) Nearing the end of the flow – the ability to understand the lifecycle of a user story – Continuous Delivery Director (CDD) provides you with the insight needed to understand the flow of a user story across the SDLC – CDD provides insights into phase promotion, cycle time, idle time and enables trust and transparency for all relevant stakeholders into the lifecycle of the delivery of value to a customer. Not only is this a CD pipeline optimization tool, giving you fantastic visibility to manage all states of the SDLC, but it also ensures applications and code doesn’t get promoted until test cases are passed. It really is the hub of the activity, with it’s integration with the testing tools to kick each one of them off. 10) Lastly with respect to continuous improvement – CA Digital Experience Insights (DXI) allows you to understand the real user experience of the value you are delivering – across Mobile, Watch and Web – DXI allows critical insight into how users are interacting with your applications via user journey flow visibility, crash analytics and operational data, ensuring you can continue to deliver a quality experience to your users. We spin the wheel to emphasize that that you don’t just kick off automated test scripts when code hits QA. These are disciplines to be used continuously at each stage of the SDLC. Only then is it possible to achieve continuous testing, continuous deployment, continuous delivery, and record-breaking ROI and customer experience. [next]