Downloaded 15 times
Uploaded byJaco Greyling
Continuous Testing - The Final Frontier
Continuous testing is the practice of testing software throughout the development lifecycle to identify and address issues early. It involves automating test cases from requirements, simulating test environments, accessing test data on demand, starting multi-layer testing before UI, democratizing performance testing, and integrating open source testing tools. Most companies cannot currently do continuous testing but aspire to achieve it through practices like in-sprint testing automation, test scripting and execution within sprints, and analyzing test results to update requirements models. Continuous testing is important because it helps ensure high quality software and speed of delivery, both of which are critical to business success.
More Related Content
Similar to Continuous Testing - The Final Frontier
Continuous Testing - The Final Frontier
- 1. Continuous Testing – Thefinal frontier of DevOps DevOpsDays Cape Town 2017 Jaco Greyling, CTO Enterprise DevOps @jaco_greyling
- 2. Testing Experiences Ambiguous or no requirements Testingis the last line of defense Lack of test environments creating delays Test data? Pfft, I create my own! QA is not Agile, always one Sprint behind Manual, manual, manual testing… 2 * TestMatick Software Testing Company
- 3. Copyright © 2017CA. All rights reserved. Eliminating the barriers between ideas and outcomes? How good are you at … 3
- 4. Copyright © 2017CA. All rights reserved. Your Software Factory must be an accelerator. Is yours, or is it an inhibitor? ProductConcept EFFICIENT, HIGH-QUALITY, PRODUCTIVE PIPELINE MORE CUSTOMER VALUE BUSINESS ALIGNMENT 4
- 5. Copyright © 2017CA. All rights reserved. CREATE AN AGILE BUSINESS THE MODERN SOFTWARE FACTORY The capabilities you need, from mobile to mainframe, to deliver the experiences customers want. Agile DevOps Security BUILD BETTER APPS, FASTER MAKE SECURITY A COMPETITIVE ADVANTAGE MAXIMIZE APPLICATION PERFORMANCEMAXIMIZE APPLICATION PERFORMANCE 5
- 6. Copyright © 2017CA. All rights reserved. Continuous Testing How do you build better apps, faster? 6
- 7. Copyright © 2017CA. All rights reserved. What is your reality? * Actual customer example if their testing strategy Dev Tools TOOLSET TEST PYRAMID UI Tools Dev Tools TOOLSET TEST PYRAMID API Tools UI Tools TESTING GOAL API Tools 94% 3% 3% 5% 45% 50% Non-functional: static, performance, security, etc… API Unit UI API Unit UI TESTING REALITY 7
- 8. Copyright © 2017CA. All rights reserved. 32% 16% 22% 63% 21% 23% 30% QA is STILL a Bottleneck… Even in DevOps Shops http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf …of delays were occurring at the Test/QA stage of the cycle. “Where are the main hold-ups in the software production process?” 63%A 2017 survey of self- proclaimed DevOps practitioners found that … 8
- 9. Copyright © 2017CA. All rights reserved. Challenges to Achieving Continuous Delivery & Testing Ave. cost of downtime per minute 53% percent of visits are abandoned if a mobile site takes more than three seconds to load Requirements Development QA / Testing Operations Release 70% of all testing is still manual4 63% of testers admit they can’t test across all the different devices and OS versions5 50% of time spent looking for test data6 79% of teams face restrictions, time limits or access fees on 3rd party services3 X X X 64% of total defect cost originate in the requirements analysis and design phase1 50% of developers time is spent fixing and finding defects2 ? 80% of teams experience delays in development and QA due to unavailable dependencies3 ? 57% are dissatisfied with the time it takes to deploy new features7 ! ! Greater than 50% production outages from configuration management ! = $4,700 1 Hyderabad Business School GITAM University Quality Flaws: Issues and Challenges in Software Development – 2012 2 Study conducted by the Judge Business School at Cambridge University – January 2013 3 Voke Market Snapshot Service Virtualization – January 2015 4 Bloor Report Automated test case generation - September 2014 5 Perfecto Mobile 2014 Benchmark Survey, Why Mobile Apps Fail - 2014 6 Forrester Global Modern Service Delivery Benchmark Online Survey - Q4 2014 7 BlazeMeter “Black Friday” study, 2016 9
- 10. Copyright © 2017CA. All rights reserved. What is Continuous Testing? IT’S NOT… TEST AUTOMATION IT’S NOT… CONTINUOUS INTEGRATION IT’S NOT… UNIT TESTING IT’S NOT… DONE ONLY IN THE TEST ENVIRONMENT IT’S NOT… A QA-LED ACTIVITY 1 2 3 4 5 10 First we need to understand what it’s not!
- 11. The practice oftesting across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected. 11 What is Continuous Testing?
- 12. Copyright © 2017CA. All rights reserved. Key Elements of Continuous Testing Automatically generate test automation scripts from requirements 1 2 Simulate test environment 3 Access test data on- demand & ensure PII compliance 4 Start multi-layer / backend “requests & response” testing prior to UI 5 Democratize performance testing Integrate open source testing tools 6 7 Ensure comprehensive, cloud-based API testing 8 Built-in automated application security testing 9 Promote artifacts when tests pass by orchestrating & automating the pipeline 10 Harness application insight across the SDLC to improve user experience 12
- 13. Most Customers CannotDo Continuous Testing Today, But They Aspire To Get There….
- 14. Copyright © 2017CA. All rights reserved. If This is Your Current SDLC You are waterfall Dev Test X WEEKS X WEEKS X WEEKS Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
- 15. Copyright © 2017CA. All rights reserved. If This is Your Current SDLC You are water-SCRUM-fall Sprint 1 Sprint 2 Sprint 3 X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
- 16. Copyright © 2017CA. All rights reserved. If This is Your Current SDLC You are not “really” agile Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint N Sprint 3 Sprint N User Stories Dev and Manual Testing Automated Regression Testing X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 1 Sprint 2 X WEEKS X WEEKS Sprint 3 Release
- 17. Copyright © 2017CA. All rights reserved. What Testing at the Speed of Agile Looks Like! In-Sprint everything to achieve “Continuous Testing” No Sprint-lag effect Release in end of Sprint In-Sprint Test Automation Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint N Sprint 3 Sprint N User Stories Dev and Exploratory Manual Testing Test Automated Scripting & Execution X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 1 Sprint 2 Release Sprint 3 ReleaseReleaseRelease HIGHLIGHTS
- 18. Copyright © 2017CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING TEST EXECUTION MONITORING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Flowchart Diagrams (Model-Driven) DEVELOPMENT TEST DESIGN
- 19. Copyright © 2017CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Test-Driven Development DEVELOPMENT REQUIREMENTS MODELING TEST EXECUTION MONITORINGTEST DESIGN
- 20. Copyright © 2017CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Test Cases Test Automation Scripts Test Data TEST DESIGN DEVELOPMENTREQUIREMENTS MODELING TEST EXECUTION MONITORING
- 21. Copyright © 2017CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Test Environments Test Automation Engines Test Frameworks DEVELOPMENTREQUIREMENTS MODELING TEST EXECUTION MONITORINGTEST DESIGN
- 22. Copyright © 2017CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Analyze Learn Update Model DEVELOPMENTREQUIREMENTS MODELING MONITORING TEST DESIGN TEST EXECUTION
- 23. Can You Afford Not To? AnyFocus On Speed… Without Quality Will Result in Their Company’s Future DEPENDS on the QUALITY of their software * “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 2/3 Of Business Leaders Say
- 24.
- 25. CTO Enterprise DevOps jaco.greyling@caafrica.co.za JacoGreyling @jaco_greyling slideshare.net/CAinc linkedin.com/in/jaco-greyling/ ca.com/za
Editor's Notes
- #8 Today UI Testing is estimated to have only 45% Automation according to Forrester. 100% manual UI testing – large QA departments Toolsets + UI Focus = High Script Maintenance Leading to unsustainable QA organic growth (HP suite) Most companies Automation Roadmap was UI focused due to tooling focus same Tool & automation experience was almost non-existence Future 90% of regression automation is in the CI/CD framework Developers can build and maintain for their unit testing UI testing for new features only* Transition automation team to service delivery focusing on engineering solutions Synthetic data generation for all lower environments Data Virtualization