Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Testing - The Final Frontier

100 views

Published on

A Continuous Testing presentation, presented at DevOpsDays Cape Town 2017.

Published in: Software
  • Be the first to comment

Continuous Testing - The Final Frontier

  1. 1. Continuous Testing – The final frontier of DevOps DevOpsDays Cape Town 2017 Jaco Greyling, CTO Enterprise DevOps @jaco_greyling
  2. 2. Testing Experiences Ambiguous or no requirements Testing is the last line of defense Lack of test environments creating delays Test data? Pfft, I create my own! QA is not Agile, always one Sprint behind Manual, manual, manual testing… 2 * TestMatick Software Testing Company
  3. 3. Copyright © 2017 CA. All rights reserved. Eliminating the barriers between ideas and outcomes? How good are you at … 3
  4. 4. Copyright © 2017 CA. All rights reserved. Your Software Factory must be an accelerator. Is yours, or is it an inhibitor? ProductConcept EFFICIENT, HIGH-QUALITY, PRODUCTIVE PIPELINE MORE CUSTOMER VALUE BUSINESS ALIGNMENT 4
  5. 5. Copyright © 2017 CA. All rights reserved. CREATE AN AGILE BUSINESS THE MODERN SOFTWARE FACTORY The capabilities you need, from mobile to mainframe, to deliver the experiences customers want. Agile DevOps Security BUILD BETTER APPS, FASTER MAKE SECURITY A COMPETITIVE ADVANTAGE MAXIMIZE APPLICATION PERFORMANCEMAXIMIZE APPLICATION PERFORMANCE 5
  6. 6. Copyright © 2017 CA. All rights reserved. Continuous Testing How do you build better apps, faster? 6
  7. 7. Copyright © 2017 CA. All rights reserved. What is your reality? * Actual customer example if their testing strategy Dev Tools TOOLSET TEST PYRAMID UI Tools Dev Tools TOOLSET TEST PYRAMID API Tools UI Tools TESTING GOAL API Tools 94% 3% 3% 5% 45% 50% Non-functional: static, performance, security, etc… API Unit UI API Unit UI TESTING REALITY 7
  8. 8. Copyright © 2017 CA. All rights reserved. 32% 16% 22% 63% 21% 23% 30% QA is STILL a Bottleneck… Even in DevOps Shops http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf …of delays were occurring at the Test/QA stage of the cycle. “Where are the main hold-ups in the software production process?” 63%A 2017 survey of self- proclaimed DevOps practitioners found that … 8
  9. 9. Copyright © 2017 CA. All rights reserved. Challenges to Achieving Continuous Delivery & Testing Ave. cost of downtime per minute 53% percent of visits are abandoned if a mobile site takes more than three seconds to load Requirements Development QA / Testing Operations Release 70% of all testing is still manual4 63% of testers admit they can’t test across all the different devices and OS versions5 50% of time spent looking for test data6 79% of teams face restrictions, time limits or access fees on 3rd party services3 X X X 64% of total defect cost originate in the requirements analysis and design phase1 50% of developers time is spent fixing and finding defects2 ? 80% of teams experience delays in development and QA due to unavailable dependencies3 ? 57% are dissatisfied with the time it takes to deploy new features7 ! ! Greater than 50% production outages from configuration management ! = $4,700 1 Hyderabad Business School GITAM University Quality Flaws: Issues and Challenges in Software Development – 2012 2 Study conducted by the Judge Business School at Cambridge University – January 2013 3 Voke Market Snapshot Service Virtualization – January 2015 4 Bloor Report Automated test case generation - September 2014 5 Perfecto Mobile 2014 Benchmark Survey, Why Mobile Apps Fail - 2014 6 Forrester Global Modern Service Delivery Benchmark Online Survey - Q4 2014 7 BlazeMeter “Black Friday” study, 2016 9
  10. 10. Copyright © 2017 CA. All rights reserved. What is Continuous Testing? IT’S NOT… TEST AUTOMATION IT’S NOT… CONTINUOUS INTEGRATION IT’S NOT… UNIT TESTING IT’S NOT… DONE ONLY IN THE TEST ENVIRONMENT IT’S NOT… A QA-LED ACTIVITY 1 2 3 4 5 10 First we need to understand what it’s not!
  11. 11. The practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected. 11 What is Continuous Testing?
  12. 12. Copyright © 2017 CA. All rights reserved. Key Elements of Continuous Testing Automatically generate test automation scripts from requirements 1 2 Simulate test environment 3 Access test data on- demand & ensure PII compliance 4 Start multi-layer / backend “requests & response” testing prior to UI 5 Democratize performance testing Integrate open source testing tools 6 7 Ensure comprehensive, cloud-based API testing 8 Built-in automated application security testing 9 Promote artifacts when tests pass by orchestrating & automating the pipeline 10 Harness application insight across the SDLC to improve user experience 12
  13. 13. Most Customers Cannot Do Continuous Testing Today, But They Aspire To Get There….
  14. 14. Copyright © 2017 CA. All rights reserved. If This is Your Current SDLC You are waterfall Dev Test X WEEKS X WEEKS X WEEKS Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
  15. 15. Copyright © 2017 CA. All rights reserved. If This is Your Current SDLC You are water-SCRUM-fall Sprint 1 Sprint 2 Sprint 3 X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
  16. 16. Copyright © 2017 CA. All rights reserved. If This is Your Current SDLC You are not “really” agile Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint N Sprint 3 Sprint N User Stories Dev and Manual Testing Automated Regression Testing X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 1 Sprint 2 X WEEKS X WEEKS Sprint 3 Release
  17. 17. Copyright © 2017 CA. All rights reserved. What Testing at the Speed of Agile Looks Like! In-Sprint everything to achieve “Continuous Testing” No Sprint-lag effect Release in end of Sprint In-Sprint Test Automation Sprint 1 Sprint 2 Sprint 3 Sprint N Sprint 2 Sprint N Sprint 3 Sprint N User Stories Dev and Exploratory Manual Testing Test Automated Scripting & Execution X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 1 Sprint 2 Release Sprint 3 ReleaseReleaseRelease HIGHLIGHTS
  18. 18. Copyright © 2017 CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING TEST EXECUTION MONITORING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Flowchart Diagrams (Model-Driven) DEVELOPMENT TEST DESIGN
  19. 19. Copyright © 2017 CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Test-Driven Development DEVELOPMENT REQUIREMENTS MODELING TEST EXECUTION MONITORINGTEST DESIGN
  20. 20. Copyright © 2017 CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Test Cases Test Automation Scripts Test Data TEST DESIGN DEVELOPMENTREQUIREMENTS MODELING TEST EXECUTION MONITORING
  21. 21. Copyright © 2017 CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Test Environments Test Automation Engines Test Frameworks DEVELOPMENTREQUIREMENTS MODELING TEST EXECUTION MONITORINGTEST DESIGN
  22. 22. Copyright © 2017 CA. All rights reserved. Simple View of Continuous Testing REQUIREMENTS MODELING Continuous Integration Continuous Testing Continuous Delivery Continuous Deployment WhatHow Analyze Learn Update Model DEVELOPMENTREQUIREMENTS MODELING MONITORING TEST DESIGN TEST EXECUTION
  23. 23. Can You Afford Not To? Any Focus On Speed… Without Quality Will Result in Their Company’s Future DEPENDS on the QUALITY of their software * “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 2/3 Of Business Leaders Say
  24. 24. Thank You.
  25. 25. CTO Enterprise DevOps jaco.greyling@caafrica.co.za Jaco Greyling @jaco_greyling slideshare.net/CAinc linkedin.com/in/jaco-greyling/ ca.com/za

×