Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Digital Testing: How to enable Continuous Adaptive Testing (EN)

1,400 views

Published on

In the digital economy, everyone’s in the software business, and the landscape is constantly changing. The traditional approach to providing business value through lengthy plan–build–test–deploy cycles no longer delivers value in an increasingly competitive, consumer–centric marketplace. Merely compressing these cycles by adopting iterative or agile methods doesn’t address the innovation, speed–to–market, quality and resilience challenges organisations face.

Shifting to a more dynamic, adaptive and continuous model calls for new tools and practices that accelerate communication, collaboration, integration, measurement and automation. Whether labelled this as digital, shift–left or DevOps, understanding the detail of this journey is an essential part to becoming digital–testing ready.

Join Jonathon as he explores the fundamentals of continuous adaptive testing that will enable the continuous delivery pipeline, and to provide instant, smart, cognitive adaptive technology that will power the next generation of insight and value–driven delivery capable platforms. Revolutionising how you test in the digital future will drive smart business decisions and end costly, reactive rework and errors.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Digital Testing: How to enable Continuous Adaptive Testing (EN)

  1. 1. 1 © 2017 CA. ALL RIGHTS RESERVED. Digital Testing Driving Cognitive Adaptive Testing Practices Jonathon Wright Chief Technology Evangelist DevOps, Automation & Digital Assurance (R&D) © 2017 CA. ALL RIGHTS RESERVED. www.slideshare.com/Jonathon_Wright
  2. 2. 2 © 2017 CA. ALL RIGHTS RESERVED. Disclaimer Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of March 1st and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion. Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if- available basis. The information in this presentation is not deemed to be incorporated into any contract. Copyright © 2017 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY. CA assumes no responsibility for the accuracy or completeness of the information. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENT “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. In no event will CA be liable for any loss or damage, direct or indirect, in connection with this presentation, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised in advance of the possibility of such damages.
  3. 3. 3 © 2017 CA. ALL RIGHTS RESERVED. Abstract In the digital economy, everyone’s in the software business, and the landscape is constantly changing. The traditional approach to providing business value through lengthy plan–build–test– deploy cycles no longer delivers value in an increasingly competitive, consumer–centric marketplace. Merely compressing these cycles by adopting iterative or agile methods doesn’t address the innovation, speed–to–market, quality and resilience challenges organisations face. Shifting to a more dynamic, adaptive and continuous model calls for new tools and practices that accelerate communication, collaboration, integration, measurement and automation. Whether labelled this as digital, shift–left or DevOps, understanding the detail of this journey is an essential part to becoming digital–testing ready. Join Jonathon as he explores the fundamentals of continuous adaptive testing that will enable the continuous delivery pipeline, and to provide instant, smart, cognitive adaptive technology that will power the next generation of insight and value–driven delivery capable platforms. Revolutionising how you test in the digital future will drive smart business decisions and end costly, reactive rework and errors. Jonathon Wright CA Chief Technology Evangelist | DevOps, Automation & Digital Assurance (R&D)
  4. 4. 5 © 2017 CA. ALL RIGHTS RESERVED. The success of our technologies largely depend on the fact that, while we were speculating about the possibility of ultra-intelligence, we increasingly enveloped the world in so many devices, sensors, application and data that it became an IT-friendly environment, where technologies could replace us without having any understanding, intentions, interpretations, emotional states, semantic skills, consciousness, self-awareness or flexible intelligence. Memory as in algorithms and immense datasets now outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015 “ ” THE NEED FOR SPEED 94%of executives face increased pressure to release apps more quickly “ Journey to the Digital Enterprise © 2017 CA. ALL RIGHTS RESERVED. 5 Based on 2014 Vanson Bourne study commissioned by CA
  5. 5. 6 © 2017 CA. ALL RIGHTS RESERVED. VALUE-DRIVEN TIME 1960s 1990s TODAY Digital Enterprise - ‘Evolution, over Revolution’ 1 – “The Battle for Competitive Advantage in the App Economy”, Oxford Economics, 2015 of enterprises believe that they need algorithmic business models to respond to disruption in the next 3 years. 78% First Wave Legacy is your Legacy (API) Four Wave Digitalization of your Core (SOA) Eight Wave Cognitive Adaptive Adoption
  6. 6. 7 © 2017 CA. ALL RIGHTS RESERVED. QUALITY 2/3 of business leaders say the future of their business depends on the quality of their software2 VELOCITY 94% of executives face increased pressure to release apps more quickly1 1. 2014 Vanson Bourne study commissioned by CA 2. “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 3. “DevOps and the Cost of Downtime” – IDC 2014 “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 © 2017 CA. ALL RIGHTS RESERVED.7 COST 25% of a single application’s development and operations costs is wasteful3 While Ensuring a Superior Digital Experience! Digital Enterprise Delivering Quality at Speed SHIFT RIGHT
  7. 7. 8 © 2017 CA. ALL RIGHTS RESERVED. Digital Engineering Cognitive Adaptive Technology Machine Learning Artificial Intelligence A B forces-destiny.com © 2017 CA. ALL RIGHTS RESERVED.8
  8. 8. 9 © 2017 CA. ALL RIGHTS RESERVED. © 2017 CA. ALL RIGHTS RESERVED.9 Digital Future(ist) Dr. James Whittaker (www.docjamesw.com) 7 stages (http://a.co/gQk9ckx) Internet of Everything (IoT Hot Tub)
  9. 9. 10 © 2017 CA. ALL RIGHTS RESERVED. Deep reinforcement learning Digital Technology Cognitive Adaptive Learning Machine learning with neural dictionaries https://www.luis.ai “We are not at the centre of the universe (Copernicus), of the biological kingdom (Darwin) or of the realm of rationality (Freud). After Turing, we are no longer the centre of the infosphere, we share the infosphere with digital technologies”. The ethics of artificial intelligence, Luciano Floridi
  10. 10. 11 © 2017 CA. ALL RIGHTS RESERVED. Mode 0 - Cognitive Cognitive Adaptive Testing (CAT): 1. “How long will it take to test these new features?” 2. “What is the associated risk rating of the new platform?” 3. “What is the defect density and prediction for this solution?” 4. “What is the current conference rating of the solution?” 5. “What is the value of this capability?” How will Machine Learning help? Digital Testing Example Case Studies of Machine Learning
  11. 11. 12 © 2017 CA. ALL RIGHTS RESERVED. Emergent embedded AI for neural networks Digital Technology Cognitive Adaptive Intelligence (iCA) Singularity Ultra-Intelligence (2030) “Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s
  12. 12. 13 © 2017 CA. ALL RIGHTS RESERVED. Mode 1 - Adaptive Cognitive Adaptive Testing (CAT): 1. “Can you create me all the tests I need to fully test this system?” 2. “How do I successfully test this new feature?” 3. “How much testing has been done already?” 4. “Have you detected any anomalies within the system?” 5. “How much testing will be enough?” How will Artificial Intelligence help? Digital Testing Example Case Studies of Artificial Intelligence
  13. 13. 14 © 2017 CA. ALL RIGHTS RESERVED. Ideas Beliefs, Values, Culture Heuristics Behaviors Value Agile Lean Digital Testing Solution Thinking – Continuous Testing 101 – Episode I Copyright © 2017 nuCognitive LLC. All Rights Reserved. Methods and Skills Ray Arell
  14. 14. 15 © 2017 CA. ALL RIGHTS RESERVED. DesignOps Create and deliver Ideas to Outcomes Pervasive Security Redefine how work is planned, managed and executed Agile Management Provide a flawless app experience optimized for performance Agile Operations Accelerate and streamline development, testing and release Continuous Delivery Continuous Development Continuous Adaptive Delivery Continuous Release Digital Testing Shift Left introducing DesignOps
  15. 15. 16 © 2017 CA. ALL RIGHTS RESERVED. DevOps Hub  Support the pace and scale of Agile  Fully Automated & Repeatable  Model based Continuous Delivery supporting highly heterogeneous SDLC tool ecosystems  Remove ambiguity from requirements & testing  Abstraction of dependent services/APIs radically accelerate speed and quality  “Pre instrumented” app releases  Flexibility/portability across modern infrastructure & cloud platforms Continuous Adaptive Delivery Release Everywhere
  16. 16. 17 © 2017 CA. ALL RIGHTS RESERVED. DesignOps – “Shift Left” (AND Right) CONTINUOUS ADAPTIVE TESTING CODE SHIFT LEFT SHIFT RIGHT BUILD DELIVERY PRODUCTION
  17. 17. 18 © 2017 CA. ALL RIGHTS RESERVED. CODE SHIFT LEFT SHIFT RIGHT BUILD DELIVERY PRODUCTION CONTINUOUS ADAPTIVE TESTING DesignOps – “Shift Left” (AND Right)
  18. 18. 19 © 2017 CA. ALL RIGHTS RESERVED. Build Deliver MonitorMeasure Learn Design Make Check Think Proactive over Reactive Governance over Ownership Maturity over Mastery Predictive over Continuous Evolution over Revolution Engineering over Agility Intelligence over Things designops.net
  19. 19. 20 © 2017 CA. ALL RIGHTS RESERVED. DIGITAL ASSURANCE VALUE-DRIVEN DELIVERY 1 Cognitive Adaptive Learning 2 Cognitive Adaptive Testing 3 Cognitive Adaptive Technology 4 Cognitive Adaptive Delivery Digital Testing Cognitive Adaptive Adoption (CA-A)
  20. 20. 21 © 2017 CA. ALL RIGHTS RESERVED. Erik Simmons Copyright © 2017 nuCognitive LLC. All Rights Reserved. Digital Testing Value-Driven Delivery – Continuous Testing 101 – Episode II
  21. 21. 22 © 2016 CA. ALL RIGHTS RESERVED.22 © 2017 CA. ALL RIGHTS RESERVED. Portfolio Management Unify long term strategy, investment and portfolio planning. Collaboratively plan, prioritize and track work across the enterprise. Project Management Develop & Test Synchronize efforts to dramatically speed app development & increase quality Manage & Monitor Make a great customer experience a competitive advantage Release & Deploy Control the release process, to continuously advance application quality, improve the customer experience and reduce costs. Continuous Delivery Minimum Viable Experience Minimum Viable Quality Value-Driven Delivery BUSINESS INITIATIVE Complexity- Informed Organizational Change BUSINESS VALUE Minimum Viable, Validate, Value Product CUSTOMER THINK MAKE Digital Testing Continuous Adaptive Adoption (CA-A)
  22. 22. 23 © 2017 CA. ALL RIGHTS RESERVED. REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS CONTINUOUS DELIVERY MAINFRAME SECURITY 23 © 2016 CA. ALL RIGHTS RESERVED. API analyticsRelease management dashboard Access incident responseReal-time app performance Digital Testing Continuous Adaptive Delivery (CA-D)
  23. 23. 24 © 2017 CA. ALL RIGHTS RESERVED.  Automate manual steps and scripts—eliminating repetitive, error-prone tasks  Design reusable deployment workflows  Integrate best-in-class tools dev through production  Gain visibility and transparency  Close feedback loops throughout the pipeline  Efficiently release multi- team, cross-app, composite releases  Define and enforce continuous delivery best practices OPTIMIZED AUTOMATED CONTINUOUS  Monitor, measure and analyze release processes  Maximize quality as the velocity and volume grows  Improve business execution  Measure and improve team productivity and maturity 3 2 1 Digital Testing Navigate the Continuous Adaptive Delivery Journey
  24. 24. 25 © 2017 CA. ALL RIGHTS RESERVED. The ability to reliably release high-quality solutions at any time CONTINUOUS ADAPTIVE TESTING Accelerate Deliveries Reduce Errors Manage Complexity Increase Visibility Drive Collaboration Continually Improve Digital Testing Continuous Adaptive Testing (CAT)
  25. 25. 26 © 2017 CA. ALL RIGHTS RESERVED. SOI Visualization (Big) Data Analytics Inference (Machine Learning & AI) Intelligent Process Automation Digital Technology Integrations Digital Testing Systems of Intelligence © 2017 CA. ALL RIGHTS RESERVED.26
  26. 26. 27 © 2017 CA. ALL RIGHTS RESERVED. AUTOMATION MAINTENANCE STATIC RULES BIG (un-structured) DATA IOT, IOE CX / DX SECURITY STRUCTURED AUTOMATION Digital Testing Key Challenges Posed by Adaptive Digital Systems
  27. 27. 28 © 2017 CA. ALL RIGHTS RESERVED. AUTOMATED & SELF- HEALING AUTOMATION DYNAMIC/ ADAPTIVE RULES BIG DATA ANALYTICS IOT-Ops DX-Ops SEC-Ops INTELIGENT / ADAPTIVE / COGNITIVE AUTOMATION + + + + + + + Digital Testing What we need for next-gen DesignOps
  28. 28. 29 © 2017 CA. ALL RIGHTS RESERVED.  Failure detection and prevention  Scripted automation  Limited process automation based on static rules  Individual metrics based on structured data in lifecycle silo  Limited insight and perspectives  Reactive: deals with “Past”  Failure prediction  Automate the automation  Intelligent process automation based on dynamic inference  Analytics based on large scale structured and unstructured data across the lifecycle  Focus on Insights  Predictive: from “What did happen?” to “What would happen? TRADITIONAL AUTOMATION COGNITIVE AUTOMATION Digital Testing Intelligent Automation Characteristics
  29. 29. 30 © 2017 CA. ALL RIGHTS RESERVED. DESIGN DEVELOP TEST DEPLOY OPERATEDESIGNOPS DATA HUBS DEVOPS HUB Digital Testing Cognitive Adaptive Delivery (CA-D)
  30. 30. 31 © 2017 CA. ALL RIGHTS RESERVED. AUTOMATE THE AUTOMATION Autonomic Root Cause Tests Creation Autonomic Test Creation Autonomic Virtual Service Creation Provision Configure Deploy TestBuild/CIDev Dev Test Prod Environments Digital Testing CA Agile Requirements Designer + CA ArD Server + CA Automator
  31. 31. 32 © 2017 CA. ALL RIGHTS RESERVED. Please can you build tests directly into the pipeline? Model-Driven Design of Tests from Use Cases1 2 3 Generate ‘Automation as Code’ from Model Add ‘CA Automator’ Task to ‘Pipeline as Code’ “Sure, just assign me the tasks and I will automatically generate the necessary artefacts within whichever release automation platform you want!” Digital Testing Test in DevOps (TiD)
  32. 32. 33 © 2017 CA. ALL RIGHTS RESERVED. CHANGE: Model-Driven Design (Code, Requirements & Tests) Dynamically generated Virtual Automation Environment Provisioning Dynamically generated Tests based on Cognitive Adaptive Testing LEARN: Model-Driven Delivery (Results, Defects & Tests) DESIGN OPSDEV TEST Digital Testing Plan Define Unit test Code Check in Build Provision Test Release Infinite Probability Drive Automation as Code Pipeline as Code Infinite Probability Drive Cognitive Adaptive Learning
  33. 33. 34 © 2017 CA. ALL RIGHTS RESERVED. Use Case 1. Requirement / Use case / User story (Rally) 2. Generate Model-Driven Design flows (ArD) 3. Optimize & Generate Automation (ArD) 4. Workflow ‘Automation as Code’ (AD Automator) 5. Populate ‘Data as Code’ (ArD Databuilder) 6. Customise ‘Configuration as Code’ (YAML) 7. Build ‘Application as Code’ (Jenkins) 8. Provision ‘Platform as Code’ (Docker) 9. Virtualize ‘Infrastructure as Code’ (SV, NV & NFV) 10. Deliver ‘Pipeline as Code’ (RA CDE, Puppet & Chef) 11. Deploy Headless Test Runners (MicroContainers) 12. Deprovision ‘Environment as Code’ (GIT / Blob) 13. Analyse ‘Results as Code’ (ArD Server) 14. Pinpoint Failure (Optimizer NLP) 15. Generate new Model-Driven Delivery flows (CLI) Pinpoint Failure Analysis Model Flows Optimize Generate Workflow CodeProvision ExecuteAnalyzePinpointIterate Coverage
  34. 34. 35 © 2017 CA. ALL RIGHTS RESERVED. Pipeline Automation Configuration Data Code Test Use Case Infrastructure Platform Microservices Release Environment Applications Model CommandLineInterfaceforTesters Digital Testing ‘Automation of Everything (AoE)’
  35. 35. 36 © 2017 CA. ALL RIGHTS RESERVED. *CA Agile Requirement Designer - Automation Builder has native support for EggPlant, Ranorex, Selenium 3.0 and Cucumber. Digital Testing ‘Automation as Code’
  36. 36. 37 © 2017 CA. ALL RIGHTS RESERVED. Cognitive Adaptive Intelligence (CAi) Release Management Environment Management Application Management Network Management Infrastructure Management Repository Management Data Management Test Data LearnProbe CA App Experience Analytics Digital Testing Cognitive Adaptive Intelligence (CAi)
  37. 37. 38 © 2017 CA. ALL RIGHTS RESERVED. Portfolio backlog prioritization and value analysis Improved scheduling and capacity Validated requirements in backlog Improved estimates Predict failures based on code commits Dynamic code promotion criteria Predict defects/failures Optimized Tests Scenario identification based on trends DX Scores Anomaly detection PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase Cognitive Adaptive Intelligence (CAi) Examples of Cognitive Adaptive Intelligence Examples of Outcomes Investment decision analytics Code quality analytics Test Optimization & Defect prediction A/B Testing & Canary Log & DX Analytics DX-based requirements Digital Testing Cognitive Adaptive Intelligence (CAi)
  38. 38. 39 © 2017 CA. ALL RIGHTS RESERVED. > 50% large organizations globally will compete using advanced analytics & proprietary algorithms 40% enterprises' net-new analytics investment will go to predictive / prescriptive 75% technology-oriented business intelligence competency centers will evolve to focus on information value generation Source: Gartner Digital Testing Intelligent Analytics is the New Battleground
  39. 39. 40 © 2017 CA. ALL RIGHTS RESERVED. What were the past defect trends? What were the root cause of the past defects? What defects are we likely to find? What preventive measures should we take? Have we overlooked anything for defect prevention? Predictive analytics Descriptive Analytics Diagnostic Analytics Prescriptive Analytics Deep Learning Digital Testing Examples of Analytics Use cases
  40. 40. 41 © 2017 CA. ALL RIGHTS RESERVED. * CA’s Analytics Reference Architecture Data Collectors DataAggregator Agents … Probes Machine/Log File Data Insight-Driven Existing Platforms Custom Platforms New Platforms Partner Platforms Analytics Engine DataInjectionAPI DataAccessAPI Service Layer DataExtraction Data Processing (Speed) Data Processing (Batch) Data Bus ‘Shift Right’ Becoming an Insight-Driven Business Digital Testing
  41. 41. 42 © 2017 CA. ALL RIGHTS RESERVED.© 2016 CA. ALL RIGHTS RESERVED.© 2017 CA. ALL RIGHTS RESERVED. BRINGS TRUE ARTIFICIAL INTELLIGENCE (AI) TO LIFECYCLE VIRTUALIZATION  Virtualize services without requiring any knowledge or decoding of the service protocols  Applies a genome sequence alignment algorithm, discovers byte-level patterns in message protocols  Now virtualize a much wider range of protocols without requiring a new DPH HIGH ACCURACY: 99.6 – 100%  Increased speed and accuracy with Entropy Weighting + Message Clustering  The more data a service observes the data, more intelligent it becomes  Perfect for performance testing where we deal with tons of data Digital Testing Cognitive Adaptive Learning (CA-L)
  42. 42. 43 © 2017 CA. ALL RIGHTS RESERVED. Digital Testing Cognitive Adaptive Technology (CA-t) 1. Digital Mobile, PerfectoMobile, Chapter 22, Jonathon Wright
  43. 43. 44 © 2017 CA. ALL RIGHTS RESERVED. Where next?
  44. 44. 45 © 2017 CA. ALL RIGHTS RESERVED. Where next? Digital Assurance - Pocketbook Blogs - #ExcuseFreeTesting Cognitive Adaptive Testing CA Technologies
  45. 45. 46 © 2017 CA. ALL RIGHTS RESERVED. Digital Assurance Pocketbook Paul Gerrard NEW exclusive Pocketbook authored by Paul Gerrard, eminent Test-Transformation consultant, thought leader, teacher and international conference speaker. Order your FREE copy today from http://bit.ly/digital-assurance
  46. 46. 47 © 2017 CA. ALL RIGHTS RESERVED. Digital Assurance Pocketbook – World Tour TORONTO LOS ANGELES SILCON VALLEY
  47. 47. 48 © 2017 CA. ALL RIGHTS RESERVED. Blogs #ExcuseFreeTesting – Legacy is your Legacy! https://dzone.com/articles/legacy-is-our-legacy-1
  48. 48. 49 © 2017 CA. ALL RIGHTS RESERVED. Q & A
  49. 49. Why CA: Don’t Take Our Word For it CA is Positioned as a Leader in the Gartner MQ for ARA … Forrester Wave SV Leader Faster time to deploy releases* 20x Reduced testing costs & improved efficiency* 25% Reduction in production errors* 98% Differentiation Customer Success 5 of the top 5 Telecom Companies 4 of the top 5 Retailers 4 of the top 5 Banks Market Validation Nominated in the “Best Use of Technology in a Project” category for work with the London Metal Exchange *Based on Global 500*Forrest TEI Research
  50. 50. 51 © 2017 CA. ALL RIGHTS RESERVED. CA CONFIDENTIAL AND PROPRIETARY INFORMATION; FOR INTERNAL USE ONLY. NO UNAUTHORIZED USE, COPYING OR DISTRIBUTION. Continuous Adaptive Delivery Ecosystem Write user stories (requirements) Right data for the right test environment Dev Test QA Prod Continuous integration (Build-Deploy- Test) Continuous Delivery Workflow Build / deployment / promotion dashboard and reports Adaptive, dynamic releases Automated & manual testing/test data Deployment workflow, rollback, pre/post tasks Automated test kick-off, tracking Testing coverage & progress reporting Trigger build & deploy in Dev Runboo k Deployment automation Release & artifact tracking, environment mgmt, baselining Application knowledge Link application to user stories To Do Doing Done ManageAuthorize Trigger release deployment Plan Releases Plan Release Trains Promote Define release workflow, gates & environments Watch progress (tracing, auditing, reporting) Create release package with build /deploy status Get app definition Release Manager Product Owner Business User Scrum Master Enterprise Architect Dev Developers QA Test/QA Deployment Manager Customer Implement components Define app: Components & versions Infra provisioning / environment mgmt / PaaS / containerization / configuration mgmt Agile Central Use app Simulate unavailable systems for full testing
  51. 51. 52 © 2017 CA. ALL RIGHTS RESERVED. © 2017 CA. All rights reserved. Thank you! #ExcuseFreeTesting

×