Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing

935 views

Published on

Jonathon Wright, Chief Technology Evangelist CA: Cognitive Adaptive Testing: Driving ‘Shift Right’ based on analytics-driven Digital Experiences (DX)
Continuous Adaptive Testing means harnessing the power of analytics and autonomics for continuous delivery. We all know digital technologies are rapidly transforming businesses. This is driven by trends towards omni-channel content delivery, utilization of big data and an improved customer experience. Digital business systems need to be extremely responsive to fickle customer sentiment, work across a variety of devices, be resilient in the face of unpredictable failure modes, and process huge amounts of unstructured data. Such scenarios put extreme pressure on IT systems and processes to be not only more responsive, but adaptive to meet digital business needs.
While automation solutions within current testing implementations help to address agility need, such automation is typically driven by static rules using conventional scripting and orchestration techniques. Such techniques incur high maintenance overhead to keep updated relative to changing circumstances. The recent emergence of predictive analytics and cognitive technologies (such as autonomics) have opened the possibility to drive adaptive automation within testing implementations. Such automation is able to self-heal and self-configure based on changing situations. In this session, Jonathon will present how analytics and autonomics technologies can be leveraged to power the next generation of continuous adaptive testing implementations, and how they can support the needs of digital businesses. He will share:
How you can automate the automation to eliminate error-prone tasks
Leverage the power of machine learning to self-heal automation through diagnosis analysis
Introduce big data analytics across the delivery pipeline to close feedback loops and provide actionable insight-driven analytics
Demonstrate how to enable these capabilities within you own organisation  
Jonathon is a strategic thought leader and distinguished technology evangelist. He specializes in emerging technologies, innovation and automation, and has more than 18 years of international commercial experience within global organizations. He is currently the Director of Product Management for CA R&D based in Oxford in the UK.
 Jonathon combines his practical experience and leadership with insights into real-world applications of the core principles and methodologies underpinning DevOps, Digital Assurance and Enterprise Digital (Smart Cities, IoT and AI). Thus, he is frequently in demand as a speaker at international conferences such as Gartner, Oracle, Unicom, EuroSTAR, STAREast and STARWest. Jonathon is the author of several award-winning books.

Published in: Technology
  • If you’re struggling with your assignments like me, check out ⇒ www.WritePaper.info ⇐. My friend sent me a link to to tis site. This awesome company. After I was continuously complaining to my family and friends about the ordeals of student life. They wrote my entire research paper for me, and it turned out brilliantly. I highly recommend this service to anyone in my shoes. ⇒ www.WritePaper.info ⇐.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1lite.top/fNdi5 ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing

  1. 1. 1 Cognitive Adaptive Testing based on analytics-driven digital experiences Jonathon Wright Automation, DevOps & Digital Assurance jlwright@iee.org | linkedin.com/in/automation
  2. 2. 2 Abstract Jonathon Wright DevOps, Automation & Digital Assurance
  3. 3. 3 The success of our technologies largely depend on the fact that, while we were speculating about the possibility of ultra-intelligence, we increasingly enveloped the world in so many devices, sensors, application and data that it became an IT-friendly environment, where technologies could replace us without having any understanding, intentions, interpretations, emotional states, semantic skills, consciousness, self-awareness or flexible intelligence. Memory as in algorithms and immense datasets now outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015 “ ” THE NEED FOR SPEED 94%of executives face increased pressure to release apps more quickly “ Journey to the Digital Testing 3 Based on 2014 Vanson Bourne study commissioned by CA
  4. 4. 4 Digital Testing, Quality & Assurance DIGITAL TESTING, QUALITY & ASSURANCE http://bit.ly/digital-assurancehttp://a.co/jlxMHWp
  5. 5. 5 DIGITAL TESTING, QUALITY & ASSURANCE Digital Testing, Quality & Assurance http://www.docjamesw.com https://youtu.be/e3PzRnY-zTg
  6. 6. 6 ‘Shift Right’ – Performance Lifecycle
  7. 7. 7 Ideas Beliefs, Values, Culture Heuristics Behaviors Value Agile Lean Continuous Testing 101 Webcast Series – Value-Driven Delivery – Episode I Methods and Skills http://catechnologies.podbean.com Ray Arell
  8. 8. 8 Shift Left - Research Model Driven Delivery (MDD) SHIFT LEFT https://www.ca.com/content/dam/ca/us/files/industry-analyst-report/continuous-testing-as-a-digital-business-enabler.pdf Number of custom AUT
  9. 9. 9 Digital Testing Urban 4.0 - Predictive Crime feat. video analytics SHIFT RIGHT https://www.slideshare.net/Jonathon_Wright/oracle-digital-evolution-enterprise-of-things
  10. 10. 10 Cognitive Adaptive Testing (CAT)
  11. 11. 11 Digital Testing TESTING IN THE DIGITAL WORLD COGNITIVE ADAPTIVE ADOPTION (DESIGNOPS) COGNITIVE ADAPTIVE DELIVERY (VALUE-DRIVEN) COGNITIVE ADAPTIVE LEARNING (ANALYTICS-DRIVEN) COGNITIVE ADAPTIVE TECHNOLOGY (INSIGHT-DRIVEN) WHERE NEXT? 1 2 3 4 5 6
  12. 12. 12 © 2016 CA. ALL RIGHTS RESERVED.12 Portfolio Management Unify long term strategy, investment and portfolio planning. Collaboratively plan, prioritize and track work across the enterprise. Project Management Develop & Test Synchronize efforts to dramatically speed app development & increase quality Manage & Monitor Make a great customer experience a competitive advantage Release & Deploy Control the release process, to continuously advance application quality, improve the customer experience and reduce costs. Continuous Delivery Minimum Viable Experience Minimum Viable Quality Value-Driven Delivery BUSINESS INITIATIVE Complexity- Informed Organizational Change BUSINESS VALUE Minimum Viable Product CUSTOMER THINK MAKE Digital Testing Complexity-Informed Organizational Change
  13. 13. 13 DesignOps Create and deliver Ideas to Outcomes Pervasive Security Redefine how work is planned, managed and executed Agile Management Provide a flawless app experience optimized for performance Agile Operations Accelerate and streamline development, testing and release Continuous Delivery Continuous Development Continuous Adaptive Testing Continuous Release Digital Testing Value-Driven Delivery (Shift-Left & Shift-Right)
  14. 14. 14 DIGITAL ASSURANCE & TESTING VALUE-DRIVEN DELIVERY 1 Cognitive Adaptive Delivery 2 Cognitive Adaptive Learning 4 Cognitive Adaptive Technology 3 Cognitive Adaptive Testing Digital Testing Cognitive Adaptive Adoption (CA-A)
  15. 15. 16 The ability to reliably release high-quality solutions at any time COGNITIVE ADAPTIVE TESTING Accelerate Deliveries Reduce Errors Manage Complexity Increase Visibility Drive Collaboration Continually Improve Digital Testing Continuous Adaptive Testing (CAT)
  16. 16. 17 > 50% large organizations globally will compete using advanced analytics & proprietary algorithms 40% enterprises' net-new analytics investment will go to predictive / prescriptive 75% technology-oriented business intelligence competency centers will evolve to focus on information value generation Source: Gartner Digital Testing Intelligent Analytics is the New Battleground
  17. 17. 18 SOI Visualization (Big) Data Analytics Driven Inference (Machine Learning & AI) Intelligent Process Automation Digital Technology Integrations Digital Testing Systems of Intelligence 18 Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  18. 18. 19 Digital Testing What is analytics-driven quality? Model Progressive Refinement Analytics-Driven ActionsPredictionsMeta Data Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  19. 19. 20 Portfolio backlog prioritization and value analysis Improved scheduling and capacity Validated requirements in backlog Improved estimates Predict failures based on code commits Dynamic code promotion criteria Predict defects/failures Optimized Tests Scenario identification based on trends DX Scores Anomaly detection PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase Examples of Cognitive Adaptive Intelligence Examples of Outcomes Investment decision analytics Code quality analytics Test Optimization & Defect prediction A/B Testing & Canary Log & DX Analytics DX-based requirements Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas Digital Testing Cognitive Adaptive Intelligence (CAi) Systems of Intelligence
  20. 20. 21 TEST MAINTENANCE STATIC RULES BIG (un-structured) DATA IOT, IOE CX / DX SECURITY STRUCTURED AUTOMATION Digital Testing Key Challenges Posed by Adaptive Digitalization Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  21. 21. 22 ▪ Failure detection and prevention ▪ Scripted ▪ Limited process automation based on static rules ▪ Individual metrics based on structured data in lifecycle silo ▪ Limited insight and perspectives ▪ Reactive: deals with “Past” ▪ Failure prediction ▪ Automate the automation ▪ Intelligent process automation based on dynamic inference ▪ Analytics based on large scale structured and unstructured data across the lifecycle ▪ Predictive: from “What did happen?” to “What would happen? TRADITIONAL TESTING COGNITIVE TESTING Digital Testing Intelligent Analytics - Characteristics Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  22. 22. 23 AUTOMATED & SELF- HEALING DYNAMIC/ ADAPTIVE RULES BIG DATA ANALYTICS IOT-Ops DX-Ops SEC-Ops INTELIGENT / ADAPTIVE / COGNITIVE + + + + + + + Digital Testing Challenges addressed by Cognitive Adaptive Delivery (CA-D) Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  23. 23. 24 © 2017 CA. ALL RIGHTS RESERVED. Digital Testing – Cognitive Adaptive Insight
  24. 24. 25 Deep reinforcement learning Digital Testing Cognitive Adaptive Learning (CA-L) Machine learning with neural dictionaries https://www.luis.ai “We are not at the centre of the universe (Copernicus), of the biological kingdom (Darwin) or of the realm of rationality (Freud). After Turing, we are no longer the centre of the infosphere, we share the infosphere with digital technologies”. The ethics of artificial intelligence, Luciano Floridi
  25. 25. 26 Test Data Set Test Algorithms Set Test Meta Data Source - IEEE Transactions on Pattern Analysis and Machine Intelligence, 21st September 2016, http://arxiv.org/abs/1609.06647 Source - Google AI APIs - https://cloud.google.com/prediction/docs/apis Digital Testing Pattern analysis and machine intelligence
  26. 26. 27 REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS CONTINUOUS DELIVERY MAINFRAME SECURITY 27 API analyticsRelease management dashboard Access incident responseReal-time app performance Digital Testing Continuous Adaptive Delivery (CA-D) Cognitive Adaptive Insight Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  27. 27. 31 Digital Testing – Cognitive Adaptive Learning
  28. 28. 32 Emergent embedded AI for neural networks Digital Technology Cognitive Adaptive Intelligence (iCA) Singularity Ultra-Intelligence (2030) “Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s
  29. 29. 33 Mode 0 - Descriptive Cognitive Adaptive Testing (CAT): 1. “How long will it take to test these new features?” --- How long historically (test cases / execution time) 2. “What is the associated risk rating of the new platform?” --- Failure rate (cyclomatic complexity > 13) 3. “How much technical debt do we have?” 4. “What is the defect density and prediction for this solution?” 5. “What is the current conference rating of the solution?” 6. “What is the value of this capability?” How will Intelligent Analytics help? Digital Testing Example use cases for Intelligent Analytics
  30. 30. 34 What were the past defect trends? What were the root cause of the past defects? What defects are we likely to find? What preventive measures should we take? Have we overlooked anything for defect prevention? Predictive analytics Descriptive Analytics Diagnostic Analytics Prescriptive Analytics Deep Learning Diagnosis Analytics Engine Examples of Analytics use cases Cognitive Adaptive Intelligence (CAi) Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  31. 31. 35 How do you pin point the failures within the release pipeline? Model-Driven Design of Tests from Use Cases1 2 3 Execute ‘Automation as Code’ from the Model Pinpoint Failure and Diagnose Issue from Results “Easy, soon as you check in the new features ready for test, the automation will run then the results are analysed automatically and if we have any failures then it identified them and raise them as bugs! Cognitive Adaptive Learning Test in DevOps (TiD)
  32. 32. 36 Mode 2 - Predictive Cognitive Adaptive Testing (CAT): 1. “Can you create me all the tests I need to fully test this system?” 2. “How do I successfully test this new feature?” 3. “How much testing has been done already?” 4. “Have you detected any anomalies within the system?” 5. “How much testing will be enough?” How will Intelligent Analytics help? Digital Testing Example use cases for Intelligent Analytics
  33. 33. 51 Where next?
  34. 34. 54 Cognitive Adaptive Technology (CA-T) Next Generation of Digital Experiences (DX) 1. Digital Mobile, Chapter 2, Digital Experiences (DX), April 2017, Jonathon Wright
  35. 35. 55 Q & A
  36. 36. 58 Blogs #ExcuseFreeTesting – Legacy is your Legacy! https://dzone.com/articles/legacy-is-our-legacy-1

×