Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Cognitive Adaptive Testing
based on analytics-driven digital experiences
Jonathon Wright
Automation, DevOps & Digital As...
2
Abstract
Jonathon
Wright
DevOps, Automation &
Digital Assurance
3
The success of our technologies largely depend
on the fact that, while we were speculating about the
possibility of ultr...
4
Digital Testing, Quality & Assurance
DIGITAL TESTING, QUALITY & ASSURANCE
http://bit.ly/digital-assurancehttp://a.co/jlx...
5
DIGITAL TESTING, QUALITY & ASSURANCE
Digital Testing, Quality & Assurance
http://www.docjamesw.com https://youtu.be/e3Pz...
6
‘Shift Right’ – Performance Lifecycle
7
Ideas
Beliefs,
Values,
Culture
Heuristics
Behaviors
Value
Agile
Lean
Continuous Testing 101
Webcast Series – Value-Drive...
8
Shift Left - Research
Model Driven Delivery (MDD)
SHIFT
LEFT
https://www.ca.com/content/dam/ca/us/files/industry-analyst...
9
Digital Testing
Urban 4.0 - Predictive Crime feat. video analytics
SHIFT
RIGHT
https://www.slideshare.net/Jonathon_Wrigh...
10
Cognitive Adaptive Testing (CAT)
11
Digital Testing
TESTING IN THE DIGITAL WORLD
COGNITIVE ADAPTIVE ADOPTION (DESIGNOPS)
COGNITIVE ADAPTIVE DELIVERY (VALUE...
12 © 2016 CA. ALL RIGHTS RESERVED.12
Portfolio Management
Unify long term strategy,
investment and portfolio
planning.
Col...
13
DesignOps
Create and deliver Ideas to Outcomes
Pervasive Security
Redefine how work is
planned, managed and
executed
Ag...
14
DIGITAL ASSURANCE & TESTING
VALUE-DRIVEN DELIVERY
1
Cognitive
Adaptive
Delivery
2
Cognitive
Adaptive
Learning
4
Cogniti...
16
The ability to reliably release high-quality solutions at any time
COGNITIVE ADAPTIVE TESTING
Accelerate
Deliveries
Red...
17
> 50%
large organizations
globally will compete
using advanced analytics
& proprietary algorithms
40%
enterprises' net-...
18
SOI
Visualization
(Big) Data
Analytics
Driven
Inference
(Machine
Learning &
AI)
Intelligent
Process
Automation
Digital
...
19
Digital Testing
What is analytics-driven quality?
Model Progressive
Refinement
Analytics-Driven
ActionsPredictionsMeta ...
20
Portfolio backlog
prioritization and
value analysis
Improved
scheduling and
capacity
Validated
requirements in
backlog
...
21
TEST MAINTENANCE STATIC RULES BIG (un-structured) DATA
IOT, IOE
CX / DX SECURITY
STRUCTURED
AUTOMATION
Digital Testing
...
22
▪ Failure detection and prevention
▪ Scripted
▪ Limited process automation based on static rules
▪ Individual metrics b...
23
AUTOMATED & SELF-
HEALING
DYNAMIC/
ADAPTIVE RULES
BIG DATA ANALYTICS
IOT-Ops
DX-Ops SEC-Ops
INTELIGENT / ADAPTIVE /
COG...
24 © 2017 CA. ALL RIGHTS RESERVED.
Digital Testing – Cognitive Adaptive Insight
25
Deep reinforcement
learning
Digital Testing
Cognitive Adaptive Learning (CA-L)
Machine learning with
neural dictionarie...
26
Test Data Set
Test Algorithms Set
Test Meta Data
Source - IEEE Transactions on Pattern Analysis and Machine Intelligenc...
27
REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO
AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS
CO...
31
Digital Testing – Cognitive Adaptive Learning
32
Emergent embedded AI
for neural networks
Digital Technology
Cognitive Adaptive Intelligence (iCA)
Singularity
Ultra-Int...
33
Mode 0 - Descriptive
Cognitive Adaptive Testing (CAT):
1. “How long will it take to test these new features?” --- How l...
34
What were the past defect trends?
What were the root cause
of the past defects?
What defects are we likely to find?
Wha...
35
How do you pin point the
failures within the release
pipeline?
Model-Driven Design of Tests from Use Cases1
2
3
Execute...
36
Mode 2 - Predictive
Cognitive Adaptive Testing (CAT):
1. “Can you create me all the tests I need to fully test this sys...
51
Where next?
54
Cognitive Adaptive Technology (CA-T)
Next Generation of Digital Experiences (DX)
1. Digital Mobile, Chapter 2, Digital ...
55
Q & A
58
Blogs #ExcuseFreeTesting – Legacy is your Legacy!
https://dzone.com/articles/legacy-is-our-legacy-1
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1

Share

Download to read offline

Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing

Download to read offline

Jonathon Wright, Chief Technology Evangelist CA: Cognitive Adaptive Testing: Driving ‘Shift Right’ based on analytics-driven Digital Experiences (DX)
Continuous Adaptive Testing means harnessing the power of analytics and autonomics for continuous delivery. We all know digital technologies are rapidly transforming businesses. This is driven by trends towards omni-channel content delivery, utilization of big data and an improved customer experience. Digital business systems need to be extremely responsive to fickle customer sentiment, work across a variety of devices, be resilient in the face of unpredictable failure modes, and process huge amounts of unstructured data. Such scenarios put extreme pressure on IT systems and processes to be not only more responsive, but adaptive to meet digital business needs.
While automation solutions within current testing implementations help to address agility need, such automation is typically driven by static rules using conventional scripting and orchestration techniques. Such techniques incur high maintenance overhead to keep updated relative to changing circumstances. The recent emergence of predictive analytics and cognitive technologies (such as autonomics) have opened the possibility to drive adaptive automation within testing implementations. Such automation is able to self-heal and self-configure based on changing situations. In this session, Jonathon will present how analytics and autonomics technologies can be leveraged to power the next generation of continuous adaptive testing implementations, and how they can support the needs of digital businesses. He will share:
How you can automate the automation to eliminate error-prone tasks
Leverage the power of machine learning to self-heal automation through diagnosis analysis
Introduce big data analytics across the delivery pipeline to close feedback loops and provide actionable insight-driven analytics
Demonstrate how to enable these capabilities within you own organisation  
Jonathon is a strategic thought leader and distinguished technology evangelist. He specializes in emerging technologies, innovation and automation, and has more than 18 years of international commercial experience within global organizations. He is currently the Director of Product Management for CA R&D based in Oxford in the UK.
 Jonathon combines his practical experience and leadership with insights into real-world applications of the core principles and methodologies underpinning DevOps, Digital Assurance and Enterprise Digital (Smart Cities, IoT and AI). Thus, he is frequently in demand as a speaker at international conferences such as Gartner, Oracle, Unicom, EuroSTAR, STAREast and STARWest. Jonathon is the author of several award-winning books.

Related Books

Free with a 30 day trial from Scribd

See all

Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing

  1. 1. 1 Cognitive Adaptive Testing based on analytics-driven digital experiences Jonathon Wright Automation, DevOps & Digital Assurance jlwright@iee.org | linkedin.com/in/automation
  2. 2. 2 Abstract Jonathon Wright DevOps, Automation & Digital Assurance
  3. 3. 3 The success of our technologies largely depend on the fact that, while we were speculating about the possibility of ultra-intelligence, we increasingly enveloped the world in so many devices, sensors, application and data that it became an IT-friendly environment, where technologies could replace us without having any understanding, intentions, interpretations, emotional states, semantic skills, consciousness, self-awareness or flexible intelligence. Memory as in algorithms and immense datasets now outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015 “ ” THE NEED FOR SPEED 94%of executives face increased pressure to release apps more quickly “ Journey to the Digital Testing 3 Based on 2014 Vanson Bourne study commissioned by CA
  4. 4. 4 Digital Testing, Quality & Assurance DIGITAL TESTING, QUALITY & ASSURANCE http://bit.ly/digital-assurancehttp://a.co/jlxMHWp
  5. 5. 5 DIGITAL TESTING, QUALITY & ASSURANCE Digital Testing, Quality & Assurance http://www.docjamesw.com https://youtu.be/e3PzRnY-zTg
  6. 6. 6 ‘Shift Right’ – Performance Lifecycle
  7. 7. 7 Ideas Beliefs, Values, Culture Heuristics Behaviors Value Agile Lean Continuous Testing 101 Webcast Series – Value-Driven Delivery – Episode I Methods and Skills http://catechnologies.podbean.com Ray Arell
  8. 8. 8 Shift Left - Research Model Driven Delivery (MDD) SHIFT LEFT https://www.ca.com/content/dam/ca/us/files/industry-analyst-report/continuous-testing-as-a-digital-business-enabler.pdf Number of custom AUT
  9. 9. 9 Digital Testing Urban 4.0 - Predictive Crime feat. video analytics SHIFT RIGHT https://www.slideshare.net/Jonathon_Wright/oracle-digital-evolution-enterprise-of-things
  10. 10. 10 Cognitive Adaptive Testing (CAT)
  11. 11. 11 Digital Testing TESTING IN THE DIGITAL WORLD COGNITIVE ADAPTIVE ADOPTION (DESIGNOPS) COGNITIVE ADAPTIVE DELIVERY (VALUE-DRIVEN) COGNITIVE ADAPTIVE LEARNING (ANALYTICS-DRIVEN) COGNITIVE ADAPTIVE TECHNOLOGY (INSIGHT-DRIVEN) WHERE NEXT? 1 2 3 4 5 6
  12. 12. 12 © 2016 CA. ALL RIGHTS RESERVED.12 Portfolio Management Unify long term strategy, investment and portfolio planning. Collaboratively plan, prioritize and track work across the enterprise. Project Management Develop & Test Synchronize efforts to dramatically speed app development & increase quality Manage & Monitor Make a great customer experience a competitive advantage Release & Deploy Control the release process, to continuously advance application quality, improve the customer experience and reduce costs. Continuous Delivery Minimum Viable Experience Minimum Viable Quality Value-Driven Delivery BUSINESS INITIATIVE Complexity- Informed Organizational Change BUSINESS VALUE Minimum Viable Product CUSTOMER THINK MAKE Digital Testing Complexity-Informed Organizational Change
  13. 13. 13 DesignOps Create and deliver Ideas to Outcomes Pervasive Security Redefine how work is planned, managed and executed Agile Management Provide a flawless app experience optimized for performance Agile Operations Accelerate and streamline development, testing and release Continuous Delivery Continuous Development Continuous Adaptive Testing Continuous Release Digital Testing Value-Driven Delivery (Shift-Left & Shift-Right)
  14. 14. 14 DIGITAL ASSURANCE & TESTING VALUE-DRIVEN DELIVERY 1 Cognitive Adaptive Delivery 2 Cognitive Adaptive Learning 4 Cognitive Adaptive Technology 3 Cognitive Adaptive Testing Digital Testing Cognitive Adaptive Adoption (CA-A)
  15. 15. 16 The ability to reliably release high-quality solutions at any time COGNITIVE ADAPTIVE TESTING Accelerate Deliveries Reduce Errors Manage Complexity Increase Visibility Drive Collaboration Continually Improve Digital Testing Continuous Adaptive Testing (CAT)
  16. 16. 17 > 50% large organizations globally will compete using advanced analytics & proprietary algorithms 40% enterprises' net-new analytics investment will go to predictive / prescriptive 75% technology-oriented business intelligence competency centers will evolve to focus on information value generation Source: Gartner Digital Testing Intelligent Analytics is the New Battleground
  17. 17. 18 SOI Visualization (Big) Data Analytics Driven Inference (Machine Learning & AI) Intelligent Process Automation Digital Technology Integrations Digital Testing Systems of Intelligence 18 Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  18. 18. 19 Digital Testing What is analytics-driven quality? Model Progressive Refinement Analytics-Driven ActionsPredictionsMeta Data Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  19. 19. 20 Portfolio backlog prioritization and value analysis Improved scheduling and capacity Validated requirements in backlog Improved estimates Predict failures based on code commits Dynamic code promotion criteria Predict defects/failures Optimized Tests Scenario identification based on trends DX Scores Anomaly detection PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase Examples of Cognitive Adaptive Intelligence Examples of Outcomes Investment decision analytics Code quality analytics Test Optimization & Defect prediction A/B Testing & Canary Log & DX Analytics DX-based requirements Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas Digital Testing Cognitive Adaptive Intelligence (CAi) Systems of Intelligence
  20. 20. 21 TEST MAINTENANCE STATIC RULES BIG (un-structured) DATA IOT, IOE CX / DX SECURITY STRUCTURED AUTOMATION Digital Testing Key Challenges Posed by Adaptive Digitalization Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  21. 21. 22 ▪ Failure detection and prevention ▪ Scripted ▪ Limited process automation based on static rules ▪ Individual metrics based on structured data in lifecycle silo ▪ Limited insight and perspectives ▪ Reactive: deals with “Past” ▪ Failure prediction ▪ Automate the automation ▪ Intelligent process automation based on dynamic inference ▪ Analytics based on large scale structured and unstructured data across the lifecycle ▪ Predictive: from “What did happen?” to “What would happen? TRADITIONAL TESTING COGNITIVE TESTING Digital Testing Intelligent Analytics - Characteristics Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  22. 22. 23 AUTOMATED & SELF- HEALING DYNAMIC/ ADAPTIVE RULES BIG DATA ANALYTICS IOT-Ops DX-Ops SEC-Ops INTELIGENT / ADAPTIVE / COGNITIVE + + + + + + + Digital Testing Challenges addressed by Cognitive Adaptive Delivery (CA-D) Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  23. 23. 24 © 2017 CA. ALL RIGHTS RESERVED. Digital Testing – Cognitive Adaptive Insight
  24. 24. 25 Deep reinforcement learning Digital Testing Cognitive Adaptive Learning (CA-L) Machine learning with neural dictionaries https://www.luis.ai “We are not at the centre of the universe (Copernicus), of the biological kingdom (Darwin) or of the realm of rationality (Freud). After Turing, we are no longer the centre of the infosphere, we share the infosphere with digital technologies”. The ethics of artificial intelligence, Luciano Floridi
  25. 25. 26 Test Data Set Test Algorithms Set Test Meta Data Source - IEEE Transactions on Pattern Analysis and Machine Intelligence, 21st September 2016, http://arxiv.org/abs/1609.06647 Source - Google AI APIs - https://cloud.google.com/prediction/docs/apis Digital Testing Pattern analysis and machine intelligence
  26. 26. 27 REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS CONTINUOUS DELIVERY MAINFRAME SECURITY 27 API analyticsRelease management dashboard Access incident responseReal-time app performance Digital Testing Continuous Adaptive Delivery (CA-D) Cognitive Adaptive Insight Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  27. 27. 31 Digital Testing – Cognitive Adaptive Learning
  28. 28. 32 Emergent embedded AI for neural networks Digital Technology Cognitive Adaptive Intelligence (iCA) Singularity Ultra-Intelligence (2030) “Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s
  29. 29. 33 Mode 0 - Descriptive Cognitive Adaptive Testing (CAT): 1. “How long will it take to test these new features?” --- How long historically (test cases / execution time) 2. “What is the associated risk rating of the new platform?” --- Failure rate (cyclomatic complexity > 13) 3. “How much technical debt do we have?” 4. “What is the defect density and prediction for this solution?” 5. “What is the current conference rating of the solution?” 6. “What is the value of this capability?” How will Intelligent Analytics help? Digital Testing Example use cases for Intelligent Analytics
  30. 30. 34 What were the past defect trends? What were the root cause of the past defects? What defects are we likely to find? What preventive measures should we take? Have we overlooked anything for defect prevention? Predictive analytics Descriptive Analytics Diagnostic Analytics Prescriptive Analytics Deep Learning Diagnosis Analytics Engine Examples of Analytics use cases Cognitive Adaptive Intelligence (CAi) Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas
  31. 31. 35 How do you pin point the failures within the release pipeline? Model-Driven Design of Tests from Use Cases1 2 3 Execute ‘Automation as Code’ from the Model Pinpoint Failure and Diagnose Issue from Results “Easy, soon as you check in the new features ready for test, the automation will run then the results are analysed automatically and if we have any failures then it identified them and raise them as bugs! Cognitive Adaptive Learning Test in DevOps (TiD)
  32. 32. 36 Mode 2 - Predictive Cognitive Adaptive Testing (CAT): 1. “Can you create me all the tests I need to fully test this system?” 2. “How do I successfully test this new feature?” 3. “How much testing has been done already?” 4. “Have you detected any anomalies within the system?” 5. “How much testing will be enough?” How will Intelligent Analytics help? Digital Testing Example use cases for Intelligent Analytics
  33. 33. 51 Where next?
  34. 34. 54 Cognitive Adaptive Technology (CA-T) Next Generation of Digital Experiences (DX) 1. Digital Mobile, Chapter 2, Digital Experiences (DX), April 2017, Jonathon Wright
  35. 35. 55 Q & A
  36. 36. 58 Blogs #ExcuseFreeTesting – Legacy is your Legacy! https://dzone.com/articles/legacy-is-our-legacy-1
  • Kelvin-Liew

    Mar. 29, 2018

Jonathon Wright, Chief Technology Evangelist CA: Cognitive Adaptive Testing: Driving ‘Shift Right’ based on analytics-driven Digital Experiences (DX) Continuous Adaptive Testing means harnessing the power of analytics and autonomics for continuous delivery. We all know digital technologies are rapidly transforming businesses. This is driven by trends towards omni-channel content delivery, utilization of big data and an improved customer experience. Digital business systems need to be extremely responsive to fickle customer sentiment, work across a variety of devices, be resilient in the face of unpredictable failure modes, and process huge amounts of unstructured data. Such scenarios put extreme pressure on IT systems and processes to be not only more responsive, but adaptive to meet digital business needs. While automation solutions within current testing implementations help to address agility need, such automation is typically driven by static rules using conventional scripting and orchestration techniques. Such techniques incur high maintenance overhead to keep updated relative to changing circumstances. The recent emergence of predictive analytics and cognitive technologies (such as autonomics) have opened the possibility to drive adaptive automation within testing implementations. Such automation is able to self-heal and self-configure based on changing situations. In this session, Jonathon will present how analytics and autonomics technologies can be leveraged to power the next generation of continuous adaptive testing implementations, and how they can support the needs of digital businesses. He will share: How you can automate the automation to eliminate error-prone tasks Leverage the power of machine learning to self-heal automation through diagnosis analysis Introduce big data analytics across the delivery pipeline to close feedback loops and provide actionable insight-driven analytics Demonstrate how to enable these capabilities within you own organisation   Jonathon is a strategic thought leader and distinguished technology evangelist. He specializes in emerging technologies, innovation and automation, and has more than 18 years of international commercial experience within global organizations. He is currently the Director of Product Management for CA R&D based in Oxford in the UK.  Jonathon combines his practical experience and leadership with insights into real-world applications of the core principles and methodologies underpinning DevOps, Digital Assurance and Enterprise Digital (Smart Cities, IoT and AI). Thus, he is frequently in demand as a speaker at international conferences such as Gartner, Oracle, Unicom, EuroSTAR, STAREast and STARWest. Jonathon is the author of several award-winning books.

Views

Total views

1,055

On Slideshare

0

From embeds

0

Number of embeds

135

Actions

Downloads

27

Shares

0

Comments

0

Likes

1

×