Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AI driven classification framework for advanced Test Automation

540 views

Published on

by Shubhradeep Nandi, Head of Digital, MSys Technologies at STeP-IN SUMMIT 2018 15th International Conference on Software Testing on August 31, 2018 at Taj, MG Road, Bengaluru

Published in: Technology
  • Be the first to comment

  • Be the first to like this

AI driven classification framework for advanced Test Automation

  1. 1. STeP-IN SUMMIT 2018 - Machine Learning AI driven test automation in the AI first world Shubhradeep Nandi
  2. 2. AI First World ❖ Technology projects predominantly are now Datascience Projects ➡ Guided by data ➡ once live requires no manual intervention 2 Images belong to their respective owners and this is just for illustration purposes only ❖ What needs to be done for adoption ❖ Exhaustive testing on the claims of the product(QA team) ❖ Explicability of the underlying model(Both DEV & QA team)
  3. 3. ❖ Machine Learning ❖ Deep Learning but Why ? ❖ Classification in the AI world 3 Refreshing a few basic concepts Images belong to their respective owners and this is just for illustration purposes only
  4. 4. ❖ Before - We were testing an Application. ❖ Now - We are still going to test an Application. 4 What did not change? Images belong to their respective owners and this is just for illustration purposes only What got changed? ❖ Before - Rule based AI(Automation with pure test/train) for testing a rule based workflow driven application. ❖ Now - Self Learning AI for testing a combination of ‣ Deterministic -The workflows that are rule driven in the app ‣ Non-Deterministic -The workflow that are driven by AI with learning elements The ‘Shift’ in Testing Paradigm
  5. 5. Deterministic vs Non- Deterministic 5 Images belong to their respective owners and this is just for illustration purposes only What????
  6. 6. Insurance driven by AI 6 Open up the App Upload your supporting docs Pay the Premium Customer Onboarding Auto Verification Deterministic rule based Non Deterministic driven by AI Images belong to their respective owners and this is just for illustration purposes only Register by filling up the form
  7. 7. Insurance driven by AI 7 Claim Settlement Upload image of damage Real time validation and estimation using AI Claims are settled Open up the App Deterministic rule based Non Deterministic driven by AI Images belong to their respective owners and this is just for illustration purposes only
  8. 8. The AI mindset ❖ What is AI? ❖ Is it all Math? ❖ Is it all Algorithm? ❖ Is it all Programming? ❖ Does it all depends on the tool/team that builds it? 8
  9. 9. Creating the holistic test framework in the AI first World ❖ Identify and Understand the scope of the Application ❖ Segregate the Deterministic Workflow from the Non Deterministic Workflow ❖ AI for testing the Deterministic workflow of the App ❖ AI for testing the Non Deterministic workflow of the App 9
  10. 10. 5 blocks to understand the scope of application What is the input for the application? 1. Define the possible inputs captured 2. Define the order of capture 10 Understand the application flow 1. The business requirement 2. The behavioural workflow Segregate the Non-Deterministic Portion 1. Learning models involved 2.Learning model behaviour matrix What is the hierarchy? 1. Identify the primary/secondary workflow 2. Identify the learning workflow Quantify the learning property 1. Data Properties 1. Size of Data 2. Noise Level 2. Degree of Automation - Automatic/Semi-Automatic 3. Supervision - Supervised/UnSupervised/Reinforced 4. Time(Online/offline or Lazy/Eager)
  11. 11. Segregating the Deterministic Workflow from non- Deterministic Workflow 11 Deterministic Non Deterministic Images belong to their respective owners and this is just for illustration purposes only
  12. 12. AI for testing the Deterministic workflow(non AI) of the App 12 1 2 3 1. Expose deterministic steps of ML 2. The application Blueprint is created 3. The Cognitive Script generation Automation with cognitive approach Images belong to their respective owners and this is just for illustration purposes only
  13. 13. AI for testing the Deterministic workflow(non AI) of the App ❖ Recorder and Meta Modelling to automate scripts generation ❖ Elastically scale functional, load and Performance Testing using deep forecast models ❖ Self Healing Tests with Deep Learning ❖ Analytical reports and Visualisation for explanations 13 Automation with Deep Learning approach Images belong to their respective owners and this is just for illustration purposes only
  14. 14. AI for testing the Non Deterministic workflow(AI) of the App 14 Images belong to their respective owners and this is just for illustration purposes only
  15. 15. Object Classification and Detection - The most sought after AI usecase ❖ There is an enormous rise of autonomous vehicles, smart video surveillance, facial detection and various entity identification applications 15 Images belong to their respective owners and this is just for illustration purposes only
  16. 16. 16 ❖ Some use cases are very critical - ❖ Outcome should be highly accurate ❖ Objects has to be detected, classified, and, delivered in fraction of a second Object Classification and Detection - The most sought after AI usecase Images belong to their respective owners and this is just for illustration purposes only
  17. 17. 17 ❖ Deep learning framework CNN(Convolutional Neural Network) has achieved the state of start in Object Classification and Detection Object Classification and Detection - The most sought after AI usecase Images belong to their respective owners and this is just for illustration purposes only
  18. 18. How to build a AI framework to test this Deep Learning usecase ? 18
  19. 19. Approach to build the framework 1.Checking the CNN robustness using Perturbations using Generative Neural Networks - Fellow et al. 2.Neural Network correctness with Linear Programming or SMT Solvers - Katz et al. 19 Images belong to their respective owners and this is just for illustration purposes only
  20. 20. Approach to build the framework 3. Systematic approach with Synthetically generated Datasets 20 Scalable Realistic instead of perturbations Images belong to their respective owners and this is just for illustration purposes only
  21. 21. Modules of the Framework 21 Generator Sampler Visualizer Generate realistic Images of objects Provide modification points to the Image generator Sampled Modifications against Metrics of Interest
  22. 22. Generator ❖ Modification functions are used to represent a subset of feature space. ❖ Low dimensional modification allow us to test Convolutional Neural Networks on a compact domain 22 Mathematical Relationship f : X —> Y X’ ⊆ X Generator (y : M —> X’) Every Modification(m) m ∈ M y(m) ∈ X’ X —> Feature Space Y—> Output M —> Modification Space y —> Modification Function m —> Individual Modifications Images belong to their respective owners and this is just for illustration purposes only
  23. 23. Sampler ❖ Identify a low discrepancy sequence methodology to competently produce sample sequences that provide high coverage of the abstract space. ➡ D(K,X) = abs(#(K)/m - vol(K)) ❖ Capitalise on the Active Learning capability to reduce process expense. ➡ Gaussian Progression for non parametric regression 23 D—> Dimension space K —> subset of Dimension Space, m —> modification as tupple vol —> Volume
  24. 24. Visualizer ❖ Intersection over Union and CNN Confidence Score - A standard approach to measure accuracy for Object Detection and Classification. 24 Images belong to their respective owners and this is just for illustration purposes only
  25. 25. Connecting the 3 Modules function ANALYZERCNN repeat p ← sampler(P)
 x ← generator(p) y ← f(x)
 D.add(m, x, y) until condition(D) visualizer(D) end function 25
  26. 26. Where can I apply this? 26 Images belong to their respective owners and this is just for illustration purposes only
  27. 27. Few of the Many… ❖ Healthcare - Radiology, Cardiology, Dermatology… ❖ Insurance - Claims, Customer on-boarding… ❖ Life Sciences - Drug Discovery, Pharmacovigilance… ❖ Manufacturing - Industrial Vision, Quality Inspection… 27
  28. 28. Thank You 28 Email :- shunandi@gmail.com or shubhradeepn@msystechnologies.com Images belong to their respective owners and this is just for illustration purposes only

×