Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
#ATAGTR2019
Assuring Quality for AI based applications
Vinod Sundararaju Antony, Senthilkumar
Thirumalaisamy, Santhosh Kum...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
#ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow...
Upcoming SlideShare
Loading in …5
×

#ATAGTR2019 Presentation "Assuring Quality for AI based applications" By Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy & Santhosh Kumar Vasudevan

50 views

Published on

Vinod Sundararaju Antony who is Director at Cognizant Technology Solutions along with Senthilkumar Thirumalaisamy who is a Manager Automation Architect at Cognizant Technology Solutions and Santhosh Kumar Vasudevan who is a Lead System Architect at Cognizant Technology Solutions took a Session on "Assuring Quality for AI based applications" at Global Testing Retreat #ATAGTR2019

Please refer our following post for session details:

https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-vinod-antony-sundaraju-as-our-esteemed-speaker/

https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-senthilkumar-thirumalaisamy-as-our-esteemed-speaker/

Published in: Technology
  • Be the first to comment

#ATAGTR2019 Presentation "Assuring Quality for AI based applications" By Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy & Santhosh Kumar Vasudevan

  1. 1. #ATAGTR2019 Assuring Quality for AI based applications Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy, Santhosh Kumar Vasudevan 14th 15th Dec 2019
  2. 2. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Abstract Key challenges include getting the right data sets for testing, predicting and test designing the expected outcome and behaviour, identifying apt algorithms AI IS NEARER TO MAINSTREAM Intelligent software and applications are omnipresent and are changing the way we engage Artificial Intelligence (AI) technology is drastically finding its way into conventional software development A Robust test strategy which increases the level of confidence in AI Apps is the need of the hour 2
  3. 3. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) AI Is Here To Stay Global AI Business Value to Reach $1.2 Trillion in 2018 and $3.9 trillion in 2022 AI has already picked up pace across various industry sectors BANKING & FINANCE HEALTHCARE MARKETING AUTOMOBILE Travel concierge Personalized platform for Airlines TRAVEL & CUSTOMER RELATION SMART HOME INFORMATION, MEDIA & ENTERTAINMENT VIRTUAL ASSISTANT RETAIL 3
  4. 4. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) AI Is Nearer To Mainstream Democratized AI – AI will become more widely available due to cloud computing, open source and the “maker” community (Deep Neural Nets is just 2-5 years from mainstream adoption) 4
  5. 5. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) But There Is Still A Long Way To Go Google Apologizes for Photo App’s Racist Blunder Tesla Car that crashed was running on Autopilot Microsoft silences its new AI bot Tay after Twitter users teach it racism 5
  6. 6. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) The spectrum of AI & the challenges in assuring Quality MACHINE LEARNING Deep Learning Predictive Analytics NATURAL LANGUAGE PROCESSING (NLP) Translation Classification & Clustering Information Extraction SPEECH Speech to Text Text to Speech Expert Systems Robotics Vision Image Recognition Machine Vision Artificial Intelligence • Non-deterministic and probabilistic - No defined input and output • Ever-changing Behaviour – AI systems are always learning • Non-Linear inputs – e.g. Voice, conversational/free-flowing text 6
  7. 7. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 7  Detecting credit card fraud  Determining whether a customer is likely to switch to a competitor  Deciding when to do preventive maintenance on a factory robot What is Machine Learning? Machine Learning focuses on data-driven predictions as opposed to following strictly static program instructions Uses the patterns to predict the futureFind patterns in data
  8. 8. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 8 Machine Learning – How does it work ? Machine Learning Algorithm Model Application Data Contains Patterns Finds Patterns Recognizes Patterns Supplies new data to see if it matches known patterns Why is it gaining traction ? Machine Learning requires Lots of data, Lots of compute power, Effective Machine Learning Algorithms. All of these are more available than ever with the evolution of Big Data, Cloud etc.
  9. 9. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 9 What are the testing challenges? ML applications are intended to learn properties of data sets, expected output is NOT already known to users Periodic learning leading to changing behaviour over period of time Prediction of all scenarios is a time consuming process Dependency on humungous amount of data?
  10. 10. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 10 Testing a Machine Learning Model Features make the most important part of a machine learning application or model. Testing features are key set of Test tasks which needs to be performed for ensuring the high performance of machine learning.
  11. 11. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 11 1. Testing Features of Machine Learning models Feature Thresholds Feature Relevance Feature Relationship Feature Suitability Feature Compliance Features Unit Testing Feature Static Review Test if the feature relationship with outcome variable in terms of correlation coefficients. Test whether value of features lies between the threshold values Age of Human (Y) - Threshold - 0 (X1) to 100 (X2). Test if Y lies between X1 and X2 (or) Y > X1 Test whether the feature importance changed with respect to previous test run. Test/review the static code analysis outcome of code generating features Test/review the code coverage of the code generating features Test/review if the generated feature violates the data compliance related issues Test the feature unsuitability by testing memory usage, inference latency and more.
  12. 12. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 12 2. Testing for metamorphic relations Test Case 1 X=X1 Machine Learning System Under Test Output 1 Y1 Test Case 2 X=G(X1) Output 2 Y2 Input Output Metamorphic Relations Input Relation Output Relation AN AI BASED APPLICATION TO PREDICT THE RISK OF DIABETES USE CASE Age BMI Predict Diabetes (Y/ N) 30 30 Y 40 32 N Indicates failure in Metamorphic relation • Validation of relations between the outputs of multiple inputs can help detecting defects in ML algorithm • Follow-up test cases and validation of the test results and could be fully automated.
  13. 13. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 13 2. Testing For Metamorphic Relations – Orientation & Color Testing ORIENTATION COLOR • RGB Channels of an image constitute pixel values for ‘red’, ‘green’ and ‘blue’ colors • Permutation of RGB input channels helps in identification of metamorphic relations and detect failures • Core property of CNN is NOT violated by RGB permutation Original, RGB BGR CNN SUT BOATTest case #1 Convolutional Neural Network Test case #1 CNN SUT BOAT Convolutional Neural Network Test Case #2 – MR1 Test Case #3 – MR2 90o 180o • Facebook uses CNN for automatic tagging algorithms, • Amazon  for generating product recommendations and • Google  for search through among users’ photos Test Case #2 – MR1 Test Case #3 – MR2
  14. 14. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 14 Use Case & Demo – Image Based Search in Social Media / Retail 34% 85% Consumers will spend more money online when AI is deployed effectively Customer interactions in retail will be managed by artificial intelligence, by 2020 Pinterest LensHow it should ideally work ? A Real life scenario
  15. 15. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 15 Use Case & Demo pickle Serialize the data Trained Model Train dataset Test Dataset for prediction Shoe Images 60K predict ML Application CNN - ResNet TESTING FOR MACHINE LEARNING BASED IMAGE CLASSIFIER FOR SHOE BRAND RECOGNITION USING CNN What are the metamorphic relations in this use case ? Train & Test whether the CNN algorithm is able to detect the image as a shoe and the brand • Test Case 1: Happy Path Testing – Subset of the images used for training the algorithm • Test Case 2: Metamorphic Relation 1 - Input image as a rotated image of the shoe • Test Case 3: Metamorphic Relation 2 - Input image with BGR (instead of RGB) • Test Case 4: Testing with the data outside of the trained data set
  16. 16. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 16 Use Case & Demo
  17. 17. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 17 2. Testing For Metamorphic Relations – Testing for Unstructured Data Continuous Information flow Big Data Software Actionable Intelligence Sentiment Analysis L1 - Supervised L2 - Semi L3 - Unsupervised Use Case: Twitter Sentiment Analysis ML software Vocabulary Synonyms Antonyms Negations Test case #1 – MR1: Classification using should result Test case #2 – MR2: Classification using should result Test case #3 – MR3: Classification using should result Test case #4 – MR4: Classification using should result Testing entails validation of continuous information flow
  18. 18. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 18 3. Adversarial Scenario Testing • Two-sample statistical hypothesis testing are conducted to determine both originate from the same distribution • Where there is high variation in distribution concludes influence of Adversial Inputs Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake Adversarial Example: Adversarial Input + ML Model = High accuracy (>80%) or 150% more than normal ML model predictions Statistical Hypothesis Testing Attackers could target autonomous vehicles by using stickers or paint to create an adversarial stop sign that the vehicle would interpret as a ‘yield’ or other sign USE CASE
  19. 19. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 19 4. Dual Coding Testing for Machine Learning Applications & Models DCT to validate Correctness of predictions by ML applications & models Dataset ML Model 1 ML Model 2 Trained with same or common set of features Compare predictions for correctness Prediction 1 using RF Prediction 2 using SVM Random Forest Support Vector Machine USE CASELEARNING MANAGEMENT SYSTEM - VDI MACHINE PREDICTION AI enabled LMS which has to predict & allocate VDI machines based on various factors including Course,# of Course Completed, # of pending course, Machine required. Test Data is passed through the 2 ML models namely Support Vector Machine & Random Forest. Random Forest consistently reported higher levels of accuracy and was chosen as the base model for implementation of LMS.
  20. 20. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 20 Best Practices for Testing ML Applications ML TESTING BEST PRACTICES Acceptance criteria, with amount of error stakeholders are willing to accept Determine the level of outcomes acceptable for each scenario Communicate the level of confidence in the results Test with new data post training Scenario classification - Expected best case, Average case, worst case Defects will be reflected in the inability of the model to achieve the goals
  21. 21. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 21 Goodbye GUI, Hello VUI • Voice First Device - Smart device designed to get tasks done conversationally • Always-ON and Always-listening - VPA enabled Wireless Speakers Analyst Corner (Gartner): • By 2021, early adopter brands that redesign their websites to support visual and voice search will increase digital commerce revenue by 30% • By 2019, half of major commerce companies and retailers with online stores will have redesigned their commerce sites to accommodate voice searches and voice navigation Voice Applications What are the Challenges in testing Voice applications? - Graphical User Interface (GUI) - Voice User Interface (VUI)  Prescriptive vs Descriptive  Limited data input vs indefinite # of test cases  Testing for languages vs Testing for languages & accents
  22. 22. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 22 Testing the Voice User Interface Alexa Skill Test FrameworkTools & Frameworks Service Simulator Echosim.io Skill metrics “Alexa, Open Domino’s?” “. . . ” User Alexa enabled Devices Amazon – Echo, Dot, Tap, Fire TV Triby, Pebble Watch Alexa Voice Service (AVS) Alexa Skills Kit (Custom) Services Request Voice – device sends to AVS Response Voice Request HTTPS/JSON – POST request to Skill End point Response HTTPS/JSON Speech-to-Text technology AUT Alexa Skills Kit - create commerce driven skill Use Case – Leading Pizza retail chain using AI Voice Application Avoid giving word clues, Avoid think- aloud techniques, Avoid accidental wake-ups Certifies application functionality as per design specifications in all possible pathway and prompts Holistic, experimental review of a speech application; Test fully developed application through pre-determined use- cases Alexa Skills Kit (ASK) - Collection of self-service APIs, tools & more. Enables designers, developers & brands to build engaging skills & reach customers through tens of millions of Alexa Enabled Devices (AED). Dialog Traversal Testing (DTT) VUI Review Testing (VRT) Usability Inspection Testing
  23. 23. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 23 Testing the Voice User Interface – Automated Voice Testing (AVT) Test Case Start Next End Text to Alexa Text-to-speech Speech-to-text Text-to-speech Text from Alexa Text to Alexa “Alexa, open Domino’s” “Welcome to Domino’s, what would you like to order?” “. . .” Speech Recognition Engine Echo  Codeless Automation tool - TestArchitect – Extend to C#, Java or Python  Based on Speech Recognition Engine (SRE)  Action-based, Interaction data table • Device Agnostic and platform agnostic • AVT can be applied to Alexa apps, Google Home apps, voice enabled web sites Features Benefits Sample Utterance • Variations of the sample utterances with different slot values and slightly different phrases • Slots for user errors: Test different permutations, such as missing or incorrect values • Provide multi accent intent, verify its capabilities to process the required ‘Intents’ & respond with the appropriate ‘Utterances’ Wake Word Alexa, Intent Slot To Plan my trip, For next Friday Test Scenarios Invocation name Ask Yatra,
  24. 24. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 24 Automated Voice Testing (AVT) Demo
  25. 25. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 25 Testing Strategy for Other Areas - Chatbots, AR/VR
  26. 26. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 26 NLP : Chatbot Overview & Challenges A computer program that simulates human conversation through voice commands or text chats or both. What are the Challenges in testing a Chatbot?  Unstructured Input - There are no barriers for users in terms of asking questions  Multitude of user interactions - 100% of Test coverage is not possible  Non-deterministic Behavior - behavior keeps changing based on learnings What is a Chatbot ?
  27. 27. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 27 Chatbot Testing • Conversational flow - Chatbots are based on conversations & very important to test conversational flow of a Chabot • Understanding - Process any kind of messages • Error Management – how does a chatbot react to errors • Conversational steps - minimize the number of steps in conversation • Bot speed - Speed at which the bot replies to the messages. • Bot Accuracy - Out of predicted utterances, # of utterances the bot gets correct will be accuracy of bot. Tools & Frameworks Functional Non Functional USE CASES Travel & Hospitality Bots can help customers plan and book trips, push personalized offers based on browsing history and preferences Content Distribution and News Push personalized news content, manage polls, unleash predictive content delivery based on behavior trends Health Care Patients leverage bots for appointments , receive personal updates based on treatment history etc.
  28. 28. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 28 AR/VR Applications and Testing Challenges • Augmented Reality - Augments virtual elements into the real world • Virtual Reality – Digital creation of virtual environment around the user • AR/VR has a market potential of $95B by 2025 AR / VR Applications What are the Challenges in testing AR/VR applications?  Wide varieties of device conditions to be dealt with and Multiple compatibility Issues  High performing system – Performance challenges to create virtual / augmented word  Need for consistently rich user experience  Test Automation of AR/VR systems Magazine ads instantly shoppable - consumers purchase straight off the page Project their selected item (through video) in their living room to check suitability Alibaba offers VR shopping experience (BUY+) through mobile & VR Headsets - users get the experience of shopping in an actual store
  29. 29. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 29 Testing Strategy for AR/VR Applications FUNCTIONAL TESTING USABILITY TESTING PERFORMANCE TESTING SYSTEM INTERACTION ASSURANCE • Validate Navigation, interactive buttons • Validate interaction with voice commands etc. AUGMENTATION ASSURANCE • Validate/Captures/ registers accurate dimensions of a 2D/3D object • Real time tracking NETWORK ASSURANCE • Test for varied network related issues (Wi- Fi/2G/3G/4G/5G ) GEO LOCATION – LOCAL AND GLOBAL • Ensure obtains local and global coordinates • Marker less apps integration with SLAM (simultaneous localization and mapping) COMPATIBILITY TESTING • For integrated and/or hosting devices and bandwidth EFFECTIVENESS • Match between system and the real world • Visibility of system status (real time) • User control and freedom EFFICIENCY • Consistency and standards • Error prevention • Accuracy • Recognition rather than recall (learnability) , Help users recognize, diagnose, and recover from errors • Environment configuration SATISFACTION • Aesthetic and minimalist design • Help and documentation • Customer experience with the system LATENCY TESTING • Test the quality of CGI and framerate for the visuals on stress conditions SCALABILITY TESTING • For cloud based AR/VR systems - dynamically scale the application resource to support increased demand PERFORMANCE UNDER DIFFERENT INPUTS • Assure performance for change in angle , orientation, distance between AR image and smart device/smart phone etc.
  30. 30. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 30 AR/VR Automation AR/VR automation can be possible by building tools with latest technologies Computer Vision Robotics Device technologies will be integrated to Automation tools automate AR/VR Systems that functions based motion detection, to simulate user interactions AI based Computer vision libraries are going to help Automation Tools/Bots understand the visuals / response from AR/VR systems to automate the interactions. Robotic Motion Simulator Natural Language processing will be another Key element for the automation tools/bot to handle; AR/VR systems that accepts voice commands or allow voice interactions Natural Language Processing AI based bots that can monitor and learn human interactions with systems captured from Functional / Crowd / Beta testing and replicate for regression automation. Following are the key technologies that will allow the Tool to achieve AR/VR automation
  31. 31. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) 31 In Summary 1 2 3 4 1 . AI I S N E AR I N G M AI N S T R E A M AD O P T I O N AI has already picked up pace across various industry sectors with the aim of deriving business value in terms of enhanced customer experience, new sources of revenue and cost reduction 2 . T E S T I N G O F AI AP P S I S A C H AL L E N G E Non-deterministic and probabilistic nature, ever-changing behavior based on learnings and non-linear inputs (e.g. voice, conversational text, images/videos etc.). 3 . R O B U S T S T R AT E G Y TO AS S U R E AI Machine Learning Models (Feature Testing, Metamorphic Testing, Adversarial Scenario testing, Dual Coded ML etc.), Voice User Interfaces Testing and Chatbots Testing 4 . M E AS U R E T H R O U G H L E V E L O F C O N F I D E N C E Achieve the defined acceptance criteria and confidence levels to make the application available for wide scale adoption

×