This document discusses the evolution of quality and the role of QA/test engineers in the context of artificial intelligence systems. It covers topics like the different types of AI systems, the challenges of testing non-deterministic intelligent systems, skills needed by QA professionals for AI testing, and how AI/ML can be used to improve testing processes and capabilities. The role of QA is shifting from traditional testing to helping build intelligent systems by focusing on data, features, metrics and gaining expertise in AI tools and methodologies.

1. Evolution of Quality and
Role of QA/Test Engineers
Roopesh Kohad

2. Introduction
● Name: Roopesh Kohad
● Designation: Sr. Project Manager
● Company: Synerzip Softech India Pvt. Ltd.
● Contact: +91-9970010578
● Email: roopesh.kohad@synerzip.com
● Topic: Evolution of Quality and Role of QA/Test Engineers
● Profile: Currently working at Synerzip Softech as Sr. Project Manager,
managing multiple accounts and part of Presales team from offshore.
Enthusiast on newer set of technologies - Cloud Computing, Big Data,
AI/Machine Learning, Blockchain.
● Interests: Likes to read books, travel and interact with people.

3. Past PSTC Themes
● 2015: Assure your Testing Agility
● 2016: Transformation of Testing Techniques
● 2017: Way beyond Digital Assurance
● 2018: Connecting Machine Learning, AI, IoT, Blockchain
and Big Data together
○ Neither “testing” not “assurance” figure in the theme!!

4. Agenda
● Artificial Intelligence overview
● AI / ML / DL / Conventional Systems
● Challenge of Testing Intelligent System
● Software Quality & Testability
● Claim about Testing Non-Functional Attributes
● Difference in approach of Testing Intelligent vs Conventional Systems
● Role of QA/Test Professional in testing Intelligent Systems
● Intelligent Testing

5. Artificial Intelligence is here

6. Artificial Intelligence Stages
Source: AI: What's Working, What's Not

7. Weak or Narrow AI

8. Kind of Intelligent Systems we are going to encounter
● Weak or Narrow AI i.e. in narrow field of application
● Examples
○ Recommendation Systems
○ Chatbots
○ Virtual Assistants
○ Robots
● Weak or Narrow AI is what is leading to most of the Automation !!

9. Artificial Intelligence Scope
Source: What's required for a machine to be intelligent

10. Artificial Intelligence / ML / DL
Source: Cousins of AI

11. ML vs DL
Source: ML vs DL

12. Conventional System
Source: Flowchart Tutorial

13. Conventional System
● Explicitly coded with the logic by humans or developers
● They are stateful and number of states of system could be enormous
● As long as State is frozen, same inputs would result in same output
● Functionality of a Conventional System is “Deterministic”
● Possible to have a “Test Oracles”!
● Conventional System clubbed with its maintainers could be called
“Intelligent”!!

14. Machine Learning Process
Source: What is Machine Learning?

15. Machine Learning System
● Learning Algorithm is trained and a model is developed which is
approximation of underlying problem space
● Humans are integral part of learning - providing data, learning algorithm and
measuring performance before it is put out
● As more and more data is provided, the system improves itself
● Same input at different stages of training could yield different output even if
State of system is frozen
● Not possible to have a Test Oracle!
● Functionality of a Machine Learning system is “Non-Deterministic”

16. Machine Learning Types
Source: What is machine learning?

17. Challenge of testing an Intelligent System
Non-Deterministic or approximation nature of
Intelligent Systems

18. Software Quality
● Software functional quality reflects how well it complies with or conforms to a
given design, based on functional requirements or specifications.
● Software structural quality refers to how it meets non-functional requirements
that support the delivery of the functional requirements, such as performance,
reliability.
Source: Wikipedia

19. Non-functional Quality Attributes
● Performance
● Security
● Usability
● Reliability
● Maintainability
● ….
● Effectiveness
● Accuracy
● ….
● List of Quality Attributes

20. Testability
Source: SEI CMU

21. Testing of Non-functional Attributes
● Application or System Under Test is actually a “System of Systems” of
Hardware/Software systems like Cloud, VMs, OSes, Networking, Internet.
● Testability of Systems reduce to the extent that Quality of non-functional
aspects become Non-deterministic
● Non-functional Quality is “Perceptible” or “Experiential”
● We can never be 100% sure of quality of Non-functional attributes
● Absence of “Test Oracles”
● Testing of Non-functional Attributes is Non-Deterministic !!

22. Intelligent System v/s Non-Functional Attributes
Testing of “Intelligent System” is EQUIVALENT to testing for
“Non-functional” attributes

23. Testing Deterministic v/s Non-Deterministic
Inherent difference in testing Deterministic vs
Nondeterministic is whether Testing is a Destructive or
Constructive process

24. Role of QA/Test Professional
QA / Test professional has to play “constructive” role in
building an “Intelligent” system

25. Steps to build Machine Learning System
Source: Building a Machine Learning Model from A-Z

26. Data Preparation
Steps:
● Query Data
● Clean Data
○ Deal with missing values
○ Remove outliers
● Format Data
More like an ETL step!!

27. Feature Engineering
“Process of transforming raw data into features that better represent the
underlying problem to the predictive models, resulting in improved model accuracy
on unseen data.”
Steps:
● Brainstorm features
● Create features
● Check how the features work with the model
● Start again from first until the features work perfectly

28. Data Modelling
Source: How to become a Data Scientist

29. Performance Measure - Metrics
Mathematical / Statistical way of measuring performance of ML Model
● Classification Accuracy
● Logarithmic Loss
● Confusion Matrix
● Area under Curve
● F1 Score
● Mean Absolute Error
● Mean Squared Error

30. Performance Measure - Other Approaches
● Testing by End User or Crowd testing
○ Test with real users
● Equivalence classes or ranges of output or tolerance
○ Assert (somewhat expected ~ actual)
● Ranking of output
○ Instead of Pass/Fail, rank outputs
● Comparison Test
○ Compare with a competing system

31. Skills for a ML QA/Test Engineer
● All qualities of QA/Test Engineer
● ...
● Academic or Learning mindset
● Understand Machine Learning methodology (except Data Science)
● Bit of Maths - Probability and Stats
● Business Domain expertise especially User Behavior and Factors in play
● Expertise with ML tools - Python and ML modules
● Expertise with Data - SQL, NoSQL or ETL

32. What can QA/Test Engineer do before testing AI?
Start working on non-functional requirements right away -
Performance, Security to gain the mindset.

33. Intelligent Testing
Source: The era of Intelligent Testing

34. AI/ML for Testing purposes
● Test Automation
○ Authoring Automated UI Testing
○ Self-Healing Automated Tests
○ API Testing
○ Visual Inspection of UI
● Test Suite optimization
○ Optimize test cases
● Defect Analytics

35. AI/ML Testing vendors
● Test Automation
○ Test.ai
○ mabl.com
○ testim.io
● Visual Inspection
○ Applitools.com
● API Testing
○ Parasoft Smart API Test Generator
● Service Offering
○ Infosys
○ Accenture

36. IoT, Big Data and Artificial Intelligence
Source: AI, Big Data, IoT