An introduction to AI in Test Engineering

An Introduction to AI in Test
Engineering
foohm71@gmail.com https://www.linkedin.com/in/heemeng/
Heemeng Foo

© 2018 Heemeng Foo
What this talk is about
1. The case of AI/ML in Testing / DevOps
2. Broad Overview of AI and Machine Learning
3. Survey of Testing / DevOps tools utilizing AI / ML
4. Why AI is the future of Testing?

© 2018 Heemeng Foo
The Dream of Agile and Lean

© 2018 Heemeng Foo
Typical Agile / Lean Development Process
1. Product dreams of product
2. Product defines the MVP
3. Dev team builds the product
4. Product is dog-fed internally
5. Product is sent to beta testers for feedback
6. Product is shipped to production (+ telemetry / crash
management)
7. Product gathers feedback, telemetry data to refine
8. Product designs A/B test to test hypothesis on refinements
9. Go To 3

© 2018 Heemeng Foo
Continuous Testing
• Idea is to test early and keep testing
• Test at each commit
• Automated frequent integration tests
• Nightly regression
• 24/7 test cycle
• If a bug is found, fix it fast, don’t let it fester
• Everything checked in must include tests
• Requires Test Automation

© 2018 Heemeng Foo
How Successful is Test Automation?
• Unit Tests
• gets updated regularly but depends on developer discipline
• Integration level tests
• depends on engineering resources
• API Level tests
• depends on test resources
• UI / Client tests
• Most of the time behind 0.5 or 1 sprint
• Flaky, false positives
• Frustrating for SDETs for rapidly changing UX designs
• Not enough time for test automation – resort to manual testing
• 52% of teams who have adopted DevOps say large portion of testing still done manually
slowing them down (SD Times survey) – http://bit.ly/2zDD2IW
• 40% teams ship code without testing – TestPlant developer survey 2017

© 2018 Heemeng Foo
What Makes Mobile Testing Hard
• Simulators/Emulators cover only functional aspects and
idealized hardware
• What’s missing are h/w related issues eg. screen size,
sensors, SoC, GPU and how they work together
• “Bloatware” introduced by OEMs
• Devices with low end hardware eg. iPod Touch, Moto E reveal
issues you don’t see on flagship devices
• Users are impatient ie. Cold Start
• Battery Consumption is becoming an issue

© 2018 Heemeng Foo
Can AI Help?
• Broad Overview of AI
• Survey of AI Testing Tools/Services

© 2018 Heemeng Foo
AI: the 10,000 ft view

© 2018 Heemeng Foo
AI, Machine Learning, Deep Learning
• From http://bit.ly/AIMLDeepLearning
• AI (Artificial intelligence) is a subfield of computer science, that was created in the 1960s,
and it was (is) concerned with solving tasks that are easy for humans, but hard for
computers. In particular, a so-called Strong AI would be a system that can do anything a
human can (perhaps without purely physical things). This is fairly generic, and includes
all kinds of tasks, such as planning, moving around in the world, recognizing objects and
sounds, speaking, translating, performing social or business transactions, creative work
(making art or poetry), etc.
• Machine learning is concerned with one aspect of this: given some AI problem that can
be described in discrete terms (e.g. out of a particular set of actions, which one is the
right one), and given a lot of information about the world, figure out what is the
“correct” action, without having the programmer program it in. Typically some
outside process is needed to judge whether the action was correct or not. In
mathematical terms, it’s a function: you feed in some input, and you want it to to produce
the right output, so the whole problem is simply to build a model of this mathematical
function in some automatic way. To draw a distinction with AI, if I can write a very clever
program that has human-like behavior, it can be AI, but unless its parameters are
automatically learned from data, it’s not machine learning.

© 2018 Heemeng Foo
AI, Machine Learning, Deep Learning
• Deep learning is one kind of machine learning that’s very popular now. It involves
a particular kind of mathematical model that can be thought of as a
composition of simple blocks (function composition) of a certain type, and where
some of these blocks can be adjusted to better predict the final outcome.
• Data science is much more than machine learning though. Data, in data science,
may or may not come from a machine or mechanical process (survey data could
be manually collected, clinical trials involve a specific type of small data) and it
might have nothing to do with learning as I have just discussed. But the main
difference is the fact that data science covers the whole spectrum of data
processing, not just the algorithmic or statistical aspects. In particular, data
science also covers cleaning data, data exploration, data engineering, data
visualization etc.
• (7 Myths of AI: http://bit.ly/7AIMyths)

© 2018 Heemeng Foo
Broad Areas in AI
• Traditional Techniques
• Machine (or statistical) Learning
• Computer Vision
• Natural Language Processing/Understanding (NLP/NLU)
• Robotics
• Intelligent Agents (or Bots)
• Deep Learning

© 2018 Heemeng Foo
Traditional AI (ie. Old)
• Artificial Neural Networks (ANN)
• Mimic how the brain “neurons” work
• Rule Based / Expert Systems
• Reproduce what “intelligence looks like” on the surface
• If-then rules, Knowledge base, Inference Engine
• Intelligent logic tree traversal (Prolog, BFS, DFS, pruning), LISP
engine
• Fuzzy Logic
• Heuristic Techniques
• Hill climbing
• Simulated annealing
• Genetic Algorithms

© 2018 Heemeng Foo
Rule Based Systems (Expert Systems)
• Source:
http://www.cse.unsw.edu.au/~billw/cs9414/notes/kr/rules/rules.html
• See https://youtu.be/kmRLeI1P9zw

© 2018 Heemeng Foo
Intelligent Tree Traversal
• Source:
https://www.researchgate.net/figure/262672371_fig1_Fig-1-
Game-tree-for-Tic-Tac-Toe-game-using-MiniMax-algorithm

© 2018 Heemeng Foo
Hill Climbing (Heuristic Optimization)
• Source:
http://computing.dcu.ie/~humphrys/Notes/GA/ex.ga.html
• See: https://youtu.be/oSdPmxRCWws

© 2018 Heemeng Foo
Artificial Neural Networks – The Perceptron
• Source: Wikipedia
• https://www.youtube.com/watch?v=1XkjVl-j8MM
• http://ataspinar.com/2016/12/22/the-perceptron/ (Perceptron in
Python code)

© 2018 Heemeng Foo
Artificial Neural Networks - ”Learning”
• Source: http://bit.ly/AIANNBackPropExplained
• This is called Back Propagation
• Explained (Python) – http://bit.ly/AIANNBackProp
• Animation: http://www.emergentmind.com/neural-network

© 2018 Heemeng Foo
Machine (or Statistical) Learning
• Regression
• Classification
• Clustering
• Recommender Systems
• Decision Tree Systems
• Ensemble Algorithms eg. Boosting, Bagging, Random Forests
• (Good resource: http://bit.ly/MLTaxonomy)
• (Another good resource: http://bit.ly/MLPicture)
• Coursera courses:
• https://www.coursera.org/learn/ml-foundations/
• https://www.coursera.org/learn/practical-machine-learning

© 2018 Heemeng Foo
• Source: https://statistics.laerd.com/spss-tutorials/linear-regression-
using-spss-statistics.php
• Example: Given house prices by zip code, sq ft in whole of USA,
can you predict house price given a zip code, sq ft.
Linear Regression

© 2018 Heemeng Foo
• Source: Quora
• Example: Given emails tagged spam/nonspam, predict if a new email is
Classification
Decision Boundary

© 2018 Heemeng Foo
Clustering
• Source: http://dataillumination.blogspot.com/
• Example: Given database of user profiles, which groups of
users are most likely to be friends

© 2018 Heemeng Foo
Recommender Systems
• Amazon, Netflix
• Technique: Collaborative Filtering (https://youtu.be/KeqVL-
0vSQg)
• Example: Customers who watched this also watched …

© 2018 Heemeng Foo
Decision Tree Systems
• Source: https://intelligentjava.wordpress.com/2015/04/28/machine-
learning-decision-tree/
• Example: http://graphics8.nytimes.com/images/2008/04/16/us/0416-
nat-subOBAMA.jpg

© 2018 Heemeng Foo
What is Overfitting?
• Source: https://vitalflux.com/machine-learning-diagnose-
underfittingoverfitting-learning-algorithm/

© 2018 Heemeng Foo
Ensemble Methods – Dealing with
Overfitting
• Source: https://youtu.be/IICBHZSIsNs
• Explanation: https://youtu.be/IICBHZSIsNs

© 2018 Heemeng Foo
The Supervised Machine Learning
“Pattern”
• Input: A set of data points with “features” and “labels”
• Preparation: Clean data
• Split into (a) Training set (b) Test set (eg. 70/30 split)
• Run training using algo on training set
• Use test set to evaluate accuracy of method

© 2018 Heemeng Foo
The Unsupervised Machine Learning
“Pattern”
• Input: A set of data points with “features”
• Preparation: Clean data
• Define a “distance metric” eg. Numerical value for
“attractiveness” for 2 people
• Define how many clusters you want
• Iterate to find clusters that maximize overall distance metric
• Used mostly for data exploration
• Example: https://i.ytimg.com/vi/_aWzGGNrcic/hqdefault.jpg
• From: https://youtu.be/_aWzGGNrcic

© 2018 Heemeng Foo
Natural Language Processing (NLP)
• Machine Translation
• Eg. English -> Spanish
• Automatic Summarization
• Eg. Summarization of customer service feedback (Salesforce)
• Sentiment Analysis
• Eg. Are customer feedback (or Tweets) generally good, neutral or bad?
• Text Classification
• Eg. Is this email spam?
• Conversational Agents
• Eg. Alexa, Siri, Customer Support Chatbots
Source: https://monkeylearn.com/blog/definitive-guide-natural-language-
processing/
Course: https://www.udacity.com/course/natural-language-processing--cx24

© 2018 Heemeng Foo
Computer Vision
• Recognition
• Object Classification eg. Blippar
• Object Identification eg. Recog my face
• Object Detection Eg. Detect cancer cells
• Motion Analysis
• Egomotion ie. Track motion of camera
• Tracking eg. Following a person
• Optical Flow ie. How object is moving relative to camera
• Scene reconstruction
• Given multiple images, create 3D model
• Image restoration
• Remove “noise” from old images
Source: https://en.wikipedia.org/wiki/Computer_vision
Course: https://www.udacity.com/course/computer-vision--cx25

© 2018 Heemeng Foo
Robotics
• Tapster Robot (Jason Huggins) -
https://www.tapster.io/
• OCR, Object Classification

© 2018 Heemeng Foo
Intelligent Agents (or Bots)
• Software that takes in information from environment (via “sensors”),
makes decision based on rules, goals or utility function (some
metric) and directs activity on the environment
• Simple Reflex Agent
• Model based Agent
• Goal based Agent
• Utility based Agent
• Learning based Agent
• Source: https://en.wikipedia.org/wiki/Intelligent_agent

© 2018 Heemeng Foo
Deep Learning
• Layers and Layers of ANN
• See http://playground.tensorflow.org/
• Pattern same as ML
• Convolution Neural Networks (CNN) – good for Computer Vision
• See https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
• Recurrent Neural Networks (RNN) – good for time series eg. Video,
audio
• Explained: https://youtu.be/WCUNPb-5EYI
• See http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-
introduction-to-rnns/
• Long-Short Term Memory (LSTM)
• Transfer Learning
• Explained: https://youtu.be/yofjFQddwHE
• Courses:
• https://www.coursera.org/learn/ml-foundations/

© 2018 Heemeng Foo
Deep Learning
• Why has Deep Learning suddenly become popular?
• Tons and tons of tagged data made available + processing power eg. GPUs
• The cat photos you post on the internet
• (Andrew Ng’s graph)
• Some Issues with Deep Learning
• You need a lot of training data and compute power
• http://bit.ly/DLDataTooSmall (Don’t use Deep Learning if your data is too small)
• http://bit.ly/DLSmallData (the rebuttal)
• Cold start problem
• “Why did the Deep Neural Network cross the road? We don’t know but it did
so 80% of the time” –Ben Evans
• Is the Deep Neural Net classifying the right features eg. WU’s Wolf
Classifier http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf
• “When all you have is a hammer, ever problem looks like a nail”
• Good article: https://www.datasciencecentral.com/profiles/blogs/when-not-
to-use-deep-learning

© 2018 Heemeng Foo
How AI is being used for Testing or DevOps?
• Using logs or telemetry to predict outages or classify issues
• Sumo Logic, Splunk, ElasticSearch, New Relic, App Dynamics
• Static Code Analysis
• Infer (Facebook) – Symbolic AI ie. Rule Based Systems
• Using Computer Vision, Intelligent Tree Traversal, Bots for UI
Testing
• Nimbledroid, AppDiff, Applitools
• Using Rule-Based, Machine Learning for Test Management
• Using AppStore/customer feedback, Twitter feed, Sentiment
Analysis
• Using Machine Learning to detect deployment failures and auto-
rollback (CI/CD)

© 2018 Heemeng Foo
Using Logs / Telemetry to Predict / Classify Issues
• Sumo Logic - https://youtu.be/WrRDK5I9iyw
• Clustering, Classification, Outlier Detection *
• Splunk MLTK - https://youtu.be/wLEG72fbWu8
• Popular ML Algos eg. Linear Regression, Decision Tree, Random Forest *
• Elasticsearch Machine Learning X-Pack - https://youtu.be/mqEqBN0_Lnc
• Unsupervised anomaly detection: https://www.elastic.co/products/x-pack/machine-
learning *
• AppDynamics - https://youtu.be/qO1M2-J4jYs
• Intelligent agents (learns what is normal baseline for traffic and acts on anomalies) *
• New Relic – https://youtu.be/KzgdP0M_V3o
Outlier detection: http://bit.ly/MLOutlierDetection
* based on marketing material or website info

© 2018 Heemeng Foo
Using Logs / Telemetry to Predict / Classify
Issues
• Sumo Logic - https://youtu.be/WrRDK5I9iyw
• Clustering, Classification, Outlier Detection
• Splunk MLTK - https://youtu.be/wLEG72fbWu8
• Popular ML Algos eg. Linear Regression, Decision Tree, Random Forest
• Elasticsearch Machine Learning X-Pack - https://youtu.be/mqEqBN0_Lnc
• Unsupervised anomaly detection: https://www.elastic.co/products/x-pack/machine-
learning
• AppDynamics - https://youtu.be/qO1M2-J4jYs
• Intelligent agents (learns what is normal baseline for traffic and acts on anomalies)
• New Relic – https://youtu.be/KzgdP0M_V3o
• Outlier detection: https://machinelearningmastery.com/how-to-identify-
outliers-in-your-data/

© 2018 Heemeng Foo
Static Code Analysis
• Facebook (Infer) - http://fbinfer.com/
• Symbolic AI (ie. NLP) *
• AI Reviewer (C++) - http://www.aireviewer.com
• Rule-Based, Heuristics *

© 2018 Heemeng Foo
Sentiment Analysis on Tweets, Customer Feedback
• Google Cloud NLP API - https://cloud.google.com/natural-language
• Using Amazon ML – http://bit.ly/2hu3Osw
• AWS Comprehend http://bit.ly/awsComprehend
• Microsoft Azure - http://bit.ly/2iWX7j9
• MeaningCloud - https://www.meaningcloud.com/products/sentiment-
analysis
• Crimson Hexagon - https://www.crimsonhexagon.com/sentiment-
analysis/
• Appbot - https://appbot.co/
• TheTool – https://thetool.io/app-ratings-reviews-tracker
• Splunk, Elasticsearch, Salesforce Einstein

© 2018 Heemeng Foo
UI Testing (Web)
• Applitools - http://applitools.com/
• Computer Vision *
• ReTest – https://retest.de/en/
• Intelligent Tree Traversal *
• Unravel - https://www.unravel.io/
• NLP, Intelligent Tree Traversal, Rule-Based *
• Functionize - https://www.functionize.com/
• Intelligent Agent, Computer Vision *
• AppVance – http://appvance.com
• Intelligent Tree Traversal, Rule-Based *
• Test.im – http://testim.io
• SauceLabs – (??)

© 2018 Heemeng Foo
UI Testing (Mobile)
• Nimbledroid - https://nimbledroid.com/
• Intelligent tree traversal, Computer Vision
• AppDiff – http://appdiff.com or http://test.ai
• Intelligent Agent (or Bots) *
• Sofy - https://www.sofy.ai
• Intelligent Agent *
• Mabl (Web too) – http://www.mabl.com
• Intelligent Agent *

© 2018 Heemeng Foo
Test Management
• InfoSys - http://bit.ly/InfoSysAITest
• Optimize test cases, regression suite optimization, failure prediction etc
• ML, Rule-Based *
• TestPlant (Eggplant AI) – https://www.testplant.com/products/eggplant-ai/
• Sealights - https://www.sealights.io/
• Rule-Based *
• SauceLabs (Test Analytics) – http://saucelabs.com/
• Data science *

© 2018 Heemeng Foo
CI/CD
• Harness - http://harness.io
• Use Machine Learning to detect deployment failures and auto-rollback

© 2018 Heemeng Foo
Where Does AI/ML Fit In
Production
Test
Backend /
Microservices
Test Backend /
Microservices
Telemetry / Crash
Management

© 2018 Heemeng Foo
Production
Test
Backend /
Microservices
Test Backend /
Microservices
Telemetry / Crash
Management
Log
Analysis
Log
Analysi
s
Log
Analysis
UI
Testing
UI
Testing
UI
Testing
UI
Testing
Sentiment
Analysis

© 2018 Heemeng Foo
Development Build
Backend /
Microservices
Test Backend
/
Microservice
s
Test Environment
Telemetry / Crash
Management
Production

© 2018 Heemeng Foo
Development Build
Backend /
Microservices
Test Backend
/
Microservice
s
Test Environment
Telemetry / Crash
Management
UI
Testing
UI
Testing
Sentiment
Analysis
Log
Analysis
Log
Analysis
Log
Analysis
Production
Test
Management
CI/CD Auto
Failure
Detection
Static
Code
Analysis

© 2018 Heemeng Foo
• Companies want faster and faster product iteration and A/B Testing
• Test automation can’t keep up
• Companies adopting lean product development want to fail fast and
fail early
• Test can’t be an obstacle
• Products are getting much much more complex.
• Complexity scales exponentially but human resource scales linearly
• Test professionals need “force multiplier” tools.
Why AI Is The Future Of Testing

© 2018 Heemeng Foo
Credits - Computer Vision
• http://www.image-net.org/challenges/LSVRC/2014/
• http://technode.com/2017/03/07/5-must-know-chinese-
computer-vision-startups/
• https://www.ibm.com/blogs/research/2016/11/identifying-skin-
cancer-computer-vision/
• https://www.pyimagesearch.com/2015/05/25/basic-motion-
detection-and-tracking-with-python-and-opencv/
• http://www.mit.edu/~kimo/publications/restoration/

An introduction to AI in Test Engineering

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to An introduction to AI in Test Engineering

Similar to An introduction to AI in Test Engineering (20)

Recently uploaded

Recently uploaded (20)

An introduction to AI in Test Engineering