A quick intro into Artificial Intelligence and Machine Learning from a Software Testing perspective. The presentation goes briefly through the history of Artificial Intelligence. It then provides a quick overview on how testing strategies need to to adapt in order to meet the needs of this new technology. It also explores how Machine Learning can be used to improve the testing process as a whole. The presentation was made in February 2018 for "Tabara de Testare" - Bucharest's Largest Software Testing Community.

1. https://www.youtube.com/watch?v=HAfLCTRuh7U&t=19s

2. Eduard MIRESCU
Feb. 2018
ARTIFICIAL INTELLIGENCE
AND THE FUTURE OF QA

3. ARTIFICIAL INTELLIGENCE (1950) – branch of computer science concerned
with making computers behave like humans.
MACHINE LEARNING (1959) – area of AI that is focused on developing
principles and techniques for automating the acquisition of knowledge ||
learn without being explicitly programmed.

4. PROJECT ALPHAGO
AlphaGo is an AI that plays Go. It is trained using deep neural networks and reinforced learning. It was shown
hundreds of thousands of Go matches and the AI learned everything there is know about Go.
2015 – AlphaGo defeated the European Go Champion 5-0.
2016 – AlphaGo defeated the Worldwide Go Champion 4-1.
PROJECT ALPHAGO ZERO
AlphaGo Zero is the next evolution of AlphaGo. Instead of learning from existing matches, the AI is given the
rules of the game and an empty board. It plays against itself millions of times and learns from its own
mistakes. In a matter of days, the AI discovered strategies that took the Go community thousands of years to
discover.
2017 – AlphaGo Zero beats AlphaGo 100-0.

5. SUPPORT/CALL
CENTER CHATBOT
VOICE RECOGNITION CHATBOTS
SIRI CORTANA ALEXA
SCIENTIFIC CHATBOTGENERALIST CHATBOT
https://www.youtube.com/watch?v=seCUU5FlWIQ

6. https://www.youtube.com/watch?v=jPajqAJWiNA

7. HEALTHCARE
- Analyze Tests Results – X-Rays, CT Scans, bloodwork
- Treatment Design – improves a patient treatment’s plan
- Virtual Nurses – chatbots that can provide customized
information for patient’s family
- Drug Creation – identify the closest medication that can be
easily modified to cure or prevent a disease
- Precision Medicine – predict cancer or cardiovascular
diseases using body scans and DNA analysis
- Health Monitoring – predict strokes

8. IMAGE RECOGNITION / SURVEILLANCE IMAGE RECOGNITION / SELF DRIVING CARS
https://www.youtube.com/watch?v=aE1kA0Jy0Xg https://www.youtube.com/watch?v=VG68SKoG7vE&

9. PREDICTIVE ANALYTICS & CLASSIFICATION
Stocks and cryptocurrencies value
Sport events result
Corruption predictions
Risks identification
Sales forecasting
House value prediction
Fraud prevention
Toxic comments and fake news analysis
Iceberg identifications and path prediction
Gifts matching
Entertainment Recommendations
Traffic predictions and optimization

10. product team
users
system
input data
output data
define system rules
feedback
system
product team
users
output data
input data
AI MODEL
DATASET
select data
configure model
define system rules
feedback
TRADITIONAL SYSTEM AI BASED SYSTEM

11. DEMO

12. SENSITIVITY TO
FEEDBACK
BIASES
UNEXPECTED
RESULTS

13. BIASES
In machine learning, a bias usually occurs when the AI model is trained using an overfitted or underfitted set of data.
The root cause may be within the data itself or how the data was prepared:
- excluded/under-represented segments
- incorrect labels
- imbalanced data preparation
- irrelevant data present
ALL EXISTING DATA DATA WE HAVE DATA WE VALIDATE WITH

14. REAL WORLD BIAS EXAMPLES
- US courtrooms use a re-offending assessment risk
algorithm that is biased against black people
- New Zealand’s passport system incorrectly interprets
Asian photos as having their eyes closed
- SAS’s performance evaluation algorithm got more
than 200 teachers fired as it used insufficient data
- YouTube’s speech-to-text system does not recognize
women’s voices
- Google Photos automatically tagged some pictures
containing black people as “gorillas”
- Flicker tagged Dachau concentration camp as “sport”
or “jungle gym”

15. UNEXPECTED RESULTS
“We can build these models, but we don’t know how they work.”
- AI algorithms are hard to understand (black box)
- Current design techniques are built around logic, where we know the expected result
- AI systems are probabilistic, as opposed to deterministic
Traditional eCommerce site:
If a product costs 10 USD and I add 2 to my shopping cart I will have to pay 20 USD.
eCommerce site that uses AI to modify prices to in real time to offer maximize sells and profits:
If a product usually costs 10 USD, the AI might sell it to you for 9 USD, and to someone else for 11 USD, but if you add 3 in the
shopping cart you might have to pay 28 USD and someone else 30 USD.

16. UNEXPECTED RESULTS
prepare the data:
- it must include all features and
classes as per model design
- all business segments are
properly represented
- have at least two sets (one for
test and validation, and another
for the actual tests)
- test data set should include
anomalies from both technical
and business perspectives
train/validate the model:
- verify if results are within the
expected boundaries
test the model:
- feed it unseen/new data
- analyze scoring
- create result visualizations to
understand biases
- verify the results of anomalous
data
- feed it more data an recheck
initial results

17. SENSITIVITY TO FEEDBACK

18. BLACK BOX ATTACKS USING ADVERSARIAL EXAMPLES
AIs are vulnerable to adversarial examples: malicious inputs modified to yield erroneous
model outputs, while appearing unmodified to a human observer.

19. SELF DRIVING CARS
Level Capabilities
0 – No Automation driver has to do everything
1 – Drive Assistance either automated steering
OR
acceleration/deceleration
2 – Partial Automation both steering AND
acceleration/deceleration
3 – Conditional Automation can drive itself, but driver
needs to interfere in some
cases depending on the
environment
4 – High Automation can drive itself even if the
driver does not
appropriately respond to
requests to interfere
5 – Full Automation can drive itself in any
conditions that a human
driver could
AI TESTING
Level Capabilities
0 – No AI tester has to do everything
1 – Automation repetitive tasks can be
scripted and executed
automatically
2 – Partial AI AI learns from tester’s data
and improves it
3 – Conditional AI test cases are generated
and executed
automatically, but the
tester needs to interfere
4 – High AI test cases are generated
and executed
automatically, tester needs
to validate results
5 – Full AI test cases are generated
and executed
automatically

20. LOCALIZATION TESTING
diacritice.ai – identifies letters that should be diacritics and automatically replaces them with the correct ones

21. UI TESTING
applitools - uses image recognition to identify humanely visible anomalies in layout

22. BUG TRIAGE
IBM DeepTriage (Research) – AI model capable of identifying the most appropriate developer to fix a defect

23. PREDICTIVE ANALYTICS / PROCESS IMPROVEMENT
- identify similar/duplicate and unique test cases
- predict defect clusters and automatically prioritize
automated test case execution on that areas
- impact assessment for future user stories
- identify risks that can appear in next iterations
- automatic traceability between requirements, test
cases and defects
https://www.youtube.com/watch?v=DnwSbLZjgdc

24. self healing automated scriptsself generating test casesautomated exploratory testing
CONCEPTS

27. 2030
+14%
16T USD
26% - 7T
15% - 4T
11% - 3T
10% - 1T
Impact 0-3 3-7 7+
3.7
3.7
3.3
3.2
3.1
3.0
2.2
2.2
37%
35%
41%
41%
47%
54%
39%
14%
23%
47%
59%
41%
36%
38%
44%
83%
40%
18%
0%
18%
17%
8%
17%
3%
6.5M 12M 25M
2017 2018 2020

28. Thank you!
Q&A

30. AI Bias Real World Examples:
- http://vamboozled.com/breaking-news-victory-in-court-in-houston/
- https://www.technologyreview.com/s/607955/inspecting-algorithms-for-bias/
- https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
- https://www.reuters.com/article/us-newzealand-passport-error/new-zealand-passport-robot-tells-applicant-of-asian-
descent-to-open-eyes-idUSKBN13W0RL
- http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html
- https://www.pcworld.com/article/209708/Is_Microsoft_Kinect_Racist.html
- http://www.businessinsider.com/google-tags-black-people-as-gorillas-2015-7
- https://www.theguardian.com/technology/2015/may/20/flickr-complaints-offensive-auto-tagging-photos
- https://mashable.com/2017/11/30/google-translate-sexism/#0ZWvyZCrZsqG
- https://mashable.com/2017/10/25/google-machine-learning-bias/?utm_cid=a-seealso#y5rC05vWfmq6
- https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/
Unexpected Behaviors:
- http://www.testingtrapezemagazine.com/wp-content/uploads/2017/04/TestingTrapeze-2017-April.pdf
- https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist
- https://arxiv.org/pdf/1312.6199.pdf
- https://blog.openai.com/adversarial-example-research/
- https://arxiv.org/pdf/1412.6572.pdf
- https://arxiv.org/pdf/1602.02697.pdf

31. Financials and Predictions:
- https://www.gartner.com/smarterwithgartner/top-trends-in-the-gartner-hype-cycle-for-emerging-technologies-2017/
- https://www.statista.com/statistics/607716/worldwide-artificial-intelligence-market-revenues/
- https://evansdata.com/reports/viewRelease.php?reportID=9
- https://www.pwc.com/gx/en/issues/analytics/assets/pwc-ai-analysis-sizing-the-prize-report.pdf
- http://www3.weforum.org/docs/FOP_Readiness_Report_2018.pdf
Applications of AI in QA:
- http://esanu.name/vitalie/?p=861
- https://diacritice.ai/
- https://applitools.com/
- https://www.infosys.com/IT-services/validation-solutions/service-offerings/Pages/machine-learning-qa.aspx
- https://www.stickyminds.com/interview/future-software-testing-ai-interview-jason-arbon
- http://www.tothenew.com/blog/artificial-intelligence-in-software-testing/
- https://en.wikipedia.org/wiki/Autonomous_car
- http://bugtriage.mybluemix.net/#chrome