A quick intro into Artificial Intelligence and Machine Learning from a Software Testing perspective. The presentation goes briefly through the history of Artificial Intelligence. It then provides a quick overview on how testing strategies need to to adapt in order to meet the needs of this new technology. It also explores how Machine Learning can be used to improve the testing process as a whole.
The presentation was made in February 2018 for "Tabara de Testare" - Bucharest's Largest Software Testing Community.
3. ARTIFICIAL INTELLIGENCE (1950) – branch of computer science concerned
with making computers behave like humans.
MACHINE LEARNING (1959) – area of AI that is focused on developing
principles and techniques for automating the acquisition of knowledge ||
learn without being explicitly programmed.
4. PROJECT ALPHAGO
AlphaGo is an AI that plays Go. It is trained using deep neural networks and reinforced learning. It was shown
hundreds of thousands of Go matches and the AI learned everything there is know about Go.
2015 – AlphaGo defeated the European Go Champion 5-0.
2016 – AlphaGo defeated the Worldwide Go Champion 4-1.
PROJECT ALPHAGO ZERO
AlphaGo Zero is the next evolution of AlphaGo. Instead of learning from existing matches, the AI is given the
rules of the game and an empty board. It plays against itself millions of times and learns from its own
mistakes. In a matter of days, the AI discovered strategies that took the Go community thousands of years to
discover.
2017 – AlphaGo Zero beats AlphaGo 100-0.
7. HEALTHCARE
- Analyze Tests Results – X-Rays, CT Scans, bloodwork
- Treatment Design – improves a patient treatment’s plan
- Virtual Nurses – chatbots that can provide customized
information for patient’s family
- Drug Creation – identify the closest medication that can be
easily modified to cure or prevent a disease
- Precision Medicine – predict cancer or cardiovascular
diseases using body scans and DNA analysis
- Health Monitoring – predict strokes
9. PREDICTIVE ANALYTICS & CLASSIFICATION
Stocks and cryptocurrencies value
Sport events result
Corruption predictions
Risks identification
Sales forecasting
House value prediction
Fraud prevention
Toxic comments and fake news analysis
Iceberg identifications and path prediction
Gifts matching
Entertainment Recommendations
Traffic predictions and optimization
10. product team
users
system
input data
output data
define system rules
feedback
system
product team
users
output data
input data
AI MODEL
DATASET
select data
configure model
define system rules
feedback
TRADITIONAL SYSTEM AI BASED SYSTEM
13. BIASES
In machine learning, a bias usually occurs when the AI model is trained using an overfitted or underfitted set of data.
The root cause may be within the data itself or how the data was prepared:
- excluded/under-represented segments
- incorrect labels
- imbalanced data preparation
- irrelevant data present
ALL EXISTING DATA DATA WE HAVE DATA WE VALIDATE WITH
14. REAL WORLD BIAS EXAMPLES
- US courtrooms use a re-offending assessment risk
algorithm that is biased against black people
- New Zealand’s passport system incorrectly interprets
Asian photos as having their eyes closed
- SAS’s performance evaluation algorithm got more
than 200 teachers fired as it used insufficient data
- YouTube’s speech-to-text system does not recognize
women’s voices
- Google Photos automatically tagged some pictures
containing black people as “gorillas”
- Flicker tagged Dachau concentration camp as “sport”
or “jungle gym”
15. UNEXPECTED RESULTS
“We can build these models, but we don’t know how they work.”
- AI algorithms are hard to understand (black box)
- Current design techniques are built around logic, where we know the expected result
- AI systems are probabilistic, as opposed to deterministic
Traditional eCommerce site:
If a product costs 10 USD and I add 2 to my shopping cart I will have to pay 20 USD.
eCommerce site that uses AI to modify prices to in real time to offer maximize sells and profits:
If a product usually costs 10 USD, the AI might sell it to you for 9 USD, and to someone else for 11 USD, but if you add 3 in the
shopping cart you might have to pay 28 USD and someone else 30 USD.
16. UNEXPECTED RESULTS
prepare the data:
- it must include all features and
classes as per model design
- all business segments are
properly represented
- have at least two sets (one for
test and validation, and another
for the actual tests)
- test data set should include
anomalies from both technical
and business perspectives
train/validate the model:
- verify if results are within the
expected boundaries
test the model:
- feed it unseen/new data
- analyze scoring
- create result visualizations to
understand biases
- verify the results of anomalous
data
- feed it more data an recheck
initial results
18. BLACK BOX ATTACKS USING ADVERSARIAL EXAMPLES
AIs are vulnerable to adversarial examples: malicious inputs modified to yield erroneous
model outputs, while appearing unmodified to a human observer.
19. SELF DRIVING CARS
Level Capabilities
0 – No Automation driver has to do everything
1 – Drive Assistance either automated steering
OR
acceleration/deceleration
2 – Partial Automation both steering AND
acceleration/deceleration
3 – Conditional Automation can drive itself, but driver
needs to interfere in some
cases depending on the
environment
4 – High Automation can drive itself even if the
driver does not
appropriately respond to
requests to interfere
5 – Full Automation can drive itself in any
conditions that a human
driver could
AI TESTING
Level Capabilities
0 – No AI tester has to do everything
1 – Automation repetitive tasks can be
scripted and executed
automatically
2 – Partial AI AI learns from tester’s data
and improves it
3 – Conditional AI test cases are generated
and executed
automatically, but the
tester needs to interfere
4 – High AI test cases are generated
and executed
automatically, tester needs
to validate results
5 – Full AI test cases are generated
and executed
automatically
21. UI TESTING
applitools - uses image recognition to identify humanely visible anomalies in layout
22. BUG TRIAGE
IBM DeepTriage (Research) – AI model capable of identifying the most appropriate developer to fix a defect
23. PREDICTIVE ANALYTICS / PROCESS IMPROVEMENT
- identify similar/duplicate and unique test cases
- predict defect clusters and automatically prioritize
automated test case execution on that areas
- impact assessment for future user stories
- identify risks that can appear in next iterations
- automatic traceability between requirements, test
cases and defects
https://www.youtube.com/watch?v=DnwSbLZjgdc
31. Financials and Predictions:
- https://www.gartner.com/smarterwithgartner/top-trends-in-the-gartner-hype-cycle-for-emerging-technologies-2017/
- https://www.statista.com/statistics/607716/worldwide-artificial-intelligence-market-revenues/
- https://evansdata.com/reports/viewRelease.php?reportID=9
- https://www.pwc.com/gx/en/issues/analytics/assets/pwc-ai-analysis-sizing-the-prize-report.pdf
- http://www3.weforum.org/docs/FOP_Readiness_Report_2018.pdf
Applications of AI in QA:
- http://esanu.name/vitalie/?p=861
- https://diacritice.ai/
- https://applitools.com/
- https://www.infosys.com/IT-services/validation-solutions/service-offerings/Pages/machine-learning-qa.aspx
- https://www.stickyminds.com/interview/future-software-testing-ai-interview-jason-arbon
- http://www.tothenew.com/blog/artificial-intelligence-in-software-testing/
- https://en.wikipedia.org/wiki/Autonomous_car
- http://bugtriage.mybluemix.net/#chrome
Editor's Notes
- prezentare
Joc de dame
Joc de X&0
IBM DeepBlue / Kasparov
Personal Assistants
Trecere catre anti-scamming
- Radiologia va disparea
- Camere foto cu Face Recognition la inceputul anilor 2000
FinTech foloseste inca din anii 90 AI pentru predictia fluctatilor la bursa, mai nou si pentru crypto
Betting industry foloseste AI pentru predictia rezultatelor meciurilor
Magazin de jucarii din SUA
Corruption in Spain
- Exemplu Booking A/B Testing pentru colectarea de feedback
Predictive analytics
Exemplu identificarea riscurilor folosind date din JIRA
Explicatie schema
Unde e Romania
Explicatie schema
Unde suntem acum
Exemplu Apple
PIB-ul Mondial
Impact in industrii
Fintech primii care vor ajunge la 100%
Numarul de dezvoltatori