SlideShare a Scribd company logo
1 of 49
Download to read offline
How AI supports
software testing?
Kari Kakkonen
Knowit
https://www.linkedin.com/in/karikakkonen
Copyright Knowit Solutions Oy 2021 1
ROLES
• Knowit Solutions Oy, Director of Training and
Competences, Lead Consultant, Trainer and
Coach
• Children’s and testing author at Dragons Out
Oy
• TMMi, Board of Directors
• Treasurer of Finnish Software Testing Board
(FiSTB)
ACHIEVEMENTS
• Tester of the Year in Finland 2021
• EuroSTAR Testing Excellence Award 2021
• Exemplary DevOps Instructor Award 2023 by
DASA
• ISTQB Executive Committee 2015-2021
• Influencing testing since 1996
• Ranked in 100 most influential IT persons in
Finland (Tivi magazine)
• Great number of presentations in Finnish and
international conferences
• TestausOSY/FAST founding member.
• Co-author of Agile Testing Foundations book
• Regular blogger in Tivi-magazine
Kari Kakkonen, Lead Testing Consultant
SERVICES
• ISTQB Advanced, Foundation, Agile Testing, AI Testing
• Knowit Quality Professional
• DASA DevOps
• Quality & Test process and organization development,
Metrics, TMMi and other assessments
• Agile testing, Scrum, Kanban, Lean
• Leadership
• Test automation, Mobile, Cloud, DevOps, AI
• Quality, cost, benefits
EDUCATION
• ISTQB Expert Level Test Management & Advanced Full
& Agile Tester certified
• DASA DevOps, Scrum Master and SAFe certified
• TMMi Professional, Assessor, Process Improver certified
• SPICE provisionary assessor certified
• M.Sc.(Eng), Helsinki University of Technology (present Aalto
University), Otaniemi, Espoo
• Marketing studies, University of Wisconsin-Madison,
the USA.
BUSINESS DOMAINS
Wide spread of business domain knowledge: Embedded,
industry, public, training, telecommunications, commerce,
Insurance, banking, pension.
28.2.2024 2
twitter.com/kkakkonen
dragonsout.com
act2lead.net
MORE INFORMATION
linkedin.com/in/karikakkonen/
© Copyright Knowit Trainings 2023
A Nordic powerhouse
for digital solutions
4 400+
Experts
6 countries
SE, NO, FI, DK, DE, PL
613 MEUR
Combined turnover 2022
4 business areas
Solutions, Experience, Connectivity and Insight
Nordic ESG-ambassador
Clear vision to accelerate the UN sustainability agenda
The book project ”Dragons Out!”
• Mission
• “Software testing brought to children”
• Book
• Author Kari Kakkonen
• Illustrator Adrienn Széll
• Text and illustration rights Dragons Out Oy
• In Finnish, English, Polish, French and growing
• For ages of 10-99
• Free “Dragon lesson in software testing”
presentation under Creative Commons –license
• Translated to 20 languages!
• More info: www.dragonsout.com
• Also other books coming out
4
ISTQB GLOBAL PRESENCE
• Number of exams
administered: over 1,2 million
• Number of certifications
issued: 914,000
• In 130 countries
TMMi for test improvement in all kinds of testing,
including agile and DevOps
• Artificial Intelligence (AI) in short
• Case of NASA code quality metrics
using AI
• Other AI and testing developments
Agenda
Copyright Knowit Solutions Oy 2021 7
Why right now?
Four drivers behind AI revolution
28.2.2024 8
© Copyright Knowit Solutions 2020 | Version 2.0
Computation growth due to general purpose GPUs The rise of Big data
Community based achievements in Deep learning Open source tools and frameworks
Quality management as top use-case for AI
Source: AI software platform adoption survey, IDC, February 2019 [Percent of respondents: N=505]
Copyright Knowit Solutions Oy 2021 9
• When talking about AI, it is
important to ask if we are
talking about modelling
humans or how to work in
ideal way.
• Machine learning is a sub-
category of AI, which
considers algorithms that
enable AI to learn ideal way to
work.
Artificial intelligence == machine learning?
Figure: Example of concepts related to AI.
Copyright Knowit Solutions Oy 2021 10
Three methods of machine learning
Machine learning methods
Supervised Learning
• The aim is to learn
mapping from input to
output.
• Output is known by
supervisor.
Unsupervised Learning
• The aim is to find
”something interesting”.
• Output is not known
(known also as density
estimation in statistics).
Reinforcement Learning
• Seemingly good
solutions are
encouraged to
develop further.
AI literature review[1]
1 Reagan, A., et al., The emotional arcs of stories are dominated by six basic shapes, EPJ Data Science (2016).
Copyright Knowit Solutions Oy 2021 11
No one tool for all
Copyright Knowit Solutions Oy 2021
Performance
Amount of data, time, and computing power
• Linear discriminant
analysis
• Logistic regression
• Medium-sized ANNs
• Genetic algorithms
• Extreme learning
machines
• Convolution neural
networks
• Generative neural
networks
• Deep belief networks
• Recurrent neural
networks
• Simple neural networks
12
Planning
descriptions
Code Unit testing
Integration testing
System testing
Acceptance
testing
Definition
descriptions
Requirements
• Static methods
• Reviews
• Dynamic methods
• Test runs
Remember there are different kinds of testing
Copyright Knowit Solutions Oy 2021 13
AI in software testing
Copyright Knowit Solutions Oy 2021
• In static testing, several machine learning
algorithms can be used to measure code
quality.[1] NASA’s code review.
• In dynamic testing, model-based regression
tests can be replaced by algorithms that
generate test flows on-the-fly.[2] Chaos
monkey of Netflix.
• In maintenance testing, machine learning can
be used to plan more realistic statistical
models of how application is actually used.
Figure: Margaret Hamilton standing next
to source code of guidance system of
Apollo spacecraft.
www.nasa.gov/
14
Toolbox for AI
Copyright Knowit Solutions Oy 2021 15
Anomaly detection use cases
Copyright Knowit Solutions Oy 2021 16
• Detecting:
• abnormal behaviour
• new kind of customers
• etc.
Anomaly detection
Copyright Knowit Solutions Oy 2021
Figure: Example of time series
17
AI
AI review
Requirement
management
Code quality
analysis
Version
control
Test
management
Defect
management
AI code quality metrics
Copyright Knowit Solutions Oy 2021 18
Developer
Product
Owner
Scrum team
Management
AI
AI review
AI code quality metrics
Copyright Knowit Solutions Oy 2021 19
• Goal is to understand technical quality and test coverage of modules.
Also, goal is to guide and focus testing:
• Focusing technical debt reduction
• Found defects per module
• Guide for test coverage in automating system level testing.
AI code quality metrics
Copyright Knowit Solutions Oy 2021
Requires that there is link from components (version control) and tests (test management) to feature descriptions and from reported defects to modules (observation control)
Module Developer Benchmark
(order.numb.)
Relative
risk
Number of
reported
bugs
Module_1 Jaakko 50 58 4
Module_2 Jaakko 4 5 1
Module_3 Mikko 180 95 7
SCRUM TEAM
20
• Goal is to provide overview:
• How fast the development is advancing
• Maturity for release
• Quality and technical debt of the software
• Defect probabilities
• Risks related to modifications of the
software.
AI code quality metrics
Copyright Knowit Solutions Oy 2021
Interpretation:
• Module 5 has high defect probability and is
related to five most important features of
the application. This results into high risk in
release at this moment.
• Defect probability of module 10 has fallen
to acceptable level.
Start-up
phase
Development
phase
Maturing
phase
MANAGEMENT
21
Stages of execution
4. DEPLOYMENT AND UTILIZATION OF THE ALGORITHM
▪ Monitoring and analysing results
2. DEFINING QUALITY METRICS
▪ Defining quality and defect metrics
▪ Building the metrics
3. DEFINING HOW RESULTS ARE PRESENTED
▪ Defining who will use results and how
▪ Building data visualization
1. DEFINING DATA SOURCES
▪ From where data is collected
▪ How will the data be collected
Copyright Knowit Solutions Oy 2021 22
Most
effort
Code quality metrics: NASA case
Copyright Knowit Solutions Oy 2021 23
• Several code quality metrics are currently
being used. The most popular ones have
been introduced in 1970.[1,2]
1 T.J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering (1976) SE-2, 4, 308.
2 M.H. Halstead, Elements of Software Science, Elsevier (1977).
3 N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous & Practical Approach, International Thompson Press (1997).
4 S.J. Sayyad, T.J. Menzies, The PROMISE Repository of Software Engineering Databases (2005).
• Goal is to provide feedback:
1. for developers about code quality.
2. for testers about needed test coverage.
3. for management about the development
project
• Although traditional metrics are extensively
used, benefit of using them has been
questioned.[3]
0
50
100
150
200
250
300
350
400
450
0 20 40 60 80 100 120
Number
of
lines.
Cyclomatic complexity
Scattering of defects in NASA CM1 project
OK Defected
Figure: NASA Metrics Data Program, CM1 Project[4]
In real applications defected modules do not separate from
non-defected by standard metrics.
AI code quality metrics
Copyright Knowit Solutions Oy 2021 24
1
2
𝑤 2
+ 𝐶 ෍
𝑖=1
𝑁
ξ𝑖
1-𝑦𝑖 𝑤, 𝑓(𝑥𝑖) + 𝑏 ≤ ξ𝑖
Minimize
Subject to with all values of i.
Quality metric 1
Quality
metric
2
• Machine learning approach was tested for NASA’s CM1
software repository[1].
• Support vector type of machine learning algorithm was
developed for this demonstration.
• This task can be solved by supervised machine learning
methods because both input (quality metrics) and output
(defects) is known.
• Goal is to find such hyperplane in multidimensional data
space that separate defected modules from non-
defected.
• Mathematically, problem can be described with algorithm:
AI-guided software testing with NASA data
Copyright Knowit Solutions Oy 2021 25
0,0
0,2
0,4
0,6
0,8
Tested on real data against NASA CM1 project[1]
Performance
(F)
Training set: 1595
Test set: 400
𝐹 =
2 ∗ 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 ∗ 𝑟𝑒𝑐𝑎𝑙𝑙
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 + 𝑟𝑒𝑐𝑎𝑙𝑙
AI-guided software testing with NASA data
Copyright Knowit Solutions Oy 2021 26
Other interesting developments in AI
and testing
Copyright Knowit Solutions Oy 2021 27
• Enhanced automation
• Test automation
• Robotic Process Automation -> Intelligent automation
• AI-powered test automation
• Efficiency by Generative AI
• Test script creation
• Manual test case creation
• Test data creation
• Enhanced test selection
• Risk analysis
• Test case selection
• Code analysis
General direction at businesses
on AI and testing
Copyright Knowit Solutions Oy 2021 28
• Start from screenshots
• Detect layout issues
• Cross-browser testing for
differences
• Ignore invisible rendering, size,
position differences
• Case: Applitools
• https://applitools.com / Visual AI
• Also works as an AI library for test
automation frameworks like Robot
Framework
AI-powered comparison of visual elements
Copyright Knowit Solutions Oy 2021 29
• AI-based test generation
• Checking functional differences
• Checking visual differences
• Automatically learning the test
automation
• For: regression testing after changes
• Case: retest
• https://retest.de/ai-based-test-
generation/
AI with regression testing
Copyright Knowit Solutions Oy 2021 30
• “World’s first”
• Go automatically through mobile apps
• Use general test cases
• Reinforcement learning
• Case: test.ai
• https://www.test.ai/
AI-powered mobile test automation platform
Copyright Knowit Solutions Oy 2021 31
• Default tests to web sites
• Teach your own tests
• Running on containers in virtual
machines
• Use for regression testing, cross-
browser testing
• Case: Mabl
• https://www.mabl.com/
Auto-healing automated testing of web
Copyright Knowit Solutions Oy 2021 32
• Build test through interface backed by ML
• Plain English
• New life for “record-playback”
• ML algorithms maintain the tests
• Case: Functionize
• https://www.functionize.com/
Intelligent testing
Copyright Knowit Solutions Oy 2021 33
• Automatically scope and generate optimized test cases
• Generate test scripts for automated execution from models and
manual tests
• Display the test coverage requirements
• Integrate with existing tools
• Case: Conformiq
• https://www.conformiq.com/
AI-enabled test creation from models
Copyright Knowit Solutions Oy 2021 34
• Enhance core test automation
• Develop automated test scripts
• Regression testing
• Build test environments rapidly
• Service focus
• Case: Mammoth
• https://www.mammoth-ai.com/
• https://www.mammoth-ai.com/artificial-intelligence/
Enhancing test automation with AI
Copyright Knowit Solutions Oy 2021 35
• Support test automation engine data selection with AI
• Can be used for testing with large quantities of data
• Can be used automating production as Robotic Process Automation
(RPA)
• Cases:
• Robot Framework test automation framework
• Aito AI tool
• https://robotframework.org/
• https://aito.ai/
• https://aito.ai/blog/machine-learning-in-robot-framework/
Automating tests or production with lots of
repetitive data
Copyright Knowit Solutions Oy 2021 36
• Supports test automation object recognition
• Helps with typical GUI recognition problems for test automation
• Can be used as a support library for existing test automation
• Cases:
• ImageHorizonLibrary is a cross-platform library for Robot Framework.
• Eficode, Oulu University and Business Finland
• https://www.eficode.com/projects/testomat
Image recognition with AI
for open source test automation
Copyright Knowit Solutions Oy 2021 37
• Python-editor PyCharm has
Refact.ai library that uses ChatGPT
• Understands Python and Robot
Framework test automation setup
• Refact.ai is labelled as an AI coding
assistant software
• https://refact.ai/
Python enhanced with ChatGPT
Copyright Knowit Solutions Oy 2021 38
• Assist test automation script
creation
• Suggest what the user might want
• What kind of code
• What kind of test script
• User reviews and approves
suggestions
• Based on Generative AI
• https://github.com/features/copilot
Github Copilot
Copyright Knowit Solutions Oy 2021 39
• Python-editor PyCharm can use
Dev Tools AI library
• Asks which element to click on the
screen and then keeps it running
• Suggests how to fix
semiautomatically a changed test
script that doesn’t pass any more
• Reduces the need for test script
maintenance
• Supports e.g. Cypress, Selenium,
Appium, Playwright
• https://www.dev-tools.ai/
AI library helps pass Python test scripts
Copyright Knowit Solutions Oy 2021 40
• Assist extending existing test
automation sets
• With e.g. Robot Framework,
Playwright, Selenium, Cypress
• Use tests as self-driving test-
bots
• Exploring web apps
• Regression testing
• Autonomous testing
• https://wopee.io/
Test automation assistants and bots
Copyright Knowit Solutions Oy 2021 41
• Derive tests from specifications
• Create tests based on user input
• Enabled by good prompt
engineering
• Challenge: repeated teaching
• User reviews AI proposals and uses
in their tests
• https://chat.openai.com/ (ChatGPT)
• https://gemini.google.com/
• https://copilot.microsoft.com/ (Bing)
Generative AI helps in test creation
Copyright Knowit Solutions Oy 2021 42
• Go through functionalities
• Identify risky areas
• Focus testing
• Case: Eggplant
• https://www.eggplantsoftwar
e.com/
Predict quality issues
Copyright Knowit Solutions Oy 2021 43
• Analyze risk in commits to the code
• Select the tests that test the risky areas
• Reduce number of test cases needed to run
• Case: Appsurify
• https://appsurify.com/
Select most suitable test cases
Copyright Knowit Solutions Oy 2021 44
• Power of open source
developers
• Machine learning
algorithms learn from the
community automatically
• Analyse code and
propose improvements
• Case: DeepCode
• https://www.deepcode.ai/
Code analysis using AI
Copyright Knowit Solutions Oy 2021 45
• Trainings and certifications
• https://www.istqb.org/certification-path-root/ai-testing.html
• Standards
• ISO
• AI/ML standard https://www.iso.org/standard/74438.html
• Testing of AI-based systems https://www.iso.org/standard/79016.html
• IEEE
• What is AI Software Testing https://ieeexplore.ieee.org/document/8705808
• AI testing perspectives https://ieeexplore.ieee.org/document/9514942
• Application of AI in Testing https://ieeexplore.ieee.org/document/9676244
Promoting and teaching AI and testing
Copyright Knowit Solutions Oy 2021 46
• AI in Testing: Impact, Problems, Challenges and Prospect
https://www.researchgate.net/publication/357876318_Artificial_Inte
lligence_in_Software_Testing_Impact_Problems_Challenges_and_Pro
spect
• Utilizing AI in Software Testing
https://www.theseus.fi/handle/10024/263992
• AI Applied to Software Testing
https://dl.acm.org/doi/10.1145/3616372
• AI Applied to Testing: A Literature Review
https://ieeexplore.ieee.org/abstract/document/9141124
• ChatGPT helps testing https://www.linkedin.com/pulse/gpt-4-sdlcs-
secret-weapon-reinventing-testing-phase-andy-abbott/
Some more research notes
Copyright Knowit Solutions Oy 2021 47
• AI-enabled testing is already a reality
• Many companies are enhancing their solutions with AI
• New companies are set up around AI
• AI can provide simplicity to complex software development projects.
• With Generative AI, manual and automated test script creation is
more productive
• With AI, testing activities can be focused on high-risk areas.
• With AI, test automation becomes more autonomous
• Testers are freed to create new tests
Conclusion
Copyright Knowit Solutions Oy 2021 48
kari.kakkonen@knowit.fi
https://linkedin.com/in/karikakkonen/
https://info.knowit.fi/qualitysummit
17-18.4.2024 Quality and AI
Thank you!

More Related Content

What's hot

SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...
SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...
SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...septianm
 
Materi penjasorkes kelas x smstr 1 dan 2
Materi penjasorkes kelas x smstr 1 dan 2Materi penjasorkes kelas x smstr 1 dan 2
Materi penjasorkes kelas x smstr 1 dan 2bimo kontaning
 
1 genap peta, atlas, dan globe
1 genap peta, atlas, dan globe1 genap peta, atlas, dan globe
1 genap peta, atlas, dan globeEddy Soesanto
 
Hubungan antara atsmosfer
Hubungan antara atsmosferHubungan antara atsmosfer
Hubungan antara atsmosferOky de Holmeas
 
pengenalan GNSS
pengenalan GNSSpengenalan GNSS
pengenalan GNSSirfanade1
 
Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)
Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)
Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)Nurul Afdal Haris
 
1. Fenomena Geosfer.pptx
1. Fenomena Geosfer.pptx1. Fenomena Geosfer.pptx
1. Fenomena Geosfer.pptxNunungJuniarti2
 
CONTOH LAPORAN KKN/KPM STAIN CURUP
CONTOH LAPORAN KKN/KPM STAIN CURUPCONTOH LAPORAN KKN/KPM STAIN CURUP
CONTOH LAPORAN KKN/KPM STAIN CURUPTatik prisnamasari
 
Sepak Bola
Sepak BolaSepak Bola
Sepak BolaRus Mala
 
REGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptx
REGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptxREGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptx
REGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptxchristin84
 
Konsep Dasar Penginderaan Jauh.pptx
Konsep Dasar Penginderaan Jauh.pptxKonsep Dasar Penginderaan Jauh.pptx
Konsep Dasar Penginderaan Jauh.pptxichsan41
 
Surat keterangan telah mengawasi ujian
Surat keterangan telah mengawasi ujianSurat keterangan telah mengawasi ujian
Surat keterangan telah mengawasi ujianayrus riz
 
Makalah geosat vlbi
Makalah geosat vlbiMakalah geosat vlbi
Makalah geosat vlbifebrina11
 

What's hot (20)

Bab 2 pengetahuan dasar pemetaan
Bab 2 pengetahuan dasar pemetaanBab 2 pengetahuan dasar pemetaan
Bab 2 pengetahuan dasar pemetaan
 
Contoh Surat Resmi
Contoh Surat ResmiContoh Surat Resmi
Contoh Surat Resmi
 
Proposal futsal
Proposal futsalProposal futsal
Proposal futsal
 
Badminton
BadmintonBadminton
Badminton
 
SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...
SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...
SOP PENYELENGGARAAN PEMETAAN PARTISIPATIF DAN PENGENDALIAN KUALITAS PETA PART...
 
Materi penjasorkes kelas x smstr 1 dan 2
Materi penjasorkes kelas x smstr 1 dan 2Materi penjasorkes kelas x smstr 1 dan 2
Materi penjasorkes kelas x smstr 1 dan 2
 
1 genap peta, atlas, dan globe
1 genap peta, atlas, dan globe1 genap peta, atlas, dan globe
1 genap peta, atlas, dan globe
 
Hubungan antara atsmosfer
Hubungan antara atsmosferHubungan antara atsmosfer
Hubungan antara atsmosfer
 
Bola bakar
Bola bakarBola bakar
Bola bakar
 
Makalah tenis meja
Makalah tenis mejaMakalah tenis meja
Makalah tenis meja
 
pengenalan GNSS
pengenalan GNSSpengenalan GNSS
pengenalan GNSS
 
Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)
Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)
Materi Mata Kuliah Gemorfologi Indonesia (Geomorfologi Maluku)
 
1. Fenomena Geosfer.pptx
1. Fenomena Geosfer.pptx1. Fenomena Geosfer.pptx
1. Fenomena Geosfer.pptx
 
CONTOH LAPORAN KKN/KPM STAIN CURUP
CONTOH LAPORAN KKN/KPM STAIN CURUPCONTOH LAPORAN KKN/KPM STAIN CURUP
CONTOH LAPORAN KKN/KPM STAIN CURUP
 
Sepak Bola
Sepak BolaSepak Bola
Sepak Bola
 
REGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptx
REGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptxREGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptx
REGIONALISASI KAWASAN DUNIA BERDASARKAN PUSAT PERTUMBUHAN EKONOMI.pptx
 
contoh surat pemberitahuan kegiatan
contoh surat pemberitahuan kegiatancontoh surat pemberitahuan kegiatan
contoh surat pemberitahuan kegiatan
 
Konsep Dasar Penginderaan Jauh.pptx
Konsep Dasar Penginderaan Jauh.pptxKonsep Dasar Penginderaan Jauh.pptx
Konsep Dasar Penginderaan Jauh.pptx
 
Surat keterangan telah mengawasi ujian
Surat keterangan telah mengawasi ujianSurat keterangan telah mengawasi ujian
Surat keterangan telah mengawasi ujian
 
Makalah geosat vlbi
Makalah geosat vlbiMakalah geosat vlbi
Makalah geosat vlbi
 

Similar to AI improves software testing to be more fault tolerant, focused and efficient

AI improves software testing by Kari Kakkonen at TQS
AI improves software testing by Kari Kakkonen at TQSAI improves software testing by Kari Kakkonen at TQS
AI improves software testing by Kari Kakkonen at TQSKari Kakkonen
 
Test-Driven Development in the Corporate Workplace
Test-Driven Development in the Corporate WorkplaceTest-Driven Development in the Corporate Workplace
Test-Driven Development in the Corporate WorkplaceAhmed Owian
 
Can we induce change with what we measure?
Can we induce change with what we measure?Can we induce change with what we measure?
Can we induce change with what we measure?Michaela Greiler
 
History Class - For software testers
History Class - For software testersHistory Class - For software testers
History Class - For software testersJoris Meerts
 
Software Analytics - Achievements and Challenges
Software Analytics - Achievements and ChallengesSoftware Analytics - Achievements and Challenges
Software Analytics - Achievements and ChallengesTao Xie
 
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...Curiosity Software Ireland
 
A Roadmap to Enterprise Quality
A Roadmap to Enterprise QualityA Roadmap to Enterprise Quality
A Roadmap to Enterprise QualityJeff Bramwell
 
Software system design sample
Software system design sampleSoftware system design sample
Software system design sampleNorman K Ma
 
Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013SmartBear
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand
 
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011TEST Huddle
 
Software engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research resultsSoftware engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research resultsNikolai Avteniev
 
How AI and ML Can Accelerate and Optimize Software Development and Testing
How AI and ML Can Accelerate and Optimize Software Development and TestingHow AI and ML Can Accelerate and Optimize Software Development and Testing
How AI and ML Can Accelerate and Optimize Software Development and TestingAggregage
 
Does Agile Really Work For Business Critical Applications?
Does Agile Really Work For Business Critical Applications?Does Agile Really Work For Business Critical Applications?
Does Agile Really Work For Business Critical Applications?CAST
 
Bridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to ProductionBridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to ProductionFlorian Wilhelm
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Kari Kakkonen
 
Pm soln9416141129710
Pm soln9416141129710Pm soln9416141129710
Pm soln9416141129710Nikhil Todkar
 

Similar to AI improves software testing to be more fault tolerant, focused and efficient (20)

AI improves software testing by Kari Kakkonen at TQS
AI improves software testing by Kari Kakkonen at TQSAI improves software testing by Kari Kakkonen at TQS
AI improves software testing by Kari Kakkonen at TQS
 
Test-Driven Development in the Corporate Workplace
Test-Driven Development in the Corporate WorkplaceTest-Driven Development in the Corporate Workplace
Test-Driven Development in the Corporate Workplace
 
Can we induce change with what we measure?
Can we induce change with what we measure?Can we induce change with what we measure?
Can we induce change with what we measure?
 
History Class - For software testers
History Class - For software testersHistory Class - For software testers
History Class - For software testers
 
Software Analytics - Achievements and Challenges
Software Analytics - Achievements and ChallengesSoftware Analytics - Achievements and Challenges
Software Analytics - Achievements and Challenges
 
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
Curiosity and Xray present - In sprint testing: Aligning tests and teams to r...
 
A Roadmap to Enterprise Quality
A Roadmap to Enterprise QualityA Roadmap to Enterprise Quality
A Roadmap to Enterprise Quality
 
Software system design sample
Software system design sampleSoftware system design sample
Software system design sample
 
Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013Cross functional peer review preso 10-01-2013
Cross functional peer review preso 10-01-2013
 
QM in Software Projects
QM in Software ProjectsQM in Software Projects
QM in Software Projects
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial Intelligence
 
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
Ben Walters - Creating Customer Value With Agile Testing - EuroSTAR 2011
 
Software engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research resultsSoftware engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research results
 
How AI and ML Can Accelerate and Optimize Software Development and Testing
How AI and ML Can Accelerate and Optimize Software Development and TestingHow AI and ML Can Accelerate and Optimize Software Development and Testing
How AI and ML Can Accelerate and Optimize Software Development and Testing
 
Does Agile Really Work For Business Critical Applications?
Does Agile Really Work For Business Critical Applications?Does Agile Really Work For Business Critical Applications?
Does Agile Really Work For Business Critical Applications?
 
Bridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to ProductionBridging the Gap: from Data Science to Production
Bridging the Gap: from Data Science to Production
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
 
1
11
1
 
Resume
ResumeResume
Resume
 
Pm soln9416141129710
Pm soln9416141129710Pm soln9416141129710
Pm soln9416141129710
 

More from Kari Kakkonen

Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
AI improves software testing through test automation, test creation and test ...
AI improves software testing through test automation, test creation and test ...AI improves software testing through test automation, test creation and test ...
AI improves software testing through test automation, test creation and test ...Kari Kakkonen
 
Kari Kakkonen Climate Impact of Software Testing
Kari Kakkonen Climate Impact of Software TestingKari Kakkonen Climate Impact of Software Testing
Kari Kakkonen Climate Impact of Software TestingKari Kakkonen
 
Climate Impact of Software Testing
Climate Impact of Software TestingClimate Impact of Software Testing
Climate Impact of Software TestingKari Kakkonen
 
Insights about children testing
Insights about children testingInsights about children testing
Insights about children testingKari Kakkonen
 
Climate Impact of Software Testing.pdf
Climate Impact of Software Testing.pdfClimate Impact of Software Testing.pdf
Climate Impact of Software Testing.pdfKari Kakkonen
 
Knights of Quality: Immersive talk about software testing
Knights of Quality: Immersive talk about software testingKnights of Quality: Immersive talk about software testing
Knights of Quality: Immersive talk about software testingKari Kakkonen
 
Climate Impact of Software Testing
Climate Impact of Software TestingClimate Impact of Software Testing
Climate Impact of Software TestingKari Kakkonen
 
Becoming MultiTalented Tester
Becoming MultiTalented TesterBecoming MultiTalented Tester
Becoming MultiTalented TesterKari Kakkonen
 
Becoming a Multitalented Tester - at KDS
Becoming a Multitalented Tester - at KDSBecoming a Multitalented Tester - at KDS
Becoming a Multitalented Tester - at KDSKari Kakkonen
 
How to test an AI application
How to test an AI applicationHow to test an AI application
How to test an AI applicationKari Kakkonen
 
How children learn software testing
How children learn software testingHow children learn software testing
How children learn software testingKari Kakkonen
 
Ohjelmistotestauksen opetuksen kokemuksia fantasiatarinan avulla
Ohjelmistotestauksen opetuksen kokemuksia fantasiatarinan avullaOhjelmistotestauksen opetuksen kokemuksia fantasiatarinan avulla
Ohjelmistotestauksen opetuksen kokemuksia fantasiatarinan avullaKari Kakkonen
 
Becoming Multitalented Tester - Kari Kakkonen
Becoming Multitalented Tester - Kari KakkonenBecoming Multitalented Tester - Kari Kakkonen
Becoming Multitalented Tester - Kari KakkonenKari Kakkonen
 
How Children Learn Testing Kari Kakkonen Odin conference.pdf
How Children Learn Testing Kari Kakkonen Odin conference.pdfHow Children Learn Testing Kari Kakkonen Odin conference.pdf
How Children Learn Testing Kari Kakkonen Odin conference.pdfKari Kakkonen
 
Insights from children about testing at NTD2022
Insights from children about testing at NTD2022Insights from children about testing at NTD2022
Insights from children about testing at NTD2022Kari Kakkonen
 
Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...Kari Kakkonen
 
Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...Kari Kakkonen
 
Insights from children about testing; how children who've read the Dragons Ou...
Insights from children about testing; how children who've read the Dragons Ou...Insights from children about testing; how children who've read the Dragons Ou...
Insights from children about testing; how children who've read the Dragons Ou...Kari Kakkonen
 
Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)
Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)
Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)Kari Kakkonen
 

More from Kari Kakkonen (20)

Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
AI improves software testing through test automation, test creation and test ...
AI improves software testing through test automation, test creation and test ...AI improves software testing through test automation, test creation and test ...
AI improves software testing through test automation, test creation and test ...
 
Kari Kakkonen Climate Impact of Software Testing
Kari Kakkonen Climate Impact of Software TestingKari Kakkonen Climate Impact of Software Testing
Kari Kakkonen Climate Impact of Software Testing
 
Climate Impact of Software Testing
Climate Impact of Software TestingClimate Impact of Software Testing
Climate Impact of Software Testing
 
Insights about children testing
Insights about children testingInsights about children testing
Insights about children testing
 
Climate Impact of Software Testing.pdf
Climate Impact of Software Testing.pdfClimate Impact of Software Testing.pdf
Climate Impact of Software Testing.pdf
 
Knights of Quality: Immersive talk about software testing
Knights of Quality: Immersive talk about software testingKnights of Quality: Immersive talk about software testing
Knights of Quality: Immersive talk about software testing
 
Climate Impact of Software Testing
Climate Impact of Software TestingClimate Impact of Software Testing
Climate Impact of Software Testing
 
Becoming MultiTalented Tester
Becoming MultiTalented TesterBecoming MultiTalented Tester
Becoming MultiTalented Tester
 
Becoming a Multitalented Tester - at KDS
Becoming a Multitalented Tester - at KDSBecoming a Multitalented Tester - at KDS
Becoming a Multitalented Tester - at KDS
 
How to test an AI application
How to test an AI applicationHow to test an AI application
How to test an AI application
 
How children learn software testing
How children learn software testingHow children learn software testing
How children learn software testing
 
Ohjelmistotestauksen opetuksen kokemuksia fantasiatarinan avulla
Ohjelmistotestauksen opetuksen kokemuksia fantasiatarinan avullaOhjelmistotestauksen opetuksen kokemuksia fantasiatarinan avulla
Ohjelmistotestauksen opetuksen kokemuksia fantasiatarinan avulla
 
Becoming Multitalented Tester - Kari Kakkonen
Becoming Multitalented Tester - Kari KakkonenBecoming Multitalented Tester - Kari Kakkonen
Becoming Multitalented Tester - Kari Kakkonen
 
How Children Learn Testing Kari Kakkonen Odin conference.pdf
How Children Learn Testing Kari Kakkonen Odin conference.pdfHow Children Learn Testing Kari Kakkonen Odin conference.pdf
How Children Learn Testing Kari Kakkonen Odin conference.pdf
 
Insights from children about testing at NTD2022
Insights from children about testing at NTD2022Insights from children about testing at NTD2022
Insights from children about testing at NTD2022
 
Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...
 
Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...Insights from children about testing; how children who've read the Dragons O...
Insights from children about testing; how children who've read the Dragons O...
 
Insights from children about testing; how children who've read the Dragons Ou...
Insights from children about testing; how children who've read the Dragons Ou...Insights from children about testing; how children who've read the Dragons Ou...
Insights from children about testing; how children who've read the Dragons Ou...
 
Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)
Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)
Dragons Out Ohjelmistotestauksen oppia fantasian keinoilla (ITK-konferenssi)
 

Recently uploaded

Software Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey HightowerSoftware Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey HightowerAnchore
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MIRomil Mishra
 
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2DianaGray10
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...stockholm university
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfROWELL MARQUINA
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementDianaGray10
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023Joshua Flannery
 
Tecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for RotogravureTecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for RotogravureAntonio de Llamas
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Dynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationDynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationBuild Intuit
 

Recently uploaded (20)

Software Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey HightowerSoftware Security in the Real World w/Kelsey Hightower
Software Security in the Real World w/Kelsey Hightower
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MI
 
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...The work to make the piecework work: An ethnographic study of food delivery w...
The work to make the piecework work: An ethnographic study of food delivery w...
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdf
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions management
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
 
Tecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for RotogravureTecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for Rotogravure
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Dynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationDynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientation
 

AI improves software testing to be more fault tolerant, focused and efficient

  • 1. How AI supports software testing? Kari Kakkonen Knowit https://www.linkedin.com/in/karikakkonen Copyright Knowit Solutions Oy 2021 1
  • 2. ROLES • Knowit Solutions Oy, Director of Training and Competences, Lead Consultant, Trainer and Coach • Children’s and testing author at Dragons Out Oy • TMMi, Board of Directors • Treasurer of Finnish Software Testing Board (FiSTB) ACHIEVEMENTS • Tester of the Year in Finland 2021 • EuroSTAR Testing Excellence Award 2021 • Exemplary DevOps Instructor Award 2023 by DASA • ISTQB Executive Committee 2015-2021 • Influencing testing since 1996 • Ranked in 100 most influential IT persons in Finland (Tivi magazine) • Great number of presentations in Finnish and international conferences • TestausOSY/FAST founding member. • Co-author of Agile Testing Foundations book • Regular blogger in Tivi-magazine Kari Kakkonen, Lead Testing Consultant SERVICES • ISTQB Advanced, Foundation, Agile Testing, AI Testing • Knowit Quality Professional • DASA DevOps • Quality & Test process and organization development, Metrics, TMMi and other assessments • Agile testing, Scrum, Kanban, Lean • Leadership • Test automation, Mobile, Cloud, DevOps, AI • Quality, cost, benefits EDUCATION • ISTQB Expert Level Test Management & Advanced Full & Agile Tester certified • DASA DevOps, Scrum Master and SAFe certified • TMMi Professional, Assessor, Process Improver certified • SPICE provisionary assessor certified • M.Sc.(Eng), Helsinki University of Technology (present Aalto University), Otaniemi, Espoo • Marketing studies, University of Wisconsin-Madison, the USA. BUSINESS DOMAINS Wide spread of business domain knowledge: Embedded, industry, public, training, telecommunications, commerce, Insurance, banking, pension. 28.2.2024 2 twitter.com/kkakkonen dragonsout.com act2lead.net MORE INFORMATION linkedin.com/in/karikakkonen/ © Copyright Knowit Trainings 2023
  • 3. A Nordic powerhouse for digital solutions 4 400+ Experts 6 countries SE, NO, FI, DK, DE, PL 613 MEUR Combined turnover 2022 4 business areas Solutions, Experience, Connectivity and Insight Nordic ESG-ambassador Clear vision to accelerate the UN sustainability agenda
  • 4. The book project ”Dragons Out!” • Mission • “Software testing brought to children” • Book • Author Kari Kakkonen • Illustrator Adrienn Széll • Text and illustration rights Dragons Out Oy • In Finnish, English, Polish, French and growing • For ages of 10-99 • Free “Dragon lesson in software testing” presentation under Creative Commons –license • Translated to 20 languages! • More info: www.dragonsout.com • Also other books coming out 4
  • 5. ISTQB GLOBAL PRESENCE • Number of exams administered: over 1,2 million • Number of certifications issued: 914,000 • In 130 countries
  • 6. TMMi for test improvement in all kinds of testing, including agile and DevOps
  • 7. • Artificial Intelligence (AI) in short • Case of NASA code quality metrics using AI • Other AI and testing developments Agenda Copyright Knowit Solutions Oy 2021 7
  • 8. Why right now? Four drivers behind AI revolution 28.2.2024 8 © Copyright Knowit Solutions 2020 | Version 2.0 Computation growth due to general purpose GPUs The rise of Big data Community based achievements in Deep learning Open source tools and frameworks
  • 9. Quality management as top use-case for AI Source: AI software platform adoption survey, IDC, February 2019 [Percent of respondents: N=505] Copyright Knowit Solutions Oy 2021 9
  • 10. • When talking about AI, it is important to ask if we are talking about modelling humans or how to work in ideal way. • Machine learning is a sub- category of AI, which considers algorithms that enable AI to learn ideal way to work. Artificial intelligence == machine learning? Figure: Example of concepts related to AI. Copyright Knowit Solutions Oy 2021 10
  • 11. Three methods of machine learning Machine learning methods Supervised Learning • The aim is to learn mapping from input to output. • Output is known by supervisor. Unsupervised Learning • The aim is to find ”something interesting”. • Output is not known (known also as density estimation in statistics). Reinforcement Learning • Seemingly good solutions are encouraged to develop further. AI literature review[1] 1 Reagan, A., et al., The emotional arcs of stories are dominated by six basic shapes, EPJ Data Science (2016). Copyright Knowit Solutions Oy 2021 11
  • 12. No one tool for all Copyright Knowit Solutions Oy 2021 Performance Amount of data, time, and computing power • Linear discriminant analysis • Logistic regression • Medium-sized ANNs • Genetic algorithms • Extreme learning machines • Convolution neural networks • Generative neural networks • Deep belief networks • Recurrent neural networks • Simple neural networks 12
  • 13. Planning descriptions Code Unit testing Integration testing System testing Acceptance testing Definition descriptions Requirements • Static methods • Reviews • Dynamic methods • Test runs Remember there are different kinds of testing Copyright Knowit Solutions Oy 2021 13
  • 14. AI in software testing Copyright Knowit Solutions Oy 2021 • In static testing, several machine learning algorithms can be used to measure code quality.[1] NASA’s code review. • In dynamic testing, model-based regression tests can be replaced by algorithms that generate test flows on-the-fly.[2] Chaos monkey of Netflix. • In maintenance testing, machine learning can be used to plan more realistic statistical models of how application is actually used. Figure: Margaret Hamilton standing next to source code of guidance system of Apollo spacecraft. www.nasa.gov/ 14
  • 15. Toolbox for AI Copyright Knowit Solutions Oy 2021 15
  • 16. Anomaly detection use cases Copyright Knowit Solutions Oy 2021 16
  • 17. • Detecting: • abnormal behaviour • new kind of customers • etc. Anomaly detection Copyright Knowit Solutions Oy 2021 Figure: Example of time series 17
  • 19. Developer Product Owner Scrum team Management AI AI review AI code quality metrics Copyright Knowit Solutions Oy 2021 19
  • 20. • Goal is to understand technical quality and test coverage of modules. Also, goal is to guide and focus testing: • Focusing technical debt reduction • Found defects per module • Guide for test coverage in automating system level testing. AI code quality metrics Copyright Knowit Solutions Oy 2021 Requires that there is link from components (version control) and tests (test management) to feature descriptions and from reported defects to modules (observation control) Module Developer Benchmark (order.numb.) Relative risk Number of reported bugs Module_1 Jaakko 50 58 4 Module_2 Jaakko 4 5 1 Module_3 Mikko 180 95 7 SCRUM TEAM 20
  • 21. • Goal is to provide overview: • How fast the development is advancing • Maturity for release • Quality and technical debt of the software • Defect probabilities • Risks related to modifications of the software. AI code quality metrics Copyright Knowit Solutions Oy 2021 Interpretation: • Module 5 has high defect probability and is related to five most important features of the application. This results into high risk in release at this moment. • Defect probability of module 10 has fallen to acceptable level. Start-up phase Development phase Maturing phase MANAGEMENT 21
  • 22. Stages of execution 4. DEPLOYMENT AND UTILIZATION OF THE ALGORITHM ▪ Monitoring and analysing results 2. DEFINING QUALITY METRICS ▪ Defining quality and defect metrics ▪ Building the metrics 3. DEFINING HOW RESULTS ARE PRESENTED ▪ Defining who will use results and how ▪ Building data visualization 1. DEFINING DATA SOURCES ▪ From where data is collected ▪ How will the data be collected Copyright Knowit Solutions Oy 2021 22 Most effort
  • 23. Code quality metrics: NASA case Copyright Knowit Solutions Oy 2021 23
  • 24. • Several code quality metrics are currently being used. The most popular ones have been introduced in 1970.[1,2] 1 T.J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering (1976) SE-2, 4, 308. 2 M.H. Halstead, Elements of Software Science, Elsevier (1977). 3 N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous & Practical Approach, International Thompson Press (1997). 4 S.J. Sayyad, T.J. Menzies, The PROMISE Repository of Software Engineering Databases (2005). • Goal is to provide feedback: 1. for developers about code quality. 2. for testers about needed test coverage. 3. for management about the development project • Although traditional metrics are extensively used, benefit of using them has been questioned.[3] 0 50 100 150 200 250 300 350 400 450 0 20 40 60 80 100 120 Number of lines. Cyclomatic complexity Scattering of defects in NASA CM1 project OK Defected Figure: NASA Metrics Data Program, CM1 Project[4] In real applications defected modules do not separate from non-defected by standard metrics. AI code quality metrics Copyright Knowit Solutions Oy 2021 24
  • 25. 1 2 𝑤 2 + 𝐶 ෍ 𝑖=1 𝑁 ξ𝑖 1-𝑦𝑖 𝑤, 𝑓(𝑥𝑖) + 𝑏 ≤ ξ𝑖 Minimize Subject to with all values of i. Quality metric 1 Quality metric 2 • Machine learning approach was tested for NASA’s CM1 software repository[1]. • Support vector type of machine learning algorithm was developed for this demonstration. • This task can be solved by supervised machine learning methods because both input (quality metrics) and output (defects) is known. • Goal is to find such hyperplane in multidimensional data space that separate defected modules from non- defected. • Mathematically, problem can be described with algorithm: AI-guided software testing with NASA data Copyright Knowit Solutions Oy 2021 25
  • 26. 0,0 0,2 0,4 0,6 0,8 Tested on real data against NASA CM1 project[1] Performance (F) Training set: 1595 Test set: 400 𝐹 = 2 ∗ 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 ∗ 𝑟𝑒𝑐𝑎𝑙𝑙 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 + 𝑟𝑒𝑐𝑎𝑙𝑙 AI-guided software testing with NASA data Copyright Knowit Solutions Oy 2021 26
  • 27. Other interesting developments in AI and testing Copyright Knowit Solutions Oy 2021 27
  • 28. • Enhanced automation • Test automation • Robotic Process Automation -> Intelligent automation • AI-powered test automation • Efficiency by Generative AI • Test script creation • Manual test case creation • Test data creation • Enhanced test selection • Risk analysis • Test case selection • Code analysis General direction at businesses on AI and testing Copyright Knowit Solutions Oy 2021 28
  • 29. • Start from screenshots • Detect layout issues • Cross-browser testing for differences • Ignore invisible rendering, size, position differences • Case: Applitools • https://applitools.com / Visual AI • Also works as an AI library for test automation frameworks like Robot Framework AI-powered comparison of visual elements Copyright Knowit Solutions Oy 2021 29
  • 30. • AI-based test generation • Checking functional differences • Checking visual differences • Automatically learning the test automation • For: regression testing after changes • Case: retest • https://retest.de/ai-based-test- generation/ AI with regression testing Copyright Knowit Solutions Oy 2021 30
  • 31. • “World’s first” • Go automatically through mobile apps • Use general test cases • Reinforcement learning • Case: test.ai • https://www.test.ai/ AI-powered mobile test automation platform Copyright Knowit Solutions Oy 2021 31
  • 32. • Default tests to web sites • Teach your own tests • Running on containers in virtual machines • Use for regression testing, cross- browser testing • Case: Mabl • https://www.mabl.com/ Auto-healing automated testing of web Copyright Knowit Solutions Oy 2021 32
  • 33. • Build test through interface backed by ML • Plain English • New life for “record-playback” • ML algorithms maintain the tests • Case: Functionize • https://www.functionize.com/ Intelligent testing Copyright Knowit Solutions Oy 2021 33
  • 34. • Automatically scope and generate optimized test cases • Generate test scripts for automated execution from models and manual tests • Display the test coverage requirements • Integrate with existing tools • Case: Conformiq • https://www.conformiq.com/ AI-enabled test creation from models Copyright Knowit Solutions Oy 2021 34
  • 35. • Enhance core test automation • Develop automated test scripts • Regression testing • Build test environments rapidly • Service focus • Case: Mammoth • https://www.mammoth-ai.com/ • https://www.mammoth-ai.com/artificial-intelligence/ Enhancing test automation with AI Copyright Knowit Solutions Oy 2021 35
  • 36. • Support test automation engine data selection with AI • Can be used for testing with large quantities of data • Can be used automating production as Robotic Process Automation (RPA) • Cases: • Robot Framework test automation framework • Aito AI tool • https://robotframework.org/ • https://aito.ai/ • https://aito.ai/blog/machine-learning-in-robot-framework/ Automating tests or production with lots of repetitive data Copyright Knowit Solutions Oy 2021 36
  • 37. • Supports test automation object recognition • Helps with typical GUI recognition problems for test automation • Can be used as a support library for existing test automation • Cases: • ImageHorizonLibrary is a cross-platform library for Robot Framework. • Eficode, Oulu University and Business Finland • https://www.eficode.com/projects/testomat Image recognition with AI for open source test automation Copyright Knowit Solutions Oy 2021 37
  • 38. • Python-editor PyCharm has Refact.ai library that uses ChatGPT • Understands Python and Robot Framework test automation setup • Refact.ai is labelled as an AI coding assistant software • https://refact.ai/ Python enhanced with ChatGPT Copyright Knowit Solutions Oy 2021 38
  • 39. • Assist test automation script creation • Suggest what the user might want • What kind of code • What kind of test script • User reviews and approves suggestions • Based on Generative AI • https://github.com/features/copilot Github Copilot Copyright Knowit Solutions Oy 2021 39
  • 40. • Python-editor PyCharm can use Dev Tools AI library • Asks which element to click on the screen and then keeps it running • Suggests how to fix semiautomatically a changed test script that doesn’t pass any more • Reduces the need for test script maintenance • Supports e.g. Cypress, Selenium, Appium, Playwright • https://www.dev-tools.ai/ AI library helps pass Python test scripts Copyright Knowit Solutions Oy 2021 40
  • 41. • Assist extending existing test automation sets • With e.g. Robot Framework, Playwright, Selenium, Cypress • Use tests as self-driving test- bots • Exploring web apps • Regression testing • Autonomous testing • https://wopee.io/ Test automation assistants and bots Copyright Knowit Solutions Oy 2021 41
  • 42. • Derive tests from specifications • Create tests based on user input • Enabled by good prompt engineering • Challenge: repeated teaching • User reviews AI proposals and uses in their tests • https://chat.openai.com/ (ChatGPT) • https://gemini.google.com/ • https://copilot.microsoft.com/ (Bing) Generative AI helps in test creation Copyright Knowit Solutions Oy 2021 42
  • 43. • Go through functionalities • Identify risky areas • Focus testing • Case: Eggplant • https://www.eggplantsoftwar e.com/ Predict quality issues Copyright Knowit Solutions Oy 2021 43
  • 44. • Analyze risk in commits to the code • Select the tests that test the risky areas • Reduce number of test cases needed to run • Case: Appsurify • https://appsurify.com/ Select most suitable test cases Copyright Knowit Solutions Oy 2021 44
  • 45. • Power of open source developers • Machine learning algorithms learn from the community automatically • Analyse code and propose improvements • Case: DeepCode • https://www.deepcode.ai/ Code analysis using AI Copyright Knowit Solutions Oy 2021 45
  • 46. • Trainings and certifications • https://www.istqb.org/certification-path-root/ai-testing.html • Standards • ISO • AI/ML standard https://www.iso.org/standard/74438.html • Testing of AI-based systems https://www.iso.org/standard/79016.html • IEEE • What is AI Software Testing https://ieeexplore.ieee.org/document/8705808 • AI testing perspectives https://ieeexplore.ieee.org/document/9514942 • Application of AI in Testing https://ieeexplore.ieee.org/document/9676244 Promoting and teaching AI and testing Copyright Knowit Solutions Oy 2021 46
  • 47. • AI in Testing: Impact, Problems, Challenges and Prospect https://www.researchgate.net/publication/357876318_Artificial_Inte lligence_in_Software_Testing_Impact_Problems_Challenges_and_Pro spect • Utilizing AI in Software Testing https://www.theseus.fi/handle/10024/263992 • AI Applied to Software Testing https://dl.acm.org/doi/10.1145/3616372 • AI Applied to Testing: A Literature Review https://ieeexplore.ieee.org/abstract/document/9141124 • ChatGPT helps testing https://www.linkedin.com/pulse/gpt-4-sdlcs- secret-weapon-reinventing-testing-phase-andy-abbott/ Some more research notes Copyright Knowit Solutions Oy 2021 47
  • 48. • AI-enabled testing is already a reality • Many companies are enhancing their solutions with AI • New companies are set up around AI • AI can provide simplicity to complex software development projects. • With Generative AI, manual and automated test script creation is more productive • With AI, testing activities can be focused on high-risk areas. • With AI, test automation becomes more autonomous • Testers are freed to create new tests Conclusion Copyright Knowit Solutions Oy 2021 48