The document discusses using artificial intelligence and mathematical models in software testing. It proposes using a neural network trained on test case data to act as an automated test oracle that classifies test results as passed or failed. A mathematical model is introduced to represent the test case execution process. An algorithm is also constructed for a comparison tool to analyze results from the neural network test oracle and the actual tested software. The approach aims to help with regression testing of software by automating some of the decision making.

1. Taras Lytvyn
http://testers.lviv.ua/
https://twitter.com/djlicker
http://ami.lnu.edu.ua/kdais/employees.html
© 2012GlobalLogic Inc. 1

2. Mathematical models and
artificial intelligence in
software testing
Taras Lytvyn
processes
© 2012GlobalLogic Inc. 2

3. Contents
- What is artificial intelligence?
- Problem statement in general
- Analysis of recent researches & technologies
- Main idea of artificial intelligence approach
- Math model
- Test oracle based on AI
- Result evaluation and classification.
Comparison tool algorithm.
© 2012GlobalLogic Inc. 3

4. What is artificial intelligence ?
Semiotic Biological
thinking, Neural networks
judgment,
language,
emotions,
creativity, etc. Intellectual
mental behavior
processes
© 2012GlobalLogic Inc. 4

5. Problem statement in general
SIMPLE TEST PROCESS
How well an evaluated app. conforms to its specs.
3 stages:
- test data generation,
- testing, Regression
- result evaluation testing
PROBLEM OF HIDDEN ERRORS
© 2012GlobalLogic Inc. 5

6. Analysis of recent researches & techs
WHERE AI CAN BE USED ?
- metric’s analysis
- cost of testing
- reliability of testing in
general
- optimization processes &
data optimization
© 2012GlobalLogic Inc. 6

7. Main idea
Decisions during
Regression Testing
MANUAL AUTOMATION
Assert functions
(Actual is equal/not
equal to expected)
Intellectual automation
Decision System during
Regression Testing
© 2012GlobalLogic Inc. 7

8. Main idea
Decision System is based on artificial intelligent approach and is
build on algorithm with NEURAL NETWORK usage.
Neural Network training process
© 2012GlobalLogic Inc. 8

9. Main idea
Tested
inputs it works ?
Software
Test outputs
cases
Neural Trained neural network
inputs
Network that will emulate our
SOFTWARE workability
© 2012GlobalLogic Inc. 9

10. Math Modeling usage
The test case execution process should be presented as
Complex Math Model
- Neural Network needs digits (not text)
- Input and output data should be normalized in specific way
- Math Model will give some limitations
Theorem & Limitations
V ji couldn’t be empty Functions
There is no intermediate step, TCE(t ji ) : Sl ji Vl ji , l k (k m)
if a corresponding stage of verification exist TCP(Sl ji ) : Sl 1 ji Vl 1 ji ,...,S1 ji V1 ji , l 1.
Intermediate stage of verification is possible TCP(S1 ji ) : S1 ji V1 ji .
© 2012GlobalLogic Inc. 10

11. AI Test Oracle
Test oracle – is a classifier that shows us
whether test was passed or not
Tested
Software
Comparison
Test Tool Result
Cases Algorithm classifications
Trained
Neural
Network
Test oracle classify the result
of test case execution
© 2012GlobalLogic Inc. 11

12. Result classification & Comparison Tool
The Comparison Tool is employed as an independent method of
comparing results from neural network and the results of the
tested versions of app.
App. result
Neural
Network
result Correct Incorrect
Correct 1 True Positive 2 True Negative
Incorrect 4 False Negative 3 False Positive
Comparison of outputs
Output type Same Different
Both correct ANN correct
Binary
Both wrong APP correct
ANN correct
Both Correct
Continuous APP correct
Both wrong
Both wrong
© 2012GlobalLogic Inc. 12

13. Example
Input Data (test case) should be NORMALIZED !!!
TC Test Verification (test
TC Title (test Test steps (test case
id case verification) /
case name) steps) / Action
(№) Expected result
1. Input Regular 1. Regular Hours
Hours should be
2. Input Age displayed
3. Input Rate of Pay 2. Age should be
Gross Pay
1.2 4. Click Calculate displayed
View
3. Rate of Pay should
be displayed
4. Gross Pay should
be displayed
Structure of
ANN is based Training phase with 50 – 5000
on Kolmogorov records
Arnold
approach
© 2012GlobalLogic Inc. 13

14. - Introduced a mathematical model describing the
test case execution for software that is tested
- Constructed a new algorithmic model of test
oracle based on neural networks
- This Model can be used in the processes of
regression testing software
- Comparison Tool as result analyzer was provided
© 2012GlobalLogic Inc. 14

15. <Q&A?>
© 2012GlobalLogic Inc. 15

16. Taras Lytvyn
automation QC in Global Logic inc.
email : taras.lytvyn@globallogic.com
web: http://testers.lviv.ua/automation_blog
skype: tasryk
© 2012GlobalLogic Inc. 16