Testify AS provides model-based testing and machine learning-based testing services. They presented a concept for smart test optimization using historical test execution data from the ecFeed platform. Their approach applies search-based algorithms like NSGA-II to prioritize test cases based on cost and effectiveness measures derived from the data to find an optimal test set. They implemented a prototype and plan to evaluate it on real customer data from ecFeed, and to explore additional machine learning techniques going forward.

1. 03.12.2021 1
Testify AS
Testify - ML-enthusiasts
Shuai Wang - Senior Test Engineer
 PhD. from Simula Research Lab & UiO
 Search-based software testing; Model-
based testing; Machine learning based
testing.
Minh Nguyen - Principal Test Engineer
 PhD. from NTNU
 Test Automation; Model-based
testing; Machine Learning based
testing.

2. 03.12.2021 2
Model-based testing (MBT)
Testify AS
Requirements
Model Test
Oracle
Test Specification
(abstract test
cases)
Test Script
(executable
test cases)
SUT
Manual
Automated
specification
Automated
derivation
Automated
generation
Automated
execution
Automated
evaluation
UML
Java
XML
etc.
(+) Automatic and systematic
generation of test cases
(+) Adjustable test coverage level
(+) Traceability from requirement to
test case
(+) Low test maintenance cost
(-) Complex modeling notations
(-) High lisence cost – tightly
integrated with comprehensive
tool sets
(-) Often support test generation
only

3. Smart Test Optimization
with ecFeed
Odin 2020
Shuai and Minh

4.  Problem definition
 Concept and implementation
 Further work
03.12.2021 4
Agenda
Testify AS

5. 03.12.2021 5
Problem definition
Testify AS
03.12.2021 5
Testify AS
Test case optimization:
- Input:
- A set of test cases to be executed
- Historical test cases execution data
- Output: An optimal set of test cases based on
pre-defined cost and effectiveness measures for a
given context.
That includes:
 Test case selection
 Test suite minimization
 Test case prioritization
TC1
TC2
TC3
TC4
TC5
TC6
TC7
TC8
TC9
TC10
TC11
TC12
TC13
TC14
TC15
TC1
TC5
TC7
TC10
TC13
TC12
TC15
TC2
Changes
Optimal set
TC5 TC1 TC7
TC12 TC2
Optimal and ordered set

6. 03.12.2021 6
ecFeed Platform http://ecfeed.com/
Model
Intuitive,
powerful and
expressive
Modeling
Test cases
Intelligent algorithms,
Optimal or scalable
test coverage
Test Generation
Test Runners
Test Execution
Standard or
customized formats
Data Export
Standard or customized
test execution data points
Collect & Analysis

7. Concept and Implementation
03.12.2021 Testify AS 7
Customer SUTs
Test execution
data points
Execute test cases
(step 1)
Data repository (step 2)
Test optimization
applications (step 3)
Support smart
testing
 Execution time
 Execution verdict (pass/fail)
 Detailed fault info
 Coverage (e.g., code)
 Configuration info
 …
 Cost and effectiveness measures
 Data characteristics
 Data statistics
 …
 Test case prioritization
 Test suite minimization
 Test selection
 …
Store and analyze
execution data
Input historical data

8. 03.12.2021 8
Search-based test case prioritization
Testify AS
Fitness
Function
https://www.researchgate.net/publication/228671024_Search_Based_Software_Engineering_A_Comprehensive_Analysis_and_Revie
w_of_Trends_Techniques_and_Applications

9. • Cost Measure:
 TET: Total execution time for the prioritized test cases, 𝑇𝐸𝑇 = 𝑖=1
𝑛𝑡𝑠
𝐸𝑇𝑡𝑐𝑖
• Effectiveness Measures:
 PD: Prioritization density to measure how many test cases have been prioritized, 𝑃𝐷 =
𝑛𝑡𝑠
𝑛𝑡
 FDC: Fault detection capability, 𝐹𝐷𝐶 = 𝑖=1
𝑛𝑡𝑠 𝐹𝑎𝑖𝑙𝑅𝑡𝑐𝑖
𝑛𝑡𝑠
o Rate of fail executions within given period/context (e.g., a week, cycle, sprint)
• The three objectives are integrated into various search algorithms such
as Non-dominated Sorting Genetic Algorithm II (NSGA-II)
• The technique was developed based on open-source multi-objective
optimization framework jMetal (http://jmetal.sourceforge.net/algorithms.html)
03.12.2021 9
Search-based test case prioritization
Testify AS

10.  Conclusion
 Applied intelligent search algorithms to solve the test optimization problem
(test case prioritization).
 Based on constructed historical test execution data.
 Future
 Ongoing: extract real historical test execution data extracted from ecFeed.
 Near future:
 Apply other machine learning techniques (e.g., reinforcement learning).
 Seek more comprehensive industrial customers (case studies).
03.12.2021 10
Wrap-up
Testify AS