DS3 is a technique that decomposes complex system test cases into simpler test cases using log-based program slicing. It performs the following steps: 1. Uses static slicing to initially decompose test cases based on assertions. 2. Analyzes test case execution logs to identify dependencies between statements and global resources. 3. Refines the static slices by adding missing dependencies identified in the logs. 4. Merges slices that share all non-assertion statements to minimize duplication. An evaluation on two large systems found that DS3 produced well-formed test case slices and significantly reduced test case execution times compared to the original complex test cases and standard static slicing. The sliced test cases also maintained similar

1. Log-Based Slicing for
System-Level Test Cases
Salma Messaoudi1, Donghwan Shin1, Annibale Panichella1,2,
Domenico Bianculli1, and Lionel Briand1,3
1
2
3

2. 2
Regression Testing
Introdution
• Arguably one of the most important activities in software testing
• However, its cost-effectiveness can be largely impaired by the cost of individual test cases
Quality assurance technique applied when changes are merged to an existing codebase
Test Case x
Test Case y
Test Case z
2.5 hours
4 hours
2 hours
Even if a perfect test case prioritization technique is applied,
no faults could be detected during the first few hours

3. 3
Observation: Complex System Test Cases
Introduction
• Often very expensive (e.g., took 1h to run a single test case)
• Why?
• Trigger (and wait for) physical components that take a lot of time to complete
• Poorly designed (e.g., containing multiple test scenarios combined into a single test case)
From a collaborative industrial research project
• The execution time of the decomposed test cases would decrease
• The cost-effectiveness of test case prioritization could improve
• Furthermore, providing finer-grained information about test results would facilitate debugging
activities, such as fault localization
What if we decompose complex system test cases?!

4. 4
Simple Idea: Program Slicing on System Test Cases
Introduction
• Decompose a complex system test case containing multiple test scenario into separate ones,
each of them with only one test scenario and its corresponding assertions
Static Slicing on line (4)
Static Slicing on line (7)
Decomposed
Test Case 1?
Decomposed
Test Case 2?
def test_example():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Simplified Example System Test Case

5. 5
Limitation of Static Slicing
Introduction
‘output.csv’ is written by line (1)
and read by lines (2) and (3)
Static Slicing on line (4)
Not executable due to
the missing “hidden” dependency!
def test_example():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Example System Test Case
def test_static_01():
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
Sliced Test Case

6. 6
Other Program Slicing Approaches?
Introduction
• Dynamic slicing*
• Require to instrument the source code and to collect coverage information
• Not feasible for a system composed of 3rd-party components for which the source code is not available
• Does not address the problem of “hidden” dependencies as they are not captured by code coverage
• Observation-based slicing**
• Require to run each system test case multiple times
• Not applicable for time-consuming system test cases (e.g., the ones developed by our industrial partner)
* Bogdan Korel and Janusz Laski. 1988. Dynamic program slicing. Information processing letters 29, 3 (1988), 155–163
** David Binkley, Nicolas Gold, Mark Harman, Syed Islam, Jens Krinke, and Shin Yoo. 2014. ORBS: Language-independent program slicing. In
Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 109–120.

7. 7
New Idea: Leaveraging Existing Logs to Refine Static Slices
Approach
• There are test case execution logs, obtained from past regression testing sessions
• The logs include run-time information about the system under test
• There is a traceability between test case statements and log messages (if not, test cases can be instrumented)
• The log contains information on the usage of global resources (if not, a watchdog process can be used)
• We can extract the global resources accessed (e.g., files, network connections) and the actions
performed (e.g., read/write files, open/close connections) upon executing each test statement
...
20210526:10:00:01 INFO [test_example.py:line 1] write ‘output.csv’
20210526:10:00:01 INFO [test_example.py:line 1] fdm initialized
20210526:10:00:02 INFO [test_example.py:line 2] read ‘output.csv’
20210526:10:00:05 INFO [test_example.py:line 3] read ‘output.csv’
...
Example log

8. 8
Our Approach: Decomposing System teSt caSes (DS3)
Approach
System
Test Case
System Test
Cases Logs
Candidate Sliced
Test Cases
Static Slicing
Perform static slice on assertions
1
Def-Use Info for
Global Resources
Def-Use Analysis for
Global Resources
Identify (stmt, entity, def/use)
2
Refined Sliced
Test Cases
Log-based
Slice Refinement
Add missing statements
3
Slice
Minimization
4
Final Sliced
Test Cases
DS3

9. 9
Step 1: Backward Static Slicing
Approach
Using standard static slicing approaches
def test_example():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Example System Test Case
def test_static_01():
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
Sliced Test Case 1 (based on line 4)
def test_static_02():
(2) ref = read_csv(‘output.csv’)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Sliced Test Case 2 (based on line 7)

10. 10
Step 2: Def-Use Analysis for Global Resources
Approach
Based on the execution logs
...
20210526:10:00:01 INFO [test_example.py:line 1] write ‘output.csv’
20210526:10:00:01 INFO [test_example.py:line 1] fdm initialized
20210526:10:00:02 INFO [test_example.py:line 2] read ‘output.csv’
20210526:10:00:05 INFO [test_example.py:line 3] read ‘output.csv’
...
Example log
(line 1, ‘output.csv’, def)
(line 2, ‘output.csv’, use)
(line 3, ‘output.csv’, use)
Identify (statement, global resources, def/use)
…
Analysis Results
based on keywords
Lines 2 and 3 depends on (à) line 1

11. 11
Step 3: Log-based Slice Refinement
Approach
To add missing “hidden” dependencies into static slices
def test_static_01():
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
Sliced Test Case 1 (based on line 4)
def test_static_02():
(2) ref = read_csv(‘output.csv’)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Sliced Test Case 2 (based on line 7)
def test_static_01():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
Refefined Sliced Test Case 1
def test_static_02():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Refined Sliced Test Case 2
Lines 2, 3 à line 1
Line 2 à line 1

12. 12
Step 4: Slice Minimization
Approach
Based on our observations in real-world codebases
Observation: If all non-assertion statements of a slice are
also in another slice, both slices share the same test
scenario, and therefore should be merged
def test_static_01():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(3) sim = deploy_proc(‘output.csv’)
(4) self.assertEqual(ref, sim)
Refefined Sliced Test Case 1
def test_static_02():
(1) fdm = create_fdm_setup()
(2) ref = read_csv(‘output.csv’)
(5) new = run_ic()
(6) diff = FindDiffs(ref, new, 1E-8)
(7) self.assertEqual(len(diff), 0)
Refined Sliced Test Case 2
Not a subset of the other
The slices are final!

13. 13
Evaluation
Evaluation
• Aim to evaluate following aspects of DS3:
• Slicing effectiveness (compared to vanilla static slicing)
• Execution time of sliced test cases (compared to its original)
• Test effectiveness of sliced test cases (compared to its original)
• Evaluation subjects
Programming Language Number of Test Cases
Prop Python 30
JSBSim C++ and Python 81 (76)
Table 1. Evaluation Subjects Summary

14. 14
Research Questions
Evaluation
• RQ1: How effective is DS3 in slicing system test cases compared to standard static slicing?
• RQ2: How efficient are the slices produced by DS3 compared to the original test cases?
• RQ3: What is the code coverage and fault detection capability of the slices produced by DS3
compared to the original test cases?

15. 15
RQ1: Slicing Effectiveness
Evaluation
• The standard static slicer has significantly lower slicing effectiveness than DS3
• Leveraging the global resource usages reported in the logs, DS3 is able to generate well-formed
slices in all cases
System # TCs Appr.
# Slices
SlicingEff
(Pass/Total)
Total Pass Fail
Prop 30
DS3 137 137 0 1.00
Static Slicer 166 40 126 0.24
JSBSim 76
DS3 84 84 0 1.00
Static Slicer 169 56 113 0.33
Table 2. Slicing Effectiveness Comparision Results

16. 16
RQ2: Execution Time
Evaluation
• On average, the execution time of a sliced test case
(239.6s) is much less than that of the original test
case (3196.9s)
• There are some cases where the total execution
time of all slices for a system test case is greater
than that of the original test case
• Because there are the same “fixtures” shared by
multiple slices
• For some cases (e.g., TC30), even the total
execution time of all sliced test cases is much less
than that of the original
• Because the original has “expensive” statements that
actually do not have any dependencies on others
System
Test Case #Slices
Execution Time (s)
Original S-Avg S-Total
TC1 2 1139 520.0 1040
TC2 2 498 342.5 685
TC3 3 245 132.6 398
TC4 1 330 201 201
… … … … …
TC29 10 680 99.1 991
TC30 5 23231 200.8 1004
Average 4.9 3195.9 239.6 967.1
Table 3. Execution Time Results

17. 17
RQ3: Test Effectiveness
Evaluation
• There is no significant difference, meaning sliced test cases are as effective as their original test
cases in terms of coverage and fault detection
• The minor difference is because, again, some of the original test cases include statements that actually
do not have any dependencies on others
Test Case Covered
Branches
Covered
Functions
Killed
Mutants
JSBSim
Original 5736 1831 289
Sliced 5698 1831 288
Table 4. Test Effectiveness Results

18. 18
Conclusion
Conclusion
• DS3 can automatically slice expensive system test cases into far less expensive sliced test cases
• Slicing test cases rarely decrease the effectiveness compared to their original test cases
• Sliced test cases can be further utilized by, for example, test case prioritization and test suite
selection, to increase the cost-effectiveness of regression testing

19. 19
Log-Based Slicing for System-Level Test Cases
@SnT_uni_lu
SnT, Interdisciplinary Centre for
Security, Reliability and Trust
Connect with us
Donghwan Shin
Research Scientist
donghwan.shin@uni.lu
Salma Messaoudi, Donghwan Shin, Annibale Panichella,
Domenico Bianculli, and Lionel Briand
Contact: