The document details a guest lecture on test amplification in Python, presented by Prof. Serge Demeyer, emphasizing the importance of software testing in detecting errors. It discusses the implementation of test amplification techniques, such as mutation coverage and examples of how to strengthen test cases, resulting in improved mutation scores and code coverage. Key findings include the success of the Amplifier tool in enhancing test validation in various software projects.

1. Test Amplification in Python
An Industrial Experience Report
Prof. Serge Demeyer
Guest lecture, Mälardalens Universitet, Västerås, Sweden
October2022

2. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Testing
2
Program
Under
Test
Input
Expected output
Software Testing is the process of executing a program or system with
the intent of finding errors.
(Myers, Glenford J., The art of software testing. Wiley, 1979

3. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Coverage
3
Program
Under
Test
Input Expected output
(mutation)
coverage

4. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Test Amplification
4
Program
Under
Test
Input Expected output
(mutation)
coverage
+coverage
Extra Input
Extra Input
+Extra Input +Extra output

5. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Example - testDeposit
5
1 def testDeposit (self) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.get_balance(), 10)
5 self.b.deposit(100)
6 self.b.deposit(100)
7 self.assertEqual(self.b.get_balance() , 210)
I
n
p
u
t
Expected
output

6. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Example - testDeposit_amplified (1/2)
6
1 def testDeposit_amplified (self) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse(self.b.is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
…
Assertion
Am
plification

7. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Example - testDeposit_amplified (2/2)
7
1 def testDeposit_amplified (self) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse(self.b.is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
9 with self.assertRaises(Exception):
10 self.b.deposit(−56313)
11 self.b.deposit(100)
12 self.b.set_owner(’Guido van Rossum’)
13 self.assertEqual(self.b.
14 get_transactions(), [10])
…
Input Am
plification

8. Guest Lecture - Mälardalens Universitet © Serge Demeyer
AmPyfier Engineering
8
AmPyfier: Test amplification in Python
Journal of Software Evolution and Process 2022
https://doi.org/10.1002/smr.2490

9. Guest Lecture - Mälardalens Universitet © Serge Demeyer
AmPyfier: Validation
9
Strengthened
37
out of 54
test classes

10. Guest Lecture - Mälardalens Universitet © Serge Demeyer
SmallAmp: Validation
10
Strengthened
86
out of 156
test classes

11. 11 pull requests
8 merged
3 pending

12. Data
The planner
AI

15. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Python Project
15
20%
80%
Project
Test Suite
• Start-up established in 2020
• Fast-growing
- more customers
- more investors
- Single variant
• AI engine is vital
(Unit) Tests

16. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Data Driven Test Amplification
16
Amplification operators must manipulate .csv files
import ampyfier
@ampyfier.amplify_csv ( "CSVPATH" )
def test_something(amp_file="something.csv"):
...

17. 360 minutes

18. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Experimental Set-Up
18
Daily
Ampyfier executes
tests on main
branch
Weekly
Ampyfier
generates new
tests

19. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Initial Amplification
19
Improvement for 13/25 files
• Mutation Score: 20.29%
• Code Coverage: 41.39%

20. Guest Lecture - Mälardalens Universitet © Serge Demeyer
Continuous Amplification
20
Improvement for all
• Mutation score: 7/8 ≈ 87%
• Code coverage: 5/8 ≈ 63%
• Both: 4/8 ≈ 50%