Talk from IWST 2023

1. 1
1
1
Evref
fervE
A Unit Test Metamodel
for Test Generation
Gabriel Darbord¹
Anne Etien¹
Nicolas Anquetil¹
Benoit Verhaeghe²
Mustapha Derras²
¹Univ. Lille, CNRS, Inria, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France
²Berger-Levrault, France
IWST 2023
Evref
fervE

2. 2
2
2
Evref
fervE
The Importance of Testing
§ Nowadays, when developping new software systems:
§ 20-50% time spent on testing
§ We test because we want:
§ Bug detection and prevention
§ Quality assurance
§ User satisfaction
§ Non-regression
§ Confidence
§ Etc.

3. 3
3
3
Evref
fervE
Legacy Software System Lack Tests
§ In 2022, Berger-Levrault owns 150 software programs
§ Three-tier architectures (client, server, database)
§ Millions of lines of code
§ Different legacy technologies that can be up to 25 years old
§ Severe lack of tests
§ Developers fear changes

4. 4
4
4
Evref
fervE
Towards Automated Test Generation
§ Berger-Levrault wants tests for their legacy software
§ Impossible to write them manually
§ Time consuming
§ Difficult and error-prone
§ Lack of resources

5. 5
5
5
Evref
fervE
Our Test Generation Approach
§ Using software models and execution traces
§ static and dynamic analysis
§ Our objective is to generate tests that are:
§ Relevant
§ Readable
§ Maintainable
§ Not relying on existing tests

6. 6
6
6
Evref
fervE
Different Criteria
Paper Relevant Readable Maintainable
J. Pires et al. ~ ~ +
G. Fraser et al. ~ - ~
A. C. R. Paiva et al. + ~ ~
M. Tufano et al. ~ + ~
LLM approaches need to be trained on codebase and existing tests

7. 7
7
7
Evref
fervE
About the Moose Platform
§ Moose is a platform for software analysis
§ It allows to:
§ Represent a software system in a model
§ Query, manipulate, transform, and visualize models

8. 8
8
8
Evref
fervE
An Approach Based on Metamodels
Codebase Traces
Value
Model
Code
Model
Unit Test
Model
Unit Test
M0
Reality
M1
Models
M2
Metamodels
Value
Metamodel
Code
Metamodel
Unit Test
Metamodel
analyze extract generate
build
conformsTo conformsTo conformsTo
produce

9. 9
9
9
Evref
fervE
Unit Test Metamodel
TestSuite TestCase TestMethod
Fixture
SetUp
TearDown Assertion
before
beforeAll
beforeEach
cleanups
arrangements
*
Value
*
* *
*
*
expected
arguments
valuesToHandle
Class Method
*
testedClass testedMethod
* unitTests
Act
after
afterAll
afterEach
* *
*

10. 10
10
10
Evref
fervE
Value Metamodel
Value
Dictionary Primitive
Collection Object
EnumValue
Unknown
Association
dictionaryValue
dictionaryKey
* *
*
*
attributes
*
references
*
*
*
*
capturedValues
UnknownType
Type
TypedEntity
EnumValue
*
elements
source

11. 11
11
11
Evref
fervE
Example Value Model
[{"name": "John Doe"}]
Set<User> getUsers()
Produces trace Modelized
Code
Value
"John Doe"
:Primitive
:Collection
:Object
HashSet<User>
:Type
getUsers()
:TypedEntity
User :Type
String :Type
name
source
type
type
type

12. 12
12
12
Evref
fervE
Build Unit Test Using Model Transformations
Code
Model
Method
Parameter
Class
Unit Test
Model
TestCase
TestMethod
Arrange
Act
Assert
Value
Model
Argument
Result

13. 13
13
13
Evref
fervE
Build Unit Test Using Model Transformations
Code
Model
Method
Parameter
Class
Unit Test
Model
TestCase
TestMethod
Arrange
Act
Assert
Value
Model
Argument
Result

14. 14
14
14
Evref
fervE
Build Unit Test Using Model Transformations
Code
Model
Method
Parameter
Class
Unit Test
Model
TestCase
TestMethod
Arrange
Act
Assert
Value
Model
Argument
Result

15. 15
15
15
Evref
fervE
Build Unit Test Using Model Transformations
Code
Model
Method
Parameter
Class
Unit Test
Model
TestCase
TestMethod
Arrange
Act
Assert
Value
Model
Argument
Result

16. 16
16
16
Evref
fervE
Build Unit Test Using Model Transformations
Code
Model
Method
Parameter
Class
Unit Test
Model
TestCase
TestMethod
Arrange
Act
Assert
Value
Model
Argument
Result

17. 17
17
17
Evref
fervE
Build Unit Test Using Model Transformations
Code
Model
Method
Parameter
Class
Unit Test
Model
TestCase
TestMethod
Arrange
Act
Assert
Value
Model
Argument
Result

18. 18
18
18
Evref
fervE
Export Code Using Abstract Syntax Trees
M0
Reality
M1
Models
M2
Metamodels
Unit Test
Model
Unit Test
Metamodel
conformsTo
AST
Metamodel
AST
Model
Unit Test
conformsTo
export refactor
generate

19. 19
19
19
Evref
fervE
Refactoring Abstract Syntax Trees
Statements
VarDecl
Variable
name
Primitive
"John Doe"
MethodCall
setName
Variable
user
Variable
name
String name = "John Doe";
user.setName(name);

20. 20
20
20
Evref
fervE
Refactoring Abstract Syntax Trees
Statements
VarDecl
Variable
name
Primitive
"John Doe"
MethodCall
setName
Variable
user
Variable
name
Statements
Primitive
"John Doe"
MethodCall
setName
Variable
user
Inline constants
String name = "John Doe";
user.setName(name);
user.setName("John Doe");

21. 21
21
21
Evref
fervE
Example of Generated Test for JUnit
class UserManager {
Set<User> users;
Set<User> usersByName(String name) {
Set<User> result = new HashSet<>();
for (User user: users)
if (user.getName().equals(name))
result.add(user);
return result;
}
}
class UserManagerTest {
UserManager manager = new UserManager();
@Test void testUsersByName() {
Set<User> expected = new HashSet<>();
User user = new User();
user.setName("John Doe");
expected.add(user);
Set<User> actual = manager
.usersByName("John Doe");
assertEquals(expected, actual);
}
}
Application code Generated test code

22. 22
22
22
Evref
fervE
Future Work
§ Pursue test quality
§ Relevant, readable, maintainable…
§ Evaluate our approach in Pharo, thanks to our community
§ Compare existing tests to generated tests
§ Generate unit tests for our industrial partner Berger-Levrault
UT

23. 23
23
23
Evref
fervE
8
8
8
Evref
fervE
An Approach Based on Metamodels
Codebase Traces
Value
Model
Code
Model
Unit Test
Model
Unit Test
M0
Reality
M1
Models
M2
Metamodels
Value
Metamodel
Code
Metamodel
Unit Test
Metamodel
analyze extract generate
build
conformsTo conformsTo conformsTo
produce
21
21
21
Evref
fervE
Example of Generated Test for JUnit
class UserManager {
Set<User> users;
Set<User> usersByName(String name) {
Set<User> result = new HashSet<>();
for (User user: users)
if (user.getName().equals(name))
result.add(user);
return result;
}
}
class UserManagerTest {
UserManager manager = new UserManager();
@Test void testUsersByName() {
Set<User> expected = new HashSet<>();
User user = new User();
user.setName("John Doe");
expected.add(user);
Set<User> actual = manager
.usersByName("John Doe");
assertEquals(expected, actual);
}
}
Application code Generated test code
5
5
5
Evref
fervE
Our Test Generation Approach
§ Using software models and execution traces
§ static and dynamic analysis
§ Our objective is to generate tests that are:
§ Relevant
§ Readable
§ Maintainable
§ Not relying on existing tests
9
9
9
Evref
fervE
Unit Test Metamodel
TestSuite TestCase TestMethod
Fixture
SetUp
TearDown Assertion
before
beforeAll
beforeEach
cleanups
arrangements
*
Value
*
* *
*
*
expected
arguments
valuesToHandle
Class Method
*
testedClass testedMethod
* unitTests
Act
after
afterAll
afterEach
* *
*
Conclusion