With the current emphasis on DevOps, automated software tests become a necessary ingredient for continuously evolving, high-quality software systems. This implies that the test code takes a significant portion of the complete code base — test to code ratios ranging from 3:1 to 2:1 are quite common. We argue that "testware'" provides interesting opportunities for formal verification, especially because the system under test may serve as an oracle to focus the analysis. As an example we describe five common problems (mainly from the subfield of mutation testing) and how formal verification may contribute. We deduce a research agenda as an open invitation for fellow researchers to investigate the peculiarities of formally verifying testware.

1. Formal Verification of Developer Tests
a Research Agenda Inspired by Mutation Testing
Serge Demeyer, Ali Parsai, Sten Vercammen, Brent van Bladel, and Mehrdad Abdi
ISoLA 2021
9th International Symposium On
Leveraging Applications of Formal Methods, Verification and Validation

2. ISoLA 2021 © Serge Demeyer 2
Test Automation
<<Breaking the build>>

3. ISoLA 2021 © Serge Demeyer 3
50%
- 75%
of code
devoted to (unit) tests!

4. ISoLA 2021 © Serge Demeyer 4
Supporting
Team
Functional Tests
Examples
Story Tests
Prototypes
Simulation
Exploratory Testing
Scenarios
Usability Testing
Acceptance Testing
Alpha / Beta
Unit Tests
Integration Tests
Performance Testing
Load Testing
Security Testing
“ility” Testing
Technology Facing
Business Facing
Critique
Product
Automated
& Manual
Manual
Autom
ated
Tools

5. ISoLA 2021 © Serge Demeyer 5
Quis custodiet ipsos custodes?
(Who will guard the guardians?)

6. ISoLA 2021 © Serge Demeyer 6
Quis custodiet ipsos custodes?
(Who will guard the guardians?)
Formal Verification Mutation Testing

7. ISoLA 2021 © Serge Demeyer 7
Quis custodiet ipsos custodes?
(Who will guard the guardians?)
Formal Verification Mutation Testing

8. ISoLA 2021 © Serge Demeyer
Unit Under Test
8
int findLast(std::vector<int> x, int y) {
if (x.size() == 0)
return -1;
for (int i = x.size() - 1; i >= 0; i--)
if (x[i] == y)
return i;
return -1;
}
y

9. ISoLA 2021 © Serge Demeyer
Test Suite
9
TEST(FindLastTests, emptyVector) {
EXPECT_EQ(-1, findLast({}, 3));
}
TEST(FindLastTests, doubleOccurrence) {
EXPECT_EQ(3, findLast({1, 2, 42, 42, 63}, 42));
}
TEST(FindLastTests, noOccurrence) {
EXPECT_EQ(-1, findLast({1, 2, 42, 42, 63}, 99));
int findLast(std::vector<int> x, int y) {
if (x.size() == 0)
return -1;
for (int i = x.size() - 1; i >= 0; i--)
if (x[i] == y)
return i;
return -1;
}

10. ISoLA 2021 © Serge Demeyer 10
100% line coverage
100% statement coverage
100% branch coverage
all tests passed

11. ISoLA 2021 © Serge Demeyer
Inject Mutant (Killed)
11
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i < 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08
}
[ PASSED ] 2 tests
.
[ FAILED ] 1 test, listed below
:
[ FAILED ] FindLastTests.doubleOccurrence
Relational Operator Replacement (ROR)
“i >= 0” becomes “i < 0”

12. ISoLA 2021 © Serge Demeyer
Inject Mutant (Survived - Live)
12
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i > 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08
}
[==========] 3 tests from 1 test suite ran. (0 ms total
)
[ PASSED ] 3 tests
.
Relational Operator Replacement (ROR)
“i >= 0” becomes “i > 0”

13. ISoLA 2021 © Serge Demeyer
TEST(FindLastTests, occurrenceOnBoundary) {
EXPECT_EQ(0, findLast({1, 2, 42, 42, 63}, 1));
}
[==========] 4 tests from 1 test suite ran. (0 ms total
)
[ PASSED ] 3 tests
.
[ FAILED ] 1 test, listed below
:
[ FAILED ] FindLastTests.occurrenceOnBoundar
y
Strengthening the Test Suite
13
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i > 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08
}

14. ISoLA 2021 © Serge Demeyer 14
Quis custodiet ipsos custodes?
(Who will guard the guardians?)
Formal Verification Mutation Testing

15. ISoLA 2021 © Serge Demeyer 15
The VeriFast Program Verifier: A Tutorial
Bart Jacobs Jan Smans
Frank Piessens
imec-DistriNet, Department of Computer Science, KU Leuven - University of Leuven, Belgium
November 28, 2017
Contents
1 Introduction
2 Example: illegal access.c
3 malloc block Chunks
4 Functions and Contracts
5 Patterns
6 Predicates
7 Recursive Predicates
8 Loops

16. ISoLA 2021 © Serge Demeyer
Pre and postconditions
16
void stack_push(struct stack *stack, int value
)
//@ requires stack(stack, ?count)
;
//@ ensures stack(stack, count + 1)
;
{
//@ open stack(stack, count)
;
struct node *n = malloc(sizeof(struct node))
;
if (n == 0) { abort();
}
n->next = stack->head
;
n->value = value
;
stack->head = n
;
//@ close nodes(n, count + 1)
;
//@ close stack(stack, count + 1);
}
int stack_pop(struct stack *stack)
//@ requires stack(stack, ?count) &*& 0 < count
;
//@ ensures stack(stack, count - 1);
{
//@ open stack(stack, count)
;
struct node *head = stack->head
;
//@ open nodes(head, count);
int result = head->value
;
stack->head = head->next
;
free(head);
//@ close stack(stack, count - 1);
return result;
}
Confirm
ed
or Counterexam
ple

17. ISoLA 2021 © Serge Demeyer
Helpers — Intermediate Assertions
17
void stack_filter(struct stack *stack, int_predicate *p
)
//@ requires stack(stack, _) &*& is_int_predicate(p) == true
;
//@ ensures stack(stack, _);
{
//@ open stack(stack, count)
;
struct node *head = stack->head
;
//@ open nodes(head, count)
;
int result = head->value
;
stack->head = head->next
;
free(head);
//@ close stack(stack, count - 1);
//@ open stack(stack, _);
struct node *head = nodes_filter(stack->head, p)
;
//@ assert nodes(head, ?count)
;
stack->head = head
;
//@ open nodes(head, count);
//@ close nodes(head, count);
//@ close stack(stack, count)
;
}
Remove all elements where the predicate function pointer p marks false

18. ISoLA 2021 © Serge Demeyer 18
Quis custodiet ipsos custodes?
(Who will guard the guardians?)
Formal Verification Mutation Testing
+

19. ISoLA 2021 © Serge Demeyer
Equivalent Mutant
19
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() <= 0)
03 return -1;
04 for (int i = x.size() - 1; i < 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08
}
[==========] 4 tests from 1 test suite ran. (0 ms total
)
[ PASSED ] 4 tests
.
Relational Operator Replacement (ROR)
“== 0” becomes “<= 0”
Some mutants have the same semantics as original program.
They cannot be detected by any test suite.
They waste test engineers time

20. ISoLA 2021 © Serge Demeyer
Verify Mutant Equivalence
20
1 int findLast(std::vector<int> x, int y)
{
2
if (x.size() == 0) return -1
;
… (identical code here
)
9 int findLastEquivalentCandidate(std::vector<int> x, int y)
{
1
0
if (x.size() <= 0) return -1
;
… (identical code here
)
20 TEST(FindLastTests, emptyVector)
{
2
1
int res1, res2
;
2
2
EXPECT_EQ(-1, res1 = findLast({}, 3))
;
2
3
//@ assert res1 == -1
;
2
4
EXPECT_EQ(-1, res2 = findLastEquivalent({}, 3))
;
2
5
//@ assert res2 == -1
;
2
6
//@ ensures res1 == res2
;
2
7
}
Confirm
ed
or Counterexam
ple

21. ISoLA 2021 © Serge Demeyer
Research Agenda
21
Equivalent Mutants
Infinite Loops
Flakey Tests
Test Clones
Test Amplification

22. ISoLA 2021 © Serge Demeyer 22
Quis custodiet ipsos custodes?
(Who will guard the guardians?)
Testcode provides interesting
opportunities for formal verification
EXPECT_EQ ≈ ASSERT
FOCUS ANALYSIS
UNIT UNDER TEST
LIMIT SEARCH SPACE