Presentation at IEEE ICST 2015

1. .lusoftware verification & validation
VVS
Generating Complex and Faulty Test Data
Through Model-Based Mutation Analysis
University of Luxembourg
Interdisciplinary Centre for Security, Reliability and Trust
Software Verification and Validation Lab
Daniel Di Nardo, Fabrizio Pastore, Lionel Briand
April 9, 2015

2. Data
Processing
System
Multiple fields
Nested structure
Constraints among fields
Huge amount
Invalid
data
Accepted
& processed
Discarded
Valid
data

3. Data
Processing
System
Invalid
data
Accepted
& processed
Discarded
Valid
data
Real world
data

4. Data
Processing
System
Invalid
data
Accepted
& processed
Discarded
Valid
data
Real world
data

5. Data
Processing
System
Invalid
data
Accepted
& processed
Discarded
Valid
data
Output in log files

6. Data
Processing
System
Multiple fields
Nested structure
Constraints among fields
Huge amount
Invalid
data
Accepted
& processed
Discarded
Valid
data
Software Engineers manually
handcraft erroneous inputs that
cover the fault model of the system
Expensive!

7. Configurations
for Operators
Data
Modelling
Generic
Mutation
Operators
Through
How to generate
test data?
How to validate
results?
How to satisfy
the fault model?
Test Automation

8. Input
Output
Data Modelling using Class Diagrams + OCL Constraints
(ASE 2013)
context Vcdu inv:
let
frameCount : Integer = self.vcFrameCount,
previousFrameCount : Integer = self.vcFrameCount
in
frameCount <> previousFrameCount + 1
implies
VcduEvents.allInstances()
->exists(e | e.eventType = COUNTER_JUMP)

9. Satellite Transmission
Header Packet 1 Packet 2 Packet 3 Packet 4...
Packet 1 Packet 2 Packet 3
VCDU 1
VCDU 2
...Packet 4 Packet 5 Packet 6...
VCDU 3
...Packet 6...
VCDU 4
VCDU 5
...Packet 6 Packet 7
VCDU 6
SYNSYNSYNSYNSYNSYNSYN
VCID=1
Header
Header
Header
Header
Header
VCID=2
VCID=1
VCID=1
VCID=1
VCID=0 Idle Packet Pattern

10. Input Data Model (simplified)
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
data : Bytesequence
1..*
1
1..*
1
1
1
1

11. Input Data Model (simplified)
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1

12. Software
Under Test
Output
Constraints
Checking
Test Input
Data Model
Violated
Constraints
Model Instance
Field Data
Mutation
Based
Generation

13. Mutation & Validation
Output
Data Loading
Data Writing
Data Mutation
Execution
Validation
Violated
Constraints
Field Data
Input/Output
constraints
Mutated Field Data
SUT
Instantiated
Objects
Mutated Instantiated
Objects
Input/Output
structure
Data Model

14. 14
Not useful to test
any kind of
(trivially) invalid input
Important to generate
test inputs that fit
a Fault Model

15. Mutation & Validation
Output
Data
Modelling
Data
Loading
Data
Wri3ng
Data
Muta3on
Execu3on
Valida3on
Violated
Constraints
Transmission
FileInput/Output
structure
Input/Output
constraints
Faulty Transmission
File
SES-­‐DAQ
Instantiated
Objects
Mutated Instantiated
Objects

16. Fault Model of SES DAQ
Fault
Description
Duplicate VCDU/Packet
A VCDU/Packet appears twice in a transmission.
Missing VCDU/Packet
A VCDU/Packet is omitted during transmission
Wrong Sequence
VCDUs/Packets are sent out of order.
Incorrect Identifier
Several transmission data fields have fixed values, e.g.
fields identifying the transmitting satellite.
Hardware/software errors may assign incorrect identifiers.
Incorrect Checksum
Hardware/software errors may result in an incorrect
checksum for a Packet or VCDU.
Incorrect Counter
Counters are used to track Packet or VCDU ordering.
Hardware/software errors may assign incorrect counter
values
Flipped Data Bits
Physical channel noise may flip one or more bits in the data
transmission.

17. Covering The Fault Model
Generic Mutation Operators
Configurations for the Operators

18. 18
Class Instance
Operators
Attribute Instance
Operators

19. Data Mutation Operators
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1

20. Class Instance Operators
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Class Instances
Swapping
Class Instance
Removal
Class Instance
Duplication

21. Class Instance Operators
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Class Instances
Swapping
Class Instance
Removal
Class Instance
Duplication

22. Class Instance Operators
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Class Instances
Swapping
Class Instance
Removal
Class Instance
Duplication

23. Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Transmission
Vcdu
ActiveZone
IdleZone
VirtualChannel
1
1
1..*
1
1
data : Bytesequence
«InputData»
PacketZone
1
1
Selecting Targets
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
1..*
1..*
«InputData»
1
1
Class Instances
Swapping
Class Instance
Removal
Class Instance
Duplication

24. Attribute Operators
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1

25. Attribute Operators
Transmission
Vcdu
Header
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer

26. Attribute Operators
Transmission
Vcdu
Packet
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
Header

27. Attribute Operators
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
spaceCraftId : Integer
«Identifier» checksum : Integer
Replacement with
Boundary Condition
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence

28. Attribute Operators
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
spaceCraftId : Integer
«Identifier» checksum : Integer
Replacement with
Boundary Condition
versionNumber : Integer
type : Integer
«Measure» dataLength : Integer
data : Bytesequence

29. Attribute Operators
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
«Identifier» spaceCraftId : Integer
«Identifier» checksum : Integer
Replacement with
Boundary Condition
«Identifier» versionNumber : Integer
«Identifier» type : Integer
«Measure» dataLength : Integer
data : Bytesequence

30. Attribute Operators
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
«Identifier» spaceCraftId : Integer
«Identifier» checksum : Integer
Replacement with
Boundary Condition
Bit Flipping
data : Bytesequence
«Identifier» versionNumber : Integer
«Identifier» type : Integer
«Measure» dataLength : Integer
data : Bytesequence

31. Avoiding Trivial Errors

32. Keeping Dependencies
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
«Identifier» spaceCraftId : Integer
«Identifier» checksum : Integer
Replacement with
Boundary Condition
Bit Flipping
data : Bytesequence
«Identifier» versionNumber : Integer
«Identifier» type : Integer
«Measure» dataLength : Integer
data : Bytesequence

33. Keeping Dependencies
Transmission
Vcdu
Header
Packet
PacketZone
ActiveZone
IdleZone
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Replacement
with Random
«Identifier» versionNumber : Integer
«Identifier» spaceCraftId : Integer
«Derived, Identifier» checksum : Integer
Replacement with
Boundary Condition
data : Bytesequence
«Identifier» versionNumber : Integer
«Identifier» type : Integer
«Measure» dataLength : Integer
data : Bytesequence
Bit Flipping

34. Refining Target Selection
Transmission
Vcdu
Header
versionNumber : Integer
spaceCraftId : Integer
checksum : Integer
versionNumber : Integer
type : Integer
dataLength : Integer
data : Bytesequence
PacketZone
ActiveZone
IdleZone
data : Bytesequence
VirtualChannel
1
1
1..*
1
1..*
1..*
1
1
1
1
1
Class Instances
Swapping
«InputData»
«InputData»
Packet
OCL queries to configure the targets
on which an operator may work.
E.g. swapping of packets that belong
to the same virtual channel

35. Mapping Between Fault Model and"
Mutation Operators
Fault
Mutation Operator
Configuration
Duplicate VCDU
Class Instance Duplication
«InputData»
Duplicate Packet
Class Instance Duplication
«InputData», «Derived»
Missing VCDU
Class Instance Removal
«InputData»
Missing Packet
Class Instance Removal
«InputData», «Derived»
Wrong VCDU Sequence
Class Instances Swapping
«InputData»
Wrong Packet Sequence
Class Instances Swapping
«InputData», «Derived»,
Query to select packets
Incorrect Identifier
Attribute Replacement with Random
«Identifier», «Derived»
Incorrect Checksum
Attribute Replacement with Random
«Identifier»
Incorrect Counter
Attribute Replacement using Boundary Condition
«Measure», «Derived»
Flipped Data Bits
Attribute Bit Flipping
none

36. Instantiated
Objects
Mutated Instantiated
Objects
Field Data
Mutated Field Data
Mutation & Validation
Output
Data Modelling
Data Loading
Data Writing
Data Mutation
Execution
Validation
Violated
Constraints
Input/Output
structure
Input/Output
constraints
SES-DAQ

37. Data Mutation Strategies
• Random (RND)
• Randomly select a mutation operator and randomly applies it to
one of the possible targets.
• Tester specifies number of test cases to generate per test suite.
• All Possible Targets (APT)
• Each class or attribute of the data model is mutated at least once
by each of the mutation operators that can be applied to it.

38. Empirical Evaluation
• Evaluated the effectiveness of the technique presented by
applying it on the SES-DAQ system.
• Compare RND/APT to SES testing practice.
• 32 system test cases using synthetic input data.
• Manually written based on domain expertise.
• GOAL: Determine if our technique can automatically achieve
equivalent or better coverage than manual test cases.

39. Test Suite Generation
• Applied both the RND and APT
• APT led to the generation of 43 test cases.
• To compare the two strategies we generated 43 test cases for RND as well.
• Both APT and RND carry a degree of randomness.
• Averaged results from 10 automatically generated test suites.
• JaCoCo used to measure code coverage.
• Number of bytecode instructions covered.

40. Experimental Results
Technique
Coverage (bytecode)
minimum
maximum
average
SES Manual Test Suite
--
--
22820 (70.9%)
RND Test Suite Generation
22550 (70.1%)
23060 (71.7%)
22899 (71.2%)
APT Test Suite Generation
23226 (72.2%)
23374 (72.7%)
23283 (72.4%)
Automatically achieve
slightly better coverage than test cases
written manually by domain experts.
Identified one crash condition (unknown fault).

41. Future Work
• Better evaluate the effectiveness of our test suite generation
strategies with faulty versions of the SES-DAQ software.
• Devise a Search-Based approach to test suite generation.
• Fitness based on:
• Coverage of Model / OCL Constraints
• Fault model
• Code Coverage

42. Complex
Test Data
Difficult To
Handcraft
Automatically achieve
slightly better coverage than
manual test cases
Conclusion