The document describes a technique for automatically generating test input data for systems subject to incremental requirements changes. It involves capturing new data requirements in an updated data model, generating an incomplete updated model instance, and using slicing-based constraint solving and model transformation to produce a valid updated model instance satisfying all constraints. This allows test cases to be reused when requirements and data fields are changed.

1. .lusoftware verification & validation
VVS
Augmenting Field Data
for Testing Systems Subject to
Incremental Requirements Changes
Daniel Di Nardo, Fabrizio Pastore, Lionel Briand
SnT Centre, University of Luxembourg
Journal First - ACM Transactions on
Software Engineering and Methodology
Volume 26 Issue 1, July 2017

2. Data Processing Systems
• Aa
• aa
Data Processing Systems

3. Data Processing Systems
Processing
Workflows
Web Crawlers

4. Data Processing Systems
Processing
Workflows
Web Crawlers

5. Data Processing Systems
Processing
Workflows
Web Crawlers
Solution: Reuse Field Data[ICST’15]

6. Data Requirements Change
Changes affect
Data Fields, Data Types, Constraints
Old Test Cases Cannot be Reused
Field Data is Not Available

7. Test Input for Base Version
Test Input for Updated Version
Automatically Generate

8. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slice-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model

9. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slice-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model

10. 1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 1
1..*1
1
versionNumber : Int
spaceCraftId : Int
virtualChanId : Int
apidValue : Apid
sequenceCount : Int
dataLength : Int
PacketHeader
SarPacketHeader
coarseTime : Int
fineTime : Int
GpsrPacketHeader
coarseTime : Int
fineTime : Int
Configuration
spaceCraftId : Int
checkCrc : Bool
idleVcid : Vcid
idleApid : Apid
VcduConfig
vcid : Vcid
minPacketSize : Int
maxPacketSize : Int
apids : Apid[]
1
1
1..*
1
prev
1..*
1
1
1
«DataType»
Apid
value : Int

11. 1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 1
1..*1
1
versionNumber : Int
spaceCraftId : Int
virtualChanId : Int
apidValue : Apid
sequenceCount : Int
dataLength : Int
PacketHeader
SarPacketHeader
coarseTime : Int
fineTime : Int
GpsrPacketHeader
coarseTime : Int
fineTime : Int
Configuration
spaceCraftId : Int
checkCrc : Bool
idleVcid : Vcid
idleApid : Apid
VcduConfig
vcid : Vcid
minPacketSize : Int
maxPacketSize : Int
apids : Apid[]
1
1
1..*
1
prev
context VcduHeader inv:
self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
1..*
1
1
1
«DataType»
Apid
value : Int

12. 1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 1
1..*1
1..*
1
versionNumber : Int
spaceCraftId : Int
virtualChanId : Int
apidValue : Apid
sequenceCount : Int
dataLength : Int
PacketHeader
SarPacketHeader
coarseTime : Int
fineTime : Int
GpsrPacketHeader
coarseTime : Int
fineTime : Int
Configuration
spaceCraftId : Int
checkCrc : Bool
idleVcid : Vcid
idleApid : Apid
VcduConfig
vcid : Vcid
minPacketSize : Int
maxPacketSize : Int
apids : Apid[]
1
1
1..*
1
prev
context Packet inv:
( self.apidValue.value=1 and
self.pHeader.oclIsTypeOf(SarPacketHeader) ) or
…
1
1
1
context VcduHeader inv:
self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
«DataType»
Apid
value : Int

13. Capture
New Data Requirements
in
Updated Data Model

14. 1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 1
1..*1
1..*
1
versionNumber : Int
spaceCraftId : Int
virtualChanId : Int
apidValue : Apid
sequenceCount : Int
dataLength : Int
PacketHeader
«Replacement»
MsiPacketHeader
coarseTime : Int
fineTime : Int
opMode : Int
status : Int
GpsrPacketHeader
coarseTime : Int
fineTime : Int
prev
context Packet inv:
( self.apidValue.value=1 and
self.pHeader.oclIsTypeOf(SarPacketHeader) ) or
…
Configuration
spaceCraftId : Int
checkCrc : Bool
idleVcid : Vcid
idleApid : Apid
VcduConfig
vcid : Vcid
minPacketSize : Int
maxPacketSize : Int
apids : Apid[]
1
1
1..*
1
SarPacketHeader
coarseTime : Int
fineTime : Int
context VcduHeader inv:
self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
«DataType»
Apid
value : Int

15. 1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 1
1..*1
1..*
1
versionNumber : Int
spaceCraftId : Int
virtualChanId : Int
apidValue : Apid
sequenceCount : Int
dataLength : Int
PacketHeader
prev
context Packet inv:
( self.apidValue.value=3 or self.apidValue.value=4
and self.pHeader.oclIsTypeOf(MsiPacketHeader) )
or …
Configuration
spaceCraftId : Int
checkCrc : Bool
idleVcid : Vcid
idleApid : Apid
VcduConfig
vcid : Vcid
minPacketSize : Int
maxPacketSize : Int
apids : Apid[]
1
1
1..*
1
«Replacement»
MsiPacketHeader
coarseTime : Int
fineTime : Int
opMode : Int
status : Int
GpsrPacketHeader
coarseTime : Int
fineTime : Int
context VcduHeader inv:
self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
«DataType»
Apid
value : Int

16. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slice-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model

17. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: SarHeader
coarseTime=10
fineTime=30
pd1: PacketData
data=..
c1: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid1
minPacketSize=200
maxPacketSize=300
apids=<apid1,apid2>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
ph2: SarHeader
coarseTime=10
fineTime=30
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph3: SarHeader
coarseTime=10
fineTime=30
pd3: PacketData
data=..
Original Model Instance

18. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slicing-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model

19. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: SarHeader
coarseTime=10
fineTime=30
pd1: PacketData
data=..
c1: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid1
minPacketSize=200
maxPacketSize=300
apids=<apid1,apid2>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
ph2: SarHeader
coarseTime=10
fineTime=30
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph3: SarHeader
coarseTime=10
fineTime=30
pd3: PacketData
data=..
Replace obsolete classes

20. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c1: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid1
minPacketSize=200
maxPacketSize=300
apids=<apid1,apid2>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Replace obsolete classes

21. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc2: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Load new configuration

22. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc2: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
context VcduHeader inv: self.vcdu.trans.con g.vcduCon gs.vcid→exists(x | x = self.virtualChanId)
Updated data may invalidate constraints

23. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc2: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
context Packet inv:
…
( (self.apidValue.value=3 or self.apidValue.value=4) and self.pHeader.oclIsTypeOf(MsiPacketHeader) ) or
…
Updated constraints may not hold

24. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc2: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Incomplete Updated Model Instance

25. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slicing-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model

26. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Identify all the root-to-leaf paths

27. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Generate a slice for each

28. a1:ActiveZonet1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Generate a slice for each
All the instances “linked by an OCL”
to an element of the path
must belong to the slice
context PacketZone inv: self.packets→forAll(x : Packet |
self.vcdu.trans.config.vcduConfigs→select(y : VcduConfig |
y.vcid = self.vcdu.vcduHeader.virtualChannelId).apids→exists(z : Apid | z = x.apidValue)
…

29. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
pd1: PacketData
data=..
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
pd2: PacketData
data=..
p3: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
pd3: PacketData
data=..
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
Generate a slice for each
Elements of collections are linked to the
previous element only

30. Constraint solving
Slice with
updated values
Generate
slices
Incomplete Updated Model Instance

31. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
“Solve” a slice
//Declare classes
abstract sig Vcdu {
trans:one Transmission, packetZone:one
PacketZone }
..
//Declare class instances
one sig vc2 extends VcduConfig{}
..
//Facts for associations
fact t1.config=c2
..
//Facts for config variables
fact { vc2.apids = apid3+apid4..
//Facts for data variables
fact { p1.apidValue = apid1 }
fact { p1.sequenceCount = 1 }
//Alloy predicate for OCLs
pred PacketHeaderApid []{ all e: PacketHeader | …
Alloy Model
UMLtoAlloy
OCL constraints…

32. t1:Transmission
p1: Packet
apidValue=apid1
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
“Solve” a slice
UMLtoAlloy
OCL constraints…
Alloy Analyzer No Solution
Relax Model
//Facts for data variables
fact { p1.apidValue = apid1 }
fact { p1.sequenceCount = 1 }
//Alloy predicate for OCLs
pred PacketHeaderApid []{ all e: PacketHeader | …
Remove facts
that prevent
the generation of
a solution
Alloy Instance

33. t1:Transmission
p1: Packet
apidValue=apid3
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=1
status=1
c2: Configuration
spaceCraftId=1
checkCrc=true
idleVcid=vcid0
idleApid=apid0
vc1: VcduConfig
vcid=vcid2
minPacketSize=200
maxPacketSize=300
apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
a1:ActiveZone
vh1: VcduHeader
versionNumber=1
spaceCraftId=1
virtualChanId=vcid1
…
“Solve” a slice
UMLtoAlloy
OCL constraints…
Alloy Analyzer No Solution
Relax Model
//Facts for data variables
fact { p1.apidValue = apid1 }
fact { p1.sequenceCount = 1 }
//Alloy predicate for OCLs
pred PacketHeaderApid []{ all e: PacketHeader | …
Remove facts
that prevent
the generation of
a solution
Alloy Instance

34. Solve slice
Slice with
updated values
Copy to
Incomplete
Model Instance
Generate
slices
Incomplete Updated Model Instance

35. p2: Packet
apidValue=apid1
sequenceCount=2
dataLength=200
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
t1:Transmission
v1:Vcdu
a1:ActiveZone
vh1
c2:
..
vc1
…
p1: Packet
apidValue=apid3
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=1
status=1
“Solve” next slice

36. p2: Packet
apidValue=apid3
sequenceCount=2
dataLength=200
ph2: MsiHeader
coarseTime=10
fineTime=30
opMode=
status=
t1:Transmission
v1:Vcdu
a1:ActiveZone
vh1
c2:
..
vc1
…
p1: Packet
apidValue=apid3
sequenceCount=1
dataLength=200
ph1: MsiHeader
coarseTime=10
fineTime=30
opMode=1
status=1
“Solve” next slice
Solving algorithm
does not rewrite
variables generated
in previous steps

37. Solve slice
Slice with
updated values
Copy to
Incomplete
Model Instance
Generate
slices
Incomplete Updated Model Instance
If a value “used” in a previous slice
has been modified, restart from first

38. Solve slice
Slice with
updated values
Copy to
Incomplete
Model Instance
Generate
slices
Valid Updated Model Instance
Incomplete Updated Model Instance

39. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slice-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model

40. Empirical Evaluation
• RQ1: How does the proposed approach compare to a non-slicing
approach?
• RQ2: Does the proposed approach scale to a practical extent?
• RQ3: Does the proposed approach allow for the effective testing
of new data requirements?
• RQ4: Does the proposed approach allow for efficient and effective
system testing compared to manual approach?

41. Case Study
• Data Processing System for Satellite Data developed by SES
• Data Model
• 82 classes, 322 attributes, 56 associations, 52 OCL constraints
• Manual testing is expensive
• 130 class instances, 261 attributes, 1152 constrained attribute
values

42. RQ1: How does the proposed approach
compare to a non-slicing approach?
• To be justified the proposed approach should provide an
advantage over one that does not use slicing
• Created a simplified technique that does not apply slicing
• All attribute values are generated from scratch from a single execution
of the Alloy Analyzer
• Generated test inputs from 1 to 40 VCDUs
• 10 inputs for each size

43. Proposed approach (using slicing)
vs a non-slicing approach
Scales better
than non-slicing
approach

44. RQ2: Does the proposed approach
scale to a practical extent?
• Generation of new data should be fast enough and scale
effectively as file size increases
• Generated test inputs from 50 to 500 VCDUs
• 10 inputs for each size

45. Execution time required to generate inputs
vs number of VCDUs
Non-trivial inputs
generated in reasonable time

46. RQ3: Does the proposed approach allow for the
effective testing of new data requirements?
• Automatically generate new data intended for use in
robustness test cases
• Integrate a mutation approach (ICST’15) to generate
robustness test cases
• Measure code coverage of the source code that implements
the new data requirements
• Compare with test cases written by engineers
Field Data
(Version 0)
Updated
Data
(Version 1)
Robustness
Test Inputs
(Version 1)
Slicing Based
Generation
Mutation Based
Generation

47. RQ3: Does the proposed approach allow for the
effective testing of new data requirements?
Test
suite
S2 Instructions Covered S2 Branches Covered # of
Tests
Avg Min Max Avg Min Max
Auto 74
(77.9%)
74
(77.9%)
74
(77.9%)
9
(81.8%)
9
(81.8%)
9
(81.8%)
103.1
Test
suite
S2 Instructions Covered S2 Branches Covered # of
Tests
Avg Min Max Avg Min Max
Auto 74
(77.9%)
74
(77.9%)
74
(77.9%)
9
(81.8%)
9
(81.8%)
9
(81.8%)
103.1
Manual 68
(71.6%)
- - 8
(72.7%)
- - 32
More effective than manual testing

48. RQ4: Does the proposed approach allow for
effective system testing?
• In the presence of a new software version engineers are
interested in testing the whole system
• Necessary to test unmodified functions when they process a
new a data type (e.g. packet data extraction in the presence
of a new packet type)
• Same approach of RQ3: robustness testing
• Measure code coverage of the whole system
• Compare with 32 expensive manually crafted test inputs

49. RQ4: Does the proposed approach allow for
effective system testing?
More effective than manual testing
Test
suite
Instructions Covered Branches Covered # of
Tests
Avg Min Max Avg Min Max
Auto 23,432.1
(72.2%)
23,273
(71.7%)
23,529
(72.5%)
978.7
(51.3%)
957
(50.2%)
987
(51.8%)
103.1
Manual 23,046
(71.0%)
- - 950
(49.8%)
- - 32

50. RQ4: Does the proposed approach allow for
efficient and effective system testing?
• Both data modelling and manual testing are expensive
• Data Model: 82 classes, 322 attributes, 56 associations, 52 OCL constraints
• Manual test: 130 class instances, 261 attributes, 1152 constrained attribute values
• Data model useful for other purposes (e.g., discuss with system engineers)
• Manual and automatically generate tests have negligible
execution time:
• Manual, 10 mins – Automatically generated, 30 mins
• Automatically generated tests are more effective

51. Original
Model Instance
Incomplete Updated
Model Instance
Data Loading
Slice-based
Constraint
Solving
Automatic
Model
Transformation
Data Writing
Valid Updated
Model Instance
Updated
DataField
Data
Data Model