Synthesis of Research works on testing variability intensive systems presented in "DiVERSE Coffee", INRIA Rennes, March 2015.

1. www.unamur.be
Software Testing for Variability-
Intensive Systems
Gilles Perrouin
PReCISE, University of Namur
gilles.perrouin@unamur.be
INRIA DiVERSE, March 2015

2. www.unamur.be gilles.perrouin@unamur.be
About Me… (for newcomers)
2
2

3. www.unamur.be gilles.perrouin@unamur.be
About Me… (for newcomers)
2
2

4. www.unamur.be gilles.perrouin@unamur.be
About Me… (for newcomers)
PhD in SE jointly from UNamur and U. of Luxembourg
(2007)
Supervisors: P. Heymans, N. Guelfi
Topic: Flexible modelling & Method for SPL derivation
Postdoc in INRIA Triskell team (now DiVERSE, 2007-2009)
Automated SPL derivation
``Variability modelling everywhere” (Dynamically Adaptive
Systems, Metamodels)
SPL Testing
Postdoc U. Luxembourg (2009-2010)
Security testing of Resilient Systems
Postdoc(s) at UNamur (2010-2015?)
3
3

5. www.unamur.be gilles.perrouin@unamur.be
Agenda
Variability-Intensive Systems ?
Structural testing
Behavioural testing
MISC (if time)
4
4

6. VARIABILITY-INTENSIVE SYSTEMS

7. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
6
6

8. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
6
6

9. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
Systematic engineering of families of products
6
6

10. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
Systematic engineering of families of products
Top-down view on variability management and development
(SEI, Fraunhofer, etc.)
6
6

11. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
Systematic engineering of families of products
Top-down view on variability management and development
(SEI, Fraunhofer, etc.)
Useful techniques (feature modelling & analysis, automated
derivation, etc.)
6
6

12. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
Systematic engineering of families of products
Top-down view on variability management and development
(SEI, Fraunhofer, etc.)
Useful techniques (feature modelling & analysis, automated
derivation, etc.)
Configurable systems have lot of variability without being
explicitly engineering following SPL paradigm
6
6

13. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
Systematic engineering of families of products
Top-down view on variability management and development
(SEI, Fraunhofer, etc.)
Useful techniques (feature modelling & analysis, automated
derivation, etc.)
Configurable systems have lot of variability without being
explicitly engineering following SPL paradigm
OSes, Unix tools (gcc, sed), CMS (Wordpress, Drupal), Eclipse,
Cloud, SOA, SOS, ….
6
6

14. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Software Product Lines (SPL)
Systematic engineering of families of products
Top-down view on variability management and development
(SEI, Fraunhofer, etc.)
Useful techniques (feature modelling & analysis, automated
derivation, etc.)
Configurable systems have lot of variability without being
explicitly engineering following SPL paradigm
OSes, Unix tools (gcc, sed), CMS (Wordpress, Drupal), Eclipse,
Cloud, SOA, SOS, ….
No reason to not consider SPL techniques for them too!
6
6

15. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
7
7

16. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
We therefore regroup all these systems under the
“variability-intensive systems” term for feature-based
software analysis…
7
7

17. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
We therefore regroup all these systems under the
“variability-intensive systems” term for feature-based
software analysis…
7
7

18. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
We therefore regroup all these systems under the
“variability-intensive systems” term for feature-based
software analysis…
7
7

19. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
We therefore regroup all these systems under the
“variability-intensive systems” term for feature-based
software analysis…
…thanks to a very liberal notion of feature!
7
7

20. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
8
8

21. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Michael Jackson: “features are behaviours”
8
8

22. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Michael Jackson: “features are behaviours”
8
Christian Kästner: “features are configuration options”
8

23. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Michael Jackson: “features are behaviours”
8
Christian Kästner: “features are configuration options”
8

24. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Michael Jackson: “features are behaviours”
8
Christian Kästner: “features are configuration options”
Features are units of variability
8

25. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Michael Jackson: “features are behaviours”
8
Christian Kästner: “features are configuration options”
Features are units of variability
8

26. www.unamur.be gilles.perrouin@unamur.be
Variability-Intensive Systems
Michael Jackson: “features are behaviours”
8
Christian Kästner: “features are configuration options”
Features are units of variability
8

27. VIS TESTING

28. www.unamur.be

29. www.unamur.be
270 Features
NASA JPL

30. STRUCTURAL VIS TESTING

31. www.unamur.be
CIT-based SPL Sampling (2010)
12
gilles.perrouin@unamur.be 12

32. www.unamur.be
CIT-based SPL Sampling (2010)
Making it easier: MDE + Alloy
Transform FM into an alloy specification
Ask alloy to filter valid tuples and compose them in
configurations
12
gilles.perrouin@unamur.be 12

33. www.unamur.be
CIT-based SPL Sampling (2010)
Making it easier: MDE + Alloy
Transform FM into an alloy specification
Ask alloy to filter valid tuples and compose them in
configurations
“Divide and conquer…”
Binary split and incremental strategies to deal with enormous
amounts of tuples…
12
gilles.perrouin@unamur.be 12

34. www.unamur.be
CIT-based SPL Sampling (2010)
Making it easier: MDE + Alloy
Transform FM into an alloy specification
Ask alloy to filter valid tuples and compose them in
configurations
“Divide and conquer…”
Binary split and incremental strategies to deal with enormous
amounts of tuples…
But recombination did not rule
Slow and redundant configurations
12
gilles.perrouin@unamur.be 12

35. www.unamur.be
CIT-based SPL Sampling (2010)
Making it easier: MDE + Alloy
Transform FM into an alloy specification
Ask alloy to filter valid tuples and compose them in
configurations
“Divide and conquer…”
Binary split and incremental strategies to deal with enormous
amounts of tuples…
But recombination did not rule
Slow and redundant configurations
12
gilles.perrouin@unamur.be 12

36. www.unamur.be
CIT-based SPL Sampling (2010)
Making it easier: MDE + Alloy
Transform FM into an alloy specification
Ask alloy to filter valid tuples and compose them in
configurations
“Divide and conquer…”
Binary split and incremental strategies to deal with enormous
amounts of tuples…
But recombination did not rule
Slow and redundant configurations
Gilles Perrouin, Sebastian Oster, Sagar Sen, Jacques Klein, Benoit Baudry, Yves Le Traon: Pairwise
testing for software product lines: comparison of two approaches. Software Quality Journal
20(3-4): 605-643 (2012)
12
gilles.perrouin@unamur.be 12

37. www.unamur.be
Similarity-driven Sampling (2012-2014)
13
gilles.perrouin@unamur.be 13

38. www.unamur.be
Similarity-driven Sampling (2012-2014)
Use SBSE to select a population of configurations
Ensure the validity of generated configurations (via SAT4J)
13
gilles.perrouin@unamur.be 13

39. www.unamur.be
Similarity-driven Sampling (2012-2014)
Use SBSE to select a population of configurations
Ensure the validity of generated configurations (via SAT4J)
Designed for Scalability
Fitness function based on distance: correlated with t-
wise coverage but easier to compute
13
gilles.perrouin@unamur.be 13

40. www.unamur.be
Similarity-driven Sampling (2012-2014)
Use SBSE to select a population of configurations
Ensure the validity of generated configurations (via SAT4J)
Designed for Scalability
Fitness function based on distance: correlated with t-
wise coverage but easier to compute
Flexibility
Tester decides generation time and # of configurations
Configurations are prioritised w.r.t fitness function: use
a subset if lack of resources
13
gilles.perrouin@unamur.be 13

41. www.unamur.be
Similarity-driven Sampling (2012-2014)
14
gilles.perrouin@unamur.be 14
Linux FM

42. www.unamur.be
Similarity-driven Sampling (2012-2014)
t=6
(1,000 confs)
0 runs 15,000 runs
eCos 94.191% 95.343%
FreeBSD 76.236% 76.494%
Linux 89.411% 90.671%
15
gilles.perrouin@unamur.be 15

43. www.unamur.be
Similarity-driven Sampling (2012-2014)
t=6
(1,000 confs)
0 runs 15,000 runs
eCos 94.191% 95.343%
FreeBSD 76.236% 76.494%
Linux 89.411% 90.671%
Ecos: 1 % more coverage implies 2,1E+15 additional 6-tuples !
15
gilles.perrouin@unamur.be 15

44. www.unamur.be
Similarity-driven Sampling (2012-2014)
t=6
(1,000 confs)
0 runs 15,000 runs
eCos 94.191% 95.343%
FreeBSD 76.236% 76.494%
Linux 89.411% 90.671%
Ecos: 1 % more coverage implies 2,1E+15 additional 6-tuples !
1,000 configurations may not be enough... 15
gilles.perrouin@unamur.be 15

45. www.unamur.be
Similarity-driven Sampling (2012-2014)
t=6
(1,000 confs)
0 runs 15,000 runs
eCos 94.191% 95.343%
FreeBSD 76.236% 76.494%
Linux 89.411% 90.671%
Ecos: 1 % more coverage implies 2,1E+15 additional 6-tuples !
1,000 configurations may not be enough...
Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and
Yves Le Traon. Bypassing the Combinatorial Explosion: Using Similarity to Generate and
Prioritize T-Wise Test Configurations for Software Product Lines. IEEE TSE, 40(7):650–670,
2014.
15
gilles.perrouin@unamur.be 15

46. www.unamur.be
Test-and-Fix FMs (2013)
16
gilles.perrouin@unamur.be 16

47. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
16
gilles.perrouin@unamur.be 16

48. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
16
gilles.perrouin@unamur.be 16

49. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
SPLs are barely built from nothing
16
gilles.perrouin@unamur.be 16

50. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
SPLs are barely built from nothing
Set of existing products evolved as a SPL not derived directly
from the FMs
16
gilles.perrouin@unamur.be 16

51. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
SPLs are barely built from nothing
Set of existing products evolved as a SPL not derived directly
from the FMs
FM Synthesis (or reverse-engineering)
16
gilles.perrouin@unamur.be 16

52. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
SPLs are barely built from nothing
Set of existing products evolved as a SPL not derived directly
from the FMs
FM Synthesis (or reverse-engineering)
Several approaches exist [She, Acher, …]
16
gilles.perrouin@unamur.be 16

53. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
SPLs are barely built from nothing
Set of existing products evolved as a SPL not derived directly
from the FMs
FM Synthesis (or reverse-engineering)
Several approaches exist [She, Acher, …]
Tough pb: parsing code, limitations in the expressiveness of
target FMs languages, heuristics…
16
gilles.perrouin@unamur.be 16

54. www.unamur.be
Test-and-Fix FMs (2013)
Repositories (SPLOT) contain mostly small to medium
size academic FMs
may not represent actual systems
SPLs are barely built from nothing
Set of existing products evolved as a SPL not derived directly
from the FMs
FM Synthesis (or reverse-engineering)
Several approaches exist [She, Acher, …]
Tough pb: parsing code, limitations in the expressiveness of
target FMs languages, heuristics…
=> FMs may not be fully representative of the systems
you want to test
16
gilles.perrouin@unamur.be 16

55. www.unamur.be gilles.perrouin@unamur.be 17

56. www.unamur.be
FMs not
representative ?
gilles.perrouin@unamur.be 17

57. www.unamur.be
FMs not
representative ?
Missing/Infeasible/
Useless
configurations…
Get the model
right !
gilles.perrouin@unamur.be 17

58. www.unamur.be
FMs not
representative ?
Missing/Infeasible/
Useless
configurations…
Get the model
right !
Some FMs have
thousands of
features…
How to fix them?
gilles.perrouin@unamur.be 17

59. www.unamur.be
Test-and-Fix Loop (2013)
18
gilles.perrouin@unamur.be 18

60. www.unamur.be
Test-and-Fix Loop (2013)
18
gilles.perrouin@unamur.be 18

61. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
18
gilles.perrouin@unamur.be 18

62. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
18
gilles.perrouin@unamur.be 18

63. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
Try to build products from randomly extracted
configurations from FM (GCF,ORF)
18
gilles.perrouin@unamur.be 18

64. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
Try to build products from randomly extracted
configurations from FM (GCF,ORF)
Fix them using Hill-Climbing EA
18
gilles.perrouin@unamur.be 18

65. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
Try to build products from randomly extracted
configurations from FM (GCF,ORF)
Fix them using Hill-Climbing EA
Mutate the FMs to align them with real products
(alter/insert/remove constraint)
18
gilles.perrouin@unamur.be 18

66. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
Try to build products from randomly extracted
configurations from FM (GCF,ORF)
Fix them using Hill-Climbing EA
Mutate the FMs to align them with real products
(alter/insert/remove constraint)
Fitness function trying to minimize different kinds of
errors
18
gilles.perrouin@unamur.be 18

67. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
Try to build products from randomly extracted
configurations from FM (GCF,ORF)
Fix them using Hill-Climbing EA
Mutate the FMs to align them with real products
(alter/insert/remove constraint)
Fitness function trying to minimize different kinds of
errors
18
gilles.perrouin@unamur.be 18

68. www.unamur.be
Test-and-Fix Loop (2013)
Detect discrepancies in two ways
Check if existing products conform to the FM (SCF,
EWC)
Try to build products from randomly extracted
configurations from FM (GCF,ORF)
Fix them using Hill-Climbing EA
Mutate the FMs to align them with real products
(alter/insert/remove constraint)
Fitness function trying to minimize different kinds of
errors
18
gilles.perrouin@unamur.be 18
Henard, C., Papadakis, M., Perrouin, G., Klein, J., & Le Traon, Y. (2013, May). Towards automated testing and
fixing of re-engineered feature models. In Software Engineering (ICSE), 35th International Conference on (pp.
1245-1248). IEEE.

69. www.unamur.be
Test-and-Fix Loop (2013)
19
gilles.perrouin@unamur.be 19

70. www.unamur.be
Test-and-Fix Loop (2013)
Experimentation on linux kernel FM
19
gilles.perrouin@unamur.be 19

71. www.unamur.be
Test-and-Fix Loop (2013)
Experimentation on linux kernel FM
Large reverse-engineered FM + manual edits
19
gilles.perrouin@unamur.be 19

72. www.unamur.be
Test-and-Fix Loop (2013)
Experimentation on linux kernel FM
Large reverse-engineered FM + manual edits
Easy building infrastructure (kconfig+make)
19
gilles.perrouin@unamur.be 19

73. www.unamur.be
Test-and-Fix Loop (2013)
Experimentation on linux kernel FM
Large reverse-engineered FM + manual edits
Easy building infrastructure (kconfig+make)
19
FM 2K runs 3K runs 4K runs 5K runs
EWC 50 46 43 41 39
SCF 1000 885 556 498 455
ORF 2468 1646 1395 1236 1084
GCF 1000 1000 1000 1000 1000
gilles.perrouin@unamur.be 19

74. BEHAVIOURAL VIS TESTING

75. www.unamur.be
21
gilles.perrouin@unamur.be
Images credit: http://hedbonstudios.deviantart.com
21

76. www.unamur.be gilles.perrouin@unamur.be
Enumerative Way
22

77. www.unamur.be gilles.perrouin@unamur.be
Enumerative Way
22

78. www.unamur.be
23
gilles.perrouin@unamur.be
Images credit: http://hedbonstudios.deviantart.com
23

79. www.unamur.be
23
gilles.perrouin@unamur.be
Images credit: http://hedbonstudios.deviantart.com
23

80. www.unamur.be gilles.perrouin@unamur.be
Featured Transition Systems
24
SuperPower
Strength s Money mHammer h
requires
SuperHero
xor
Classen, Andreas, et al. "Featured transition systems: Foundations for verifying variability-
intensive systems and their application to LTL model checking." IEEE Transactions on
Software Engineering, 39.8 (2013): 1069-1089.
s0 s1 s2
findVillain
smash / s ∨ h
fireRocket / m
reload / m
s3
throwHammer / h
retrieveWeapon / h
finishHim

81. STATISTICAL TESTING

82. www.unamur.be
VIS Testing: Which product first ?
26
gilles.perrouin@unamur.be 26

83. www.unamur.be
VIS Testing: Which product first ?
Particularly useful during regression testing
Using weights on features [Henard, Johansen]
26
gilles.perrouin@unamur.be 26

84. www.unamur.be
VIS Testing: Which product first ?
Particularly useful during regression testing
Using weights on features [Henard, Johansen]
Does not consider behaviour
26
gilles.perrouin@unamur.be 26

85. www.unamur.be
VIS Testing: Which product first ?
Particularly useful during regression testing
Using weights on features [Henard, Johansen]
Does not consider behaviour
26
gilles.perrouin@unamur.be 26

86. www.unamur.be
VIS Testing: Which product first ?
Particularly useful during regression testing
Using weights on features [Henard, Johansen]
Does not consider behaviour
Using probabilities on transitions
26
gilles.perrouin@unamur.be 26

87. www.unamur.be
VIS Testing: Which product first ?
Particularly useful during regression testing
Using weights on features [Henard, Johansen]
Does not consider behaviour
Using probabilities on transitions
If model unavailable we may learn it from the system…
26
gilles.perrouin@unamur.be 26

88. www.unamur.be
VIS Testing: Which product first ?
Particularly useful during regression testing
Using weights on features [Henard, Johansen]
Does not consider behaviour
Using probabilities on transitions
If model unavailable we may learn it from the system…
26
gilles.perrouin@unamur.be 26

89. www.unamur.be
Usage Model
Statistical testing [Whittaker 1994]
Deterministic Time Markov Chain (DTMC)
27
gilles.perrouin@unamur.be 27

90. www.unamur.be
Usage Model
Statistical testing [Whittaker 1994]
Deterministic Time Markov Chain (DTMC)
Independent from the FTS
Allows usage of existing tools
Extraction method is agnostic of features
DTMC may be incomplete
27
gilles.perrouin@unamur.be 27

91. www.unamur.be
Usage Model
Statistical testing [Whittaker 1994]
Deterministic Time Markov Chain (DTMC)
Independent from the FTS
Allows usage of existing tools
Extraction method is agnostic of features
DTMC may be incomplete
Allows invalid paths
=>DTMC + FTS detects inconsistencies
27
gilles.perrouin@unamur.be 27

92. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
28
Family-Based Test Prioritization
gilles.perrouin@unamur.be 28

93. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
29
Family-Based Test Prioritization
1. Product
Selection
gilles.perrouin@unamur.be 29

94. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
30
Family-Based Test Prioritization
2. FTS
Projection
1. Product
Selection
gilles.perrouin@unamur.be 30

95. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
31
Family-Based Test Prioritization
2. FTS
Projection
1. Product
Selection
3. DTMC
Pruning
gilles.perrouin@unamur.be 31

96. www.unamur.be
Product-Based Test Derivation
[Samih+ 2012]
32
Family-Based Test Prioritization
2. FTS
Projection
1. Product
Selection
3. DTMC
Pruning
gilles.perrouin@unamur.be 32

97. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
33
Family-Based Test Prioritization
2. FTS
Projection
1. Product
Selection
3. DTMC
Pruning
1. Trace
Selection
DFS(lmax = 7 ; Pr min = 0; Pr max = 0.1 ; DTMC) =
{(pay, change, cancel, return; Pr = 0,01) ;
(free, cancel, return; Pr = 0,09) ;
(pay, change, tea, serveTea, open, take, close; Pr = 0,009);
(pay, change, tea, serveTea, take; Pr = 0,081) ;
(free, tea, serveTea, open, take, close; Pr = 0,081)}
gilles.perrouin@unamur.be 33

98. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
34
Family-Based Test Prioritization
2. FTS
Projection
1. Product
Selection
3. DTMC
Pruning
1. Trace
Selection
2. Trace
Filtering
and FTS
Pruning
DFS(lmax = 7 ; Pr min = 0; Pr max = 0.1 ; DTMC) =
{(pay, change, cancel, return; Pr = 0,01) ;
(free, cancel, return; Pr = 0,09) ;
(pay, change, tea, serveTea, open, take, close; Pr = 0,009);
(pay, change, tea, serveTea, take; Pr = 0,081) ;
(free, tea, serveTea, open, take, close; Pr = 0,081)}
gilles.perrouin@unamur.be 34

99. www.unamur.be
Product-Based Test Derivation
[Samih+2012]
35
Family-Based Test Prioritization
2. FTS
Projection
1. Product
Selection
3. DTMC
Pruning
1. Trace
Selection
2. Trace
Filtering
and FTS
Pruning
3. Product
Prioritization
DFS(lmax = 7 ; Pr min = 0; Pr max = 0.1 ; DTMC) =
{(pay, change, cancel, return; Pr = 0,01) ;
(free, cancel, return; Pr = 0,09) ;
(pay, change, tea, serveTea, open, take, close; Pr = 0,009);
(pay, change, tea, serveTea, take; Pr = 0,081) ;
(free, tea, serveTea, open, take, close; Pr = 0,081)}
(¬f∧t)∧
gilles.perrouin@unamur.be 35

100. www.unamur.be
Product-Based Test
Derivation [Samih+2012]
36
Family-Based Test Prioritization
[Devroey+2014]
2. FTS
Projection
1. Product
Selection
3. DTMC
Pruning
1. Trace
Selection
2. Trace
Filtering
and FTS
Pruning
3. Product
Prioritization
DFS(lmax = 7 ; Pr min = 0; Pr max = 0.1 ; DTMC) =
{(pay, change, cancel, return; Pr = 0,01) ;
(free, cancel, return; Pr = 0,09) ;
(pay, change, tea, serveTea, open, take, close; Pr = 0,009);
(pay, change, tea, serveTea, take; Pr = 0,081) ;
(free, tea, serveTea, open, take, close; Pr = 0,081)}
gilles.perrouin@unamur.be 36

101. www.unamur.be
Case-study: Claroline@UNamur
Open source online course management system (http://
www.claroline.net/)
± 7000 users
Upload/download documents, online exercises, forum,
agenda, announcements, etc.
37
http://webcampus.unamur.be
gilles.perrouin@unamur.be 37

102. www.unamur.be
Claroline: DTMC
Derived from anonymized Apache Access Log (5.26 Go)
From January 1st to October 1st 2013
12.689.033 PHP pages HTTP requests
38
gilles.perrouin@unamur.be 38

103. www.unamur.be
Claroline: DTMC
Derived from anonymized Apache Access Log (5.26 Go)
From January 1st to October 1st 2013
12.689.033 PHP pages HTTP requests
(1 PHP page ➔ 1 state) + initial state
1 request ➔ 1 transition
User session = sequence of requests (timeout = 45 min)
2-gram without smoothing [Verwer et al. 2013]
38
Pr(si,sj ) =
#occurence(si,sj )
#occurence(si,s)
s∈S
∑
gilles.perrouin@unamur.be 38

104. www.unamur.be
Claroline: models
Usage Model (DTMC)
• 96 states and 2149 transitions
• 2 hours computation
• Ubuntu Linux (Intel Core i3, 3.10 GHz, 4GB mem.)
39
gilles.perrouin@unamur.be 39

105. www.unamur.be
Claroline: models
Usage Model (DTMC)
• 96 states and 2149 transitions
• 2 hours computation
• Ubuntu Linux (Intel Core i3, 3.10 GHz, 4GB mem.)
Feature Diagram (FD)
• Built manually by inspecting a Claroline local instance
• 44 features
• Lots of optional features
39
gilles.perrouin@unamur.be 39

106. www.unamur.be
Claroline: models
Usage Model (DTMC)
• 96 states and 2149 transitions
• 2 hours computation
• Ubuntu Linux (Intel Core i3, 3.10 GHz, 4GB mem.)
Feature Diagram (FD)
• Built manually by inspecting a Claroline local instance
• 44 features
• Lots of optional features
Featured Transition System (FTS)
• Web crawler on local instance to get the pages
• (1 page ➔ 1 state) + initial state
• Every state accessible from anywhere
• Transitions tagged with feature expressions based on the knowledge of the
system
• 107 states and 11236 transitions
39
gilles.perrouin@unamur.be 39

107. www.unamur.be
Family-based vs. Product-based
Depends on needs
40
gilles.perrouin@unamur.be 40

108. www.unamur.be
Family-based vs. Product-based
Depends on needs
Decoupling the DTMC from the system model may be
inefficient
You may select invalid traces that need to be eliminated
Fortunately “web” systems do not have so many constrained
behaviours
Need to investigate dedicated algorithms (Viterbi) for some
tasks (highest probability traces)
Collaboration with Hamza Samih, SoSyM paper to come :-)
40
gilles.perrouin@unamur.be 40

109. MODEL-BASED MUTATION TESTING

110. www.unamur.be
Featured Mutants (2014)
42
gilles.perrouin@unamur.be 42

111. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
42
gilles.perrouin@unamur.be 42

112. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
42
gilles.perrouin@unamur.be 42

113. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
42
gilles.perrouin@unamur.be 42

114. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
Equivalent Mutant Problem (EMP)
42
gilles.perrouin@unamur.be 42

115. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
Equivalent Mutant Problem (EMP)
42
gilles.perrouin@unamur.be 42

116. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
Equivalent Mutant Problem (EMP)
42
Use FTS to model “mutants families” !
gilles.perrouin@unamur.be 42

117. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
Equivalent Mutant Problem (EMP)
42
Use FTS to model “mutants families” !
SPL paradigm to tackle the above problems
gilles.perrouin@unamur.be 42

118. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
Equivalent Mutant Problem (EMP)
42
Use FTS to model “mutants families” !
SPL paradigm to tackle the above problems
gilles.perrouin@unamur.be 42

119. www.unamur.be
Featured Mutants (2014)
Considered mutation testing to assess SPL test suites
[Henard+13]
Mutation analysis is effective but
Explosion of the number of mutants => costly to run them
against each test case
Equivalent Mutant Problem (EMP)
42
Use FTS to model “mutants families” !
SPL paradigm to tackle the above problems
Xavier Devroey, Gilles Perrouin, Maxime Cordy, Mike Papadakis, Axel, and Pierre-
Yves Schobbens. A Variability Perspective of Mutation Analysis. In ESEC/SIGSOFT
FSE - Visions and Challenges Track, Hong Kong, China, 2014. ACM
gilles.perrouin@unamur.be 42

120. www.unamur.be
Featured Mutants (2014)
43
gilles.perrouin@unamur.be 43
1 2 3
4
5
6
7 8 9
pay/m change/m
free / m
take / m
close/m
open/m take/m
cancel / ¬sm_4return/¬sm_4
soda/m serveSoda /m
tea /¬eex_t_o
serveTea /
m
Mutants
m
StateMissing
sm
EventExchanged
eex
Mutants Family Model
(FTS)
Variability Model
(FM)
Optional
sm_4
open /eex_t_o
eex_t_o
WrongStartState
wss
wss_5
wss_5
¬wss_5
1 2 3
4
5
6
7 8 9
pay change
free
take
close
open take
cancelreturn
soda serveSoda
tea serveTea
Original Transition System
Mutants
1 2 3
5
6
7 8 9
pay change
free
take
close
open take
soda serveSoda
tea serveTea
1 2 3
4
5
6
7 8 9
pay change
free
take
close
open take
cancelreturn
soda serveSoda
open serveTea
1 2 3
4
5
6
7 8 9
pay change
free
take
close
open take
cancelreturn
soda serveSoda
tea serveTea
a) StateMissing
b) EventExchanged
b) WrongStartState
+ =
Mandatory
Figure 1: Approach Overview

121. www.unamur.be
Featured Mutants (2015)
gilles.perrouin@unamur.be 44
44

122. www.unamur.be
Featured Mutants (2015)
gilles.perrouin@unamur.be 44
Fast Mutation Analysis: Run analysis once against all
mutants ≃ Mutant Schemata at the model level
44

123. www.unamur.be
Featured Mutants (2015)
gilles.perrouin@unamur.be 44
Fast Mutation Analysis: Run analysis once against all
mutants ≃ Mutant Schemata at the model level
44

124. www.unamur.be
Featured Mutants (2015)
gilles.perrouin@unamur.be 44
Fast Mutation Analysis: Run analysis once against all
mutants ≃ Mutant Schemata at the model level
Selective Mutation: Use the feature model to
configure your mutants at any order you want
44

125. www.unamur.be
Featured Mutants (2015)
gilles.perrouin@unamur.be 44
Fast Mutation Analysis: Run analysis once against all
mutants ≃ Mutant Schemata at the model level
Selective Mutation: Use the feature model to
configure your mutants at any order you want
44

126. www.unamur.be
Featured Mutants (2015)
gilles.perrouin@unamur.be 44
Fast Mutation Analysis: Run analysis once against all
mutants ≃ Mutant Schemata at the model level
Selective Mutation: Use the feature model to
configure your mutants at any order you want
EMP: FTS being formal, EMP can be mathematically
defined using trace equivalence relations (expensive
currently Monte-Carlo simulation is used instead)
44

127. www.unamur.be
Featured Mutants (2015)
45
gilles.perrouin@unamur.be 45
Minimal Val. 11.00 msec. < 0.01 msec. < 0.01 msec. 5.37 ⇥ 107
msec. 3.69 ⇥ 106
msec. < 0.01 msec.
Maximal Val. 105.00 msec. 14.00 msec. 7.00 msec. 2.30 ⇥ 108
msec. 3.01 ⇥ 107
msec. 40.00 msec.
Median 12.00 msec. 2.00 msec. < 0.01 msec. 8.49 ⇥ 107
msec. 6.05 ⇥ 106
msec. < 0.01 msec.
Mean 12.45 msec. 2.63 msec. 0.04 msec. 9.18 ⇥ 107
msec. 7.14 ⇥ 106
msec. 0.30 msec.
Std. Dev. 3.93 msec. 1.35 msec. 0.21 msec. 2.68 ⇥ 107
msec. 3.30 ⇥ 106
msec. 1.81 msec.
●●●●
●
●●
●
●
●
●●●●●●●●●
●●
●●
●
●
●●●
●
●●
●
●
●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●●●
●●●
●
●
●
●●●●●●
●
●●●●
●●
●●●●●●●
●
●
●●●
●●●●
●
●●●●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●
●●●
●●
●
●●●
●
●●●●
●
●●
●
●●●●●●●
●
●
●
●
●
●●●●●
●●
●●●●
●
●●●
●●
●
●●
●●●
●
●
●●●
●●
●
●●●
●
●
●
●●●●●●●●
●
●
●●●●●●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●●●
●●●●●●●●●●●●●●●●●
●●●
●●
●●●
●
●●●●●
●
●●
●
●●●●●●●●●●
●
●
●●
●
●●●
●●●●
●●●●
●●
●●●●
●
●
●
●●●●●●●
●
●
●
●●
●
●●●●●●●
●
●
●
●●●
● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
020406080100 Minepump
Time(inmillisec.)
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●●
●
●●●●●●●
●
●
●●
●
●●
●●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●●●●
●
●●●
●●
●
●
●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
0.0e+005.0e+071.0e+081.5e+082.0e+08
Claroline
Time(inmillisec.)
Figure 4: Mutants Execution Time by Model and Mutant Type

128. www.unamur.be
Featured Mutants (2015)
45
gilles.perrouin@unamur.be 45
Minimal Val. 11.00 msec. < 0.01 msec. < 0.01 msec. 5.37 ⇥ 107
msec. 3.69 ⇥ 106
msec. < 0.01 msec.
Maximal Val. 105.00 msec. 14.00 msec. 7.00 msec. 2.30 ⇥ 108
msec. 3.01 ⇥ 107
msec. 40.00 msec.
Median 12.00 msec. 2.00 msec. < 0.01 msec. 8.49 ⇥ 107
msec. 6.05 ⇥ 106
msec. < 0.01 msec.
Mean 12.45 msec. 2.63 msec. 0.04 msec. 9.18 ⇥ 107
msec. 7.14 ⇥ 106
msec. 0.30 msec.
Std. Dev. 3.93 msec. 1.35 msec. 0.21 msec. 2.68 ⇥ 107
msec. 3.30 ⇥ 106
msec. 1.81 msec.
●●●●
●
●●
●
●
●
●●●●●●●●●
●●
●●
●
●
●●●
●
●●
●
●
●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●●●
●●●
●
●
●
●●●●●●
●
●●●●
●●
●●●●●●●
●
●
●●●
●●●●
●
●●●●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●
●●●
●●
●
●●●
●
●●●●
●
●●
●
●●●●●●●
●
●
●
●
●
●●●●●
●●
●●●●
●
●●●
●●
●
●●
●●●
●
●
●●●
●●
●
●●●
●
●
●
●●●●●●●●
●
●
●●●●●●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●●●
●●●●●●●●●●●●●●●●●
●●●
●●
●●●
●
●●●●●
●
●●
●
●●●●●●●●●●
●
●
●●
●
●●●
●●●●
●●●●
●●
●●●●
●
●
●
●●●●●●●
●
●
●
●●
●
●●●●●●●
●
●
●
●●●
● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
020406080100 Minepump
Time(inmillisec.)
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●●
●
●●●●●●●
●
●
●●
●
●●
●●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●●●●
●
●●●
●●
●
●
●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
0.0e+005.0e+071.0e+081.5e+082.0e+08
Claroline
Time(inmillisec.)
Figure 4: Mutants Execution Time by Model and Mutant Type
Mean speedup:
100 times

129. www.unamur.be
Featured Mutants (2015)
45
gilles.perrouin@unamur.be 45
Minimal Val. 11.00 msec. < 0.01 msec. < 0.01 msec. 5.37 ⇥ 107
msec. 3.69 ⇥ 106
msec. < 0.01 msec.
Maximal Val. 105.00 msec. 14.00 msec. 7.00 msec. 2.30 ⇥ 108
msec. 3.01 ⇥ 107
msec. 40.00 msec.
Median 12.00 msec. 2.00 msec. < 0.01 msec. 8.49 ⇥ 107
msec. 6.05 ⇥ 106
msec. < 0.01 msec.
Mean 12.45 msec. 2.63 msec. 0.04 msec. 9.18 ⇥ 107
msec. 7.14 ⇥ 106
msec. 0.30 msec.
Std. Dev. 3.93 msec. 1.35 msec. 0.21 msec. 2.68 ⇥ 107
msec. 3.30 ⇥ 106
msec. 1.81 msec.
●●●●
●
●●
●
●
●
●●●●●●●●●
●●
●●
●
●
●●●
●
●●
●
●
●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●●●
●●●
●
●
●
●●●●●●
●
●●●●
●●
●●●●●●●
●
●
●●●
●●●●
●
●●●●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●
●●●
●●
●
●●●
●
●●●●
●
●●
●
●●●●●●●
●
●
●
●
●
●●●●●
●●
●●●●
●
●●●
●●
●
●●
●●●
●
●
●●●
●●
●
●●●
●
●
●
●●●●●●●●
●
●
●●●●●●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●●●
●●●●●●●●●●●●●●●●●
●●●
●●
●●●
●
●●●●●
●
●●
●
●●●●●●●●●●
●
●
●●
●
●●●
●●●●
●●●●
●●
●●●●
●
●
●
●●●●●●●
●
●
●
●●
●
●●●●●●●
●
●
●
●●●
● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
020406080100 Minepump
Time(inmillisec.)
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●●
●
●●●●●●●
●
●
●●
●
●●
●●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●●●●
●
●●●
●●
●
●
●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
0.0e+005.0e+071.0e+081.5e+082.0e+08
Claroline
Time(inmillisec.)
Figure 4: Mutants Execution Time by Model and Mutant Type
Mean speedup:
100 times
Mean speedup:
1,000,000 times

130. www.unamur.be
Featured Mutants (2015)
45
gilles.perrouin@unamur.be 45
Minimal Val. 11.00 msec. < 0.01 msec. < 0.01 msec. 5.37 ⇥ 107
msec. 3.69 ⇥ 106
msec. < 0.01 msec.
Maximal Val. 105.00 msec. 14.00 msec. 7.00 msec. 2.30 ⇥ 108
msec. 3.01 ⇥ 107
msec. 40.00 msec.
Median 12.00 msec. 2.00 msec. < 0.01 msec. 8.49 ⇥ 107
msec. 6.05 ⇥ 106
msec. < 0.01 msec.
Mean 12.45 msec. 2.63 msec. 0.04 msec. 9.18 ⇥ 107
msec. 7.14 ⇥ 106
msec. 0.30 msec.
Std. Dev. 3.93 msec. 1.35 msec. 0.21 msec. 2.68 ⇥ 107
msec. 3.30 ⇥ 106
msec. 1.81 msec.
●●●●
●
●●
●
●
●
●●●●●●●●●
●●
●●
●
●
●●●
●
●●
●
●
●●●●●●●●●●●●●●●
●
●●●●●●●●●●
●
●
●●●
●●●
●
●
●
●●●●●●
●
●●●●
●●
●●●●●●●
●
●
●●●
●●●●
●
●●●●●
●
●●
●
●●●●●●●●●●●●●●●
●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●
●●●
●●
●
●●●
●
●●●●
●
●●
●
●●●●●●●
●
●
●
●
●
●●●●●
●●
●●●●
●
●●●
●●
●
●●
●●●
●
●
●●●
●●
●
●●●
●
●
●
●●●●●●●●
●
●
●●●●●●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●●●
●●●●●●●●●●●●●●●●●
●●●
●●
●●●
●
●●●●●
●
●●
●
●●●●●●●●●●
●
●
●●
●
●●●
●●●●
●●●●
●●
●●●●
●
●
●
●●●●●●●
●
●
●
●●
●
●●●●●●●
●
●
●
●●●
● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
020406080100 Minepump
Time(inmillisec.)
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●●
●
●●●●●●●
●
●
●●
●
●●
●●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●●●●
●
●●●
●●
●
●
●
●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
Alive Killed Mutants FTS
0.0e+005.0e+071.0e+081.5e+082.0e+08
Claroline
Time(inmillisec.)
Figure 4: Mutants Execution Time by Model and Mutant Type
Mean speedup:
100 times
Mean speedup:
1,000,000 times
Devroey et al. Featured Mutants: a Mutant Family Approach to Model-Based Mutation Testing
Submitted to ISSTA’15

131. CONCLUSIONS

132. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
47

133. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
47

134. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
47

135. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
Not only fun but may turn useful even for single systems
47

136. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
Not only fun but may turn useful even for single systems
47

137. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
Not only fun but may turn useful even for single systems
Flexibility is key to be scalable (also you need to admit
you won’t be exact => heuristics)
47

138. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
Not only fun but may turn useful even for single systems
Flexibility is key to be scalable (also you need to admit
you won’t be exact => heuristics)
47

139. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
Not only fun but may turn useful even for single systems
Flexibility is key to be scalable (also you need to admit
you won’t be exact => heuristics)
Main future work remains getting automatically
(behavioural) models from VIS, stay tuned :)
47

140. www.unamur.be
Conclusion
gilles.perrouin@unamur.be 47
Few experiences in testing VIS
Triskell saw metamodels everywhere, now I see VIS
everywhere :)
Not only fun but may turn useful even for single systems
Flexibility is key to be scalable (also you need to admit
you won’t be exact => heuristics)
Main future work remains getting automatically
(behavioural) models from VIS, stay tuned :)
47

141. www.unamur.be
Thanks
Benoit Baudry, Sagar Sen, Jacques Klein, Yves Le Traon,
Sebastian Oster, Christopher Hénard, Mike Papadakis
Xavier Devroey, Maxime Cordy, Patrick Heymans, Pierre-
Yves Schobbens, Axel Legay, Andreas Classen
48gilles.perrouin@unamur.be

142. MISC COLLABORATIVE WORK IN
PROGRESS

143. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
50

144. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
50

145. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
50

146. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
Cross-cutting contraints (why this option is disappearing if I
choose this ?)
50

147. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
Cross-cutting contraints (why this option is disappearing if I
choose this ?)
Multiple stakeholders
50

148. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
Cross-cutting contraints (why this option is disappearing if I
choose this ?)
Multiple stakeholders
Visualisation techniques helpful in this respect
50

149. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
Cross-cutting contraints (why this option is disappearing if I
choose this ?)
Multiple stakeholders
Visualisation techniques helpful in this respect
Quite unexplored area (Botterweck, Pleuss)
50

150. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
Cross-cutting contraints (why this option is disappearing if I
choose this ?)
Multiple stakeholders
Visualisation techniques helpful in this respect
Quite unexplored area (Botterweck, Pleuss)
Most VM IDE (pure::variants, FeatureIDE) offers only tree based
representation => inefficient in many respect (size, constraints)
50

151. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
Configuring a VIS is cognitively difficult
Many features (options)
Cross-cutting contraints (why this option is disappearing if I
choose this ?)
Multiple stakeholders
Visualisation techniques helpful in this respect
Quite unexplored area (Botterweck, Pleuss)
Most VM IDE (pure::variants, FeatureIDE) offers only tree based
representation => inefficient in many respect (size, constraints)
Working with visualisation experts (Nicolas Biri, Celine
Sauvage-Thomase) at LIST (Luxembourg) to derive a way
of choosing visualisations for configurations tasks
50

152. www.unamur.be gilles.perrouin@unamur.be
Visualisation of VIS Configurations
51
C. Sauvage-Thomase et al. Towards a systematic
approach for cognitively efficient configuration
visualizations, JLDP 2014

153. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
52

154. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
Generalisation of the preceding for Graphical DSL
concrete syntaxes
52

155. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
Generalisation of the preceding for Graphical DSL
concrete syntaxes
Based on the “Physics of Notations” by D.L. Moody (2009)
and “Semiology of Graphics” by Bertin (1967)
52

156. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
Generalisation of the preceding for Graphical DSL
concrete syntaxes
Based on the “Physics of Notations” by D.L. Moody (2009)
and “Semiology of Graphics” by Bertin (1967)
52

157. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
Generalisation of the preceding for Graphical DSL
concrete syntaxes
Based on the “Physics of Notations” by D.L. Moody (2009)
and “Semiology of Graphics” by Bertin (1967)
PhD Thesis of Nicolas Genon (UNamur)
52

158. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
Generalisation of the preceding for Graphical DSL
concrete syntaxes
Based on the “Physics of Notations” by D.L. Moody (2009)
and “Semiology of Graphics” by Bertin (1967)
PhD Thesis of Nicolas Genon (UNamur)
P. Caire, et al. "Visual notation design 2.0: Towards user
comprehensible requirements engineering notations."
Requirements Engineering Conference (RE), 2013
52

159. www.unamur.be gilles.perrouin@unamur.be
Cognitively Efficient Visual Languages
Generalisation of the preceding for Graphical DSL
concrete syntaxes
Based on the “Physics of Notations” by D.L. Moody (2009)
and “Semiology of Graphics” by Bertin (1967)
PhD Thesis of Nicolas Genon (UNamur)
P. Caire, et al. "Visual notation design 2.0: Towards user
comprehensible requirements engineering notations."
Requirements Engineering Conference (RE), 2013
We are looking for more complex examples in various domains
(e.g. Airbus Designs)
52