Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Variability-Intensive Systems, tutorial SPLC 2017, part I

6 views

Published on

Tutorial presented at the 21st International Systems and Software Product Line Conference

Published in: Software
  • Be the first to comment

  • Be the first to like this

Testing Variability-Intensive Systems, tutorial SPLC 2017, part I

  1. 1. www.unamur.be Testing Variability-intensive Systems Gilles Perrouin Xavier Devroey Maxime Cordy SPLC ’17, Sevilla, Spain @GPerrouin @xdevroey gilles.perrouin@unamur.be xavier.devroey@unamur.be maxime.cordy@unamur.be
  2. 2. www.unamur.be Testing Variability-intensive Systems SPLC ’17, Sevilla, Spain
  3. 3. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  4. 4. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  5. 5. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  6. 6. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  7. 7. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  8. 8. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  9. 9. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf) 9,999,999
  10. 10. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  11. 11. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf) ?!?
  12. 12. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  13. 13. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf) -1 ?!?
  14. 14. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  15. 15. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf) "sfdeljknesv" ?!?
  16. 16. System
 under
 test
  17. 17. Menu System
 under
 test
  18. 18. Menu System
 under
 testSpecification
  19. 19. Menu System
 under
 testSpecification
  20. 20. Menu System
 under
 testSpecification Test case
  21. 21. Menu System
 under
 testSpecification Test case
  22. 22. Menu System
 under
 testSpecification Test case Output
  23. 23. Menu System
 under
 testSpecification Test case Output
  24. 24. Menu System
 under
 testSpecification Test case Output Pass or fail?
  25. 25. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. [SWEBOK v3] Software-intensive Systems Testing
  26. 26. www.unamur.be Testing Variability-intensive Systems SPLC ’17, Sevilla, Spain
  27. 27. [A Software Product Line (SPL) is]
 a set of software-intensive systems that share a common, managed set of features satisfying the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way P. C. Clements and L. Northrop — ‘‘Software Product Lines:
 Practices and Patterns’’.
 SEI Series in Software Engineering. Addison-Wesley, 2001. Variability-Intensive Systems: SPL
  28. 28. Hall of Fame splc.net/fame.html
  29. 29. Hall of Fame splc.net/fame.html
  30. 30. gilles.perrouin@unamur.be 19 Variability-Intensive Systems: Configurable Systems
  31. 31. 20
  32. 32. 20
  33. 33. 20
  34. 34. 20
  35. 35. 20
  36. 36. 20
  37. 37. 20
  38. 38. 21 gilles.perrouin@unamur.be 21 We therefore regroup all these systems under the “variability-intensive systems” umbrella for feature- based testing…. …thanks to a very liberal notion of feature! Variability-Intensive Systems
  39. 39. gilles.perrouin@unamur.be 22 Variability-Intensive Systems Michael Jackson: “features are behaviours”
  40. 40. gilles.perrouin@unamur.be Christian Kästner: “features are configuration 22 Variability-Intensive Systems Michael Jackson: “features are behaviours”
  41. 41. gilles.perrouin@unamur.be Christian Kästner: “features are configuration 22 Variability-Intensive Systems Michael Jackson: “features are behaviours”
  42. 42. gilles.perrouin@unamur.be Christian Kästner: “features are configuration Features are units of variability 22 Variability-Intensive Systems Michael Jackson: “features are behaviours”
  43. 43. gilles.perrouin@unamur.be Christian Kästner: “features are configuration Features are units of variability 22 Variability-Intensive Systems Michael Jackson: “features are behaviours”
  44. 44. gilles.perrouin@unamur.be Christian Kästner: “features are configuration Features are units of variability 22 Variability-Intensive Systems Michael Jackson: “features are behaviours”
  45. 45. Feature Model (FM)
  46. 46. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  47. 47. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  48. 48. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  49. 49. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  50. 50. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  51. 51. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  52. 52. Feature Model (FM) CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.
  53. 53. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. Software-intensive Systems Testing [SWEBOK v3]
  54. 54. Variability-intensive systems testing consists of the dynamic verification 
 that a a set of software-intensive systems programs provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. Variability-Intensive Systems Testing
  55. 55. Features 0
  56. 56. 33 Features 0
  57. 57. 33 320 Features 0
  58. 58. 33 320 6,888 Features 0
  59. 59. 32
  60. 60. 32 Set of programs can be HUUUGE!
  61. 61. 32 Set of programs can be HUUUGE! # test cases for one program already huge
  62. 62. 32 Set of programs can be HUUUGE! # test cases for one program already huge So we have a huge*HUUGE problem!!!
  63. 63. 33 gilles.perrouin@unamur.be 33 We would like to provide you an overview of the model-based techniques to test variability- intensive systems, considering both structural and behavioural concerns. Tutorial Objective
  64. 64. 34 gilles.perrouin@unamur.be 34 Part I (Until 16h) : sample configurations to test Feature modelling Sampling techniques Discussion on the JHipster case Part II (16h30-18h) : select behaviours to test A (short) introduction to behaviour modelling Behaviour-driven test case selection Test concretisation Open Discussion (18h-18h30) Tutorial Agenda
  65. 65. www.unamur.be Part I: Sampling Techniques SPLC ’17, Sevilla, Spain
  66. 66. 36 gilles.perrouin@unamur.be 36 Feature Modelling Ecosystem Academic Marcilio Mendonca, Moises Branco, Donald Cowan: S.P.L.O.T. - Software Product Lines Online Tools. OOPSLA Companion, October 2009, Orlando, Florida, USA. Christian Kästner, Thomas Thüm, Gunter Saake, Janet Feigenspan, Thomas Leich, Fabian Wielgorz, and Sven Apel. FeatureIDE: Tool Framework for Feature-Oriented Software Development. ICSE 2009. Formal Demonstration paper. Mathieu Acher, Philippe Collet, Philippe Lahire, Robert France. FAMILIAR: A Domain- Specific Language for Large Scale Management of Feature Models. Science of Computer Programming, Elsevier, 2013, 78 (6) Antkiewicz, M., K. Bąk, A. Murashkin, R. Olaechea, J. Liang, and K. Czarnecki, "Clafer Tools for Product Line Engineering", Software Product Line Conference, Tokyo, Japan, 2013. root Website { group [0..2] { Foo, Bar, Baz, } Baz requires Bar; }Andreas Classen, Quentin Boucher, Patrick Heymans, A text-based approach to feature modelling: Syntax and semantics of TVL, In Science of Computer Programming, Volume 76, Issue 12, 2011 TVL Haugen, Ø., Wasowski, A., & Czarnecki, K. (2012, September). CVL: common variability language. In SPLC (2) (pp. 266-267).
  67. 67. 36 gilles.perrouin@unamur.be 36 Feature Modelling Ecosystem Academic Marcilio Mendonca, Moises Branco, Donald Cowan: S.P.L.O.T. - Software Product Lines Online Tools. OOPSLA Companion, October 2009, Orlando, Florida, USA. Christian Kästner, Thomas Thüm, Gunter Saake, Janet Feigenspan, Thomas Leich, Fabian Wielgorz, and Sven Apel. FeatureIDE: Tool Framework for Feature-Oriented Software Development. ICSE 2009. Formal Demonstration paper. Mathieu Acher, Philippe Collet, Philippe Lahire, Robert France. FAMILIAR: A Domain- Specific Language for Large Scale Management of Feature Models. Science of Computer Programming, Elsevier, 2013, 78 (6) Antkiewicz, M., K. Bąk, A. Murashkin, R. Olaechea, J. Liang, and K. Czarnecki, "Clafer Tools for Product Line Engineering", Software Product Line Conference, Tokyo, Japan, 2013. root Website { group [0..2] { Foo, Bar, Baz, } Baz requires Bar; }Andreas Classen, Quentin Boucher, Patrick Heymans, A text-based approach to feature modelling: Syntax and semantics of TVL, In Science of Computer Programming, Volume 76, Issue 12, 2011 TVL Haugen, Ø., Wasowski, A., & Czarnecki, K. (2012, September). CVL: common variability language. In SPLC (2) (pp. 266-267). Most Influential Paper Award SPLC 2017 !
  68. 68. 37 gilles.perrouin@unamur.be 37 Feature Modelling Ecosystem Commercial Tutorial: Using Feature Models to Manage Variability and Requirements Reuse - Danilo Beuche SPLC 2017 Tutorial: Feature-Based Systems and Software Product Line Engineering: PLE for the Enterprise - Charles Krueger and Paul Clements SPLC 2017
  69. 69. AIM HIGHER WITH SKALUP Discover what we can do for you VISIT US @ SKALUP.COM MAXIME.CORDY@SKALUP.COM PATRICK.HEYMANS@SKALUP.COM
  70. 70. Workbench SkalUP ecosystem
  71. 71. Workbench TDL TVL edits SkalUP ecosystem
  72. 72. Workbench TDL TVL edits Engine used by SkalUP ecosystem
  73. 73. Workbench TDL TVL edits Configurator invokes Engine used by SkalUP ecosystem
  74. 74. Workbench GUI
 Generator invokes generates TDL TVL edits Configurator invokes Engine used by SkalUP ecosystem
  75. 75. Workbench GUI
 Generator invokes generates TDL TVL edits Configurator invokes Engine used by FCSS TVDL edits used by SkalUP ecosystem
  76. 76. 40 gilles.perrouin@unamur.be 40 Source code Existing configurators Product Descriptions Architecture … Reverse Engineering of Feature Models
  77. 77. 40 gilles.perrouin@unamur.be 40 Source code Existing configurators Product Descriptions Architecture … Reverse Engineering of Feature Models J. Sincero et al. Efficient extraction and analysis of preprocessor-based variability. GPCE '10.
  78. 78. 40 gilles.perrouin@unamur.be 40 Source code Existing configurators Product Descriptions Architecture … Reverse Engineering of Feature Models J. Sincero et al. Efficient extraction and analysis of preprocessor-based variability. GPCE '10. S. She et al., Reverse engineering feature models, ICSE 2011
  79. 79. 40 gilles.perrouin@unamur.be 40 Source code Existing configurators Product Descriptions Architecture … Reverse Engineering of Feature Models J. Sincero et al. Efficient extraction and analysis of preprocessor-based variability. GPCE '10. S. She et al., Reverse engineering feature models, ICSE 2011 JM. Davril et al. Feature model extraction from large collections of informal product descriptions. ESEC/FSE 2013.
  80. 80. 40 gilles.perrouin@unamur.be 40 Source code Existing configurators Product Descriptions Architecture … Reverse Engineering of Feature Models J. Sincero et al. Efficient extraction and analysis of preprocessor-based variability. GPCE '10. S. She et al., Reverse engineering feature models, ICSE 2011 JM. Davril et al. Feature model extraction from large collections of informal product descriptions. ESEC/FSE 2013. M. Acher, et al . Reverse engineering architectural feature models. ECSA'11
  81. 81. 40 gilles.perrouin@unamur.be 40 Source code Existing configurators Product Descriptions Architecture … Reverse Engineering of Feature Models REVE 2017 5th International Workshop on Reverse Variability Engineering J. Sincero et al. Efficient extraction and analysis of preprocessor-based variability. GPCE '10. S. She et al., Reverse engineering feature models, ICSE 2011 JM. Davril et al. Feature model extraction from large collections of informal product descriptions. ESEC/FSE 2013. M. Acher, et al . Reverse engineering architectural feature models. ECSA'11
  82. 82. 41 gilles.perrouin@unamur.be 41 Sampling
  83. 83. Menu Menu Menu Menu Menu Menu
  84. 84. Menu Menu Menu Menu Menu Menu
  85. 85. Menu Menu Menu Menu Menu Menu
  86. 86. Menu Menu Menu Menu Menu
  87. 87. Menu Menu Menu Menu
  88. 88. Menu Menu Menu
  89. 89. Menu Menu
  90. 90. Menu Menu Test case
  91. 91. Menu Menu Test case Output
  92. 92. Menu Menu Test case Output Pass or fail?
  93. 93. Menu Menu Test case Output Pass or fail? Test case
  94. 94. Menu Menu Test case Output Pass or fail? Test case Output
  95. 95. Menu Menu Test case Output Pass or fail? Test case Output Pass or fail?
  96. 96. 44 gilles.perrouin@unamur.be 44 Assumption Most of the faults are due to the undesired interactions of a limited number of options Core idea Extract configurations so that all combinations of options are covered (at least) once Combinatorial Interaction Testing D. M. Cohen et al., The AETG system: an approach to testing based on combinatorial design. TSE, 1997 D. Kuhn et al, Software fault interactions and implications for software testing. TSE 2004
  97. 97. 45 CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt. Combinatorial Interaction Testing
  98. 98. 45 CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.{t, ps}, {t, c}, {dd, cr}, {dd,on}, {dd, off}, … Combinatorial Interaction Testing
  99. 99. 45 CPTerminal t Identification id DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig requires PaymentSchema ps Connectivity c [1..*] opt.{t, ps}, {t, c}, {dd, cr}, {dd,on}, {dd, off}, … Combinatorial Interaction Testing not only for 2-tuples (pairwise), interaction strength=[1(?),6]
  100. 100. 46 gilles.perrouin@unamur.be 46 CIT for Variability Intensive Systems
  101. 101. 46 gilles.perrouin@unamur.be 46 Huge reduction of the number of configurations to consider (2-wise): Printers: 172 features, 1.14E27 configs => 180 configs Linux: 6,888 features, ? configs => 480 configs CIT for Variability Intensive Systems
  102. 102. 46 gilles.perrouin@unamur.be 46 Huge reduction of the number of configurations to consider (2-wise): Printers: 172 features, 1.14E27 configs => 180 configs Linux: 6,888 features, ? configs => 480 configs CIT for Variability Intensive Systems Martin Fagereng Johansen et al. An algorithm for generating t-wise covering arrays from large feature models. SPLC ‘12
  103. 103. 46 gilles.perrouin@unamur.be 46 Huge reduction of the number of configurations to consider (2-wise): Printers: 172 features, 1.14E27 configs => 180 configs Linux: 6,888 features, ? configs => 480 configs The main issue is that CIT extraction is an NP- complete problem when there are constraints… CIT for Variability Intensive Systems Martin Fagereng Johansen et al. An algorithm for generating t-wise covering arrays from large feature models. SPLC ‘12
  104. 104. 47 gilles.perrouin@unamur.be 47 Alloy
  105. 105. 47 gilles.perrouin@unamur.be 47 Alloy Software Product Line Feature Diagram 1. Transformation FeatureDiagram2Alloy 2. Generation of initial T-wise tuples l 3. Detection of Valid tuples 4. Creating and Solving Conjunctions of Tuples BinarySplit Incremental Growth 5. Analysis Value of T Alloy Feature Diagram AF Set of Valid Tuples V [Min,Max] Scope [Min,Max] Duration Selection Strategy Set of Configurations covering all Valid Tuples P
  106. 106. 47 gilles.perrouin@unamur.be 47 Alloy Software Product Line Feature Diagram 1. Transformation FeatureDiagram2Alloy 2. Generation of initial T-wise tuples l 3. Detection of Valid tuples 4. Creating and Solving Conjunctions of Tuples BinarySplit Incremental Growth 5. Analysis Value of T Alloy Feature Diagram AF Set of Valid Tuples V [Min,Max] Scope [Min,Max] Duration Selection Strategy Set of Configurations covering all Valid Tuples P G. Perrouin et al., Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines, ICST 2010
  107. 107. 48 gilles.perrouin@unamur.be 48 MoSo-Polite 4. MOSO-POLITE MoSo-PoLiTe combines the aforementioned techniques Reu- se Techniques and Subset-Heuristics providing a framework to generate a subset of products including an automatic model-based test case generation for each product of the subset [22]. Within this contribution we only focus on the subset generation and ignore the model-based test case ge- neration. Feature Model Flat Feature Model Flat Feature Model CSP Flattening Parameter Extraction Subset Extractor Set of Configurations Feature Model Parsing Variant Models Products Feature Model Combinatorial Testing Configuration Derivation Product Derivation select Figure 2: MoSo-PoLiTe Framework S. Oster et al., "Automated incremental pairwise testing of software product lines.”SPLC 2010. M. Steffens et al., Industrial evaluation of pairwise spl testing with moso-polite. VAMOS 2012
  108. 108. 49 gilles.perrouin@unamur.be 49 Another constraint based approach based on prolog Generates 30% less configurations than MoSo- Polite PACOGEN Hervieu et al., PACOGEN: Automatic Generation of Pairwise Test Configurations from Feature Models. ISSRE’ 11
  109. 109. 50 gilles.perrouin@unamur.be 50 Use search-based techniques (Simulated Annealing) to minimise the size of covering array Usually gives very small covering array compared to others CASA M. B. Cohen et al., “Augmenting simulated annealing to build interaction test suites,” ISSRE 2003
  110. 110. 51 gilles.perrouin@unamur.be 51 Use an incremental way of accommodating tuples Deterministic algorithm always produce the same answer for a given FM Focus on scalability Industrial version: pairwiser (see pairwiser.org) SPLCAT M. F. Johansen, Ø. Haugen, and F. Fleurey, “An algorithm for generating t- wise covering arrays from large feature models,” SPLC 2012.
  111. 111. 52 gilles.perrouin@unamur.be 52 Open Issues in CIT
  112. 112. 52 gilles.perrouin@unamur.be 52 Open Issues in CIT
  113. 113. 52 gilles.perrouin@unamur.be 52 Despite the improvements in the CIT techniques that can now scale to thousands of features, there are two problems: Open Issues in CIT
  114. 114. 52 gilles.perrouin@unamur.be 52 Despite the improvements in the CIT techniques that can now scale to thousands of features, there are two problems: Higher order interaction remains expensive/ intractable Open Issues in CIT
  115. 115. 52 gilles.perrouin@unamur.be 52 Despite the improvements in the CIT techniques that can now scale to thousands of features, there are two problems: Higher order interaction remains expensive/ intractable 100% coverage implies no flexibility on your testing budget (time + number of configurations) Open Issues in CIT
  116. 116. 53 gilles.perrouin@unamur.be 53 Mimicking CIT with Dissimilarity
  117. 117. 53 gilles.perrouin@unamur.be 53 Diversity has been proposed as a successful technique to find bugs Mimicking CIT with Dissimilarity H. Hemmati et al., Achieving scalable model-based testing through test case diversity, TOSEM 2012.
  118. 118. 53 gilles.perrouin@unamur.be 53 Diversity has been proposed as a successful technique to find bugs Intuition Dissimilar configurations cover more t-tuples than similar ones Mimicking CIT with Dissimilarity H. Hemmati et al., Achieving scalable model-based testing through test case diversity, TOSEM 2012.
  119. 119. 53 gilles.perrouin@unamur.be 53 Diversity has been proposed as a successful technique to find bugs Intuition Dissimilar configurations cover more t-tuples than similar ones Mimicking CIT with Dissimilarity H. Hemmati et al., Achieving scalable model-based testing through test case diversity, TOSEM 2012. C. Henard et al., "Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines," TSE 2014
  120. 120. 54 gilles.perrouin@unamur.be 54 Mimicking CIT with Dissimilarity
  121. 121. 54 gilles.perrouin@unamur.be 54 SBSE well-suited for large configurations spaces Generated configurations are valid (via SAT4J) Designed for Scalability Fitness function based on distance: correlated with t-wise coverage but easier to compute Flexibility Testers decide on generation time and # configurations Configurations are prioritized w.r.t fitness function: use a subset if lack of resources Mimicking CIT with Dissimilarity
  122. 122. 55 Linux FM Mimicking t-wise ?
  123. 123. 56 120 FMs ∈ [11;1,000] features Prioritization
  124. 124. 56 Prioritization
  125. 125. 56 More efficient for high values of t Prioritization
  126. 126. Ecos: 1 % more coverage implies 2,1E+15 additional 6-tuples ! 1,000 configurations may not be enough... Taming large FMs 57 t=6 (1,000 confs) 0 runs 15,000 runs eCos (1,244 features) 94.191% 95.343% FreeBSD (1,396 features) 76.236% 76.494% Linux (6,888 features) 89.411% 90.671%
  127. 127. 58 gilles.perrouin@unamur.be 58 Tool Support C. Henard et al., PLEDGE: a product line editor and test generation tool SPLC 2013 (Vol.2) https://github.com/christopherhenard/pledge/
  128. 128. 59 gilles.perrouin@unamur.be 59 Other Sampling Techniques
  129. 129. 59 gilles.perrouin@unamur.be 59 Multi-objective Combining similarity and cost Optimisation w.r.t. to defects, costs, # unused/ selected features, correctness Other Sampling Techniques C. Henard, et al. Multi-objective test generation for software product lines. SPLC 2013 C. Henard et al. Combining multi-objective search and constraint solving for configuring large software product lines. ICSE 2015.
  130. 130. 59 gilles.perrouin@unamur.be 59 Multi-objective Combining similarity and cost Optimisation w.r.t. to defects, costs, # unused/ selected features, correctness Random Other Sampling Techniques C. Henard, et al. Multi-objective test generation for software product lines. SPLC 2013 C. Henard et al. Combining multi-objective search and constraint solving for configuring large software product lines. ICSE 2015. Andrea Arcuri et al. Formal Analysis of the Probability of Interaction Fault Detection Using Random Testing. TSE 2012
  131. 131. 59 gilles.perrouin@unamur.be 59 Multi-objective Combining similarity and cost Optimisation w.r.t. to defects, costs, # unused/ selected features, correctness Random Min-Max selection Other Sampling Techniques C. Henard, et al. Multi-objective test generation for software product lines. SPLC 2013 C. Henard et al. Combining multi-objective search and constraint solving for configuring large software product lines. ICSE 2015. Andrea Arcuri et al. Formal Analysis of the Probability of Interaction Fault Detection Using Random Testing. TSE 2012 I. Abal, et al. 42 variability bugs in the Linux kernel: A qualitative analysis. ASE 2014 F. Medeiros et al., A Comparison of 10 Sampling Algorithms for Configurable Systems, ICSE 16
  132. 132. 60 gilles.perrouin@unamur.be 60 Direct comparison Using mutation analysis Comparing/Evaluating Sampling Techniques F. Medeiros et al., A Comparison of 10 Sampling Algorithms for Configurable Systems, ICSE 16 G. Perrouin et al., Pairwise testing for software product lines: comparison of two approaches, Softw. Qual. Journal 2012. C. Henard et al., Assessing software product line testing via model-based mutation: An application to similarity testing, ICST Workshops 2013
  133. 133. 61 gilles.perrouin@unamur.be 61 Conclusion on Sampling Techniques
  134. 134. 61 gilles.perrouin@unamur.be 61 Many sampling ways CIT effective at finding bugs but expensive in some cases SBSE useful for fitting test budget while ensuring decent coverage Conclusion on Sampling Techniques
  135. 135. 61 gilles.perrouin@unamur.be 61 Many sampling ways CIT effective at finding bugs but expensive in some cases SBSE useful for fitting test budget while ensuring decent coverage Conclusion on Sampling Techniques
  136. 136. 61 gilles.perrouin@unamur.be 61 Many sampling ways CIT effective at finding bugs but expensive in some cases SBSE useful for fitting test budget while ensuring decent coverage More research is required to compare techniques… Conclusion on Sampling Techniques
  137. 137. 61 gilles.perrouin@unamur.be 61 Many sampling ways CIT effective at finding bugs but expensive in some cases SBSE useful for fitting test budget while ensuring decent coverage More research is required to compare techniques… Conclusion on Sampling Techniques
  138. 138. 61 gilles.perrouin@unamur.be 61 Many sampling ways CIT effective at finding bugs but expensive in some cases SBSE useful for fitting test budget while ensuring decent coverage More research is required to compare techniques… …on various case studies Conclusion on Sampling Techniques
  139. 139. www.unamur.be Part I: Sampling in Practice: The JHipster Case SPLC ’17, Sevilla, Spain
  140. 140. Axel Halin Alexandre Nuttinck Dr. Mathieu
 Acher
  141. 141. 64 gilles.perrouin@unamur.be 64 Web-apps generator Spring-Boot Bootstrap / AngularJS 100 % Open Source Yeoman Bower, npm yo Used all over the world Large companies (HBO, Google, Adobe)1 Independent developers GitHub 5000+ stars 100+ releases (JHipster 3.6.1, 18 Aug 2016) JHipster Introduction 1 https://jhipster.github.io/companies-using-jhipster
  142. 142. 66 gilles.perrouin@unamur.be 66 JHipster variability prompts.js Questions Constraints Conditional compilation (EJS) Reverse Engineering JHipster’s Variability
  143. 143. 67 gilles.perrouin@unamur.be 67 162508 different configurations 2 weeks of manual effort (2 master students) Reverse Engineering JHipster’s Variability
  144. 144. • + 26,000 products • 4,376 hours (182 days) processor time • 8 person/month setup testing infrastructure [Halin et al., VaMoS ’17] Inclusive Testing Infrastructure CSV file containing the results of the 26,000 configurations Data analysis in progress…
  145. 145. 69 gilles.perrouin@unamur.be 69 Looks interesting as a case for research… …but how do the JHipster team test it ?
  146. 146. 69 gilles.perrouin@unamur.be 69 Looks interesting as a case for research… …but how do the JHipster team test it ? How would you do?
  147. 147. 70 gilles.perrouin@unamur.be 70 They like the idea of being able to run all JHipster configurations (26,000) and finding all the 6 faults reported via the analysis of the failing configurations…. …but they only have the infrastructure to test only 12 configurations (now expanded to 20) ! JHipster Team Feedback (Work In Progress)
  148. 148. 70 gilles.perrouin@unamur.be 70 They like the idea of being able to run all JHipster configurations (26,000) and finding all the 6 faults reported via the analysis of the failing configurations…. …but they only have the infrastructure to test only 12 configurations (now expanded to 20) ! JHipster Team Feedback (Work In Progress) W hich sampling strategy would you use?

×