Sampling, Re-use and Incremental Testing in
Software Product Lines
Myra B. Cohen
myra@cse.unl.edu
University of Nebraska-L...
Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introducti...
Family-Based Approach to
Development
Nokia
Mobile phones
Boeing BoldStroke Future Combat Systems
Intuit
Philips Philips
Software Product Lines
•  A software product line (SPL) is a set of programs that share a
significant common (managed) set...
Feature Oriented Domain
Analysis
•  SEI FODA Project in Late 1980s
•  Identified features (variability) as the key to soft...
Feature Models
•  Representation of the features and
dependencies/constraints in an SPL
•  Defines what is (and is not) al...
Software Product Line
Engineering
•  Consists of two parts:
– Domain engineering
•  Determining what the re-usable compone...
Domain Testing
•  Validate and verify reusable
components
•  No running application exists at this
time - only components
...
What is Variability?
Commonality
The features shared by a set of systems
Variability
The features that differ between some...
Examples of Variability
•  Electric bulb can be lit or unlit
•  Software applications support different
languages
•  A tri...
Mechanisms for
Implementing variability
–  Compile flags
–  Properties files
–  Command-line arguments
–  Inheritance
–  I...
Mobile Phone SPL
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
Testing a Product Line
•  SPL engineering decreases time to market
•  With shorter life cycles, the percentage of
time spe...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V V
Camera
Video
C...
V V V
16MC 8MC BW
Validating at the Unit Level
Display
Type
VP
V
16MC
V V V
16MC 8MC BW
Validating at the Unit Level
Display
Type
VP
V
16MC
V
8MC
V V V
16MC 8MC BW
Validating at the Unit Level
Display
Type
VP
V
16MC
V
8MC
V
BW
Focus is on reuse of artifacts during tes...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
NASA Mars Rover: Spirit
Real Interaction Failure
•  NASA Detected failed communications on
two independent channels
•  Determined that the rover w...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
Fault Map of Configurations
V4
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14
C0
C10
C20
C30
C40
C50
C60
F1 F2 F3 F4 F5
C0...
V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
C...
Some Other Approaches to
SPL Testing
•  Extended use cases for variability [Bertolino et al., SPFE 2003]
•  Integration te...
Some SPL Resources
•  http://splot-research.org/
•  http://www.isa.us.es/fama/
•  http://sir.unl.edu
Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introducti...
A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels ...
A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels ...
A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels ...
A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels ...
A Combinatorial Problem
• 33x22=108 feasible SPL instances
• 10 features each with 5 values = 510 or
9,765,625 possible in...
A Combinatorial Problem
• 33x22=108 feasible SPL instances
• 10 features each with 5 values = 510 or
9,765,625 possible in...
Combinatorial Interaction Testing
(CIT)
•  Test all pairs or t-way combinations of
feature-values, where t is a defined
st...
Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None N...
Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None N...
Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None N...
Combinatorial Interaction Testing
(CIT)
•  Based on statistical design of experiments (DOE)
–  Manufacturing
–  Drug test ...
Empirical Results on CIT
for System Inputs
•  CIT has been well studied on testing input
parameters for programs
•  [Brown...
CIT for Configurable
Systems
•  Some studies on configurable
systems
•  [Kuhn, Wallace,Gallo 04]
–  Medical devices, web b...
CIT vs. Default
Configuration of vim in SIRFaultDetection(%)
020406080100
v2 v3 v4 v5 v6 v7
Version default configuration
How do we Generate the
Samples?
•  Greedy approaches
– AETG, IPOG
– Search-based (simulated annealing, GAs)
•  Use of cons...
Meta-Heuristic Search
∑ -Set of feasible solutions
0 1 1 1 1
1 0 1 0 0
1 0 1 0 0
1 0 1 1 1
0 0 0 1 1
1 1 0 1 0
0 1 1 1 1
1...
Problems with Approach
•  May get stuck in local optima
Meta-heuristic Algorithms
•  Provide mechanisms to escape local
optima
•  Sometimes the algorithm can accept a
worse choic...
Examples
•  Genetic Algorithms
•  Simulated Annealing
•  Tabu Search
Search Based Software Engineering
•  Meant to simulate the coolng of metal
•  Physical annealing is a process of slowly cooling
a metal, in baths of decreas...
•  The basic strategy is the same as a hill climb,
but the meta-heuristic creates a gentle decrease
in the probability for...
•  Start with a solution S
•  Select a new solution S’ that is a neighbor of S
•  If S’-S ≤ 0 then S’ is accepted
•  Other...
Simulated Annealing
0 1 1 1 1
1 0 1 0 0
1 0 1 0 0
1 0 1 1 1
0 0 0 1 1
1 1 0 1 0
0 1 1 1 1
1 0 1 0 0
0 0 1 0 0
1 0 1 1 1
0 ...
Lower bound L
Binary search for “best” solution
Upper bound U
(L+U)/2
Finding “N”
The Real System
Constraints on Valid Configurations:
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
TAROT2013 Testing School - Myra Cohen presentation
Upcoming SlideShare
Loading in …5
×

TAROT2013 Testing School - Myra Cohen presentation

928 views
855 views

Published on

TAROT 2013 9th International Summer School on Training And Research On Testing, Volterra, Italy, 9-13 July, 2013

These slides summarize Myra Cohen's presentation about "Sampling, Re-use and Incremental Testing in Software Product Lines"

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
928
On SlideShare
0
From Embeds
0
Number of Embeds
55
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

TAROT2013 Testing School - Myra Cohen presentation

  1. 1. Sampling, Re-use and Incremental Testing in Software Product Lines Myra B. Cohen myra@cse.unl.edu University of Nebraska-Lincoln
  2. 2. Agenda SPL System Testing: the combinatorics SPL Integration Testing Summary Continuous Test Suite Augmentation Introduction to Software Product Lines
  3. 3. Family-Based Approach to Development Nokia Mobile phones Boeing BoldStroke Future Combat Systems Intuit Philips Philips
  4. 4. Software Product Lines •  A software product line (SPL) is a set of programs that share a significant common (managed) set of features satisfying a market segment or mission developed from a common set of core assets in a prescribed way Product 1 Commonality Variability .. .Product 2 Product n Core Assets
  5. 5. Feature Oriented Domain Analysis •  SEI FODA Project in Late 1980s •  Identified features (variability) as the key to software product lines •  Identified the need for artifact-independent modeling of the features in an SPL •  •  Introduced the feature diagram which led to what we know today a the feature model
  6. 6. Feature Models •  Representation of the features and dependencies/constraints in an SPL •  Defines what is (and is not) allowable in the SPL •  Can be written as a logic equation and can use SAT solvers to reason about model •  Several different notations for feature models
  7. 7. Software Product Line Engineering •  Consists of two parts: – Domain engineering •  Determining what the re-usable components/ architecture is for the SPL •  Defines the commonality and variability of the SPL – Application engineering •  Deriving products from the platform created in the domain engineering step •  Makes use of re-use and exploits variability/ commonality
  8. 8. Domain Testing •  Validate and verify reusable components •  No running application exists at this time - only components •  This differs from system/application testing where the entire system is tested as a unit
  9. 9. What is Variability? Commonality The features shared by a set of systems Variability The features that differ between some pair of systems
  10. 10. Examples of Variability •  Electric bulb can be lit or unlit •  Software applications support different languages •  A triple band mobile phone supports three different network standards •  Can load Choco vs. MiniSat to solve constraints
  11. 11. Mechanisms for Implementing variability –  Compile flags –  Properties files –  Command-line arguments –  Inheritance –  Interface definition (and information hiding) –  Design patterns (e.g., strategy) –  Connectors (e.g., in architecture)
  12. 12. Mobile Phone SPL
  13. 13. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  14. 14. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  15. 15. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  16. 16. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  17. 17. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  18. 18. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  19. 19. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  20. 20. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Mobile Phone SPL Display Type VP
  21. 21. Testing a Product Line •  SPL engineering decreases time to market •  With shorter life cycles, the percentage of time spent in testing has increased – the new bottleneck •  Much of the current work on testing SPLs focuses on testing individual instances and reuse of specific test cases
  22. 22. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V V Camera Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Validating at the Unit Level Display Type VP
  23. 23. V V V 16MC 8MC BW Validating at the Unit Level Display Type VP V 16MC
  24. 24. V V V 16MC 8MC BW Validating at the Unit Level Display Type VP V 16MC V 8MC
  25. 25. V V V 16MC 8MC BW Validating at the Unit Level Display Type VP V 16MC V 8MC V BW Focus is on reuse of artifacts during testing Views testing from the feature level rather than the system level
  26. 26. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp System Level Testing Display Type VP
  27. 27. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp System Level Testing Viewer Type VP Test Case: Open email in HTML format
  28. 28. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp System Level Testing Display Type VP Test Case: Open email in HTML format
  29. 29. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Feature Interactions Viewer Type VP Failure: caused by interaction of data passed between 16MC display and textual viewer
  30. 30. NASA Mars Rover: Spirit
  31. 31. Real Interaction Failure •  NASA Detected failed communications on two independent channels •  Determined that the rover was continually re-booting – risked running out of energy and/or overheating •  Fault caused by assumptions made between two different features: (NASA component expected third party file system to deallocate memory) –  when one feature was removed the reboots stopped
  32. 32. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Possible Interactions Display Type VP
  33. 33. Fault Map of Configurations V4 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 C0 C10 C20 C30 C40 C50 C60 F1 F2 F3 F4 F5 C0 C10 C20 C30 C40 C50 C60 Faults C o n f I g u r a t I o n s
  34. 34. V V V 16MC 8MC BW V V GV TV V V 2MP 1MP Viewer Type VP Camera Type VP Phone VP V Display V Email Viewer V Camera V Video Camera V Video Ringtones requires_v_vp requires_v_vp requires_v_vp Testing the Interaction Space Viewer Type VP Number of possible interactions grows exponentially with number of variation points
  35. 35. Some Other Approaches to SPL Testing •  Extended use cases for variability [Bertolino et al., SPFE 2003] •  Integration testing a SPL based on UML activity diagrams [Reis et al., FASE 2007] •  Generate test cases incrementally for a SPL [Uzuncaova et al., ISSRE 2008] •  Applying test cases for a subset of products only [Kim et al., ASE 2010] •  Applying monitoring properties for a subset of products only [Kim et al., RV 2010]
  36. 36. Some SPL Resources •  http://splot-research.org/ •  http://www.isa.us.es/fama/ •  http://sir.unl.edu
  37. 37. Agenda SPL System Testing: the combinatorics SPL Integration Testing Summary Continuous Test Suite Augmentation Introduction to Software Product Lines
  38. 38. A Combinatorial Problem Display Email Viewer Camera Video Camera Video Ringtones 16 Million Colors Graphical 2 Megapixels Yes Yes 8 Million Colors Text 1 Megapixel No No Black and White None None
  39. 39. A Combinatorial Problem Display Email Viewer Camera Video Camera Video Ringtones 16 Million Colors Graphical 2 Megapixels Yes Yes 8 Million Colors Text 1 Megapixel No No Black and White None None 3
  40. 40. A Combinatorial Problem Display Email Viewer Camera Video Camera Video Ringtones 16 Million Colors Graphical 2 Megapixels Yes Yes 8 Million Colors Text 1 Megapixel No No Black and White None None 3 x 3
  41. 41. A Combinatorial Problem Display Email Viewer Camera Video Camera Video Ringtones 16 Million Colors Graphical 2 Megapixels Yes Yes 8 Million Colors Text 1 Megapixel No No Black and White None None • 33x22=108 feasible SPL instances 3 x 3 x 3 x 2 x 2
  42. 42. A Combinatorial Problem • 33x22=108 feasible SPL instances • 10 features each with 5 values = 510 or 9,765,625 possible instances Display Email Viewer Camera Video Camera Video Ringtones 16 Million Colors Graphical 2 Megapixels Yes Yes 8 Million Colors Text 1 Megapixel No No Black and White None None
  43. 43. A Combinatorial Problem • 33x22=108 feasible SPL instances • 10 features each with 5 values = 510 or 9,765,625 possible instances • 4 hours to run test suite = approximately 4,459 years to test GCC optimizer: configuration space ~1061 Display Email Viewer Camera Video Camera Video Ringtones 16 Million Colors Graphical 2 Megapixels Yes Yes 8 Million Colors Text 1 Megapixel No No Black and White None None
  44. 44. Combinatorial Interaction Testing (CIT) •  Test all pairs or t-way combinations of feature-values, where t is a defined strength of testing •  Uses a mathematical structure called a covering array
  45. 45. Display Email Viewer Camera Video V. Ringtones 1 16 Million Colors None 1 Megapixel Yes Yes 2 8 Million Colors Text None No No 3 16 Million Colors Text 2 Megapixels No Yes 4 Black and White None None No Yes 5 8 Million Colors None 2 Megapixels Yes No 6 16 Million Colors Graphical None Yes No 7 Black and White Text 1 Megapixel Yes No 8 8 Million Colors Graphical 1 Megapixel No Yes 9 Black and White Graphical 2 Megapixels Yes Yes 2-way CIT
  46. 46. Display Email Viewer Camera Video V. Ringtones 1 16 Million Colors None 1 Megapixel Yes Yes 2 8 Million Colors Text None No No 3 16 Million Colors Text 2 Megapixels No Yes 4 Black and White None None No Yes 5 8 Million Colors None 2 Megapixels Yes No 6 16 Million Colors Graphical None Yes No 7 Black and White Text 1 Megapixel Yes No 8 8 Million Colors Graphical 1 Megapixel No Yes 9 Black and White Graphical 2 Megapixels Yes Yes 2-way CIT
  47. 47. Display Email Viewer Camera Video V. Ringtones 1 16 Million Colors None 1 Megapixel Yes Yes 2 8 Million Colors Text None No No 3 16 Million Colors Text 2 Megapixels No Yes 4 Black and White None None No Yes 5 8 Million Colors None 2 Megapixels Yes No 6 16 Million Colors Graphical None Yes No 7 Black and White Text 1 Megapixel Yes No 8 8 Million Colors Graphical 1 Megapixel No Yes 9 Black and White Graphical 2 Megapixels Yes Yes 2-way CIT
  48. 48. Combinatorial Interaction Testing (CIT) •  Based on statistical design of experiments (DOE) –  Manufacturing –  Drug test interactions –  Chemical interactions •  For software testing –  Mandl – compiler testing –  Brownlie, Prowse, Phadke – OATS system –  D. Cohen, Dalal, Fredman, Patton, Parelius – AETG –  Williams, Probert – network node interfaces –  Yilmaz, Cohen, Porter- ACE/TAO
  49. 49. Empirical Results on CIT for System Inputs •  CIT has been well studied on testing input parameters for programs •  [Brownlie, Prowse, Phadke 92], [Burr,Young 98],[Burroughs, Jain, Erickson 94] [Cohen, Dalal, Fredman, Patton 97], [Dalal, Jain, Karunanithi, Leaton, Lott 98], [Dunietz, Erlich, Szablak, Mallows, Iannino 97], [Mandl 85] $ sort -m -c input.txt
  50. 50. CIT for Configurable Systems •  Some studies on configurable systems •  [Kuhn, Wallace,Gallo 04] –  Medical devices, web browsers, data mgt, http server •  [Yilmaz, Cohen, Porter 04,06] –  Ace/TAO Corba middleware •  [Qu, Cohen, Rothermel 08]
  51. 51. CIT vs. Default Configuration of vim in SIRFaultDetection(%) 020406080100 v2 v3 v4 v5 v6 v7 Version default configuration
  52. 52. How do we Generate the Samples? •  Greedy approaches – AETG, IPOG – Search-based (simulated annealing, GAs) •  Use of constraint solvers
  53. 53. Meta-Heuristic Search ∑ -Set of feasible solutions 0 1 1 1 1 1 0 1 0 0 1 0 1 0 0 1 0 1 1 1 0 0 0 1 1 1 1 0 1 0 0 1 1 1 1 1 0 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 1 1 1 0 1 0 ? cost(S) Randomly move through space S1 S2 S3 ….. Si
  54. 54. Problems with Approach •  May get stuck in local optima
  55. 55. Meta-heuristic Algorithms •  Provide mechanisms to escape local optima •  Sometimes the algorithm can accept a worse choice in the hope we will find a better route to a good solution
  56. 56. Examples •  Genetic Algorithms •  Simulated Annealing •  Tabu Search Search Based Software Engineering
  57. 57. •  Meant to simulate the coolng of metal •  Physical annealing is a process of slowly cooling a metal, in baths of decreasing temperatures. •  At each stage the metal molecules are allowed to come to an equilibrium. Then the bath is lowered to the next temperature. •  If the initial temperature is not high enough or if the cooling is done too quickly the metal will be brittle. Simulated Annealing
  58. 58. •  The basic strategy is the same as a hill climb, but the meta-heuristic creates a gentle decrease in the probability for making a bad move. •  At the start of our hill climb we often make a lot of bad moves. •  By the time we are close to an optimal solution we are almost in a pure hill climb (the probability drops to almost zero) Simulated Annealing (cont.)
  59. 59. •  Start with a solution S •  Select a new solution S’ that is a neighbor of S •  If S’-S ≤ 0 then S’ is accepted •  Otherwise –  Accept S’ with a probability Where T = temperature and B is the Boltzmann constant (this is a distribution which we won’t actually use for simulating the process) Simulated Annealing € e s−s' kBT
  60. 60. Simulated Annealing 0 1 1 1 1 1 0 1 0 0 1 0 1 0 0 1 0 1 1 1 0 0 0 1 1 1 1 0 1 0 0 1 1 1 1 1 0 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 1 1 1 0 1 0 ? Probability of bad choice depends on cooling schedule
  61. 61. Lower bound L Binary search for “best” solution Upper bound U (L+U)/2 Finding “N”
  62. 62. The Real System Constraints on Valid Configurations:

×