Confidential Software Product Line Testing in Practice Test Scope based on Risk Rev PA1 2011-10-26 1
Introduction Software product lines, or software product line development, refers to software engineering methods, tools and techniques for creating a collection of similar software systems from a shared set of software assets using a common means of production.  It is obvious that executing the same test scope on all these similar software systems will not be very efficient How is the scope selection for the different software systems optimized?
Overview Set Scope Identify Identify Scope Impact of Variation Risk based on Variation Points Space current Points Risk
Identify Variation Points The first step is to identify the variation points between the different software systems Which components differ between system? What dependencies are specific to the software system compared to the rest of the product line? Do the system resources differ in the product line?
Scope Impact of Variation Points What is the impact of less memory in one software system compared to the rest of the product line to the test scope? What is the impact of a customer variant of a software system with different UI, but otherwise identical system? What is the impact of including a certain feature in one software system that is not included in the rest of the product line? The answers are of course context specific, but these are questions that should be asked early – during the design of the different software systems
Identify Risk Space Risks are completely context dependant, even though it is possible to have check lists as support It is critical to identify the consequence severity and probability of each risk for the software systems and take these into account during scope selection High Probability High Risk Low Low Risk Probability Low Impact High Impact
Risk Space Change Triggers (Example) Risk Space Change Triggers Bug fix Integration Feature Integration SW Environment Changes Previous test results New Quality Criteria New requirements Known issues Time passed
Set Scope based on current Risk Variation Points Affect New System Change in Risk Trigger Scope Selection Create Specific Test Space Scope
Without variation points Not software system specific! Change in Risk New Product Line Trigger Scope Selection Create Space Test Scope Create Waste and Overlap in Scope
Solution? Tool support for identifying variation points Tool support for identifying dependencies in the software Risk identification process in place Standardized risk lists Risk matrixes Test automation to reduce risk at low cost by executing test cases to explore and reveal the risk space
Conclusion Risk-based testing  is critical for software product line testing Identifying the variation points is the core of the problem Identifying the impact of the variation points on test scope is often even more difficult to do correctly Identifying the risk space and what should trigger a new scope selection adds even more complexity Software Product Line testing is not trivial and introduces risk if we cannot accept a large overlap in test scopes between software systems Tool support is critical to manage these complexities
Reference Software Product Line [2011-11-08]http://en.wikipedia.org/wiki/Software_product_line Risk Based Testing [2011-11-08]http://en.wikipedia.org/wiki/Risk-based_testing