This document summarizes research on using pairwise testing to model variability and analyze quality of service (QoS) for web service orchestrations. Feature diagrams are used to explicitly represent variability in composite services, and pairwise testing is applied to select configurations covering all pairwise feature interactions. QoS distributions are computed for these configurations to predict overall orchestration QoS in a way that accounts for variability. The approach provides more realistic service level agreements than considering only worst-case scenarios.

1. S-Cube Learning Package
Service Level Agreements:
Variability Modeling and QoS Analysis of Web
Services Orchestrations
INRIA
Sagar Sen, Benoit Baudry , Olivier Barais,
www.s-cube-network.eu

2. Learning Package Categorization
S-Cube
SBA Quality Management
Quality Assurance and Quality Prediction
Variability Modeling and QoS Analysis of
Web Services Orchestrations
www.s-cube-network.eu

3. Learning Package Overview
• Problem Description
• Variability Modeling and QoS Analysis of
Web Services Orchestrations
• Discussion
• Conclusions
www.s-cube-network.eu

4. Feature Diagrams
Feature Diagrams (FD) introduced by Kang et al. represent all
configurations.
[1] K. Kang, S. Cohen, J. Hess, W. Novak, and S.
Peterson, “Feature-Oriented Domain Analysis (FODA)
Feasibility Study,"
Software Engineering Institute, 1990.
www.s-cube-network.eu

5. Compatibility between FD and
orchestrations
An orchestration should invoke services corresponding to primitive
nodes in a configuration (a valid instance of the FD).
www.s-cube-network.eu

6. SLA in composite services
Execution time for this car crash crisis management
service?
6
www.s-cube-network.eu

7. SLA in composite services
Execution time for this car crash crisis management
service?
7
www.s-cube-network.eu

8. QoS models for atomic services
Compute QoS distributions for atomic services
8
www.s-cube-network.eu

9. QoS models for atomic services
Compute QoS distributions for atomic services
9
www.s-cube-network.eu

10. QoS for one configuraiton
A
D
B
E F
MUX
Merge 10
www.s-cube-network.eu

11. Large number of configurations
Execution time for
Total number of
this car crash
possible crisis
configurations: management
225 service?
11
www.s-cube-network.eu

12. Learning Package Overview
• Problem Description
• Variability Modeling and QoS Analysis
of Web Services Orchestrations
• Discussion
• Conclusions
www.s-cube-network.eu

13. Proposal
Adapt pairwise selection to sample
configurations in the composite service
Compute QoS distributions for this sample
13
www.s-cube-network.eu

14. Motivating Questions
• Generate configurations covering all pairwise
interactions for a
• composite service, ensuring variability is
captured.
• From this, infer variability in QoS parameters.
• Stability with respect to the pairwise sample
selected.
• Comparison to exhaustive sampling of the
configuration space.
www.s-cube-network.eu

15. Methodology
1. The modeling inputs may be specified as a 3-
tuple (Services, Feature Diagram,
Orchestration).
2. Pairwise constraints are used to sample a set of
configurations.
3. QoS for orchestrations invoking services in the
configuration.
4. Comparisons with exhaustive sampling and
consistency over multiple sample sets.
www.s-cube-network.eu

16. Pairwise Samples
•Combinatorial interaction testing (CIT) has been shown in
network
•monitoring case studies3 to reduce tests for 75 parameters with
10^29 exhaustive combinations to only 28 tests.
•CIT used to select a minimal set of configurations for four
boolean features A, B, C, D.
• A Pairwise Sample consists of all configurations
satisfying pairwise interactions for a composite service.
• There can be many pairwise samples for a given FD
(not unique).
www.s-cube-network.eu

17. Explicit model of variability
17
www.s-cube-network.eu

18. Variability in the composite
service
18
www.s-cube-network.eu

19. Pairwise test selection for
Feature diagram
A set TC of test configurations such that
X1,…, Xn n features
 i  [1..n] Xi  {0,1}
 Xj, Xk |  Xja, Xkb |  c  TC | TC  Xja, Xkb
 c  TC, c is a valid configuration w.r.t feature
model
www.s-cube-network.eu

20. Pairwise for composite services
A
Mandatory
B C D Optional
XOR
E F
Pairwise Interaction Configurations
A¬B, A¬C, A¬D, A¬E, A¬F, ¬B¬D, ¬C¬D A
AB, AC, BC, B¬D, B¬E, C¬D, C¬E, C¬F ABC
AD, AE, C¬B, D¬B, E¬B, ¬B¬F, CD, CE, DE, E¬F ACDE
B¬C, BD, BE, B¬F, D¬C, E¬C, ¬C¬F, D¬F ABDE
AF, ¬B¬C, ¬B¬E, F¬B, ¬C¬E, F¬C, D¬E ADF
BF, CF, DF, F¬E ABCDF
www.s-cube-network.eu

21. Q1 ‘coverage’ of the pairwise
sample
22
www.s-cube-network.eu

22. Q1 ‘coverage’ of the pairwise
sample
23
www.s-cube-network.eu

23. Q2 pairwise vs. random
24
www.s-cube-network.eu

24. Q2 pairwise vs. random
25
www.s-cube-network.eu

25. Q3 stability of pairwise
Percentile 25 25(max 50(min) 50(max 75(min 75(max 90(min 90(max
(min) ) ) ) ) ) )
Std. Dev. 2.18 1.52 2.59 1.73 2.90 1.82 3.19 1.83
26
(seconds)
www.s-cube-network.eu

26. Q4 establishing classes of SLA
27
www.s-cube-network.eu

27. Learning Package Overview
• Problem Description
• Variability Modeling and QoS Analysis of
Web Services Orchestrations
• Discussion
• Conclusions
www.s-cube-network.eu

28. Discussions
• SLAs should take into account variable
configurations and probabilistic nature of QoS
parameters.
• Product line of composite services with
extensively analyzed SLAs.
• Eliminating deviating configurations from SLAs.
• Theoretical work to determine conditions when
pairwise analysis can be used to sample QoS
metrics.
www.s-cube-network.eu

29. Learning Package Overview
• Problem Description
• Variability Modeling and QoS Analysis of
Web Services Orchestrations
• Discussion
• Conclusions
www.s-cube-network.eu

30. Conclusion
Pairwise is a systematic sampling technique
Initial results for QoS prediction are
encouraging
Allows for a more realistic SLAs than current
pessismistic (worst case) SLAs
31
www.s-cube-network.eu

31. Further S-Cube Reading
Kattepur, S. Sen, B. Baudry, A. Benveniste, C. Jard, Variability Modeling and
QoS Analysis of Web Services Orchestrations, In International Conference
on Web Services, IEEE, 2010.
Sagar Sen, Automatic Effective Model Discovery, PhD Thesis, Université
de Rennes 1, June 2010
www.s-cube-network.eu

32. References
A. Kattepur, S. Sen, B. Baudry, A. Benveniste, C. Jard, Pairwise Testing of Dynamic
Composite Services, In International Symposium on Software Engineering for Adaptive
and Self Managing Systems (SEAMS), IEEE, 2011.
K. Kang, S. Cohen, J. Hess, W. Novak, and S. Peterson, “Feature-Oriented Domain
Analysis (FODA) Feasibility Study," Software Engineering Institute, 1990.
J. Misra and W. R. Cook, “Computation Orchestration: A Basis for Wide-area Computing,«
Springer J. of Software and Systems Modeling, vol. 6, no. 1, pp. 83 – 110, Mar. 2007.
D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton, “The Combinatorial Design
Approach to Automatic Test Generation," IEEE Software, vol. 13, no. 5, pp. 83–88,
Sept. 1996.
J. Kienzle, N. Guelfi, and S. Mustafiz, “Crisis Management Systems: A Case Study for
Aspect-Oriented Modeling," McGill Univ., Technical Report, 2009.
G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. le Traon, “Automatic and Scalable T-wise
Test Case Generation Strategies for Software Product Lines," Proc. of Intl. Conf. On
Software Testing, April 2010.
S. Rosario, A. Benveniste, S. Haar, and C. Jard, “Probabilistic QoS and Soft Contracts for
Transaction-Based Web Services Orchestrations," IEEE Trans. on Services
Computing, vol. 1, no. 4, pp. 187 – 200, 2008.
www.s-cube-network.eu

33. Acknowledgements
The research leading to these results has
received funding from the European
Community’s Seventh Framework Programme
[FP7/2007-2013] under grant agreement
215483 (S-Cube).
www.s-cube-network.eu