SSSG Spring 2017

1. 1
Exploiting Structure and
Behavior of Highly
Configurable Systems to
Measure Performance
Miguel Velez
Software Engineering Ph.D. student
Joint work with: Christian Kästner, Pooyan Jamshidi, and
Norbert Siegmund

2. Most software is highly configurable
2

3. Developers:
Lose track of interactions
Users:
Overwhelmed by choices
Cannot configure for their needs
Measuring performance in HCS is difficult
3

4. “For each valid configuration in a program, X is the Y
you will get”
Y = Quality Attribute
X = Actual Value
In an ideal world
4

5. Execution Time
5
Configuration Performance
A=0 B=0 C=0 D=0 6s
A=0 B=0 C=1 D=1 6s
A=1 B=0 C=0 D=1 9s
… …

6. Black-Box Approach
6
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

7. We want to learn
7
A B C D T
1
0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9
1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

8. 8
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9
1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13
1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10
We want to learn

9. 9
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13
1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10
We want to learn

10. Build performance model
10
T = 3A + 1AB + 6

11. Black-Box in practice
11
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

12. 12
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13
1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10
Black-Box in practice

13. Want to find interactions
Identify what we need to measure
Determine if we do not need to measure all
configurations
Ideally
13

14. Our Proposal: Exploit Structure and
Behavior of the Program to Measure
Performance
14

15. White-Box Analysis
15

16. Determine how configuration options are used
throughout the program
Identify interactions among options
Compare execution traces based on configurations
Know if we need to execute all configurations or subset
What We Can Do With Structure and Behavior
16

17. Approach: Combine Static Taint Analysis
with Dynamic Analysis
17

18. Goal
18

19. Static Taint Analysis
19

20. Static Taint Analysis
20

21. Static Taint Analysis
21

22. Static Taint Analysis
22

23. Static Taint Analysis
23

24. Static Taint Analysis
24

25. Static Taint Analysis
25

26. Static Taint Analysis
26

27. Static Taint Analysis
27

28. Static Taint Analysis
28

29. Static Taint Analysis
29

30. Relevant Statements and Options
30

31. 31
Relevant Statements and Options

32. Dynamic Analysis
32

33. Transform AST
33

34. 34
Building the Table

35. 35
Building the Table

36. 36
Building the Table

37. 37
Building the Table

38. 38
Building the Table

39. 39
Building the Table

40. 40
Another Example

41. Future Directions
41

42. C
Java
RQ: How often can we exploit lack of interactions to
measure performance?
Apply in Real Programs
42

43. Augment Tracking Information
43

44. • From Black-Box to White-Box
• Taint Analysis with Dynamic Analysis
• New pruning strategies in real programs
Summary
44