FOSD 2017

1. Exploiting Structure and
Behavior of Highly
Configurable Systems to
Measure Performance
Miguel Velez
Software Engineering Ph.D. student
Carnegie Mellon University
Joint work with: Christian Kästner, Pooyan Jamshidi, and
Norbert Siegmund
1

2. Most software is highly configurable
2

3. For developers and users
Overwhelmed by choices
Lose track of interactions
Measuring performance in HCS is difficult
3

4. “For each valid configuration, x is the y you will get”
y = quality attribute x = actual value
In an ideal world
4
Configuration Execution Time
A=0 B=0 C=0 D=0 6s
A=0 B=0 C=1 D=1 6s
A=1 B=0 C=0 D=1 9s
… …
T = 3A + 1AB + 6

5. Black-Box Approach
5
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

6. Black-Box Approach
6
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

7. Black-Box Approach
7
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

8. Black-Box in practice
8
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13 1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10

9. 9
A B C D T
1 0 0 0 0 6
2 0 0 0 1 6
3 0 0 1 0 6
4 0 0 1 1 6
5 0 1 0 0 6
6 0 1 0 1 6
7 0 1 1 0 6
8 0 1 1 1 6
9 1 0 0 0 9
10 1 0 0 1 9
11 1 0 1 0 9
12 1 0 1 1 9
13
1 1 0 0 10
14 1 1 0 1 10
15 1 1 1 0 10
16 1 1 1 1 10
Black-Box in practice

10. Want to find interactions
Identify what we need to measure
Determine if we do not need to measure all
configurations
Ideally
10

11. Our Proposal: Exploit Structure and
Behavior of the Program to Measure
Performance
11

12. White-Box Analysis
12

13. Determine how options are used
Identify interactions
Compare execution traces based on
configurations
Know if we need to execute all configurations or
subset
What We Can Do With Structure and Behavior
13

14. Approach: Combine Static Taint Analysis
with Dynamic Analysis
14

15. Goal
15

16. Static Taint Analysis
16

17. Static Taint Analysis
17

18. Static Taint Analysis
18

19. Static Taint Analysis
19

20. Static Taint Analysis
20

21. Static Taint Analysis
21

22. Static Taint Analysis
22

23. Static Taint Analysis
23

24. Static Taint Analysis
24

25. Static Taint Analysis
25

26. Static Taint Analysis
26

27. Relevant Statements and Options
27

28. 28
Relevant Statements and Options

29. Dynamic Analysis
29

30. Transform AST
30

31. 31
Building Performance Table

32. 32
Building Performance Table

33. 33
Building Performance Table

34. 34
Building Performance Table

35. 35
Building Performance Table

36. 36
Building Performance Table

37. Future Directions
37

38. C
Java
RQ: How often can we exploit lack of
interactions to measure performance?
Apply in Real Programs
38

39. Augment Tracking Information
39

40. Augment Tracking Information
40

41. Augment Tracking Information
41

42. • From Black-Box to White-Box
• Static Taint Analysis with Dynamic Analysis
• New pruning strategies in real programs
Summary
42
Miguel Velez
mvelezce@cs.cmu.edu

43. 43
Data-flow Example