PLMW@PLDI'16: Fighting Software Inefficiency Through Automated Bug Detection

1. Figh%ng So*ware Inefficiency
Through Automated Bug Detec%on
Shan Lu
University of Chicago
1

2. A li=le bit about myself
2

3. My life J
3

4. I learned a lot from …
4

5. Bug Detec%on
Background
5

6. Figh%ng so*ware bugs is crucial
• So3ware is everywhere
– h5p://en.wikipedia.org/wiki/List_of_so3ware_bugs
• So3ware bugs are widespread and costly
– Lead to 40% system down Eme [Blueprints 2000]
– Cost 312 Billion lost per year [Cambridge 2013]
6

7. What is your favorite bug?
• How many of you have been bothered by bugs?
7

8. How to detect bugs?
• Study & understand real-world bugs
• Discover pa5erns of common bugs
– Source code level
– Binary code level
– …
• Design pa5ern-matching program analysis
– StaEc analysis
– Dynamic analysis
– …
8

9. Bug detec%on examples
– Memory bug detecEon
• Pa5ern: over-bound writes, …
• DetecEon: check memory accesses & …
– Concurrency bug detecEon
• Pa5ern: data races, atomicity violaEons, …
• DetecEon: check memory accesses & …
9
P = malloc (10);
P[100] = ‘a’;
if (P)
*P=‘a’;
P=NULL;

10. Why Performance-Bug
Detec%on?
10

11. How did this start?
• One of our bug detectors is strangely slow
– Why not profiling?
• Lots of noises in profiling
• Measuring cost not inefficiency
• Am I able to do this?
• Where should I start?
11

12. An Empirical Study of
Real-World Performance Bugs
Are there performance bugs?
Are they important?
What types of performance bugs are there?
12
Understanding and detecting real-world performance bugs [PLDI '12]

13. Methodology
13
ApplicaEon
Apache
Chrome
GCC
Mozilla
MySQL
So3ware Type
Server So3ware
GUI ApplicaEon
GUI ApplicaEon
Compiler
Command-line UElity +
Server + Library
Language
C/Java
C/C++
C/C++
C++/JS
C/C++/C#
MLOC
1.3
Bug DB
History Tags
Compile-
Eme-hog
5.7
4.7
14.0
N/A
N/A
perf
S5
0.45
14 y
13 y
10 y
13 y
4 y
# Bugs
25
10
10
36
28
Total: 109

14. Findings
• Are there performance bugs?
– Yes
• Are they important?
– Some are
• What types of performance bugs are there?
– What are their root causes?
– Where are they typically located?
– How are they usually fixed?
14

15. Bug Examples
15
+ int i = -k.length();
- while (s.indexOf(k) == -1) {
+ while (i++<0 ||
+ s.substring(i).indexOf(k)==-1)
{s.append (nextchar());}
Patch for Apache-Ant Bug 34464
for (i = 0; i < tabs.length; i++) {
…
tabs[i].doTransact();
}
+ doAggregateTransact(tabs);
Mozilla Bug 490742 & Patch

16. What is next?
Can we detect performance bugs?
What “pa5ern” did we find?
16

17. A Patch-Based
Inefficiency Detector
17

18. Sta%c inefficiency pa=erns exist
• StaEcally checkable inefficiency pa5erns exist
+ int i = -k.length();
- while (s.indexOf(k) == -1) {
+ while (i++<0 ||
+ s.substring(i).indexOf(k)==-1)
{s.append (nextchar());}
Patch for Apache-Ant Bug 34464
for (i = 0; i < tabs.length; i++) {
…
tabs[i].doTransact();
}
+ doAggregateTransact(tabs);
Mozilla Bug 490742 & Patch

19. How to get these pa=erns?
• Manually extract from patches
19
Not Contain Rules
Dynamic Rules
LLVM Checkers
Python Checkers

20. Detec%on Results
• 17 checkers find PPPs in original buggy versions
• 13 checkers find 332 PPPs in latest versions
Found by
cross-applicaKon
checking
Inherits from
buggy versions
Introduced later
* PPP: Poten6al Performance Problem
Inefficiency pa=ern based bug detec%on is promising!

21. What is next?
Do we have to manually specify rules?
Can we build generic detectors?
21

22. Toddler
A dynamic and generic detector
targeEng inefficient nested loops
22
Toddler: Detecting Performance Problems via Similar
Memory-Access Patterns [ICSE '13]

23. What are generic inefficiency pa=erns?
23

24. Previous example
24
while (s.indexOf(k) == -1)
{s.append (nextchar());}
Apache-Ant Bug 34464
Password: abcdefg
Password: abcdefgh
Password: abcdefghi

25. What is the pa=ern?
• What type of nested loops are likely inefficient?
– Many inner loops are similar with each other
• Some instrucEons keeps reading similar sequences of values
25
abcdefg
abcdefgh
abcdefghi

26. How to detect?
• Input: Test code + system under test
• Steps:
1. Instrument the system under test
Monitor loop starts/ends & memory reads inside loops
2. Analyze trace produced by instrumentaEon
IdenEfy repeEEve memory-read sequences
• Output: Loops that are likely performance bugs
26

27. Evalua%on Subjects and New Bugs
27
Applica%on Descrip%on LOC Known Bugs New Bugs Fixed Confirmed
Ant Build tool 109,765 1 8 1 0
Apache CollecEons CollecEons library 51,516 1 20 10 4
Groovy Dynamic language 136,994 1 2 2 0
Google Core Libraries CollecEons library 156,004 2 10 1 2
JFreeChart Chart framework 64,184 1 1 0 0
Jmeter Load tesEng tool 86,549 1 0 0 0
Lucene Text search engine 320,899 2 0 0 0
PDFBox PDF framework 78,578 1 0 0 0
Solr Search server 373,138 1 0 0 0
JDK standard library 2 0 0
JUnit tesEng framework 1 1 0
9 Apps + 2 Libs 50,000 – 320,000 11 44 15 6

28. Toddler vs. HProf
28
Known Bug
Bug Detected? False P. Rank Slowdown
TODD. PROF TODD. PROF TODD. PROF
Ant ü û 0 19.3 13.7 4.2
Apache CollecEons ü ü 0 1.0 10.0 2.1
Groovy ü ü 0 3.7 15.5 3.7
Google Core Libraries #1 ü ü 0 1.8 9.0 3.8
Google Core Libraries #2 ü û 0 5.3 7.5 3.2
JFreeChart ü û 0 53.7 13.4 8.8
JMeter ü û 0 10.3 8.5 1.9
Lucene #1 ü û 0 7.7 6.8 2.5
Lucene #2 ü ü 0 3.1 25.4 3.1
PDFBox ü û 1 18.8 51.8 12.1
Solr ü û 0 178.3 114.2 7.1
11 11 4 1 n/a 15.9X 4.0X

29. What is next?
Why so many bugs are not fixed by
developers?
29

30. Caramel
A staEc and generic detector
targeEng inefficient loops with
simple patches
30
CARAMEL: Detecting and Fixing Performance Problems That
Have Non-Intrusive Fixes [ICSE'15]
Won SIGSOFT Distinguished Paper Award

31. What are perf. bugs not fixed?
31
Correctness
Maintainability
Manual effort
Potential speedup under certain workload
Can we detect bugs with simple fixes?

32. What is the pa=ern?
• What is a typical simple fix for an inefficient loop?
• What types of bugs have the above type of fix?
– We thought for a long Eme …
32
for(…)
+ if (cond) break;

33. Bug example
33
boolean alreadyPresent = false;
while (isActualEmbeddedProperty.hasNext()) {
if (alreadyPresent) break; // CondBreak FIX
if (oldVal.getStr().equals(newVal.getStr()))
alreadyPresent = true;
if ( ! alreadyPresent )
prop.container().addProp(newVal); // side effect
}
• New bug in PDFBox found by us, fixed by developers
• Developers fix bugs that have CondBreak fixes:
– Waste computaEon in loops
– Fix is non-intrusive

34. What Bugs Have CondBreak Fixes?
34
Every
Itera%on
Late
Itera%ons
Early
Itera%ons
No-Result Type 1 Type 2 Type Y
Useless-Result Type X Type 3 Type 4
Where Is Computation Wasted?How Is
Computation
Wasted?

35. Ingredient 1: Result Instruc%on
boolean alreadyPresent = false;
while (isActualEmbeddedProperty.hasNext()) {
if (alreadyPresent) break; // CondBreak FIX
if (oldVal.getStr().equals(newVal.getStr()))
alreadyPresent = true;
if ( ! alreadyPresent )
prop.container().addProp(newVal); // side effect
}
Result InstrucEon

36. Ingredient 2: Instruc%on-Condi%on
boolean alreadyPresent = false;
while (isActualEmbeddedProperty.hasNext()) {
if (alreadyPresent) break; // CondBreak FIX
if (oldVal.getStr().equals(newVal.getStr()))
alreadyPresent = true;
if ( ! alreadyPresent )
prop.container().addProp(newVal); // Result Ins.
}
InstrucEon-CondiEon
36

37. Ingredient 3: Loop-Condi%on
boolean alreadyPresent = false;
while (isActualEmbeddedProperty.hasNext()) {
if (alreadyPresent) break; // CondBreak FIX
if (oldVal.getStr().equals(newVal.getStr()))
alreadyPresent = true;
if ( ! alreadyPresent )
prop.container().addProp(newVal); // Result Ins.
}
InstrucEon-CondiEon
Also Loop-CondiEon
37

38. Evalua%on Subjects and New Bugs
• 15 applica%ons
– 11 Java, 4 C/C++
• 150 new bugs
• 116 bugs fixed
– 51 in Java
– 65 in C/C++
• Only 4 rejected
• 22 bugs in GCC fixed
• 149/150 fixed
automaEcally
• Only 23 false posiEves
Applica%on Descrip%on LOC Bugs
Ant Build tool 140,674 1
Groovy Dynamic language 161,487 9
JMeter Load tesEng tool 114,645 4
Log4J Logging framework 51,936 6
Lucene Text search engine 441,649 14
PDFBox PDF framework 108,796 10
Sling Web app. framework 202,171 6
Solr Search server 176,937 2
Struts Web app. framework 175,026 4
Tika Content extracEon 50,503 1
Tomcat Web server 295,223 4
Google Chrome Web browser 13,371,208 22
GCC Compiler 1,445,425 22
Mozilla Web browser 5,893,397 27
MySQL Database server 1,774,926 18

39. Different aspects of figh%ng bugs
39
In-house
bug detecEon
In-field
failure recovery
In-field
failure diagnosis
In-house
bug fixing
Low overhead
High accuracy High accuracy

40. Work from my group
40
In-house
bug detecEon
[ASPLOS06];
[SOSP07];
[ASPLOS09];
[ASPLOS10];
[ASPLOS11];
[OOPSLA13]
[PLDI12];
[ICSE13];
[ICSE15]
In-field
failure recovery
[ASPLOS13.A]
[FSE14]
Not yet
In-field
failure diagnosis
[OOPSLA10];
[ASPLOS13.B];
[ASPLOS14];
[OOPSLA16*]
[OOPSLA14]
In-house
bug fixing
[PLDI11];
[OSDI12];
[FSE16]
[CAV13]
concurrency
bugs
performance
bugs

41. Conclusions & Future Work
41
Constraints/Requirements
Techniques
Bugs

42. Thanks!
Ques%ons?
42
My collaborators
• Prof. Darko Marinov
• Adrian Nistor
• Linhai Song