1. Shahed Zaman
Ahmed E. Hassan
SAIL, Queen’s University
A Qualitative Study on
Performance Bugs
Bram%Adams
MCIS,&École&Polytechnique&de&Montréal

2. Bugs, Bugs, Bugs!
2
Costly Affect%
Reputa4on

3. Bugs, Bugs, Bugs!
2
482%bugs/week

4. Most%research%treats%all%bugs%equally!%
Does%this%make%sense?

5. Bugs, Bugs, Bugs!
4
Performance%Bug
problem&in&efficiency/speed&at&which&
a&system&performs&at&runDEme
=

6. 5hHp://www.pcmag.com/arEcle2/0,2817,2325772,00.asp
systemEkilling%
performance%problems

7. Security versus Performance Bugs:
A Case Study on
Shahed&Zaman,&Bram&Adams,&Ahmed&E.&Hassan
SoUware&Analysis&and&Intelligence&Lab&(SAIL),&Queen’s&University
6

8. MSR%2011%Quan4ta4ve%Study
Performance&bugs&are&fixed&
by&more&experienced&
developers
People
Performance&bug&fixes&
are&more&complex
Bug%FixTime
&&&&&Performance&bugs&take&
more%4me&to&fix
7

9. Why?
8

10. Shahed Zaman
Ahmed E. Hassan
SAIL, Queen’s University
A Qualitative Study on
Performance Bugs
Bram%Adams
MCIS,&École&Polytechnique&de&Montréal

11. Chrome®
44,997%bugs
510%performance%bugs
295,198%bugs
7,603%performance%bugs
Performance%Bugs NonEperformance%bugs
100%Bugs
100%Bugs
=
200%Bugs
+
100%Bugs
100%Bugs
=
200%Bugs
+
Study Setup

12. 11
bug%reports
‘slow’,%‘perf’,%
‘hang’
Our%heuris4cs%have:
100&±&10%%precision
83&±&10%%recall
Our%heuris4cs%have:
90&±&10%%precision
97&±&10%%recall

13. 12
analyze

14. 13
comments
bug%report
+%a]achments

15. 14
update&taxonomy
analyze
think

16. 14
Iterative
Process
update&taxonomy
analyze
think

17. 4 Dimensions
(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon

18. 4 Dimensions
(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon

19. Findings
is&regression
blocks&release
WFM&after&a&long&Time
People&Threaten&to&switch
0 5 10 15 20 25
Performance
NonDPerf.
%
sta4s4cally%
significant%
difference

20. No Traceability
17
WorksForMe

21. Findings
is&regression
blocks&release
WFM&after&a&long&Time
scares&users&away
0 5 10 15 20 25
Performance
NonDPerf.
%

22. 4 Dimensions
(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon

23. Findings
has&measurements
has&test&cases
has&stack&traces
Replication&Problems
Reported&by&Member
Duplicate&Bugs
0 10 20 30 40 50 60
Performance
NonDPerf.
%

24. 21
automa4c%performance%
regression%tes4ng

25. 21
automa4c%performance%
regression%tes4ngperformance&
regression&
idenEfied&
automaEcally

26. solicits%comparison%to%
other%browsers
22

27. Findings
has&measurements
has&test&cases
has&stack&traces
has&replication&problems
Reported&by&Member
Duplicate&Bugs
0 10 20 30 40 50 60
Performance
NonDPerf.
%

28. Replication Problems
24
incomplete&report
<empty>
<more
empty>
plugins/addDons

29. Hard to Track Bugs of
External Applications
25

30. Findings
has&measurement&data
has&test&cases
has&stack&traces
has&replication&problems
is&reported&by&developer
is&a&duplicate
0 10 20 30 40 50 60
Performance
NonDPerf.
%

31. 4 Dimensions
(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon

32. Findings
has&discussion
Depends&on&Other&Bugs
Blocks&Other&Bugs
0 10 20 30 40 50
Performance
NonDPerf.
%

33. Performance Bugs
Require Collaboration
29
understand&the&problem
idenEfy&bug&fix
reach&consensus
me&too

34. Findings
has&discussion
depends&on&other&bug
blocks&other&bug
0 10 20 30 40 50
Performance
NonDPerf.
%

35. Rebasing Performance
31
adjust&baseline

36. 4 Dimensions
(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon

37. 33
NO&difference&between&patch&discussions&and&reviews&of
performance&and&nonDperformance&bugs

38. is&regression + +
blocks&release + +
WFM&aUer&a&long&Eme + +
scares&users&away + +
has&measurements + +
has&test&cases + +
contains&stacktrace + +
has&replicaEon&problems + +
is&reported&by&a&developer ? +
is&a&duplicate E E
has&discussion + +
depends&on&other&bug + +
blocks&other&bug + E
impact&on&the&user
context&of&the&bug
bug&fix

39. is&regression + +
blocks&release + +
WFM&aUer&a&long&Eme + +
scares&users&away + +
has&measurements + +
has&test&cases + +
contains&stacktrace + +
has&replicaEon&problems + +
is&reported&by&a&developer ? +
is&a&duplicate E E
has&discussion + +
depends&on&other&bug + +
blocks&other&bug + E
impact&on&the&user
context&of&the&bug
bug&fix

40. MSR%2011%Quan4ta4ve%Study%(reprise)
Performance&bugs&are&fixed&
by&more&experienced&
developers
Performance&bug&fixes&
are&more&complex
&&&&&Performance&bugs&take&
more%4me&to&fix
36
replication problems
more dependencies
more discussion
WFM after a long time
more release
blocking
users leaving

41. Threats to Validity
37
human&bias
only&4&
dimensions
&&&20&subD
dimensions
only&2&subject
systems
‘slow’,%‘perf’,%
‘hang’
heurisEcs&for&
performance&bug&
idenEficaEon

42. Chrome®
44,997%bugs
510%performance%bugs
295,198%bugs
7,603%performance%bugs
Performance%Bugs NonEperformance%bugs
100%Bugs
100%Bugs
=
200%Bugs
+
100%Bugs
100%Bugs
=
200%Bugs
+
Study Setup
no traceability
no reproducability
discussion!
4 Dimensions
(20 subdimensions)
impact&on&the&user
context&of&the&bug
bug&fix
bug&fix&validaEon
is&regression + +
blocks&release + +
WFM&aUer&a&long&Eme + +
scares&users&away + +
has&measurements + +
has&test&cases + +
contains&stacktrace + +
has&replicaEon&problems + +
is&reported&by&a&developer ? +
is&a&duplicate E E
has&discussion + +
depends&on&other&bug + +
blocks&other&bug + E
impact&on&the&user
context&of&the&bug
bug&fix
MSR%2011%Quan4ta4ve%Study%(reprise)
Performance&bugs&are&fixed&
by&more&experienced&
developers
Performance&bug&fixes&
are&more&complex
&&&&&Performance&bugs&take&
more%4me&to&fix
37
replication problems
more dependencies
more discussion
WFM after a long time
more release
blocking
users leaving