A Qualitative Study on Performance Bugs (MSR 2012)

796 views

Published on

Software performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experi- enced developers and require changes to more code than non-performance bugs. In order to be able to improve the resolution of performance bugs, a better understanding is needed of the current practice and shortcomings of reporting, reproducing, tracking and fixing performance bugs. This paper qualitatively studies a random sample of 400 performance and non-performance bug reports of Mozilla Firefox and Google Chrome across four dimensions (Impact, Context, Fix and Fix validation). We found that developers and users face problems in reproducing performance bugs and have to spend more time discussing performance bugs than other kinds of bugs. Sometimes performance regressions are tolerated as a trade- off to improve something else.

http://sail.cs.queensu.ca/publications/pubs/MSR2012_Zaman.pdf

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
796
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Qualitative Study on Performance Bugs (MSR 2012)

  1. 1. A Qualitative Study on Performance Bugs Shahed Zaman Bram%Adams Ahmed E. Hassan MCIS,&École&Polytechnique&de&Montréal SAIL, Queen’s University
  2. 2. Bugs, Bugs, Bugs!Costly Affect% Reputa4on 2
  3. 3. Bugs, Bugs, Bugs! 482%bugs/week 2
  4. 4. Most%research%treats%all%bugs%equally!% Does%this%make%sense?
  5. 5. Bugs, Bugs, Bugs! problem&in&efficiency/speed&at&which&Performance%Bug = a&system&performs&at&runDEme 4
  6. 6. systemEkilling% performance%problemshHp://www.pcmag.com/arEcle2/0,2817,2325772,00.asp 5
  7. 7. Security versus Performance Bugs:A Case Study on Shahed&Zaman,&Bram&Adams,&Ahmed&E.&Hassan SoUware&Analysis&and&Intelligence&Lab&(SAIL),&Queen’s&University 6
  8. 8. MSR%2011%Quan4ta4ve%Study Time People Bug%Fix&&&&&Performance&bugs&take& Performance&bugs&are&fixed& Performance&bug&fixes& more%4me&to&fix by&more&experienced& are&more&complex developers 7
  9. 9. Why? 8
  10. 10. A Qualitative Study on Performance Bugs Shahed Zaman Bram%Adams Ahmed E. Hassan MCIS,&École&Polytechnique&de&Montréal SAIL, Queen’s University
  11. 11. Study Setup Chrome ® 295,198%bugs 44,997%bugs 7,603%performance%bugs 510%performance%bugs 100%Bugs 100%Bugs + + 100%Bugs 100%Bugs = = 200%Bugs 200%BugsPerformance%Bugs NonEperformance%bugs
  12. 12. bug%reports Our%heuris4cs%have: 100&±&10%%precision 83&±&10%%recall‘slow’,%‘perf’,% ‘hang’ Our%heuris4cs%have: 90&±&10%%precision 97&±&10%%recall 11
  13. 13. analyze 12
  14. 14. bug%report+%a]achments comments 13
  15. 15. update&taxonomyanalyze think 14
  16. 16. update&taxonomyanalyze thinkIterativeProcess 14
  17. 17. context&of&the&bug 4 Dimensions (20 subdimensions) bug&fix&validaEonimpact&on&the&user bug&fix
  18. 18. context&of&the&bug 4 Dimensions (20 subdimensions) bug&fix&validaEonimpact&on&the&user bug&fix
  19. 19. sta4s4cally% Findings significant% difference is&regressionPerformanceNonDPerf. blocks&release WFM&after&a&long&Time People&Threaten&to&switch 0 5 10 15 20 25 %
  20. 20. WorksForMeNo Traceability 17
  21. 21. Findings is&regressionPerformanceNonDPerf. blocks&release WFM&after&a&long&Time scares&users&away 0 5 10 15 20 25 %
  22. 22. context&of&the&bug 4 Dimensions (20 subdimensions) bug&fix&validaEonimpact&on&the&user bug&fix
  23. 23. Performance Findings NonDPerf. has&measurements has&test&cases has&stack&tracesReplication&ProblemsReported&by&Member Duplicate&Bugs 0 10 20 30 40 50 60 %
  24. 24. automa4c%performance% regression%tes4ng 21
  25. 25. automa4c%performance%performance& regression%tes4ngregression&idenEfied&automaEcally 21
  26. 26. solicits%comparison%to% other%browsers 22
  27. 27. Performance Findings NonDPerf. has&measurements has&test&cases has&stack&traceshas&replication&problems Reported&by&Member Duplicate&Bugs 0 10 20 30 40 50 60 %
  28. 28. Replication Problems <more<empty> empty>incomplete&report plugins/addDons 24
  29. 29. Hard to Track Bugs ofExternal Applications 25
  30. 30. Performance Findings NonDPerf. has&measurement&data has&test&cases has&stack&traceshas&replication&problemsis&reported&by&developer is&a&duplicate 0 10 20 30 40 50 60 %
  31. 31. context&of&the&bug 4 Dimensions (20 subdimensions) bug&fix&validaEonimpact&on&the&user bug&fix
  32. 32. Findings Performance NonDPerf. has&discussionDepends&on&Other&Bugs Blocks&Other&Bugs 0 10 20 30 40 50 %
  33. 33. Performance Bugs Require Collaborationunderstand&the&problem reach&consensusidenEfy&bug&fix me&too 29
  34. 34. Findings Performance NonDPerf. has&discussiondepends&on&other&bug blocks&other&bug 0 10 20 30 40 50 %
  35. 35. Rebasing Performance adjust&baseline 31
  36. 36. context&of&the&bug 4 Dimensions (20 subdimensions) bug&fix&validaEonimpact&on&the&user bug&fix
  37. 37. NO & diff er enpe ce rfo &b e rm tw an ee ce n &p &an atc d &n h &d on i sc Dp e us rfo si o rm ns an &an ce d &r &b u ev gs iew s&o f 33
  38. 38. is&regression + + blocks&release + + WFM&aUer&a&long&Eme + +impact&on&the&user scares&users&away + + has&measurements + + has&test&cases + + contains&stacktrace + + has&replicaEon&problems + +context&of&the&bug is&reported&by&a&developer ? + is&a&duplicate E E has&discussion + + depends&on&other&bug + + bug&fix blocks&other&bug + E
  39. 39. is&regression + + blocks&release + + WFM&aUer&a&long&Eme + +impact&on&the&user scares&users&away + + has&measurements + + has&test&cases + + contains&stacktrace + + has&replicaEon&problems + +context&of&the&bug is&reported&by&a&developer ? + is&a&duplicate E E has&discussion + + depends&on&other&bug + + bug&fix blocks&other&bug + E
  40. 40. MSR%2011%Quan4ta4ve%Study%(reprise) Performance&bugs&are&fixed&&&&&&Performance&bugs&take& Performance&bug&fixes& by&more&experienced& more%4me&to&fix are&more&complex developers replication problems more release blocking more dependencies m ore discussion users leaving WFM after a long time 36
  41. 41. Threats to Validity ‘slow’,%‘perf’,% ‘hang’human&bias heurisEcs&for& only&4& only&2&subject performance&bug& dimensions systems idenEficaEon &&&20&subD dimensions 37
  42. 42. Study Setup Chrome ® no traceability 295,198%bugs context&of&the&bug 44,997%bugs 7,603%performance%bugs 510%performance%bugs 4 Dimensions (20 subdimensions) bug&fix&validaEon impact&on&the&user 100%Bugs 100%Bugs + + 100%Bugs 100%Bugs = = 200%Bugs 200%BugsPerformance%Bugs NonEperformance%bugs bug&fix is&regression + + MSR%2011%Quan4ta4ve%Study%(reprise) blocks&release + + WFM&aUer&a&long&Eme + + Performance&bugs&are&fixed& &&&&&Performance&bugs&take& Performance&bug&fixes&impact&on&the&user by&more&experienced& scares&users&away + + more%4me&to&fix are&more&complex developers has&measurements + + has&test&cases + + contains&stacktrace no reproducability + + discussion! has&replicaEon&problems + + ? + replication problems more release is&reported&by&a&developercontext&of&the&bug is&a&duplicate E E blocking more dependencies has&discussion + + depends&on&other&bug + + more discussion users leaving bug&fix blocks&other&bug + E WFM after a long time 37

×