Your SlideShare is downloading. ×
Characterizing Verification of Bug Fixes in Two Open Source IDEs (MSR 2012)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Characterizing Verification of Bug Fixes in Two Open Source IDEs (MSR 2012)

302
views

Published on

Paper by Rodrigo Souza and Christina Chavez, presented at the 9th Working Conference on Mining Software Repositories (MSR 2012) -- http://2012.msrconf.org/program.php …

Paper by Rodrigo Souza and Christina Chavez, presented at the 9th Working Conference on Mining Software Repositories (MSR 2012) -- http://2012.msrconf.org/program.php

Experimental package and presentation video available at https://sites.google.com/site/rodrigorgs2/msr2012

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
302
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
  • First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
  • First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
  • First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
  • First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
  • First of all, this is how a bug report works.\nFirst someone reports a new bug.\nAfter some discussion, a developer submits a bug fix for the problem, and marks the bug as FIXED.\nThen, someone else verifies that the bug fix is appropriate, and marks the bug as VERIFIED.\n
  • In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
  • In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
  • In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
  • In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
  • In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
  • In this exploratory work, we try to characterize the process of verifying bug fixes by mining bug repositories.\nWe investigated three questions: when are bug fixes verified? who verifies them? and how are them verified?\n\n
  • We used data from the previous MSR Challenge, containing about 10 years of bug reports from two open source IDEs: Eclipse and NetBeans. We analyzed two subprojects for each IDE.\n
  • So, first, when are bug fixes verified?\n
  • We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
  • We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
  • We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
  • We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
  • We have plotted the accumulated number of verifications over time for the projects.\nFor NetBeans, the verification rate is almost constant, meaning that bug fixes are verified all the time.\n\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • For Eclipse/Platform, however, verifications are much more frequent just before a release,\nwhich suggests that there’s a verification phase in Eclipse’s process.\n
  • Next, who verifies the bug fixes? In some projects, there is a team dedicated to verifications: Quality Assurance team, or QA team.\n
  • We defined that the QA team of a project is formed by all developers that perform at least 10 times more verifications than bug fixes.\n
  • Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
  • Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
  • Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
  • Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
  • Using this definition, we found that in NetBeans the QA team contains 20% of its developers, who perform more than 80% of the verifications.\n
  • In Eclipse, we found no evidence of a QA team.\n
  • Finally, how are the bug fixes verified?\nThat is, what are the techniques used to verify each bug fix?\n
  • We looked at the comments developers write when they mark a bug as VERIFIED.\n
  • But it appears that most comments just state the obvious: that the bug fix was verified using some version of the software.\nUsing regular expressions, we found that less than 4% of the comments refer to automated testing or code inspection, although further research is needed.\n
  • We’d also like to share some pitfalls we’ve found during this research.\n
  • Plotting the number of verifications in NetBeans/Platform over its lifetime, we found what appears to be a huge verification effort, represented by a big rise in the graph.\nHowever, by looking at the data, we discovered that this big rise represents...\n\n
  • More than 2 thousand bugs that were verified... in just 5 hours... by only 1 guy!\nOf course, no human being can do that.\nThe developer was not actually verifying the bug fixes. It turns out, Superman was just doing some cleanup by marking old bugs as verified.\n\n2003-07-01, 2676 bugs, user 17822, 2003-07-01 10:52:14 to 2003-07-01 16:08:14 (about 5 hours)\n
  • More than 2 thousand bugs that were verified... in just 5 hours... by only 1 guy!\nOf course, no human being can do that.\nThe developer was not actually verifying the bug fixes. It turns out, Superman was just doing some cleanup by marking old bugs as verified.\n\n2003-07-01, 2676 bugs, user 17822, 2003-07-01 10:52:14 to 2003-07-01 16:08:14 (about 5 hours)\n
  • More than 2 thousand bugs that were verified... in just 5 hours... by only 1 guy!\nOf course, no human being can do that.\nThe developer was not actually verifying the bug fixes. It turns out, Superman was just doing some cleanup by marking old bugs as verified.\n\n2003-07-01, 2676 bugs, user 17822, 2003-07-01 10:52:14 to 2003-07-01 16:08:14 (about 5 hours)\n
  • So, be careful: mass verifications may represent a large part of the verifications in a project, but they are not really software verification and may bias your analyses.\nIn the previous analyses, mass verifications were discarded.\n\n-- Be careful, because such mass verifications may bias your analyses\n
  • Also, by reading a few comments, we found that, in some projects, marking a bug as VERIFIED has a special meaning. \nFor example, in Eclipse/EMF it just means that the bug fix was made available in a build).\n\n
  • Future work\n
  • People say that, if you control your process, you can control the quality of your product. \n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • So we want to investigate how particular features of the bug verification process may influence (directly or indirectly) the reopening of bugs.\n For example, is it more effective to have a verification phase or to verify bug fixes all the time?\n\nWe intend to build a causal network to investigate this question, so we are looking for variables that influence the verification process or the reopening of bugs.\n
  • Thank you very much!\n
  • Transcript

    • 1. Characterizing Verification of Bug Fixes in Two Open Source IDEs Rodrigo Souza* and Christina Chavez Software Engineering Labs Department of Computer Science - IM Universidade Federal da Bahia (UFBA), Brazil {rodrigo, flach}@dcc.ufba.br * speakerJune 2, 2012MSR 2012, Zürich
    • 2. 2
    • 3. NEW2
    • 4. NEW FI XED2
    • 5. NEW FI XED E D V ERIFI2
    • 6. Characterize the verification process by mining bug repositories VER IFIED 3
    • 7. Characterize the verification process by mining bug repositories VER IFIED When? Who? How? 3
    • 8. DataMSR 2011 Challenge data set (~ 10 years of bug reports) Platform VersionControl Platform EMF (Eclipse Modeling Framework) 4
    • 9. When?
    • 10. accumulated # of verifications ~ 800verifications ~ 2 years 6
    • 11. accumulated # of verifications ~ 800 * = releaseverifications * * * * ~ 2 years 6
    • 12. accumulated # of verifications ~ 700 * = release *verifications * * * * * * * ~ 1 year 7
    • 13. accumulated # of verifications ~ 700 * = release *verifications * * verification * phase * * * * ~ 1 year 7
    • 14. Who?
    • 15. QA developerbug fixes x ≥ 10x verifications 9
    • 16. QA team10
    • 17. 20% of developers QA team 10
    • 18. 20% of developers QA team 80% of the verifications 10
    • 19. QAteam 11
    • 20. How?
    • 21. 13
    • 22. Most comments just state the obvious. 4% refer to Less than automated testing or code inspection. Further research needed 14
    • 23. Pitfalls
    • 24. accumulated # of verifications ~ 14kverifications ~ 10 years 16
    • 25. accumulated # of verifications ~ 14kverifications ~ 10 years 16
    • 26. 17
    • 27. 2.6k bugs 17
    • 28. 2.6k bugs 17
    • 29. 2.6k bugs 17
    • 30.  Mass verifications: representrepository cleanup, not softwareverification.They may represent a large part of theverifications and bias your analyses. 18
    • 31.  Pseudo verifications: In someprojects, marking a bug as VERIFIEDmeans something else!(e.g., in Eclipse/EMF, since 2007, itmeans that the fix is available in a build) 19
    • 32. Future Work
    • 33. process product(software) 21
    • 34. process verification process* product (software)* qa team, reopeningverification phase etc. 22
    • 35. causal process (bayesian) network verification process* product (software)* qa team, reopeningverification phase etc. 22
    • 36. Thanks!Verification ✓ ✗ phase QA team ✗ ✓ Comments rarely state the verification technique. Beware of mass verifications and pseudo verifications. 23