Successfully reported this slideshow.

Assisting Code Search with Automatic Query Reformulation for Bug Localization



Loading in …3
1 of 23
1 of 23

More Related Content

Similar to Assisting Code Search with Automatic Query Reformulation for Bug Localization

Related Books

Free with a 14 day trial from Scribd

See all

Assisting Code Search with Automatic Query Reformulation for Bug Localization

  1. 1. Assisting Code Search with Automatic Query Reformulation for Bug Localization Bunyamin Sisman & Avinash Kak | Purdue University MSR’13
  2. 2. Outline I. Motivation II. Past Work on Automatic Query Reformulation III. Proposed Approach: Spatial Code Proximity (SCP) IV. Data Preparation V. Results VI. Conclusions MSR'13
  3. 3. Main Motivation It is extremely common for software developers to use arbitrary abbreviations & concatenations in software. These are generally difficult to predict when searching the code base of a project. The question is “Is there some way to automatically reformulate a user’s query so that all such relevant terms are also used in retrieval?” MSR'13
  4. 4.  We show how a query can be automatically reformulated for superior retrieval accuracy  We propose a new framework for Query Reformulation, which leverages the spatial proximity of the terms in files  The approach leads to significant improvements over the baseline and the competing Query Reformulation approaches MSR'13 Summary of Our Contribution
  5. 5.  Our approach preserves or improves the retrieval accuracy for 76% of the 4,393 bugs we analyzed for Eclipse and Chrome projects  Our approach improves the retrieval accuracy for 42% of the 4,393 bugs  Improvements are 66% for Eclipse and 90% for Chrome in terms of MAP (Mean Average Precision)  We also describe the conditions under which Query Reformulation may perform poorly. MSR'13 Summary of Our Contribution
  6. 6. Query Reformulation with Relevance Feedback 1. Perform an initial retrieval with the original query 2. Analyze the set of top retrieved documents vis- à-vis the query 3. Reformulate the query MSR'13
  7. 7. Acquiring Relevance Feedback  Implicitly: infer feedback from user interactions  Explicitly: user provides feedback [Gay et al. 2009]  Pseudo Relevance Feedback (PRF): Automatic QR  This is our work! MSR'13
  8. 8. Data Flow in the Proposed Retrieval Framework MSR'13
  9. 9. Automatic Query Reformulation  No user involvement!  It takes less than a second to reformulate a query on ordinary desktop hardware!  It is cheap!  It is effective! MSR'13
  10. 10. Previous Work on Automatic QR (for Text Retrieval) Rocchio’s Formula (ROCC) Relevance Model (RM) MSR'13
  11. 11. The Proposed Approach to QR: Spatial Code Proximity (SCP)  Spatial Code Proximity is an elegant approach to giving greater weights to terms in source code that occur in the vicinity of the terms in a users’ query  Proximities may be created through commonly used concatenations  Punctuation characters  Camel Casing etc…  Underscores: tab_strip_gtk  Camel casing: kPinnedTabAnimationDurationMs MSR'13
  12. 12. Spatial Code Proximity (SCP) (Cont’d)  Tokenize source files and index the positions of the terms in each source file:  Use the distance between terms to find relevant terms vis-à-vis a query! MSR'13
  13. 13. SCP: Bringing the Query into the Picture MSR'13  Example Query: “Browser Animation”  First perform an initial retrieval with the original query  Increase the weights of those nearby terms!
  14. 14. Research Questions  Question 1: Does the proposed QR approach improve the accuracy of source code retrieval. If so, to what extent?  Question 2: How do the QR techniques that are currently in the literature perform for source code retrieval?  Question 3: How does the initial retrieval performance affect the performance of QR?  Question 4: What are the conditions under which QR may perform poorly? MSR'13
  15. 15. Data Preparation  For evaluation, we need a set of queries and the relevant files  We use the titles of the bug reports as queries  We have to link the repository commits to the bug tracking database!  Used regular expressions to detect Bug Fix commits based on commit messages MSR'13
  16. 16. Data Preparation (Cont’d) Eclipse v3.1 Chrome v4.0 #Bugs 4,035 358 Avg. # Relevant Files 2.76 3.82 Avg. #Commits 1.36 1.23 MSR'13 1 Resulting dataset: BUGLinks1
  17. 17. Evaluation Framework  We use Precision and Recall based metrics to evaluate the retrieval accuracy.  Determine the query sets for which the proposed QR approaches lead to 1. improvements in the retrieval accuracy 2. degradation in the retrieval accuracy 3. no change in the retrieval accuracy  Analyze these sets to understand the characteristics of the queries each set contains MSR'13
  18. 18. Evaluation Framework (Cont’d)  For comparison of these sets, we used the following Query Performance Prediction (QPP) metrics [Haiduc et al. 2012, He et al. 2004]:  Average Inverse Document Frequency (avgIDF)  Average Inverse Collection Term Frequency (avgICTF)  Query Scope (QS)  Simplified Clarity Score (SCS)  Additionally, we analyzed  Query Lengths  Number of Relevant files per bug MSR'13
  19. 19. QR with Bug Report Titles ROCC RM SCP (Proposed) 0 500 1000 1500 2000 #Bugs ROCC RM SCP (Proposed) MSR'13
  20. 20. Improvements in Retrieval Accuracy (% Increase in MAP) ROCC RM SCP (Proposed) 0% 20% 40% 60% 80% 100% Eclipse Chrome ROCC RM SCP (Proposed) MSR'13
  21. 21. Conclusions & Future Work  Our framework can use a weak initial query as a jumping off point for a better query.  No user input is necessary  We obtained significant improvements over the baseline and the well-known Automatic QR methods.  Future Work includes evaluation of different term proximity metrics in source code for QR MSR'13
  22. 22. References  [1] B. Sisman and A. Kak, “Incorporating version histories in information retrieval based bug localization,” in Proceedings of the 9th Working Conference on Mining Software Repositories (MSR’12). IEEE, 2012, pp. 50–59  [2] G. Gay, S. Haiduc, A. Marcus, and T. Menzies, “On the use of relevance feedback in IR-based concept location,” in International Conference on Software Maintenance (ICSM’09), sept. 2009, pp. 351 –360.  [3] A. Marcus, A. Sergeyev, V. Rajlich, and J. I. Maletic, “An information retrieval approach to concept location in source code,” in Proceedings of the 11th Working Conference on Reverse Engineering (WCRE’04). IEEE Computer Society, 2004, pp. 214–223 MSR'13
  23. 23. References  [4] S. Haiduc, G. Bavota, R. Oliveto, A. De Lucia, and A. Marcus, “Automatic query performance assessment during the retrieval of software artifacts,” in Proceedings of the 27th International Conference on Automated Software Engineering (ASE’12) . ACM, 2012, pp. 90–99  [5] B. He and I. Ounis, “Inferring query performance using pre-retrieval predictors,” in Proc. Symposium on String Processing and Information Retrieval . Springer Verlag, 2004, pp. 43–54 MSR'13

Editor's Notes

  • Despite the naming conventions in programming languages, the textual content of software is made up by using arbitrary abbreviations and concatenations.
  • Among different approaches to QR, Relevance Feedback has been shown to be an effective method. It basically has three steps
  • It has been shown that Relevance Feedback is an effective method for Query Reformulation.
  • First of all, we index the source code. When a query comes, we do an initial first pass retrieval to get the highest ranked documents.Then the QR module takes the initial query and the first pass retrieval results to reformulate the query.Then with the reformulated query we perform the second retrieval to obtain the final results.
  • There are several previous studies on QR, the most important of which are …
  • The fact that the developer concatenated those terms to declare a variable or class name indicates that those terms are conceptually related.So for a given term nearby terms are more likely relevant
  • There is nothing to do with a query at this point
  • I am showing you an indexed file. Lets consider a query that consists of two terms. We will use a window of size 3 to capture the spatial proximity effects in this file vis-à-vis this particular query.We have a query term browser in the first line. So using a window we increase the weights of the nearby terms!
  • While all the three methods degrade about the same number of queries, SCP improves the largest number of queries.For more than 200 queries SCP does better than the next best model RM. In the paper, you will also find an analysis of the queries for which QR does not lead to improvements.
  • ×