Assisting Code Search with Automatic Query Reformulation for Bug Localization
1. Assisting Code Search with
Automatic Query Reformulation
for Bug Localization
Bunyamin Sisman & Avinash Kak
firstname.lastname@example.org | email@example.com
II. Past Work on Automatic Query Reformulation
III. Proposed Approach: Spatial Code Proximity (SCP)
IV. Data Preparation
3. Main Motivation
It is extremely common for software developers to use
abbreviations & concatenations
in software. These are generally difficult to predict when
searching the code base of a project.
The question is “Is there some way to automatically
reformulate a user’s query so that all such relevant terms
are also used in retrieval?”
4. We show how a query can be automatically
reformulated for superior retrieval accuracy
We propose a new framework for Query
Reformulation, which leverages the spatial proximity
of the terms in files
The approach leads to significant improvements over
the baseline and the competing Query Reformulation
Summary of Our Contribution
5. Our approach preserves or improves the retrieval
accuracy for 76% of the 4,393 bugs we analyzed for
Eclipse and Chrome projects
Our approach improves the retrieval accuracy for 42%
of the 4,393 bugs
Improvements are 66% for Eclipse and 90% for Chrome
in terms of MAP (Mean Average Precision)
We also describe the conditions under which Query
Reformulation may perform poorly.
Summary of Our Contribution
6. Query Reformulation with
1. Perform an initial retrieval with the original query
2. Analyze the set of top retrieved documents vis-
à-vis the query
3. Reformulate the query
9. Automatic Query Reformulation
No user involvement!
It takes less than a second to reformulate a query on
ordinary desktop hardware!
It is cheap!
It is effective!
10. Previous Work on Automatic
QR (for Text Retrieval)
Rocchio’s Formula (ROCC)
Relevance Model (RM)
11. The Proposed Approach to QR:
Spatial Code Proximity (SCP)
Spatial Code Proximity is an elegant approach to
giving greater weights to terms in source code that
occur in the vicinity of the terms in a users’ query
Proximities may be created through commonly used
Camel Casing etc…
Camel casing: kPinnedTabAnimationDurationMs
12. Spatial Code Proximity (SCP)
Tokenize source files and index the positions of the
terms in each source file:
Use the distance between terms to find relevant terms
vis-à-vis a query!
13. SCP: Bringing the Query into the Picture
Example Query: “Browser Animation”
First perform an initial retrieval with the original query
Increase the weights of those nearby terms!
14. Research Questions
Question 1: Does the proposed QR approach improve
the accuracy of source code retrieval. If so, to what
Question 2: How do the QR techniques that are
currently in the literature perform for source code
Question 3: How does the initial retrieval performance
affect the performance of QR?
Question 4: What are the conditions under which QR
may perform poorly?
15. Data Preparation
For evaluation, we need a set of queries
and the relevant files
We use the titles of the bug reports as
We have to link the repository commits
to the bug tracking database!
Used regular expressions to detect Bug Fix
commits based on commit messages
17. Evaluation Framework
We use Precision and Recall based metrics to evaluate
the retrieval accuracy.
Determine the query sets for which the proposed QR
approaches lead to
1. improvements in the retrieval accuracy
2. degradation in the retrieval accuracy
3. no change in the retrieval accuracy
Analyze these sets to understand the characteristics of
the queries each set contains
18. Evaluation Framework (Cont’d)
For comparison of these sets, we used the following Query
Performance Prediction (QPP) metrics [Haiduc et al. 2012, He
et al. 2004]:
Average Inverse Document Frequency (avgIDF)
Average Inverse Collection Term Frequency (avgICTF)
Query Scope (QS)
Simplified Clarity Score (SCS)
Additionally, we analyzed
Number of Relevant files per bug
21. Conclusions & Future Work
Our framework can use a weak initial query
as a jumping off point for a better query.
No user input is necessary
We obtained significant improvements over
the baseline and the well-known Automatic
Future Work includes evaluation of different
term proximity metrics in source code for QR
 B. Sisman and A. Kak, “Incorporating version
histories in information retrieval based bug
localization,” in Proceedings of the 9th Working
Conference on Mining Software Repositories (MSR’12).
IEEE, 2012, pp. 50–59
 G. Gay, S. Haiduc, A. Marcus, and T. Menzies, “On
the use of relevance feedback in IR-based concept
location,” in International Conference on Software
Maintenance (ICSM’09), sept. 2009, pp. 351 –360.
 A. Marcus, A. Sergeyev, V. Rajlich, and J. I.
Maletic, “An information retrieval approach to
concept location in source code,” in Proceedings of
the 11th Working Conference on Reverse Engineering
(WCRE’04). IEEE Computer Society, 2004, pp. 214–223
 S. Haiduc, G. Bavota, R. Oliveto, A. De Lucia, and
A. Marcus, “Automatic query performance assessment
during the retrieval of software artifacts,” in
Proceedings of the 27th International Conference on
Automated Software Engineering (ASE’12) .
ACM, 2012, pp. 90–99
 B. He and I. Ounis, “Inferring query performance
using pre-retrieval predictors,” in Proc. Symposium on
String Processing and Information Retrieval . Springer
Verlag, 2004, pp. 43–54
Despite the naming conventions in programming languages, the textual content of software is made up by using arbitrary abbreviations and concatenations.
Among different approaches to QR, Relevance Feedback has been shown to be an effective method. It basically has three steps
It has been shown that Relevance Feedback is an effective method for Query Reformulation.
First of all, we index the source code. When a query comes, we do an initial first pass retrieval to get the highest ranked documents.Then the QR module takes the initial query and the first pass retrieval results to reformulate the query.Then with the reformulated query we perform the second retrieval to obtain the final results.
There are several previous studies on QR, the most important of which are …
The fact that the developer concatenated those terms to declare a variable or class name indicates that those terms are conceptually related.So for a given term nearby terms are more likely relevant
There is nothing to do with a query at this point
I am showing you an indexed file. Lets consider a query that consists of two terms. We will use a window of size 3 to capture the spatial proximity effects in this file vis-à-vis this particular query.We have a query term browser in the first line. So using a window we increase the weights of the nearby terms!
While all the three methods degrade about the same number of queries, SCP improves the largest number of queries.For more than 200 queries SCP does better than the next best model RM. In the paper, you will also find an analysis of the queries for which QR does not lead to improvements.