SlideShare a Scribd company logo
The Forgotten Role of
Search Queries in IR-based
Bug Localization:
An Empirical Study
Masud Rahman*, Foutse Khomh$,
Shamima Yeasmin+, Chanchal Roy+
Dalhousie University*, Polytechnique Montréal$,
University of Saskatchewan+, Canada
Software Bugs
• 606 software bugs recorded in 2017
o $1.7 trillion costs to the global economy
o 3.7 billion users affected
o 314 companies impacted
• Developers spend ~50% of the time in
finding and fixing these bugs.
2
Bug Report & Bug Localization
Bug Localization
3
IR-Based Bug Localization
JDIValue, toString, execute,
EvaluationThread, run, NullPointerException
able cast null
Keyword
selection
127 Words
53
1
4
Query Reformulation in IR-
based Bug Localization
5
JDIValue, toString, execute,
EvaluationThread, run, NullPointerException
able cast null
88%
Research Questions
RQ1: How do the state-of-the-art approaches
perform in identifying appropriate search keywords
from bug reports for IR-based bug localization?
RQ2: Can optimal, near-optimal search queries be
constructed from the bug reports that lack bug
localization hints or simply contain natural language
only texts?
RQ3: How optimal, near-optimal, and non-optimal
search queries differ from each other in their
characteristics and performances?
6
Workflow of our Study
Bug-fixing
commits
Linking &
filtration
Refined
dataset
Bug reports
Finding
summary
7
RQ1: Frequency vs.
Graph-based keyword
selection
RQ2: Optimal vs. Baseline
search queries
RQ3: Optimal vs. Non-
optimal queries
Dataset, Metrics, & Setup
• Six subject systems: 2,320 bug reports
• Four performance metrics: Hit@K, MRR,
MAP, and Query Effectiveness (QE)
• QE: Rank of the first true positive.
• Baseline queries: title, description, title +
description
8
9
RQ1: Frequency vs. Graph-based
Keyword Selection for Search
PageRank Algorithm
RQ1: Frequency vs. Graph-based
Keywords for Bug Localization
10
Method Hit@1 Hit@10 MRR MAP
Baseline 31.98% 66.50% 0.43 42.69%
TF 24.39% 55.58% 0.34 33.94%
TF-IDF 27.02% 59.80% 0.37 36.76%
Kevic & Fritz 23.36% 52.92% 0.32 31.98%
Graph-based keyword selection
STRICT 25.82% 63.02% 0.37 37.31%
Query Min Median Average Max
Baseline 03 32 49 406
TF-IDF 03 10 10 10
STRICT 03 10 10 10
RQ1: Keyword Selection from
4 Subsets of Bug Reports
11
Method Hit@1 Hit@10 MRR MAP
Bug reports with good baseline and no localization hints (567)
Baseline 41.96% 100.00% 0.60 58.15%
STRICT 36.14% 84.83% 0.51 50.29%
Bug reports with good baseline and localization hints (954)
Baseline 50.01% 100.00% 0.66 66.63%
TF-IDF 41.59% 83.55% 0.54 55.15%
Bug reports with poor baseline and no localization hints (372)
Baseline 0% 0% 0 0%
STRICT 1.56% 16.89% 0.05 5.12%
Bug reports with poor baseline and localization hints (427)
Baseline 0% 0% 0 0%
STRICT 4.91% 25.77% 0.11 10.66%
34%
RQ2: Optimal Query
Generation from a Bug Report
12
P = {q1, q2, q3,… qn}
Selection
Crossover
Mutation
Fitness
calculation
Qop = {k1, k2……km}
QE, MAP
Primitive dialogs
RQ2: Optimal vs. Baseline
Queries in Bug Localization
13
Bug reports with poor baseline and no localization hints (372)
Baseline 0% 0% 0 0%
Optimal 50.04% 77.96% 0.58 56.47%
Bug reports with poor baseline and localization hints (427)
Baseline 0% 0% 0 0%
Optimal 80.70% 93.37% 0.85 86.19%
Method Hit@1 Hit@10 MRR MAP
All bug reports (2,320)
Baseline 31.98% 66.50% 0.43 42.69%
Optimal 87.41% 95.74% 0.90 90.00%
RQ3: Non-Optimal vs. Optimal
Search Queries
14
• Query dataset: 13,914 search queries.
• 4,893 optimal search queries
• 5,164 non-optimal queries
• QE: Rank of the first true positive.
• Optimal: QE = 1, Non-optimal: QE > 10
• 31 query characteristics from literature
• Query classification using Random
Forest algorithm.
15
RQ3: Query Classification &
Feature Importance
Query Precision Recall
Optimal 84.60% 79.80%
Non-optimal 81.90% 86.20%
Frequency, Entropy,
Location, POS
RQ3: Optimal vs. Non-Optimal
– Regression Analysis
16
Significant
Actionable Insights
• Frequency: Optimal keywords are less
frequent than non-optimal ones in a bug
report.
• Entropy: Optimal keywords are less
ambiguous than non-optimal ones.
• Location: Optimal keywords are more likely
to be found in the body section of a bug
report.
• POS: Optimal keywords are more likely to be
nouns than the non-optimal ones.
17
RQ3: Query Improvements
using Actionable Insights
18
Bug reports with poor baseline and no localization hints (372)
Method Hit@1 Hit@10 MRR MAP
Baseline 0% 0% 0 0%
Baseline + Insights 1.28% 16.09% 0.04 4.50%
STRICT 1.56% 16.89% 0.05 5.12%
Insights + STRICT 2.66% 15.18% 0.05 5.09%
Bug reports with poor baseline and localization hints (427)
Baseline 0% 0% 0 0%
Baseline + Insights 6.79% 34.29% 0.14 13.79%
STRICT 4.91% 25.77% 0.11 10.66%
STRICT + Insights 6.80% 32.83% 0.13 13.39%
Take-Away Messages
• 34% of bug reports lead to very poor search
queries (RQ1)
• Even state-of-the-art approaches are not
sufficient to detect appropriate keywords
from them (RQ1)
• Genetic algorithm (GA) shows that optimal
keywords exist in those bug reports (RQ2)
• Optimal search queries can achieve up to
50%--80% Hit@1 and 56%--86% MAP.
19
• Optimal keywords are different than non-
optimal keywords (RQ3)
• Four aspects: frequency, term entropy,
term location, and part of speech.
• Insights lead to 27%--34% higher Hit@10.
• ML: How to automatically predict the
appropriate keywords from a bug report?
• GA: How to automatically determine the
fitness of a candidate search query?
20
Take-Away Messages
Thank you! Questions?
21
Masud Rahman, PhD
masud.rahman@dal.ca
@masud2336
RQ2: Query Improvement &
Worsening Ratios
22

More Related Content

Similar to The Forgotten Role of Search Queries in IR-based Bug Localization: An Empirical Study

SurfExample- Recommendation of Exception Handling Code Examples
SurfExample- Recommendation of Exception Handling Code ExamplesSurfExample- Recommendation of Exception Handling Code Examples
SurfExample- Recommendation of Exception Handling Code Examples
Masud Rahman
 
Vanderbilt b
Vanderbilt bVanderbilt b
Vanderbilt b
Claudine Garcia
 
Machine Learning for Malware Detection: Beyond Accuracy Rates
Machine Learning for Malware Detection: Beyond Accuracy RatesMachine Learning for Malware Detection: Beyond Accuracy Rates
Machine Learning for Malware Detection: Beyond Accuracy Rates
Marcus Botacin
 
Multisensor data fusion in Food Quality Assessment
Multisensor data fusion in Food Quality AssessmentMultisensor data fusion in Food Quality Assessment
Multisensor data fusion in Food Quality Assessment
Alka Mishra
 
SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...
SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...
SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...
geraintduck
 
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...
Sagar Deogirkar
 
Results from the Enterprise Search and Findability Survey 2012
Results from the Enterprise Search and Findability Survey 2012Results from the Enterprise Search and Findability Survey 2012
Results from the Enterprise Search and Findability Survey 2012
Findwise
 
Intelligent and Automatic In vivo Detection and Quantification of Transplante...
Intelligent and Automatic In vivo Detection and Quantification of Transplante...Intelligent and Automatic In vivo Detection and Quantification of Transplante...
Intelligent and Automatic In vivo Detection and Quantification of Transplante...
Michigan State University Research
 
Genomics Crash Course for Data Engineers
Genomics Crash Course for Data EngineersGenomics Crash Course for Data Engineers
Genomics Crash Course for Data Engineers
Allen Day, PhD
 
Relevance Improvements at Cengage - Ivan Provalov
Relevance Improvements at Cengage - Ivan ProvalovRelevance Improvements at Cengage - Ivan Provalov
Relevance Improvements at Cengage - Ivan Provalov
lucenerevolution
 
Trivandrum
TrivandrumTrivandrum
Trivandrum
vgovindaraju
 
BH-US-06-Bilar.pdf
BH-US-06-Bilar.pdfBH-US-06-Bilar.pdf
BH-US-06-Bilar.pdf
MohammadRazavi17
 
Semantic Analysis to Compute Personality Traits from Social Media Posts
Semantic Analysis to Compute Personality Traits from Social Media PostsSemantic Analysis to Compute Personality Traits from Social Media Posts
Semantic Analysis to Compute Personality Traits from Social Media Posts
Giulio Carducci
 
Improving IR-Based Bug Localization with Context-Aware-Query Reformulation
Improving IR-Based Bug Localization with Context-Aware-Query ReformulationImproving IR-Based Bug Localization with Context-Aware-Query Reformulation
Improving IR-Based Bug Localization with Context-Aware-Query Reformulation
Masud Rahman
 
Hendrix 2015 composite endpoints redacted
Hendrix 2015 composite endpoints redacted Hendrix 2015 composite endpoints redacted
Hendrix 2015 composite endpoints redacted
Alzforum
 
PhD Comprehensive exam of Masud Rahman
PhD Comprehensive exam of Masud RahmanPhD Comprehensive exam of Masud Rahman
PhD Comprehensive exam of Masud Rahman
Masud Rahman
 
Evaluating the Usefulness of IR-Based Fault LocalizationTechniques
Evaluating the Usefulness of IR-Based Fault LocalizationTechniquesEvaluating the Usefulness of IR-Based Fault LocalizationTechniques
Evaluating the Usefulness of IR-Based Fault LocalizationTechniques
Alex Orso
 

Similar to The Forgotten Role of Search Queries in IR-based Bug Localization: An Empirical Study (17)

SurfExample- Recommendation of Exception Handling Code Examples
SurfExample- Recommendation of Exception Handling Code ExamplesSurfExample- Recommendation of Exception Handling Code Examples
SurfExample- Recommendation of Exception Handling Code Examples
 
Vanderbilt b
Vanderbilt bVanderbilt b
Vanderbilt b
 
Machine Learning for Malware Detection: Beyond Accuracy Rates
Machine Learning for Malware Detection: Beyond Accuracy RatesMachine Learning for Malware Detection: Beyond Accuracy Rates
Machine Learning for Malware Detection: Beyond Accuracy Rates
 
Multisensor data fusion in Food Quality Assessment
Multisensor data fusion in Food Quality AssessmentMultisensor data fusion in Food Quality Assessment
Multisensor data fusion in Food Quality Assessment
 
SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...
SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...
SMBM 2012: Ambiguity and Variability of Database and Software Names in Bioinf...
 
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...
Comparative Study of Machine Learning Algorithms for Sentiment Analysis with ...
 
Results from the Enterprise Search and Findability Survey 2012
Results from the Enterprise Search and Findability Survey 2012Results from the Enterprise Search and Findability Survey 2012
Results from the Enterprise Search and Findability Survey 2012
 
Intelligent and Automatic In vivo Detection and Quantification of Transplante...
Intelligent and Automatic In vivo Detection and Quantification of Transplante...Intelligent and Automatic In vivo Detection and Quantification of Transplante...
Intelligent and Automatic In vivo Detection and Quantification of Transplante...
 
Genomics Crash Course for Data Engineers
Genomics Crash Course for Data EngineersGenomics Crash Course for Data Engineers
Genomics Crash Course for Data Engineers
 
Relevance Improvements at Cengage - Ivan Provalov
Relevance Improvements at Cengage - Ivan ProvalovRelevance Improvements at Cengage - Ivan Provalov
Relevance Improvements at Cengage - Ivan Provalov
 
Trivandrum
TrivandrumTrivandrum
Trivandrum
 
BH-US-06-Bilar.pdf
BH-US-06-Bilar.pdfBH-US-06-Bilar.pdf
BH-US-06-Bilar.pdf
 
Semantic Analysis to Compute Personality Traits from Social Media Posts
Semantic Analysis to Compute Personality Traits from Social Media PostsSemantic Analysis to Compute Personality Traits from Social Media Posts
Semantic Analysis to Compute Personality Traits from Social Media Posts
 
Improving IR-Based Bug Localization with Context-Aware-Query Reformulation
Improving IR-Based Bug Localization with Context-Aware-Query ReformulationImproving IR-Based Bug Localization with Context-Aware-Query Reformulation
Improving IR-Based Bug Localization with Context-Aware-Query Reformulation
 
Hendrix 2015 composite endpoints redacted
Hendrix 2015 composite endpoints redacted Hendrix 2015 composite endpoints redacted
Hendrix 2015 composite endpoints redacted
 
PhD Comprehensive exam of Masud Rahman
PhD Comprehensive exam of Masud RahmanPhD Comprehensive exam of Masud Rahman
PhD Comprehensive exam of Masud Rahman
 
Evaluating the Usefulness of IR-Based Fault LocalizationTechniques
Evaluating the Usefulness of IR-Based Fault LocalizationTechniquesEvaluating the Usefulness of IR-Based Fault LocalizationTechniques
Evaluating the Usefulness of IR-Based Fault LocalizationTechniques
 

More from Masud Rahman

HereWeCode 2022: Dalhousie University
HereWeCode 2022: Dalhousie UniversityHereWeCode 2022: Dalhousie University
HereWeCode 2022: Dalhousie University
Masud Rahman
 
PhD Seminar - Masud Rahman, University of Saskatchewan
PhD Seminar - Masud Rahman, University of SaskatchewanPhD Seminar - Masud Rahman, University of Saskatchewan
PhD Seminar - Masud Rahman, University of Saskatchewan
Masud Rahman
 
PhD proposal of Masud Rahman
PhD proposal of Masud RahmanPhD proposal of Masud Rahman
PhD proposal of Masud Rahman
Masud Rahman
 
Doctoral Symposium of Masud Rahman
Doctoral Symposium of Masud RahmanDoctoral Symposium of Masud Rahman
Doctoral Symposium of Masud Rahman
Masud Rahman
 
Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...
Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...
Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...
Masud Rahman
 
ICSE2018-Poster-Bug-Localization
ICSE2018-Poster-Bug-LocalizationICSE2018-Poster-Bug-Localization
ICSE2018-Poster-Bug-Localization
Masud Rahman
 
MSR2017-Challenge
MSR2017-ChallengeMSR2017-Challenge
MSR2017-Challenge
Masud Rahman
 
MSR2017-RevHelper
MSR2017-RevHelperMSR2017-RevHelper
MSR2017-RevHelper
Masud Rahman
 
STRICT-SANER2017
STRICT-SANER2017STRICT-SANER2017
STRICT-SANER2017
Masud Rahman
 
MSR2015-Challenge
MSR2015-ChallengeMSR2015-Challenge
MSR2015-Challenge
Masud Rahman
 
MSR2014-Challenge
MSR2014-ChallengeMSR2014-Challenge
MSR2014-Challenge
Masud Rahman
 
CodeInsight-SCAM2015
CodeInsight-SCAM2015CodeInsight-SCAM2015
CodeInsight-SCAM2015
Masud Rahman
 
STRICT-SANER2015
STRICT-SANER2015STRICT-SANER2015
STRICT-SANER2015
Masud Rahman
 
CMPT-842-BRACK
CMPT-842-BRACKCMPT-842-BRACK
CMPT-842-BRACK
Masud Rahman
 
RACK-Tool-ICSE2017
RACK-Tool-ICSE2017RACK-Tool-ICSE2017
RACK-Tool-ICSE2017
Masud Rahman
 
RACK-SANER2016
RACK-SANER2016RACK-SANER2016
RACK-SANER2016
Masud Rahman
 
QUICKAR-ASE2016-Singapore
QUICKAR-ASE2016-SingaporeQUICKAR-ASE2016-Singapore
QUICKAR-ASE2016-Singapore
Masud Rahman
 
CORRECT-ToolDemo-ASE2016
CORRECT-ToolDemo-ASE2016CORRECT-ToolDemo-ASE2016
CORRECT-ToolDemo-ASE2016
Masud Rahman
 
ACER-ASE2017-slides
ACER-ASE2017-slidesACER-ASE2017-slides
ACER-ASE2017-slides
Masud Rahman
 
CMPT470-usask-guest-lecture
CMPT470-usask-guest-lectureCMPT470-usask-guest-lecture
CMPT470-usask-guest-lecture
Masud Rahman
 

More from Masud Rahman (20)

HereWeCode 2022: Dalhousie University
HereWeCode 2022: Dalhousie UniversityHereWeCode 2022: Dalhousie University
HereWeCode 2022: Dalhousie University
 
PhD Seminar - Masud Rahman, University of Saskatchewan
PhD Seminar - Masud Rahman, University of SaskatchewanPhD Seminar - Masud Rahman, University of Saskatchewan
PhD Seminar - Masud Rahman, University of Saskatchewan
 
PhD proposal of Masud Rahman
PhD proposal of Masud RahmanPhD proposal of Masud Rahman
PhD proposal of Masud Rahman
 
Doctoral Symposium of Masud Rahman
Doctoral Symposium of Masud RahmanDoctoral Symposium of Masud Rahman
Doctoral Symposium of Masud Rahman
 
Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...
Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...
Supporting Source Code Search with Context-Aware and Semantics-Driven Code Se...
 
ICSE2018-Poster-Bug-Localization
ICSE2018-Poster-Bug-LocalizationICSE2018-Poster-Bug-Localization
ICSE2018-Poster-Bug-Localization
 
MSR2017-Challenge
MSR2017-ChallengeMSR2017-Challenge
MSR2017-Challenge
 
MSR2017-RevHelper
MSR2017-RevHelperMSR2017-RevHelper
MSR2017-RevHelper
 
STRICT-SANER2017
STRICT-SANER2017STRICT-SANER2017
STRICT-SANER2017
 
MSR2015-Challenge
MSR2015-ChallengeMSR2015-Challenge
MSR2015-Challenge
 
MSR2014-Challenge
MSR2014-ChallengeMSR2014-Challenge
MSR2014-Challenge
 
CodeInsight-SCAM2015
CodeInsight-SCAM2015CodeInsight-SCAM2015
CodeInsight-SCAM2015
 
STRICT-SANER2015
STRICT-SANER2015STRICT-SANER2015
STRICT-SANER2015
 
CMPT-842-BRACK
CMPT-842-BRACKCMPT-842-BRACK
CMPT-842-BRACK
 
RACK-Tool-ICSE2017
RACK-Tool-ICSE2017RACK-Tool-ICSE2017
RACK-Tool-ICSE2017
 
RACK-SANER2016
RACK-SANER2016RACK-SANER2016
RACK-SANER2016
 
QUICKAR-ASE2016-Singapore
QUICKAR-ASE2016-SingaporeQUICKAR-ASE2016-Singapore
QUICKAR-ASE2016-Singapore
 
CORRECT-ToolDemo-ASE2016
CORRECT-ToolDemo-ASE2016CORRECT-ToolDemo-ASE2016
CORRECT-ToolDemo-ASE2016
 
ACER-ASE2017-slides
ACER-ASE2017-slidesACER-ASE2017-slides
ACER-ASE2017-slides
 
CMPT470-usask-guest-lecture
CMPT470-usask-guest-lectureCMPT470-usask-guest-lecture
CMPT470-usask-guest-lecture
 

Recently uploaded

ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdfANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
Priyankaranawat4
 
Pride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School DistrictPride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School District
David Douglas School District
 
Cognitive Development Adolescence Psychology
Cognitive Development Adolescence PsychologyCognitive Development Adolescence Psychology
Cognitive Development Adolescence Psychology
paigestewart1632
 
Digital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments UnitDigital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments Unit
chanes7
 
MARY JANE WILSON, A “BOA MÃE” .
MARY JANE WILSON, A “BOA MÃE”           .MARY JANE WILSON, A “BOA MÃE”           .
MARY JANE WILSON, A “BOA MÃE” .
Colégio Santa Teresinha
 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
TechSoup
 
How to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRMHow to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRM
Celine George
 
PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.
Dr. Shivangi Singh Parihar
 
World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024
ak6969907
 
How to Build a Module in Odoo 17 Using the Scaffold Method
How to Build a Module in Odoo 17 Using the Scaffold MethodHow to Build a Module in Odoo 17 Using the Scaffold Method
How to Build a Module in Odoo 17 Using the Scaffold Method
Celine George
 
Liberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdfLiberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdf
WaniBasim
 
The Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collectionThe Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collection
Israel Genealogy Research Association
 
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptxChapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
Mohd Adib Abd Muin, Senior Lecturer at Universiti Utara Malaysia
 
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat  Leveraging AI for Diversity, Equity, and InclusionExecutive Directors Chat  Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
TechSoup
 
How to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP ModuleHow to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP Module
Celine George
 
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Dr. Vinod Kumar Kanvaria
 
How to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 InventoryHow to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 Inventory
Celine George
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
mulvey2
 
BBR 2024 Summer Sessions Interview Training
BBR  2024 Summer Sessions Interview TrainingBBR  2024 Summer Sessions Interview Training
BBR 2024 Summer Sessions Interview Training
Katrina Pritchard
 
South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)
Academy of Science of South Africa
 

Recently uploaded (20)

ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdfANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
 
Pride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School DistrictPride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School District
 
Cognitive Development Adolescence Psychology
Cognitive Development Adolescence PsychologyCognitive Development Adolescence Psychology
Cognitive Development Adolescence Psychology
 
Digital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments UnitDigital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments Unit
 
MARY JANE WILSON, A “BOA MÃE” .
MARY JANE WILSON, A “BOA MÃE”           .MARY JANE WILSON, A “BOA MÃE”           .
MARY JANE WILSON, A “BOA MÃE” .
 
Walmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdfWalmart Business+ and Spark Good for Nonprofits.pdf
Walmart Business+ and Spark Good for Nonprofits.pdf
 
How to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRMHow to Manage Your Lost Opportunities in Odoo 17 CRM
How to Manage Your Lost Opportunities in Odoo 17 CRM
 
PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.PCOS corelations and management through Ayurveda.
PCOS corelations and management through Ayurveda.
 
World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024
 
How to Build a Module in Odoo 17 Using the Scaffold Method
How to Build a Module in Odoo 17 Using the Scaffold MethodHow to Build a Module in Odoo 17 Using the Scaffold Method
How to Build a Module in Odoo 17 Using the Scaffold Method
 
Liberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdfLiberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdf
 
The Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collectionThe Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collection
 
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptxChapter 4 - Islamic Financial Institutions in Malaysia.pptx
Chapter 4 - Islamic Financial Institutions in Malaysia.pptx
 
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat  Leveraging AI for Diversity, Equity, and InclusionExecutive Directors Chat  Leveraging AI for Diversity, Equity, and Inclusion
Executive Directors Chat Leveraging AI for Diversity, Equity, and Inclusion
 
How to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP ModuleHow to Add Chatter in the odoo 17 ERP Module
How to Add Chatter in the odoo 17 ERP Module
 
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
 
How to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 InventoryHow to Setup Warehouse & Location in Odoo 17 Inventory
How to Setup Warehouse & Location in Odoo 17 Inventory
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
 
BBR 2024 Summer Sessions Interview Training
BBR  2024 Summer Sessions Interview TrainingBBR  2024 Summer Sessions Interview Training
BBR 2024 Summer Sessions Interview Training
 
South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)
 

The Forgotten Role of Search Queries in IR-based Bug Localization: An Empirical Study

  • 1. The Forgotten Role of Search Queries in IR-based Bug Localization: An Empirical Study Masud Rahman*, Foutse Khomh$, Shamima Yeasmin+, Chanchal Roy+ Dalhousie University*, Polytechnique Montréal$, University of Saskatchewan+, Canada
  • 2. Software Bugs • 606 software bugs recorded in 2017 o $1.7 trillion costs to the global economy o 3.7 billion users affected o 314 companies impacted • Developers spend ~50% of the time in finding and fixing these bugs. 2
  • 3. Bug Report & Bug Localization Bug Localization 3
  • 4. IR-Based Bug Localization JDIValue, toString, execute, EvaluationThread, run, NullPointerException able cast null Keyword selection 127 Words 53 1 4
  • 5. Query Reformulation in IR- based Bug Localization 5 JDIValue, toString, execute, EvaluationThread, run, NullPointerException able cast null 88%
  • 6. Research Questions RQ1: How do the state-of-the-art approaches perform in identifying appropriate search keywords from bug reports for IR-based bug localization? RQ2: Can optimal, near-optimal search queries be constructed from the bug reports that lack bug localization hints or simply contain natural language only texts? RQ3: How optimal, near-optimal, and non-optimal search queries differ from each other in their characteristics and performances? 6
  • 7. Workflow of our Study Bug-fixing commits Linking & filtration Refined dataset Bug reports Finding summary 7 RQ1: Frequency vs. Graph-based keyword selection RQ2: Optimal vs. Baseline search queries RQ3: Optimal vs. Non- optimal queries
  • 8. Dataset, Metrics, & Setup • Six subject systems: 2,320 bug reports • Four performance metrics: Hit@K, MRR, MAP, and Query Effectiveness (QE) • QE: Rank of the first true positive. • Baseline queries: title, description, title + description 8
  • 9. 9 RQ1: Frequency vs. Graph-based Keyword Selection for Search PageRank Algorithm
  • 10. RQ1: Frequency vs. Graph-based Keywords for Bug Localization 10 Method Hit@1 Hit@10 MRR MAP Baseline 31.98% 66.50% 0.43 42.69% TF 24.39% 55.58% 0.34 33.94% TF-IDF 27.02% 59.80% 0.37 36.76% Kevic & Fritz 23.36% 52.92% 0.32 31.98% Graph-based keyword selection STRICT 25.82% 63.02% 0.37 37.31% Query Min Median Average Max Baseline 03 32 49 406 TF-IDF 03 10 10 10 STRICT 03 10 10 10
  • 11. RQ1: Keyword Selection from 4 Subsets of Bug Reports 11 Method Hit@1 Hit@10 MRR MAP Bug reports with good baseline and no localization hints (567) Baseline 41.96% 100.00% 0.60 58.15% STRICT 36.14% 84.83% 0.51 50.29% Bug reports with good baseline and localization hints (954) Baseline 50.01% 100.00% 0.66 66.63% TF-IDF 41.59% 83.55% 0.54 55.15% Bug reports with poor baseline and no localization hints (372) Baseline 0% 0% 0 0% STRICT 1.56% 16.89% 0.05 5.12% Bug reports with poor baseline and localization hints (427) Baseline 0% 0% 0 0% STRICT 4.91% 25.77% 0.11 10.66% 34%
  • 12. RQ2: Optimal Query Generation from a Bug Report 12 P = {q1, q2, q3,… qn} Selection Crossover Mutation Fitness calculation Qop = {k1, k2……km} QE, MAP Primitive dialogs
  • 13. RQ2: Optimal vs. Baseline Queries in Bug Localization 13 Bug reports with poor baseline and no localization hints (372) Baseline 0% 0% 0 0% Optimal 50.04% 77.96% 0.58 56.47% Bug reports with poor baseline and localization hints (427) Baseline 0% 0% 0 0% Optimal 80.70% 93.37% 0.85 86.19% Method Hit@1 Hit@10 MRR MAP All bug reports (2,320) Baseline 31.98% 66.50% 0.43 42.69% Optimal 87.41% 95.74% 0.90 90.00%
  • 14. RQ3: Non-Optimal vs. Optimal Search Queries 14 • Query dataset: 13,914 search queries. • 4,893 optimal search queries • 5,164 non-optimal queries • QE: Rank of the first true positive. • Optimal: QE = 1, Non-optimal: QE > 10 • 31 query characteristics from literature • Query classification using Random Forest algorithm.
  • 15. 15 RQ3: Query Classification & Feature Importance Query Precision Recall Optimal 84.60% 79.80% Non-optimal 81.90% 86.20% Frequency, Entropy, Location, POS
  • 16. RQ3: Optimal vs. Non-Optimal – Regression Analysis 16 Significant
  • 17. Actionable Insights • Frequency: Optimal keywords are less frequent than non-optimal ones in a bug report. • Entropy: Optimal keywords are less ambiguous than non-optimal ones. • Location: Optimal keywords are more likely to be found in the body section of a bug report. • POS: Optimal keywords are more likely to be nouns than the non-optimal ones. 17
  • 18. RQ3: Query Improvements using Actionable Insights 18 Bug reports with poor baseline and no localization hints (372) Method Hit@1 Hit@10 MRR MAP Baseline 0% 0% 0 0% Baseline + Insights 1.28% 16.09% 0.04 4.50% STRICT 1.56% 16.89% 0.05 5.12% Insights + STRICT 2.66% 15.18% 0.05 5.09% Bug reports with poor baseline and localization hints (427) Baseline 0% 0% 0 0% Baseline + Insights 6.79% 34.29% 0.14 13.79% STRICT 4.91% 25.77% 0.11 10.66% STRICT + Insights 6.80% 32.83% 0.13 13.39%
  • 19. Take-Away Messages • 34% of bug reports lead to very poor search queries (RQ1) • Even state-of-the-art approaches are not sufficient to detect appropriate keywords from them (RQ1) • Genetic algorithm (GA) shows that optimal keywords exist in those bug reports (RQ2) • Optimal search queries can achieve up to 50%--80% Hit@1 and 56%--86% MAP. 19
  • 20. • Optimal keywords are different than non- optimal keywords (RQ3) • Four aspects: frequency, term entropy, term location, and part of speech. • Insights lead to 27%--34% higher Hit@10. • ML: How to automatically predict the appropriate keywords from a bug report? • GA: How to automatically determine the fitness of a candidate search query? 20 Take-Away Messages
  • 21. Thank you! Questions? 21 Masud Rahman, PhD masud.rahman@dal.ca @masud2336
  • 22. RQ2: Query Improvement & Worsening Ratios 22