This document proposes approaches to improve the process of debugging programming errors and exceptions. It summarizes existing ad-hoc approaches and their limitations. The proposed approaches leverage context from the integrated development environment to provide context-aware web search, query recommendation, content suggestion, code examples, and exception handling support. The approaches were evaluated in experiments and user studies and showed improvements over traditional search engines and existing approaches in areas like search accuracy, recall, and time to fix exceptions. The thesis contributes techniques like SurfClipse, QueryClipse, ContentSuggest, SurfExample, and ExcClipse to address different phases of the exception handling process.

1. EXPLOITING CONTEXT IN DEALING
WITH PROGRAMMING ERRORS AND
EXCEPTIONS
Mohammad Masudur Rahman
Department of Computer Science
University of Saskatchewan

2. 2
Exception triggering point
SoftwareResearchLab,UofS

3. EXCEPTION DEBUGGING
3
Not a helpful message for
understanding or solving the
exception
SoftwareResearchLab,UofS
Web
search!!

4. SOLVING EXCEPTION
(STEP I: WEB SEARCH)
4
 The browser does not know the context (i.e., details)
of the exception.
 Not much helpful ranking
 Hundreds of search results
 Forces the developer to switch back and forth between
IDE and browser.
 Trial and error in searching
SoftwareResearchLab,UofS
Switching is
often
distracting

5. SOLVING EXCEPTION
(STEP II: QUERY SELECTION)
5
Selection of traditional search query
Switching to web browser for
web search
SoftwareResearchLab,UofS
This query may not
be sufficient enough
for most of the
exceptions

6. SOLVING EXCEPTION
(STEP III: POST-SEARCH ANALYSIS)
6
• Only the most relevant section needs to be
checked to determine the relevance of a page.
• Frequent switching between IDE and web
browser for content analysis
• Manual analysis of a number of web pages is
non-trivial and time-consuming
1
2
3
4

7. SOLVING EXCEPTION
(STEP IV: HANDLING EXCEPTIONS)
7
 Only adds a generic handler (i.e.,
printStackTrace()) for the exception.
 Not much helpful for effective handling
of the exception

8. OBSERVATIONS ON TRADITIONAL/AD-HOC
APPROACH FOR EXCEPTION SOLVING
 Step I: Web search may not be much effective or
reliable
 Step II: Support for search query formulation is
not enough
 Step III: Support for post-search analysis is
insufficient
 Step IV: Support for exception handling is not
enough
8
SoftwareResearchLab,UofS

9. DEALING WITH ERRORS & EXCEPTIONS:
PROPOSED APPROACH
9

10. CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013,
CSMR/WCRE 2014)
(2) QueryClipse
(ICSME 2014)
(3) ContentSuggest
(4) SurfExample
(SCAM 2014)
(5) ExcClipse
10
Web search
Query formulation
Post-search analysis
Exception handling
User study

11. TECHNICAL DETAILS & PROGRAMMING
CONTEXT OF AN EXCEPTION
11
Programming context (i.e.,
context code)
Error message
Stack trace
1
2
3
SoftwareResearchLab,UofS

12. SURFCLIPSE: IDE-BASED CONTEXT-AWARE
META SEARCH ENGINE
12

13. MOTIVATIONAL EXPERIMENT
13
 75 programming exceptions (details later)
 Individual engine can provide solutions for at most
58 exceptions & each has some unique results.
 Combination of content & context is always better
than content only
Search Query Common for
All
Google
Unique
Yahoo Unique Bing
Unique
Content Only 32 09 16 18
Content and
Context
47 09 11 10
SoftwareResearchLab,UofS

14. THE KEY IDEA !! META SEARCH ENGINE
14
SoftwareResearchLab,UofS
Fig: Meta search engine

15. PROPOSED IDE-BASED META SEARCH MODEL
Start
Results
Web page
15
SoftwareResearchLab,UofS

16. PROPOSED IDE-BASED META SEARCH
MODEL
 Distinguished Features (5)
 IDE-Based solution
 Web search, search result and web browsing all from IDE
 No context-switching needed
 Meta search engine
 Captures data from multiple search engines
 Also applies custom ranking techniques
 Context-Aware search
 Uses stack traces information
 Uses context-code (surroundings of exception locations)
 Software As A Service (SAAS)
 Search is provided as a web service, and can be leveraged by
an IDE. http://srlabg53-2.usask.ca/wssurfclipse/ 16
SoftwareResearchLab,UofS

17. PROPOSED IDE-BASED META SEARCH
MODEL
 Two Working Modes
 Proactive Mode
 Auto-detects the occurrence of an exception
 Initiates search for exception by client itself
 Aligned with Cordeiro et al. (RSSE’ 2012) & Ponzanelli et al.
(ICSE 2013)
 Interactive Mode
 Developer starts search using context menu
 Also facilitates keyword-based search
 Aligned with traditional web search within the IDE
17
SoftwareResearchLab,UofS

18. PROPOSED METRICS & SCORES
 Content Matching Score (Scms)
 Cosine similarity based measurement
 Stack trace Matching Score (Sstm)
 Structural and lexical similarity measurement of stack
traces
 Code context Matching Score (Sccx)
 Code snippet similarity (code clones)
 StackOverflow Vote Score (Sso)
 Total votes for all posts in the SO result link
18
SoftwareResearchLab,UofS

19. PROPOSED METRICS & SCORES
 Site Traffic Rank Score (Sstr)-- Alexa and Compete
Rank of each link
 Search Engine weight (Ssew)---Relative reliability
or importance of each search engine. Experiments
with 75 programming queries against the search
engines.
 Heuristic weights of the metrics are determined
through controlled experiments.
19
SoftwareResearchLab,UofS

20. EXPERIMENT OVERVIEW
21
SoftwareResearchLab,UofS
75 exceptions Eclipse plug-in &
Java development
Gold set solutions Peers

21. RESULTS ON DIFFERENT RANKING ASPECTS
Score Components Metrics Proactive Mode
(Top 30)
Interactive Mode
(Top 30)
Content MP
TEF
R
0.0371
56 (75)
74.66%
0.0481
65 (75)
86.66%
Content +Context MP
TEF
R
0.0376
55 (75)
73.33%
0.0514
66 (75)
88.00%
Content + Context +
Popularity
MP
TEF
R
0.0381
56 (75)
74.66%
0.0519
66 (75)
88.00%
Content +Context +
Popularity
+Confidence
MP
TEF
R
0.0380
56 (75)
74.66%
0.0538
68 (75)
90.66%
[ MP = Mean Precision, R = Recall,
TEF= Total Exceptions Fixed]
22
SoftwareResearchLab,UofS

22. COMPARISON WITH EXISTING APPROACHES
Recommender Metrics Top 10 Top 20 Top 30
Cordeiro et al.
(only stack traces)
MP
TEF
R
0.0202
15 (75)
20.00%
0.0128
18 (75)
24.00%
0.0085
18 (75)
24.00%
Proposed Method
(Proactive Mode)
MP
TEF
R
0.0886
51 (75)
68.00%
0.0529
55 (75)
73.33%
0.0380
56 (75)
74.66%
Ponzanelli et al.
(only context-code)
MP
TEF
R
0.0243
7 (37)
18.92%
0.0135
7 (37)
18.92%
0.0099
7 (37)
18.92%
Proposed Method
(Proactive Mode)
MP
TEF
R
0.1000
30 (37)
81.08%
0.0621
32 (37)
86.48%
0.0450
32 (37)
86.48%
[ MP = Mean Precision, R = Recall,
TEF= Total Exceptions Fixed]
23
SoftwareResearchLab,UofS

23. COMPARISON WITH TRADITIONAL SEARCH
ENGINES
Search Engine Metrics Top 10 Top 20 Top 30
Google MP
TEF
R
0.1571
57 (75)
76.00%
0.0864
57 (75)
76.00%
0.0580
57 (75)
76.00%
Bing MP
TEF
R
0.1013
55 (75)
73.33%
0.0533
58 (75)
77.33%
0.0364
58 (75)
77.33%
Yahoo! MP
TEF
R
0.0986
54 (75)
72.00%
0.0539
57 (75)
76.00%
0.0369
57 (75)
76.00%
StackOverflow
Search
MP
TEF
R
0.0226
14 (75)
18.66%
0.0140
17 (75)
22.66%
0.0097
17 (75)
22.66%
Proposed Method
(Interactive mode)
MP
TEF
R
0.1229
59 (75)
78.66%
0.0736
64 (75)
85.33%
0.0538
68 (75)
90.66%
24
SoftwareResearchLab,UofS

24. CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013,
CSMR/WCRE 2014)
(2) QueryClipse
(ICSME 2014)
(3) ContentSuggest
(4) SurfExample
(SCAM 2014)
(5) ExcClipse
25
Web search
Query formulation
Post-search analysis
Exception handling
User study

25. QUERYCLIPSE: CONTEXT-AWARE SEARCH
QUERY RECOMMENDER
26

26. MOTIVATING EXAMPLE
27
SoftwareResearchLab,UofS
Stack traceContext code
QueryClipse
2
1
Recommended search queries

27. PROPOSED CONTEXT-AWARE QUERY
RECOMMENDATION APPROACH
 Distinguishing Features (4)
 Context-Aware Query
 Exploits both stack trace and context code
 Extract search keywords carefully and systematically
 Ranked List for Queries
 Ranked list based on keyword importance in queries
 Automatic suggestion through auto-completion
 Custom Search Query
 Stack trace graph based on implied relationships
 Keyword importance based on network connectivity
 Query Length Customization
 Number of query keywords customizable
 Search-friendly & easily applicable for any search engines 28
SoftwareResearchLab,UofS

28. CONTENTSUGGEST: CONTEXT-AWARE PAGE
CONTENT RECOMMENDER
29

29. MOTIVATING EXAMPLE
30
1
2
3
 Only the most relevant page section displayed
 Less information overhead, less effort required
 No need to browse the page for relevance checking
SoftwareResearchLab,UofS

30. PROPOSED CONTEXT-AWARE PAGE CONTENT
SUGGESTION APPROACH
 Distinguishing Features (3)
 Relevant section(s) suggestion
 Analyzes both quality and relevance of the content
 Exploits stack trace & context code for relevance checking
 Partial automation in post-search analysis
 Less content, less overhead
 Need to analyze less content for page relevance checking
 Displayed content more useful than meta description
 Noise-free version of the page
 Removes advertisements, irrelevant widgets and so on
 Applies link-based heuristics
 Returns a noise-free version of the web page
31
SoftwareResearchLab,UofS

31. CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013,
CSMR/WCRE 2014)
(2) QueryClipse
(ICSME 2014)
(3) ContentSuggest
(4) SurfExample
(SCAM 2014)
(5) ExcClipse
32
Web search
Query formulation
Post-search analysis
Exception handling
User study

32. SURFEXAMPLE: CONTEXT-AWARE CODE EXAMPLE
RECOMMENDER FOR EXCEPTION HANDLING
33

33. MOTIVATING EXAMPLE
34
SoftwareResearchLab,UofS
Context code
Recommended code example
SurfExample

34. PROPOSED CONTEXT-AWARE CODE EXAMPLE
RECOMMENDER FOR EXCEPTION HANDLING
 Distinguishing Features (3)
 Graph-based structural relevance
 Static relationship and data dependency graph
 Graph structure matching
 Handler quality Paradigm
 Novel idea to ensure quality of exception handlers
 Based on readability, amount & quality of the handler actions
 Seamless integration of dataset
 Exploits GitHub API for data collection
 Hundreds of popular and mature open source projects from
Eclipse, Apache and others
35
SoftwareResearchLab,UofS

35. CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013,
CSMR/WCRE 2014)
(2) QueryClipse
(ICSME 2014)
(3) ContentSuggest
(4) SurfExample
(SCAM 2014)
(5) ExcClipse
36
Web search
Query formulation
Post-search analysis
Exception handling
User study

36. EXCCLIPSE: COMPARATIVE ANALYSIS
BETWEEN PROPOSED APPROACHES AND
TRADITIONAL ONES
37

37. USER STUDY OVERVIEW
38
SoftwareResearchLab,UofS
Six participants Four exceptions
Four tasks

38. USER STUDY OVERVIEW
39
SoftwareResearchLab,UofS
Questionnaire Observation checklist
Training Execution Evaluation

39. EVALUATION FEATURES
41
SoftwareResearchLab,UofS
Tool Feature Functionality Notation
Support for query formulation Web search F1
Accuracy & effectiveness of results Web search F2
Post-search content analysis Web search F3
Support for query formulation Code search F4
Relevance & accuracy of results Code search F5
Usability Overall F6
Efficiency Overall F7
Visualization support Web search F8
Visualization support Code search F9

40. FEATURE-WISE RATINGS (WEB SEARCH)
42
SoftwareResearchLab,UofS

41. FEATURE-WISE RATINGS (CODE SEARCH)
43
SoftwareResearchLab,UofS

42. FEATURE-WISE RATINGS
(NON-FUNCTIONAL)
44
SoftwareResearchLab,UofS

43. OVERALL RATINGS
45
SoftwareResearchLab,UofS

44. THREATS TO VALIDITY
 SurfClipse--Search engines constantly evolving,
same results may not be produced at later time.
 QueryClipse-- Long query generation due to
lengthy error message in the stack trace.
 ContentSuggest-- Changes in the look and feel of
the page due to removal of <style> and <script>
tags
 SurfExample-- Subjective bias in gold set
development
 ExcClipse--Limited number of participants
46
SoftwareResearchLab,UofS

45. CONCLUSION
47

46. CONCLUDING REMARKS
 SurfClipse – context-aware meta search engine
 IDE-based complete web search solution
 Outperforms two relevant existing approaches
 More recall than three search engines with precision
comparable to Google, the best performing engine.
 QueryClipse– context-aware query recommender
 More effective than traditional queries and queries by
existing approaches
 Highly applicable in terms of pyramid score
48
SoftwareResearchLab,UofS

47. CONCLUDING REMARKS
 ContentSuggest– context-aware page content
recommender
 Exploits exception details and recommends relevant
section(s) from the page
 Less information, less overhead for relevance check
 Great potential for problem solving
 SurfExample– code example recommender for
exception handling
 Graph-based structural relevance matching
 Handler Quality paradigm
 Outperforms four existing approaches in all metrics
49
SoftwareResearchLab,UofS

48. FUTURE WORK
 SurfClipse – DOM-based element extraction &
topic modeling
 QueryClipse– Semantic and customized query
recommendation
 ContentSuggest– More focused content
recommendation (e.g., paragraph of interest) for
problem solving
 SurfExample– More directed support (e.g.,
applicability of an example) for exception handling
50
SoftwareResearchLab,UofS

49. THANK YOU!!
51
SoftwareResearchLab,UofS

50. REFERENCES
[1] J. Cordeiro, B. Antunes, and P. Gomes. Context-based Recommendation to Support Problem Solving in
Software Development. In Proc. RSSE, pages 85 –89, June 2012.
[2] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: Stack Overflow in the IDE. In Proc. ICSE, pages 1295–
1298, 2013
[3] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two Studies of Opportunistic
Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proc. SIGCHI, pages 1589–1598,
2009.
[4] F. Sun, D. Song, and L. Liao. DOM Based Content Extraction via Text Density. In Proc. SIGIR, pages 245–254,
2011.
[5] T. Gottron. Content Code Blurring: A New Approach to Content Extraction. In Proc. DEXA, pages 29–33,
2008.
[6] S. Bajracharya, J. Ossher, and C. Lopes. Sourcerer: An Internet-Scale Software Repository. In Proc. SUITE,
pages 1–4, 2009
[7] E. A. Barbosa, A. Garcia, and M. Mezini. Heuristic Strategies for Recommendation of Exception Handling
Code. In Proc. SBES, pages 171–180, 2012
[8] R. Holmes and G. C. Murphy. Using Structural Context to Recommend Source Code Examples. In Proc.
ICSE, pages 117–125, 2005
[9] W. Takuya and H. Masuhara. A Spontaneous Code Recommendation Tool Based on Associative Search. In
Proc. SUITE, pages 17–20, 2011.
[10] M. M. Rahman, S. Yeasmin, and C. K. Roy. Towards a Context-Aware IDEBased Meta Search Engine for
Recommendation about Programming Errors and Exceptions. In Proc. CSMR-WCRE, pages 194–203, 2014
[11] M. M. Rahman and C.K. Roy. On the Use of Context in Recommending Exception Handling Code
Examples. In Proc. SCAM, 10 pp., 2014 (to appear)
[12] M. M. Rahman and C.K. Roy. SurfClipse: Context-Aware Meta Search in the IDE. In Proc. ICSME, 4 pp., 2014
(to appear)
[13] Slide 13, Meta Search Engine, http://en.wikipedia.org/wiki/Metasearch_engine 52
SoftwareResearchLab,UofS

51. APPENDICES
53

52. MAPPING BETWEEN PROBLEM SOLVING
STEPS & PROPOSED APPROACHES
54
Step II: Associated with
QueryClipse
Step I: Associated with SurfClipse
Step III: Associated with
ContentSuggest
Step IV: Associated with
SurfExample
Proposed
Approach

53. THREATS TO VALIDITY (SURFCLIPSE)
 Search not real time yet, generally takes about 20-
25 seconds per search. Multithreading used,
extensive parallel processing needed.
 Search engines constantly evolving, same results
may not be produced at later time.
 Experimented with common exceptions, which are
widely discussed and available in the web.
55
SoftwareResearchLab,UofS

54. TRADITIONAL SEARCH QUERIES
 Popular queries by search engines may not be
relevant all the time
 Preparing a suitable query is non-trivial
 Trial and error approach in query formulation
 Ad-hoc (e.g., error message) queries may not
reflect the context of the exception
56
SoftwareResearchLab,UofS

55. PROPOSED CONTEXT-AWARE QUERY
RECOMMENDATION APPROACH
57
SoftwareResearchLab,UofS

56. PROPOSED METRICS (3)
 Trace Token Rank (TTR)
 Trace graph developed based on implied relationship
 Calculated using Graph-based term-weighting
 An adaptation from Google’s PageRank algorithm
 Degree of Interest (DOI)
 Heuristic proximity of token to the exception location
 Associated with call references in the stack trace
 Trace Token Frequency (TTF)
 Frequency of trace token in context code
 Associated with method call and object instantiation
58
SoftwareResearchLab,UofS

57. EXPERIMENT OVERVIEW
 50 exceptions, their technical details and context
code segments collected from our first study.
 Recommended queries evaluated by searching with
Google, Bing and Yahoo!
 Queries compared with existing approaches
 Query ranks validated with experiments
 Applicability of query validated using a user study
 Performance metrics– precision, recall, % of
exceptions solved
59
SoftwareResearchLab,UofS

58. RESULTS ON DIFFERENT RANKING ASPECTS
60
Rank
Aspects
Metrics Google Bing Yahoo!
Top 10 Top 20 Top 10 Top 20 Top 10 Top 20
{DOI, TTR} MAPK
R
PTCS
36.36%
15.19%
48.00%
36.36%
15.19%
48.00%
49.24%
27.84%
70.00%
49.24%
30.68%
76.00%
51.70%
34.09%
76.00%
51.61%
35.23%
78.00%
{TTR, TTF} MAPK
R
PTCS
38.23%
15.34%
46.00%
38.23%
15.34%
46.00%
50.18%
29.55%
70.00%
50.09%
31.25%
74.00%
45.46%
30.68%
68.00%
44.60%
32.39%
70.00%
{DOI, TTF} MAPK
R
PTCS
37.26%
17.61%
50.00%
37.26%
17.61%
50.00%
49.53%
27.84%
72.00%
48.23%
30.11%
74.00%
53.49%
30.68%
78.00%
51.35%
32.95%
78.00%
{DOI, TTR,
TTF}
MAPK
R
PTCS
34.06%
13.64%
42.00%
34.06%
13.64%
42.00%
51.85%
27.84%
72.00%
50.44%
31.25%
76.00%
55.31%
31.82%
76.00%
53.40%
35.23%
80.00%
[ MP = Mean Average Precision at K, R = Recall,
PTCS= % of Exceptions Solved]

59. COMPARISON WITH EXISTING APPROACHES
61
SoftwareResearchLab,UofS
Approach Metrics Google Bing Yahoo!
Top 10 Top 20 Top 10 Top 20 Top 10 Top 20
Traditional
(Only error
message)
MAPK
R
PTCS
38.97%
19.88%
52.00%
38.97%
19.88%
52.00%
44.11%
24.43%
58.00%
43.82%
26.14%
60.00%
43.18%
25.00%
56.00%
43.18
25.00%
56.00%
Cordeiro et
al.
MAPK
R
PTCS
21.33%
10.80%
36.00%
21.17%
11.93%
38.00%
19.22%
11.93%
34.00%
19.22%
13.07%
36.00%
15.94%
10.80%
32.00%
16.60%
13.06%
40.00%
Ponzanelli et
al.
MAPK
R
PTCS
14.36%
9.09%
24.00%
14.36%
9.09%
24.00%
30.27%
12.50%
38.00%
29.98%
13.07%
38.00%
28.12%
12.50%
38.00%
28.12%
12.50%
38.00%
Proposed
approach
MAPK
R
PTCS
34.06%
13.64%
42.00%
34.06%
13.64%
42.00%
51.85%
27.84%
72.00%
50.44%
31.25%
76.00%
55.31%
31.82%
76.00%
53.40%
35.23%
80.00%
[ MP = Mean Average Precision at K, R = Recall,
PTCS= % of Exceptions Solved]

60. FINDINGS FROM USER STUDY
62
SoftwareResearchLab,UofS
Query No. 1 2 3 4 5 APS MAPS
PS (Rank I) 0.75 0.89 1.00 1.00 1.00 0.93
PS (Rank II) 0.67 0.72 0.93 1.00 0.63 0.79 0.84
PS (Rank III) 0.67 0.72 1.00 0.93 0.63 0.79
[ PS = Pyramid Score, APS = Average Pyramid Score,
MAPS= Mean Average Pyramid Score]

61. THREATS TO VALIDITY
 Long query generation due to lengthy error
message in the stack trace.
 Less user-friendly query due to complex program
tokens– class name, method name in the trace
information.
63
SoftwareResearchLab,UofS

62. PROPOSED CONTEXT-AWARE PAGE CONTENT
SUGGESTION APPROACH
64
SoftwareResearchLab,UofS

63. PROPOSED METRICS & SCORES (3)
 Content Density (CTD)
 Text Density
 Density of any textual content within a tag
 Link Density
 Density of link-based content (i.e., <a>, <input>)
 Code Density
 Density of code related content (i.e., <code>, <pre>)
 Content Relevance (CTR)
 Text Relevance
 Relevance of any textual content within a tag
 Code Relevance
 Relevance of code related content within a tag
 Content Score (CTS)
 Combines both Content Density and Content Relevance 65
SoftwareResearchLab,UofS

64. EXPERIMENT OVERVIEW
 500 web pages,150 exceptions and their details
(i.e., stack trace, context code) as dataset
 40% of the pages from StackOverflow Q & A site
 Evaluated against manually prepared gold sets
 Evaluated for both relevant and noise-free content
recommendation
 Compared with four existing approaches
 Performance metrics– precision, recall, F1-
measure
66
SoftwareResearchLab,UofS

65. RESULT ON DIFFERENT ASPECTS OF PAGE
CONTENT (RELEVANT CONTENT SUGGESTION)
67
SoftwareResearchLab,UofS
Content
Aspect
Metrics SO Pages Non-SO Pages All Pages
Content
Density
(CTD)
MP
MR
MF
50.91%
91.74%
62.32%
49.50%
75.71%
53.76%
50.07%
82.18%
57.22%
Content
Relevance
(CTR)
MP
MR
MF
86.63%
52.17%
61.07%
69.17%
57.66%
55.88%
76.23%
55.44%
57.98%
{CTD, CTR} MP
MR
MF
89.91%
74.90%
80.07%
74.12%
80.76%
73.91%
80.50%
78.39%
76.40%
[ MP = Mean Precision, MR = Mean Recall,
MF= Mean F1-measure]

66. COMPARISON WITH EXISTING APPROACHES
68
SoftwareResearchLab,UofS
Approach Metrics SO Pages Non-SO Pages All Pages
Sun et al. MP
MR
MF
80.61%
86.41%
83.14%
78.70%
75.67%
75.48%
79.49%
80.14%
78.67%
ACCB [50] MP
MR
MF
90.65%
77.32%
83.07%
93.07%
79.98%
84.64%
92.06%
78.87%
83.99%
Proposed
(noise-free content
extraction)
MP
MR
MF
91.27%
89.27%
90.00%
88.90%
86.20%
85.76%
89.88%
87.48%
87.53%
Sun et al. (adapted) MP
MR
MF
52.63%
86.49%
62.57%
38.89%
41.84%
34.49%
44.44%
59.88%
45.84%
Proposed
(relevant section
suggestion)
MP
MR
MF
89.91%
74.90%
80.07%
74.12%
80.76%
73.91%
80.50%
78.39%
76.40%
[ MP = Mean Precision, MR = Mean Recall,
MF= Mean F1-measure]

67. THREATS TO VALIDITY
 Changes in the look and feel of the page due to
removal of <style> and <script> tags
 Lack of a fully-fledged user study
69
SoftwareResearchLab,UofS

68. PROPOSED METRICS (3)
 Structural Relevance (Rstr)
 API Object Match (AOM)
 Field Access Match (FAM)
 Method Invocation Match (MIM)
 Data Dependency Match (DDM)
 Lexical Relevance (Rlex)
 Cosine Similarity
 Code Clone Measure
 Quality of Exception Handler (Qehc)
 Readability (RA)
 Average Handler Actions (AHA)
 Handler to Code Ratio (HCR)
70
SoftwareResearchLab,UofS

69. EXPERIMENT OVERVIEW
 65 exceptions and context code segments
 4400 code examples from 700+ repositories of
Eclipse, Apache, Facebook and Twitter
 Evaluated against manually prepared gold set
 Compared with four existing approaches
 Performance metrics– precision, recall, # and % of
exceptions handled
71
SoftwareResearchLab,UofS

70. RESULT ON DIFFERENT RANKING ASPECTS
72
SoftwareResearchLab,UofS
Ranking Aspects Metrics Top 5 Top 10 Top 15
Structure (Rstr) MAPK
R
PEH
38.07%
50.00%
69.23%
33.84%
61.93%
75.38%
32.64%
69.32%
81.54%
Content (Rlex) MAPK
R
PEH
35.00%
45.45%
66.15%
33.85%
63.63%
75.38%
33.08%
70.45%
81.54%
{Structure (Rstr),
Content (Rlex)}
MAPK
R
PEH
43.08%
51.70%
69.23%
38.69%
66.48%
75.38%
37.33%
74.43%
81.54%
{Structure (Rstr),
Content (Rlex),
Quality (Qehc)}
MAPK
R
PEH
41.92%
57.39%
73.85%
39.92%
68.75%
81.54%
38.64%
76.70%
86.15%
[ MAPK = Mean Average Precision at K, R = Recall,
PEH= % of exceptions handled]

71. COMPARISON WITH EXISTING APPROACHES
73
SoftwareResearchLab,UofS
Recommender Metrics Top 5 Top 10 Top 15
Barbosa et al. MAPK
R
PEH
16.15%
16.47%
27.69%
14.69%
25.57%
38.46%
13.72%
31.25%
44.62%
Holmes & Murphy MAPK
R
PEH
4.62%
11.36%
24.62%
2.31%
21.59%
38.46%
2.31%
27.84%
47.69%
Takuya & Masuhara MAPK
R
PEH
21.54%
15.34%
33.85%
20.51%
27.27%
47.69%
19.74%
30.68%
47.69%
Bajracharya et al. MAPK
R
PEH
8.46%
10.80%
18.46%
7.95%
15.91%
27.69%
6.41%
19.32%
30.77%
Proposed approach MAPK
R
PEH
41.92%
57.39%
73.85%
39.92%
68.75%
81.54%
38.64%
76.70%
86.15%
[ MAPK = Mean Average Precision at K, R = Recall,
PEH= % of exceptions handled]

72. THREATS TO VALIDITY
 Subjective bias in gold set development
 Limited size of dynamic corpus for recommendation
 Limited number of exceptions for experiments
74
SoftwareResearchLab,UofS

73. TRADITIONAL WEB SEARCH
 No ties between IDE and web
browsers
 Does not consider problem-
context
 Environment-switching is
distracting & time-consuming
 Often not much productive
(trial & error approach)
75
SoftwareResearchLab,UofS

74. TRADITIONAL SUPPORT FOR POST-SEARCH
CONTENT ANALYSIS
 Keyword highlighting in title based on search
query
 Very limited meta description using keywords or
phrases.
 Page URL, Page visit statistics
 Little clues for actual content in the web page
 Forces one to browse the page
76
SoftwareResearchLab,UofS

75. TRADITIONAL CODE SEARCH ENGINES
 No ties with the IDE
 Returns hundreds of result
pages
 Keyword matching search
 No support for search query
formulation
77
SoftwareResearchLab,UofS

76. PROPOSED CONTEXT-AWARE CODE EXAMPLE
RECOMMENDER FOR EXCEPTION HANDLING
78
SoftwareResearchLab,UofS

77. USER STUDY GROUPING
79
P1,P2,P3,P4,P5,P6
A (P1,P4,P5) B (P2,P3,P6)
EC1, EC2,
EC3, EC4
S1
(EC1,
EC2)
S2
(EC3,
EC4)
T/S1 + E/S2 T/S2 + E/S1
I (P1,P5) II (P4) I (P2) II (P3,P6)
TE ET TE ET