This document proposes approaches to improve the process of debugging programming errors and exceptions. It summarizes existing ad-hoc approaches and their limitations. The proposed approaches leverage context from the integrated development environment to provide context-aware web search, query recommendation, content suggestion, code examples, and exception handling support. The approaches were evaluated in experiments and user studies and showed improvements over traditional search engines and existing approaches in areas like search accuracy, recall, and time to fix exceptions. The thesis contributes techniques like SurfClipse, QueryClipse, ContentSuggest, SurfExample, and ExcClipse to address different phases of the exception handling process.
SurfClipse-- An IDE based context-aware Meta Search EngineMasud Rahman
Despite various debugging supports of the existing IDEs for programming errors and exceptions, software developers often look at web for working solutions or any up-to-date information. Traditional web search does not consider thecontext of the problems that they search solutions for, and thus it often does not help much in problem solving. In this paper, we propose a context-aware meta search tool, SurfClipse, that analyzes an encountered exception andits context in the IDE, and recommends not only suitable search queries but also relevant web pages for the exception (and its context). The tool collects results from three popular search engines and a programming Q & A site against the exception in the IDE, refines the results for relevance against the context of the exception, and then ranks them before recommendation. It provides two working modes--interactive and proactive to meet the versatile needs of the developers, and one can browse the result pages using a customized embedded browser provided by the tool.
Keith J. Jones, Ph.D. - MALGAZER: AN AUTOMATED MALWARE CLASSIFIER WITH RUNNIN...Keith Jones, PhD
The document describes research on developing an automated malware classifier called Malgazer. Key aspects include:
- Optimizing the running window entropy (RWE) algorithm to reduce computation time. RWE is used as a feature for training machine learning models.
- Collecting and exploring a dataset of over 25,000 malware samples classified across 6 functional categories from VirusTotal.
- Training over 200 machine learning experiments using RWE and GIST features, and algorithms including decision trees, random forests, and neural networks. The best models achieved over 94% accuracy.
- Developing Malgazer as an RWE-based malware classifier and web application to classify new samples, achieving performance superior to prior literature
This document discusses techniques for testing advanced driver assistance systems (ADAS) through physics-based simulation. It faces challenges due to the large, complex, and multidimensional test input space as well as the computational expense of simulation. The document proposes using a genetic algorithm guided by decision trees to more efficiently search for critical test cases. Classification trees are built to partition the input space into homogeneous regions in order to better guide the selection and generation of test inputs toward more critical areas.
Practical operability techniques for teams - Matthew Skelton - Conflux - Cont...Matthew Skelton
In this talk, we explore five practical, tried-and-tested, real world techniques for improving operability with many kinds of software systems, including cloud, Serverless, on-premise, and IoT.
Logging as a live diagnostics vector with sparse Event IDs
Operational checklists and ‘Run Book dialogue sheets’ as a discovery mechanism for teams
Endpoint healthchecks as a way to assess runtime dependencies and complexity
Correlation IDs beyond simple HTTP calls
Lightweight ‘User Personas’ as drivers for operational dashboards
Based on our work in many industry sectors, we will share our experience of helping teams to improve the operability of their software systems through
Required audience experience
Some experience of building web-scale systems or industrial IoT/embedded systems would be helpful.
Objective of the talk
We will share our experience of helping teams to improve the operability of their software systems. Attendees will learn some practical operability approaches and how teams can expand their understanding and awareness of operability through these simple, team-friendly techniques.
From a talk given at Continuous Lifecycle London 2018: https://continuouslifecycle.london/sessions/practical-team-focused-operability-techniques-for-distributed-systems/
Test Automation Framework Online Training by QuontraSolutionsQuontra Solutions
Quontra Solutions certified instructors with more then 12 years of industry experience deploy a proven method of Selenium implementation and training the staff for the corporate by providing an overview, demonstrating each major system capability, and giving participants step-by-step lessons.
our training courses in Selenium / QTP / Quality Center, automation technologies in USA,UK. Selenium Training for Automation testing courses will be offered by both our On-Site and Remote experts. Call us for the Next batches starting dates, Learn Selenium with highly qualified and expert professional Selenium consultants in United states , serving people in the the QA industry in United States..
Selenium Training Courses offered:
Contact us for the course details.
Quontra Solutions
Email: info@quontrasolutions.co.uk
Call Now :
US: +1 404-900-9988.
UK: (20)3734 1498.
Designing a Test Automation Framework By Quontra solutionsQUONTRASOLUTIONS
Quontra Solutions provides Selenium Testing online training by Real time IT experts. Selenium is having good demand in the market. Our Selenium online training Instructors are very much experienced and highly qualified and dedicated.
Our Selenium online training program is job oriented. After completion of Selenium training with us you should be able to work on any kind of project. After completion of Selenium online training our dedicated team will be supporting you.
Get 10 % OFF when you register for this program.
Please call us for demo on Selenium. Quontra Solutions is the best Selenium online training Institute in United States.
Highlights in our training:
* Very in depth course material with real time scenarios.
* We are providing class with highly qualified trainer.
* We will provide class and demo session at student flexible timings.
* In training case studies and real time scenarios covered.
* Each topic covers real time solutions.
* We will give every recorded session for play later.
* We are giving placement support by multiple consultancies in USA, UK etc.
* We will give full support while attending the interviews and contact us after completion of the course.
Pre-requisites:
• Manual Testing Skills
• Basic Programming Skills, OOPS Concepts
• Knowledge on Test Automation Principles and practices
The document discusses various ABAP performance analysis tools including Code Inspector (SCI), Performance Trace (ST05), and Runtime Analysis (SE30).
Code Inspector performs static code analysis to identify potential performance and security issues. Performance Trace allows recording and analysis of database access, locking activities, and remote calls. Runtime Analysis provides insight into time spent in database vs ABAP code and analysis of internal table operations.
These tools each have benefits and limitations but together provide a comprehensive set of options for evaluating SQL statements, code execution paths, and identifying optimization opportunities at both the static code and runtime levels. Regular usage of these tools should be part of the development process.
SurfClipse-- An IDE based context-aware Meta Search EngineMasud Rahman
Despite various debugging supports of the existing IDEs for programming errors and exceptions, software developers often look at web for working solutions or any up-to-date information. Traditional web search does not consider thecontext of the problems that they search solutions for, and thus it often does not help much in problem solving. In this paper, we propose a context-aware meta search tool, SurfClipse, that analyzes an encountered exception andits context in the IDE, and recommends not only suitable search queries but also relevant web pages for the exception (and its context). The tool collects results from three popular search engines and a programming Q & A site against the exception in the IDE, refines the results for relevance against the context of the exception, and then ranks them before recommendation. It provides two working modes--interactive and proactive to meet the versatile needs of the developers, and one can browse the result pages using a customized embedded browser provided by the tool.
Keith J. Jones, Ph.D. - MALGAZER: AN AUTOMATED MALWARE CLASSIFIER WITH RUNNIN...Keith Jones, PhD
The document describes research on developing an automated malware classifier called Malgazer. Key aspects include:
- Optimizing the running window entropy (RWE) algorithm to reduce computation time. RWE is used as a feature for training machine learning models.
- Collecting and exploring a dataset of over 25,000 malware samples classified across 6 functional categories from VirusTotal.
- Training over 200 machine learning experiments using RWE and GIST features, and algorithms including decision trees, random forests, and neural networks. The best models achieved over 94% accuracy.
- Developing Malgazer as an RWE-based malware classifier and web application to classify new samples, achieving performance superior to prior literature
This document discusses techniques for testing advanced driver assistance systems (ADAS) through physics-based simulation. It faces challenges due to the large, complex, and multidimensional test input space as well as the computational expense of simulation. The document proposes using a genetic algorithm guided by decision trees to more efficiently search for critical test cases. Classification trees are built to partition the input space into homogeneous regions in order to better guide the selection and generation of test inputs toward more critical areas.
Practical operability techniques for teams - Matthew Skelton - Conflux - Cont...Matthew Skelton
In this talk, we explore five practical, tried-and-tested, real world techniques for improving operability with many kinds of software systems, including cloud, Serverless, on-premise, and IoT.
Logging as a live diagnostics vector with sparse Event IDs
Operational checklists and ‘Run Book dialogue sheets’ as a discovery mechanism for teams
Endpoint healthchecks as a way to assess runtime dependencies and complexity
Correlation IDs beyond simple HTTP calls
Lightweight ‘User Personas’ as drivers for operational dashboards
Based on our work in many industry sectors, we will share our experience of helping teams to improve the operability of their software systems through
Required audience experience
Some experience of building web-scale systems or industrial IoT/embedded systems would be helpful.
Objective of the talk
We will share our experience of helping teams to improve the operability of their software systems. Attendees will learn some practical operability approaches and how teams can expand their understanding and awareness of operability through these simple, team-friendly techniques.
From a talk given at Continuous Lifecycle London 2018: https://continuouslifecycle.london/sessions/practical-team-focused-operability-techniques-for-distributed-systems/
Test Automation Framework Online Training by QuontraSolutionsQuontra Solutions
Quontra Solutions certified instructors with more then 12 years of industry experience deploy a proven method of Selenium implementation and training the staff for the corporate by providing an overview, demonstrating each major system capability, and giving participants step-by-step lessons.
our training courses in Selenium / QTP / Quality Center, automation technologies in USA,UK. Selenium Training for Automation testing courses will be offered by both our On-Site and Remote experts. Call us for the Next batches starting dates, Learn Selenium with highly qualified and expert professional Selenium consultants in United states , serving people in the the QA industry in United States..
Selenium Training Courses offered:
Contact us for the course details.
Quontra Solutions
Email: info@quontrasolutions.co.uk
Call Now :
US: +1 404-900-9988.
UK: (20)3734 1498.
Designing a Test Automation Framework By Quontra solutionsQUONTRASOLUTIONS
Quontra Solutions provides Selenium Testing online training by Real time IT experts. Selenium is having good demand in the market. Our Selenium online training Instructors are very much experienced and highly qualified and dedicated.
Our Selenium online training program is job oriented. After completion of Selenium training with us you should be able to work on any kind of project. After completion of Selenium online training our dedicated team will be supporting you.
Get 10 % OFF when you register for this program.
Please call us for demo on Selenium. Quontra Solutions is the best Selenium online training Institute in United States.
Highlights in our training:
* Very in depth course material with real time scenarios.
* We are providing class with highly qualified trainer.
* We will provide class and demo session at student flexible timings.
* In training case studies and real time scenarios covered.
* Each topic covers real time solutions.
* We will give every recorded session for play later.
* We are giving placement support by multiple consultancies in USA, UK etc.
* We will give full support while attending the interviews and contact us after completion of the course.
Pre-requisites:
• Manual Testing Skills
• Basic Programming Skills, OOPS Concepts
• Knowledge on Test Automation Principles and practices
The document discusses various ABAP performance analysis tools including Code Inspector (SCI), Performance Trace (ST05), and Runtime Analysis (SE30).
Code Inspector performs static code analysis to identify potential performance and security issues. Performance Trace allows recording and analysis of database access, locking activities, and remote calls. Runtime Analysis provides insight into time spent in database vs ABAP code and analysis of internal table operations.
These tools each have benefits and limitations but together provide a comprehensive set of options for evaluating SQL statements, code execution paths, and identifying optimization opportunities at both the static code and runtime levels. Regular usage of these tools should be part of the development process.
Toward Automatic Generation of SPARQL result set VisualizationsMarcello Leida
The problem of representing RDF data using charts, dashboards, maps and so on has become pressing, in particular to prove the value of the Semantic Web to enhance the analysis of business data.
State of the art solutions focus on mapping query results to a specific chart type or view and then manually writing the procedure that creates the final dashboard.
Whenever a different visualization model is required, the mapping process needs to be repeated.
Furthermore, even if several different mappings are pre-defined and available to the user there may be several situations which cannot be predicted because they depend on the specific analysis the user wants to carry out.
In this paper we propose a semi-automatic approach that generates various charts from SPARQL queries over data represented as RDF graphs, we introduce and describe the generic approach and present a use case scenario in the context of service monitoring.
This document provides best practices for Apex development at Salesforce, covering principles, processes, development conventions, testing, and more. It recommends tracking requirements, using source control, automating deployments, writing efficient code, handling errors properly, writing tests with assertions and different scenarios, and avoiding governor limits. The goal is to produce code that is well-structured, commented when needed, performs efficiently with queries and loops, has high test coverage, uses test data properly, and can be tested at scale.
Using Compass to Diagnose Performance Problems MongoDB
Speaker: Brian Blevins, Technical Services Engineer, MongoDB
Level: 200 (Intermediate)
Track: Performance
Since the performance of your application drives engagement and revenue, it can make or break the success of your organization. You can use the Compass graphical client from MongoDB to visualize your database schema, collect information on optimization opportunities and make database changes to improve performance. In this talk, we will briefly introduce Compass and then delve into the features supporting database performance optimization. The talk will combine instruction on the use of Compass with recommendations for performance best practices. We will also review the detection and resolution of slow queries and excessive network utilization. After attending the talk, audience members will have a better understanding of the capabilities of Compass, including how those capabilities can be used to find and correct performance bottlenecks in MongoDB databases. This session is designed for those with limited MongoDB experience. Attendees should have a basic understanding of MongoDB’s schema design, the server/database/collection layout, and how their application accesses and uses the MongoDB database.
What You Will Learn:
- Identify excessive network utilization, adjust queries appropriately and use Compass to confirm results.
- Understand how the Compass graphical client can help you improve performance in your MongoDB deployment.
- Use Compass real time statistics to identify slow queries and recognize when a query is a good candidate for adding an index.
Using Compass to Diagnose Performance Problems in Your ClusterMongoDB
Using Compass to Diagnose Performance Problems in Your Cluster
Speaker: Brian Blevins, Technical Services Engineer, MongoDB
Date/Time: June 20, 1:50 PM
Track: Performance
Since the performance of your application drives engagement and revenue, it can make or break the success of your organization. You can use the Compass graphical client from MongoDB to visualize your database schema, collect information on optimization opportunities and make database changes to improve performance. In this talk, we will briefly introduce Compass and then delve into the features supporting database performance optimization. The talk will combine instruction on the use of Compass with recommendations for performance best practices. We will also review the detection and resolution of slow queries and excessive network utilization. After attending the talk, audience members will have a better understanding of the capabilities of Compass, including how those capabilities can be used to find and correct performance bottlenecks in MongoDB databases. This session is designed for those with limited MongoDB experience. Attendees should have a basic understanding of MongoDB’s schema design, the server/database/collection layout, and how their application accesses and uses the MongoDB database.
What You Will Learn:
- Identify excessive network utilization, adjust queries appropriately and use Compass to confirm results.
- Understand how the Compass graphical client can help you improve performance in your MongoDB deployment.
- Use Compass real time statistics to identify slow queries and recognize when a query is a good candidate for adding an index.
ASP.NET MVC Best Practices malisa ncubeMalisa Ncube
This document discusses best practices for ASP.NET MVC. It begins with an introduction to ASP.NET MVC and its components: Model, View, Controller. It then outlines 9 best practices, including isolating layers, using the Post-Redirect-Get pattern, securing from forgery, making the application testable and extensible, writing clean code, using strongly typed views, optimizing JSON endpoints and performance. It also briefly discusses new features in ASP.NET MVC 4 and tips for productivity and globalization.
Share Point Development With Unit TestingJeremy Thake
This document discusses unit testing for SharePoint development. It outlines three main goals of unit testing: to run quickly, run on every developer machine with minimal configuration, and not test Microsoft code. It describes where unit testing can be done in SharePoint, such as web parts and event receivers. The document discusses challenges of unit testing SharePoint code due to tightly coupled code and sealed classes. It provides examples of approaches like mocking, wrappers, repositories, and MVP to enable unit testing. Finally, it emphasizes that unit testing helps with refactoring, catching errors from other developers, and reducing environment dependencies, but does not replace writing high quality code.
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand
1. The document discusses using artificial intelligence techniques like machine learning and natural language processing to help automate software testing. It focuses on applying these techniques to testing advanced driver assistance systems.
2. A key challenge in software testing is scalability as the input spaces and code bases grow large and complex. Effective automation is needed to address this challenge. The document describes several industrial research projects applying AI to help automate testing of advanced driver assistance systems.
3. One project aims to develop an automated testing technique for emergency braking systems in cars using a physics-based simulation. The goal is to efficiently explore complex test scenarios and identify critical situations like failures to avoid collisions.
CORRECT is a code reviewer recommendation tool that:
- Recommends appropriate code reviewers automatically by mining developers' contributions across projects
- Provides recommendation rationales that fit within developers' workflows
- Achieves over 90% accuracy in recommending reviewers based on library and technology experience
- Outperforms an existing technique (RevFinder) with 92.15% top-5 accuracy, 85.93% mean precision and 81.39% mean recall
- Performs similarly on open source projects with 85.20% top-5 accuracy, demonstrating effectiveness for public and private codebases
This document describes techniques for scaling up superoptimization to synthesize larger and more optimal code fragments. It introduces LENS, an enumerative search algorithm that uses pruning strategies like bidirectional search and selective abstraction refinement to solve more benchmarks faster than existing approaches. It also describes a context-aware window decomposition that allows LENS to optimize larger code by decomposing the problem. Finally, it discusses a cooperative superoptimizer that combines LENS with stochastic and symbolic searches to leverage their respective strengths and optimize more benchmarks.
SurfClipse-- An IDE based context-aware Meta Search Engine (ERA Track)Masud Rahman
Traditional web search forces the developers to leave their working environments and look for solutions in the web browsers. It often does not consider the context of their programming problems. The context-switching between the web browser and the working environment is time-consuming and distracting, and the keyword-based traditional search often does not help much in problem solving. In this paper, we propose an Eclipse IDE-based web search solution that collects the data from three web search APIs– Google, Yahoo, Bing and a programming Q & A site– StackOverflow. It then provides search results within IDE taking not only the content of the selected error into account but also the problem context, popularity and search engine recommendation of the result links. Experiments with 25 runtime errors and exceptions show that the proposed ap- proach outperforms the keyword-based search approaches with a recommendation accuracy of 96%. We also validate the results with a user study involving five prospective participants where we get a result agreement of 64.28%. While the preliminary results are promising, the approach needs to be further validated with more errors and exceptions followed by a user study with more participants to establish itself as a complete IDE-based web search solution.
ABAP Test Cockpit in action with Doctor ZedGe and abap2xlsxAlessandro Lavazzi
This document discusses various tools for analyzing ABAP code statically, including the ABAP Test Cockpit, Code Inspector, and a new tool called Doctor ZedGe. It provides information on installing and using these tools to analyze code complexity, violations of coding standards, and the presence of hard-coded business values. Doctor ZedGe enhances the ABAP Test Cockpit by generating reports in Excel focused on different user personas like developers, team leaders, and project managers.
The document provides an overview of software testing techniques and strategies. It discusses unit testing, integration testing, validation testing, system testing, and debugging. The key points covered include:
- Unit testing involves testing individual software modules or components in isolation from the rest of the system. This includes testing module interfaces, data structures, boundary conditions, and error handling paths.
- Integration testing combines software components into clusters or builds to test their interactions before full system integration. Approaches include top-down and bottom-up integration.
- Validation testing verifies that the software meets the intended requirements and customer expectations defined in validation criteria.
- System testing evaluates the fully integrated software system, including recovery, security, stress,
Scalable Software Testing and Verification of Non-Functional Properties throu...Lionel Briand
This document discusses scalable software testing and verification of non-functional properties through heuristic search and optimization. It describes several projects with industry partners that use metaheuristic search techniques like hill climbing and genetic algorithms to generate test cases for non-functional properties of complex, configurable software systems. The techniques address issues of scalability and practicality for engineers by using dimensionality reduction, surrogate modeling, and dynamically adjusting the search strategy in different regions of the input space. The results provided worst-case scenarios more effectively than random testing alone.
An IDE-Based Context-Aware Meta Search EngineMasud Rahman
The document proposes an IDE-based meta search engine called SurfClipse that searches multiple search engines for solutions to programming exceptions. It considers the exception context, content relevance, link popularity, and search engine confidence to score and rank results. An experiment tested SurfClipse on 25 exceptions and found correct solutions for 24 cases. A user study with 5 exceptions had 64.28% agreement between recommended and participant solutions. Future work includes improving the metrics, conducting more experiments and user studies, and developing a distributed version of SurfClipse.
The previous research has focused on quick and efficient generation of wrappers; the
development of tools for wrapper maintenance has received less attention. This is an important research
problem because Web sources often change in ways that prevent the wrappers from extracting data
correctly. Present an efficient algorithm that extract unstructured data to structural data from web. The
wrapper verification system detects when a wrapper is not extracting correct data, usually because the
Web source has changed its format. The Verification framework automatically recovers data using
Dimension Reduction Techniques from changes in the Web source by identifying data on Web pages.
After apply wrapped data to One Class Classification in Numerical features for avoid classification
problem. Finally, the result data apply in Top-K query for provide best rank based on probabilities
scores. Wrapper verification system relies on one-class classification techniques to beat previous
weaknesses to identify the problem by analysing both the signature and the classifier output. If there are
sufficient mislabelled slots, a technique to find a pattern could be explored.
Applications of Machine Learning and Metaheuristic Search to Security TestingLionel Briand
This document discusses testing web application firewalls (WAFs) for SQL injection (SQLi) vulnerabilities. It states that the testing goal is to generate test cases that result in executable malicious SQL statements that can bypass the WAF. It also notes that WAF filter rules often need customization to avoid false positives and protect against new attacks, but that customization is error-prone due to complex rules, time/resource constraints, and a lack of automated tools.
Rachid Kherrazi presented on using ASDSpec, a tool that combines model-based testing and model-driven engineering, to generate test cases from ASD interface models. The document discussed model-based testing benefits like increased productivity, reliability, and maintainability. It described the ASDSpec workflow of reusing existing ASD interface models to automatically generate Spec Explorer test models and test cases. A case study applying ASDSpec to a container terminal system was presented, showing improved productivity over traditional testing approaches while still having limitations to fully test interactions, data, and external code.
The Magic Of Application Lifecycle Management In Vs PublicDavid Solivan
The document discusses challenges with software development projects and how tools from Microsoft can help address these challenges. It notes that most projects fail or are over budget and challenges include poor requirements gathering and testing. However, tools like Visual Studio and Team Foundation Server that integrate requirements, work tracking, source control, testing and other functions can help make successful projects more possible by facilitating team collaboration. The document outlines features of these tools and how they aim to make application lifecycle management a routine part of development.
The Automation Firehose: Be Strategic and Tactical by Thomas HaverQA or the Highway
The document discusses strategies for automating software testing. It emphasizes taking a risk-based approach to determine what to automate based on factors like frequency of use, complexity, and legal risk. The document provides recommendations for test automation best practices like treating automated test code like development code, using frameworks and tools to standardize coding practices, and prioritizing unit and integration testing over UI testing. It also discusses challenges that can arise with test automation like flaky tests, long test execution times, and keeping automation in sync with changing software. Metrics for measuring the effectiveness of test automation are presented, like test coverage, defect findings and trends, and time savings.
RAISE Lab at Dalhousie University
aims to develop tools and technologies for intelligent automation in software engineering. An overview is presented by Dr. Masud Rahman, Assistant Professor, Faculty of Computer Science, Dalhousie University, Canada.
The Forgotten Role of Search Queries in IR-based Bug Localization: An Empiric...Masud Rahman
The document summarizes a study on improving search queries for bug localization using natural language text from bug reports. The study evaluated different keyword selection techniques, generated optimal search queries using a genetic algorithm, and compared optimal versus non-optimal queries. Key findings include: 1) Current approaches failed to identify keywords for 34% of bug reports, 2) A genetic algorithm produced optimal queries that achieved up to 80% higher performance than baselines, and 3) Optimal queries differed in using less frequent, less ambiguous, noun-heavy keywords located in bug report bodies.
More Related Content
Similar to Exploiting Context in Dealing with Programming Errors and Exceptions
Toward Automatic Generation of SPARQL result set VisualizationsMarcello Leida
The problem of representing RDF data using charts, dashboards, maps and so on has become pressing, in particular to prove the value of the Semantic Web to enhance the analysis of business data.
State of the art solutions focus on mapping query results to a specific chart type or view and then manually writing the procedure that creates the final dashboard.
Whenever a different visualization model is required, the mapping process needs to be repeated.
Furthermore, even if several different mappings are pre-defined and available to the user there may be several situations which cannot be predicted because they depend on the specific analysis the user wants to carry out.
In this paper we propose a semi-automatic approach that generates various charts from SPARQL queries over data represented as RDF graphs, we introduce and describe the generic approach and present a use case scenario in the context of service monitoring.
This document provides best practices for Apex development at Salesforce, covering principles, processes, development conventions, testing, and more. It recommends tracking requirements, using source control, automating deployments, writing efficient code, handling errors properly, writing tests with assertions and different scenarios, and avoiding governor limits. The goal is to produce code that is well-structured, commented when needed, performs efficiently with queries and loops, has high test coverage, uses test data properly, and can be tested at scale.
Using Compass to Diagnose Performance Problems MongoDB
Speaker: Brian Blevins, Technical Services Engineer, MongoDB
Level: 200 (Intermediate)
Track: Performance
Since the performance of your application drives engagement and revenue, it can make or break the success of your organization. You can use the Compass graphical client from MongoDB to visualize your database schema, collect information on optimization opportunities and make database changes to improve performance. In this talk, we will briefly introduce Compass and then delve into the features supporting database performance optimization. The talk will combine instruction on the use of Compass with recommendations for performance best practices. We will also review the detection and resolution of slow queries and excessive network utilization. After attending the talk, audience members will have a better understanding of the capabilities of Compass, including how those capabilities can be used to find and correct performance bottlenecks in MongoDB databases. This session is designed for those with limited MongoDB experience. Attendees should have a basic understanding of MongoDB’s schema design, the server/database/collection layout, and how their application accesses and uses the MongoDB database.
What You Will Learn:
- Identify excessive network utilization, adjust queries appropriately and use Compass to confirm results.
- Understand how the Compass graphical client can help you improve performance in your MongoDB deployment.
- Use Compass real time statistics to identify slow queries and recognize when a query is a good candidate for adding an index.
Using Compass to Diagnose Performance Problems in Your ClusterMongoDB
Using Compass to Diagnose Performance Problems in Your Cluster
Speaker: Brian Blevins, Technical Services Engineer, MongoDB
Date/Time: June 20, 1:50 PM
Track: Performance
Since the performance of your application drives engagement and revenue, it can make or break the success of your organization. You can use the Compass graphical client from MongoDB to visualize your database schema, collect information on optimization opportunities and make database changes to improve performance. In this talk, we will briefly introduce Compass and then delve into the features supporting database performance optimization. The talk will combine instruction on the use of Compass with recommendations for performance best practices. We will also review the detection and resolution of slow queries and excessive network utilization. After attending the talk, audience members will have a better understanding of the capabilities of Compass, including how those capabilities can be used to find and correct performance bottlenecks in MongoDB databases. This session is designed for those with limited MongoDB experience. Attendees should have a basic understanding of MongoDB’s schema design, the server/database/collection layout, and how their application accesses and uses the MongoDB database.
What You Will Learn:
- Identify excessive network utilization, adjust queries appropriately and use Compass to confirm results.
- Understand how the Compass graphical client can help you improve performance in your MongoDB deployment.
- Use Compass real time statistics to identify slow queries and recognize when a query is a good candidate for adding an index.
ASP.NET MVC Best Practices malisa ncubeMalisa Ncube
This document discusses best practices for ASP.NET MVC. It begins with an introduction to ASP.NET MVC and its components: Model, View, Controller. It then outlines 9 best practices, including isolating layers, using the Post-Redirect-Get pattern, securing from forgery, making the application testable and extensible, writing clean code, using strongly typed views, optimizing JSON endpoints and performance. It also briefly discusses new features in ASP.NET MVC 4 and tips for productivity and globalization.
Share Point Development With Unit TestingJeremy Thake
This document discusses unit testing for SharePoint development. It outlines three main goals of unit testing: to run quickly, run on every developer machine with minimal configuration, and not test Microsoft code. It describes where unit testing can be done in SharePoint, such as web parts and event receivers. The document discusses challenges of unit testing SharePoint code due to tightly coupled code and sealed classes. It provides examples of approaches like mocking, wrappers, repositories, and MVP to enable unit testing. Finally, it emphasizes that unit testing helps with refactoring, catching errors from other developers, and reducing environment dependencies, but does not replace writing high quality code.
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand
1. The document discusses using artificial intelligence techniques like machine learning and natural language processing to help automate software testing. It focuses on applying these techniques to testing advanced driver assistance systems.
2. A key challenge in software testing is scalability as the input spaces and code bases grow large and complex. Effective automation is needed to address this challenge. The document describes several industrial research projects applying AI to help automate testing of advanced driver assistance systems.
3. One project aims to develop an automated testing technique for emergency braking systems in cars using a physics-based simulation. The goal is to efficiently explore complex test scenarios and identify critical situations like failures to avoid collisions.
CORRECT is a code reviewer recommendation tool that:
- Recommends appropriate code reviewers automatically by mining developers' contributions across projects
- Provides recommendation rationales that fit within developers' workflows
- Achieves over 90% accuracy in recommending reviewers based on library and technology experience
- Outperforms an existing technique (RevFinder) with 92.15% top-5 accuracy, 85.93% mean precision and 81.39% mean recall
- Performs similarly on open source projects with 85.20% top-5 accuracy, demonstrating effectiveness for public and private codebases
This document describes techniques for scaling up superoptimization to synthesize larger and more optimal code fragments. It introduces LENS, an enumerative search algorithm that uses pruning strategies like bidirectional search and selective abstraction refinement to solve more benchmarks faster than existing approaches. It also describes a context-aware window decomposition that allows LENS to optimize larger code by decomposing the problem. Finally, it discusses a cooperative superoptimizer that combines LENS with stochastic and symbolic searches to leverage their respective strengths and optimize more benchmarks.
SurfClipse-- An IDE based context-aware Meta Search Engine (ERA Track)Masud Rahman
Traditional web search forces the developers to leave their working environments and look for solutions in the web browsers. It often does not consider the context of their programming problems. The context-switching between the web browser and the working environment is time-consuming and distracting, and the keyword-based traditional search often does not help much in problem solving. In this paper, we propose an Eclipse IDE-based web search solution that collects the data from three web search APIs– Google, Yahoo, Bing and a programming Q & A site– StackOverflow. It then provides search results within IDE taking not only the content of the selected error into account but also the problem context, popularity and search engine recommendation of the result links. Experiments with 25 runtime errors and exceptions show that the proposed ap- proach outperforms the keyword-based search approaches with a recommendation accuracy of 96%. We also validate the results with a user study involving five prospective participants where we get a result agreement of 64.28%. While the preliminary results are promising, the approach needs to be further validated with more errors and exceptions followed by a user study with more participants to establish itself as a complete IDE-based web search solution.
ABAP Test Cockpit in action with Doctor ZedGe and abap2xlsxAlessandro Lavazzi
This document discusses various tools for analyzing ABAP code statically, including the ABAP Test Cockpit, Code Inspector, and a new tool called Doctor ZedGe. It provides information on installing and using these tools to analyze code complexity, violations of coding standards, and the presence of hard-coded business values. Doctor ZedGe enhances the ABAP Test Cockpit by generating reports in Excel focused on different user personas like developers, team leaders, and project managers.
The document provides an overview of software testing techniques and strategies. It discusses unit testing, integration testing, validation testing, system testing, and debugging. The key points covered include:
- Unit testing involves testing individual software modules or components in isolation from the rest of the system. This includes testing module interfaces, data structures, boundary conditions, and error handling paths.
- Integration testing combines software components into clusters or builds to test their interactions before full system integration. Approaches include top-down and bottom-up integration.
- Validation testing verifies that the software meets the intended requirements and customer expectations defined in validation criteria.
- System testing evaluates the fully integrated software system, including recovery, security, stress,
Scalable Software Testing and Verification of Non-Functional Properties throu...Lionel Briand
This document discusses scalable software testing and verification of non-functional properties through heuristic search and optimization. It describes several projects with industry partners that use metaheuristic search techniques like hill climbing and genetic algorithms to generate test cases for non-functional properties of complex, configurable software systems. The techniques address issues of scalability and practicality for engineers by using dimensionality reduction, surrogate modeling, and dynamically adjusting the search strategy in different regions of the input space. The results provided worst-case scenarios more effectively than random testing alone.
An IDE-Based Context-Aware Meta Search EngineMasud Rahman
The document proposes an IDE-based meta search engine called SurfClipse that searches multiple search engines for solutions to programming exceptions. It considers the exception context, content relevance, link popularity, and search engine confidence to score and rank results. An experiment tested SurfClipse on 25 exceptions and found correct solutions for 24 cases. A user study with 5 exceptions had 64.28% agreement between recommended and participant solutions. Future work includes improving the metrics, conducting more experiments and user studies, and developing a distributed version of SurfClipse.
The previous research has focused on quick and efficient generation of wrappers; the
development of tools for wrapper maintenance has received less attention. This is an important research
problem because Web sources often change in ways that prevent the wrappers from extracting data
correctly. Present an efficient algorithm that extract unstructured data to structural data from web. The
wrapper verification system detects when a wrapper is not extracting correct data, usually because the
Web source has changed its format. The Verification framework automatically recovers data using
Dimension Reduction Techniques from changes in the Web source by identifying data on Web pages.
After apply wrapped data to One Class Classification in Numerical features for avoid classification
problem. Finally, the result data apply in Top-K query for provide best rank based on probabilities
scores. Wrapper verification system relies on one-class classification techniques to beat previous
weaknesses to identify the problem by analysing both the signature and the classifier output. If there are
sufficient mislabelled slots, a technique to find a pattern could be explored.
Applications of Machine Learning and Metaheuristic Search to Security TestingLionel Briand
This document discusses testing web application firewalls (WAFs) for SQL injection (SQLi) vulnerabilities. It states that the testing goal is to generate test cases that result in executable malicious SQL statements that can bypass the WAF. It also notes that WAF filter rules often need customization to avoid false positives and protect against new attacks, but that customization is error-prone due to complex rules, time/resource constraints, and a lack of automated tools.
Rachid Kherrazi presented on using ASDSpec, a tool that combines model-based testing and model-driven engineering, to generate test cases from ASD interface models. The document discussed model-based testing benefits like increased productivity, reliability, and maintainability. It described the ASDSpec workflow of reusing existing ASD interface models to automatically generate Spec Explorer test models and test cases. A case study applying ASDSpec to a container terminal system was presented, showing improved productivity over traditional testing approaches while still having limitations to fully test interactions, data, and external code.
The Magic Of Application Lifecycle Management In Vs PublicDavid Solivan
The document discusses challenges with software development projects and how tools from Microsoft can help address these challenges. It notes that most projects fail or are over budget and challenges include poor requirements gathering and testing. However, tools like Visual Studio and Team Foundation Server that integrate requirements, work tracking, source control, testing and other functions can help make successful projects more possible by facilitating team collaboration. The document outlines features of these tools and how they aim to make application lifecycle management a routine part of development.
The Automation Firehose: Be Strategic and Tactical by Thomas HaverQA or the Highway
The document discusses strategies for automating software testing. It emphasizes taking a risk-based approach to determine what to automate based on factors like frequency of use, complexity, and legal risk. The document provides recommendations for test automation best practices like treating automated test code like development code, using frameworks and tools to standardize coding practices, and prioritizing unit and integration testing over UI testing. It also discusses challenges that can arise with test automation like flaky tests, long test execution times, and keeping automation in sync with changing software. Metrics for measuring the effectiveness of test automation are presented, like test coverage, defect findings and trends, and time savings.
Similar to Exploiting Context in Dealing with Programming Errors and Exceptions (20)
RAISE Lab at Dalhousie University
aims to develop tools and technologies for intelligent automation in software engineering. An overview is presented by Dr. Masud Rahman, Assistant Professor, Faculty of Computer Science, Dalhousie University, Canada.
The Forgotten Role of Search Queries in IR-based Bug Localization: An Empiric...Masud Rahman
The document summarizes a study on improving search queries for bug localization using natural language text from bug reports. The study evaluated different keyword selection techniques, generated optimal search queries using a genetic algorithm, and compared optimal versus non-optimal queries. Key findings include: 1) Current approaches failed to identify keywords for 34% of bug reports, 2) A genetic algorithm produced optimal queries that achieved up to 80% higher performance than baselines, and 3) Optimal queries differed in using less frequent, less ambiguous, noun-heavy keywords located in bug report bodies.
The document outlines Masud Rahman's PhD thesis proposal on supporting source code search with context-aware, analytics-driven query reformulation. The proposal discusses three research questions: 1) evaluating term weighting techniques for keyword selection from source code and bug reports, 2) incorporating bug report quality for local code search, and 3) leveraging crowd knowledge and data analytics to deliver query keywords. The contribution summary highlights techniques for term dependence, quality-aware bug localization, and using crowd knowledge and large data analytics.
PhD Comprehensive exam of Masud RahmanMasud Rahman
This document presents a systematic literature review of automated query reformulations for source code search. It discusses seven research questions explored in the review, including the methods, algorithms, data sources, evaluation metrics, challenges, publication trends, and comparisons between local and internet-scale code search queries. The review analyzed over 50 primary studies identified through a multi-database search and filtering process. Key findings include the predominant use of term weighting, query expansion and reduction techniques, evaluations based on standard information retrieval metrics, and various challenges like vocabulary mismatch that remain unsolved. Opportunities for future work are also identified, such as leveraging bug reports for keyword selection and using semantic representations to address vocabulary issues.
This document summarizes a talk given by Masud Rahman, a PhD candidate at the University of Saskatchewan. The talk focused on Rahman's PhD thesis research, which aims to improve code search by generating context-aware, analytics-driven queries through effective reformulation. The talk outlined three research questions around improving keyword selection, incorporating bug report quality, and using crowd knowledge and data analytics. It provided an overview of Rahman's PhD thesis and publications addressing the research questions. Evaluation methods for the proposed approaches were also discussed.
This document summarizes a study on improving bug localization through considering the quality of bug reports and reformulating bug report queries. The study analyzes 5,500 bug reports from eight projects and finds that existing bug localization techniques perform poorly when bug reports lack useful information or contain excessive stack traces. Preliminary findings suggest context-aware query reformulation may help address these limitations by improving the quality and relevance of the queries used.
This document summarizes research into the impact of continuous integration (CI) on code reviews. The researchers studied over 500,000 pull requests and builds from open source projects to answer three questions: 1) Whether build status influences code review participation, 2) If frequent builds improve review quality, and 3) Predicting if a build will trigger new reviews. They found that passed builds were more associated with new reviews and comments. Projects with frequent builds received more review comments that remained steady over time, unlike less frequently built projects. Their machine learning model could predict if a build would trigger new reviews with up to 64% accuracy.
This document presents research on predicting the usefulness of code review comments using textual features and developer experience. The researchers analyzed 1,482 code review comments, manually classified as useful or non-useful. They found non-useful comments had more stop words and less code elements, while useful comments had higher conceptual similarity to changed code. More experienced reviewers provided more useful comments. The researchers also built a Random Forest model that predicts comment usefulness with 66% accuracy, outperforming baselines. Their work provides the first automated approach to assess code review comment usefulness.
The document describes a technique called STRICT that uses TextRank and POSRank algorithms to identify important terms from a software change task description to generate an effective initial search query. An experiment on 1,939 change tasks from 8 open source projects found that STRICT improved the query effectiveness in 57.84% of cases compared to baseline queries like title alone. STRICT also showed better retrieval performance based on metrics like mean average precision and mean recall compared to state-of-the-art techniques. The approach validates the use of graph-based ranking algorithms to address the challenge of generating relevant initial search queries from natural language change task descriptions.
The document analyzes why some questions on Stack Overflow remain unresolved and explores whether machine learning can predict which questions will be unresolved. It finds that unresolved questions have higher topic entropy, meaning they are less specific. Owners of unresolved questions reject answers more often, have lower reputation, and are less active on Stack Overflow. Models using features like topic entropy, answer rejection ratio, and owner reputation achieved up to 78% accuracy at predicting unresolved questions. The study aims to help improve question quality on Stack Overflow.
This document analyzes data from over 78,000 pull requests on GitHub to understand why pull request failure rates are high. It finds that 57.05% of pull requests failed, most often due to issues with recursion/refactoring, database queries, arrays/functions. Programming languages like Java, JavaScript and Ruby saw more failed pull requests on average than PHP. Projects in IDE and framework domains had the most pull request activity. Older projects, projects with more forks/developers, and projects where developers had 20-50 months of experience saw the highest numbers of pull requests and failures. The study aims to help understand and address common reasons for pull request failures on GitHub.
The document describes a technique called CodeInsight that mines insightful code comments from crowdsourced knowledge on Stack Overflow. An exploratory study of Stack Overflow discussions found that around 22% of comments discuss tips, bugs, or warnings related to code examples. CodeInsight uses heuristics like popularity, relevance, comment rank, sentiment, and word count to retrieve these insightful comments for a given code segment. An empirical evaluation showed the technique could recall over 80% of relevant comments on average. A user study with professional developers found that 80% of the comments recommended by CodeInsight were accurate and useful.
This document proposes using TextRank to identify initial search terms for software change tasks. It adapts TextRank, originally used for keyword extraction and text summarization, to build a graph of terms from development artifacts and rank them. An evaluation on 349 change tasks from two systems identifies search terms, which outperform an existing approach in solving more tasks with higher precision and recall. The approach recommends initial search queries to help developers find relevant code artifacts when performing change tasks.
This document discusses a method called BRACK for identifying bug-prone API methods using crowdsourced knowledge from Stack Overflow. BRACK ranks API method invocations based on two heuristics: API Context-Susceptibility (ACS) which estimates how context can impact an invocation, and API Error-Associativity (AEA) which calculates the co-occurrence of an invocation in defective and corrected code segments. An evaluation of BRACK on 8 open source systems found that it achieved a top-3 accuracy of 75.93% in identifying bug-prone invocations, and that ACS was more effective than AEA. The evaluation also showed BRACK had no significant bias towards system size or API package and performed comparably
The document presents research on RACK, a tool that uses crowdsourced knowledge from Stack Overflow to reformulate natural language code search queries into relevant API names. The researchers analyzed Stack Overflow data to find that answers frequently refer to APIs by name and cover a high percentage of core APIs. They also found question titles contain terms relevant to real code search queries. RACK maps query terms to API names using this data, then searches GitHub code examples. An evaluation showed RACK returns relevant examples with 79% top-10 accuracy, outperforming existing techniques.
RACK is an approach that automatically recommends relevant APIs for code search queries using crowdsourced knowledge from Stack Overflow questions, answers, and titles. An exploratory study found that accepted Stack Overflow answers frequently mention API names and cover a large percentage of standard APIs. Question titles often contain keywords relevant to code search. RACK constructs an API-token mapping database from Stack Overflow and ranks APIs for a given query based on heuristics measuring keyword-API co-occurrence and coherence. An evaluation found RACK achieved around 79% top-10 accuracy and outperformed existing techniques, demonstrating the potential of leveraging crowdsourced technical knowledge for API recommendation.
QUICKAR is a technique for automatically reformulating code search queries using crowdsourced knowledge from Stack Overflow. It constructs an adjacency list database of terms from Stack Overflow question titles. For an initial search query, it identifies reformulation candidates by comparing the query terms to terms in the adjacency list database and project source code. In experiments, QUICKAR significantly outperformed a baseline technique, improving over 50% of queries while worsening less than 50%, by leveraging vocabulary from Stack Overflow to address mismatches between developer queries and code.
The document describes CORRECT, a technique for recommending code reviewers for pull requests on GitHub based on developers' cross-project and technology experience. It evaluates CORRECT using codebases from both a commercial software company and open source projects. The results show that CORRECT achieves over 90% accuracy in recommending reviewers, outperforming a baseline technique. Library and technology experience are also found to be good proxies for code review skills. CORRECT performs equally well on both private and public codebases without bias toward any development framework.
TOPIC OF DISCUSSION: CENTRIFUGATION SLIDESHARE.pptxshubhijain836
Centrifugation is a powerful technique used in laboratories to separate components of a heterogeneous mixture based on their density. This process utilizes centrifugal force to rapidly spin samples, causing denser particles to migrate outward more quickly than lighter ones. As a result, distinct layers form within the sample tube, allowing for easy isolation and purification of target substances.
Candidate young stellar objects in the S-cluster: Kinematic analysis of a sub...Sérgio Sacani
Context. The observation of several L-band emission sources in the S cluster has led to a rich discussion of their nature. However, a definitive answer to the classification of the dusty objects requires an explanation for the detection of compact Doppler-shifted Brγ emission. The ionized hydrogen in combination with the observation of mid-infrared L-band continuum emission suggests that most of these sources are embedded in a dusty envelope. These embedded sources are part of the S-cluster, and their relationship to the S-stars is still under debate. To date, the question of the origin of these two populations has been vague, although all explanations favor migration processes for the individual cluster members. Aims. This work revisits the S-cluster and its dusty members orbiting the supermassive black hole SgrA* on bound Keplerian orbits from a kinematic perspective. The aim is to explore the Keplerian parameters for patterns that might imply a nonrandom distribution of the sample. Additionally, various analytical aspects are considered to address the nature of the dusty sources. Methods. Based on the photometric analysis, we estimated the individual H−K and K−L colors for the source sample and compared the results to known cluster members. The classification revealed a noticeable contrast between the S-stars and the dusty sources. To fit the flux-density distribution, we utilized the radiative transfer code HYPERION and implemented a young stellar object Class I model. We obtained the position angle from the Keplerian fit results; additionally, we analyzed the distribution of the inclinations and the longitudes of the ascending node. Results. The colors of the dusty sources suggest a stellar nature consistent with the spectral energy distribution in the near and midinfrared domains. Furthermore, the evaporation timescales of dusty and gaseous clumps in the vicinity of SgrA* are much shorter ( 2yr) than the epochs covered by the observations (≈15yr). In addition to the strong evidence for the stellar classification of the D-sources, we also find a clear disk-like pattern following the arrangements of S-stars proposed in the literature. Furthermore, we find a global intrinsic inclination for all dusty sources of 60 ± 20◦, implying a common formation process. Conclusions. The pattern of the dusty sources manifested in the distribution of the position angles, inclinations, and longitudes of the ascending node strongly suggests two different scenarios: the main-sequence stars and the dusty stellar S-cluster sources share a common formation history or migrated with a similar formation channel in the vicinity of SgrA*. Alternatively, the gravitational influence of SgrA* in combination with a massive perturber, such as a putative intermediate mass black hole in the IRS 13 cluster, forces the dusty objects and S-stars to follow a particular orbital arrangement. Key words. stars: black holes– stars: formation– Galaxy: center– galaxies: star formation
Exploiting Context in Dealing with Programming Errors and Exceptions
1. EXPLOITING CONTEXT IN DEALING
WITH PROGRAMMING ERRORS AND
EXCEPTIONS
Mohammad Masudur Rahman
Department of Computer Science
University of Saskatchewan
3. EXCEPTION DEBUGGING
3
Not a helpful message for
understanding or solving the
exception
SoftwareResearchLab,UofS
Web
search!!
4. SOLVING EXCEPTION
(STEP I: WEB SEARCH)
4
The browser does not know the context (i.e., details)
of the exception.
Not much helpful ranking
Hundreds of search results
Forces the developer to switch back and forth between
IDE and browser.
Trial and error in searching
SoftwareResearchLab,UofS
Switching is
often
distracting
5. SOLVING EXCEPTION
(STEP II: QUERY SELECTION)
5
Selection of traditional search query
Switching to web browser for
web search
SoftwareResearchLab,UofS
This query may not
be sufficient enough
for most of the
exceptions
6. SOLVING EXCEPTION
(STEP III: POST-SEARCH ANALYSIS)
6
• Only the most relevant section needs to be
checked to determine the relevance of a page.
• Frequent switching between IDE and web
browser for content analysis
• Manual analysis of a number of web pages is
non-trivial and time-consuming
1
2
3
4
7. SOLVING EXCEPTION
(STEP IV: HANDLING EXCEPTIONS)
7
Only adds a generic handler (i.e.,
printStackTrace()) for the exception.
Not much helpful for effective handling
of the exception
8. OBSERVATIONS ON TRADITIONAL/AD-HOC
APPROACH FOR EXCEPTION SOLVING
Step I: Web search may not be much effective or
reliable
Step II: Support for search query formulation is
not enough
Step III: Support for post-search analysis is
insufficient
Step IV: Support for exception handling is not
enough
8
SoftwareResearchLab,UofS
13. MOTIVATIONAL EXPERIMENT
13
75 programming exceptions (details later)
Individual engine can provide solutions for at most
58 exceptions & each has some unique results.
Combination of content & context is always better
than content only
Search Query Common for
All
Google
Unique
Yahoo Unique Bing
Unique
Content Only 32 09 16 18
Content and
Context
47 09 11 10
SoftwareResearchLab,UofS
14. THE KEY IDEA !! META SEARCH ENGINE
14
SoftwareResearchLab,UofS
Fig: Meta search engine
16. PROPOSED IDE-BASED META SEARCH
MODEL
Distinguished Features (5)
IDE-Based solution
Web search, search result and web browsing all from IDE
No context-switching needed
Meta search engine
Captures data from multiple search engines
Also applies custom ranking techniques
Context-Aware search
Uses stack traces information
Uses context-code (surroundings of exception locations)
Software As A Service (SAAS)
Search is provided as a web service, and can be leveraged by
an IDE. http://srlabg53-2.usask.ca/wssurfclipse/ 16
SoftwareResearchLab,UofS
17. PROPOSED IDE-BASED META SEARCH
MODEL
Two Working Modes
Proactive Mode
Auto-detects the occurrence of an exception
Initiates search for exception by client itself
Aligned with Cordeiro et al. (RSSE’ 2012) & Ponzanelli et al.
(ICSE 2013)
Interactive Mode
Developer starts search using context menu
Also facilitates keyword-based search
Aligned with traditional web search within the IDE
17
SoftwareResearchLab,UofS
18. PROPOSED METRICS & SCORES
Content Matching Score (Scms)
Cosine similarity based measurement
Stack trace Matching Score (Sstm)
Structural and lexical similarity measurement of stack
traces
Code context Matching Score (Sccx)
Code snippet similarity (code clones)
StackOverflow Vote Score (Sso)
Total votes for all posts in the SO result link
18
SoftwareResearchLab,UofS
19. PROPOSED METRICS & SCORES
Site Traffic Rank Score (Sstr)-- Alexa and Compete
Rank of each link
Search Engine weight (Ssew)---Relative reliability
or importance of each search engine. Experiments
with 75 programming queries against the search
engines.
Heuristic weights of the metrics are determined
through controlled experiments.
19
SoftwareResearchLab,UofS
27. PROPOSED CONTEXT-AWARE QUERY
RECOMMENDATION APPROACH
Distinguishing Features (4)
Context-Aware Query
Exploits both stack trace and context code
Extract search keywords carefully and systematically
Ranked List for Queries
Ranked list based on keyword importance in queries
Automatic suggestion through auto-completion
Custom Search Query
Stack trace graph based on implied relationships
Keyword importance based on network connectivity
Query Length Customization
Number of query keywords customizable
Search-friendly & easily applicable for any search engines 28
SoftwareResearchLab,UofS
29. MOTIVATING EXAMPLE
30
1
2
3
Only the most relevant page section displayed
Less information overhead, less effort required
No need to browse the page for relevance checking
SoftwareResearchLab,UofS
30. PROPOSED CONTEXT-AWARE PAGE CONTENT
SUGGESTION APPROACH
Distinguishing Features (3)
Relevant section(s) suggestion
Analyzes both quality and relevance of the content
Exploits stack trace & context code for relevance checking
Partial automation in post-search analysis
Less content, less overhead
Need to analyze less content for page relevance checking
Displayed content more useful than meta description
Noise-free version of the page
Removes advertisements, irrelevant widgets and so on
Applies link-based heuristics
Returns a noise-free version of the web page
31
SoftwareResearchLab,UofS
31. CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013,
CSMR/WCRE 2014)
(2) QueryClipse
(ICSME 2014)
(3) ContentSuggest
(4) SurfExample
(SCAM 2014)
(5) ExcClipse
32
Web search
Query formulation
Post-search analysis
Exception handling
User study
34. PROPOSED CONTEXT-AWARE CODE EXAMPLE
RECOMMENDER FOR EXCEPTION HANDLING
Distinguishing Features (3)
Graph-based structural relevance
Static relationship and data dependency graph
Graph structure matching
Handler quality Paradigm
Novel idea to ensure quality of exception handlers
Based on readability, amount & quality of the handler actions
Seamless integration of dataset
Exploits GitHub API for data collection
Hundreds of popular and mature open source projects from
Eclipse, Apache and others
35
SoftwareResearchLab,UofS
35. CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013,
CSMR/WCRE 2014)
(2) QueryClipse
(ICSME 2014)
(3) ContentSuggest
(4) SurfExample
(SCAM 2014)
(5) ExcClipse
36
Web search
Query formulation
Post-search analysis
Exception handling
User study
39. EVALUATION FEATURES
41
SoftwareResearchLab,UofS
Tool Feature Functionality Notation
Support for query formulation Web search F1
Accuracy & effectiveness of results Web search F2
Post-search content analysis Web search F3
Support for query formulation Code search F4
Relevance & accuracy of results Code search F5
Usability Overall F6
Efficiency Overall F7
Visualization support Web search F8
Visualization support Code search F9
44. THREATS TO VALIDITY
SurfClipse--Search engines constantly evolving,
same results may not be produced at later time.
QueryClipse-- Long query generation due to
lengthy error message in the stack trace.
ContentSuggest-- Changes in the look and feel of
the page due to removal of <style> and <script>
tags
SurfExample-- Subjective bias in gold set
development
ExcClipse--Limited number of participants
46
SoftwareResearchLab,UofS
46. CONCLUDING REMARKS
SurfClipse – context-aware meta search engine
IDE-based complete web search solution
Outperforms two relevant existing approaches
More recall than three search engines with precision
comparable to Google, the best performing engine.
QueryClipse– context-aware query recommender
More effective than traditional queries and queries by
existing approaches
Highly applicable in terms of pyramid score
48
SoftwareResearchLab,UofS
47. CONCLUDING REMARKS
ContentSuggest– context-aware page content
recommender
Exploits exception details and recommends relevant
section(s) from the page
Less information, less overhead for relevance check
Great potential for problem solving
SurfExample– code example recommender for
exception handling
Graph-based structural relevance matching
Handler Quality paradigm
Outperforms four existing approaches in all metrics
49
SoftwareResearchLab,UofS
48. FUTURE WORK
SurfClipse – DOM-based element extraction &
topic modeling
QueryClipse– Semantic and customized query
recommendation
ContentSuggest– More focused content
recommendation (e.g., paragraph of interest) for
problem solving
SurfExample– More directed support (e.g.,
applicability of an example) for exception handling
50
SoftwareResearchLab,UofS
50. REFERENCES
[1] J. Cordeiro, B. Antunes, and P. Gomes. Context-based Recommendation to Support Problem Solving in
Software Development. In Proc. RSSE, pages 85 –89, June 2012.
[2] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: Stack Overflow in the IDE. In Proc. ICSE, pages 1295–
1298, 2013
[3] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two Studies of Opportunistic
Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proc. SIGCHI, pages 1589–1598,
2009.
[4] F. Sun, D. Song, and L. Liao. DOM Based Content Extraction via Text Density. In Proc. SIGIR, pages 245–254,
2011.
[5] T. Gottron. Content Code Blurring: A New Approach to Content Extraction. In Proc. DEXA, pages 29–33,
2008.
[6] S. Bajracharya, J. Ossher, and C. Lopes. Sourcerer: An Internet-Scale Software Repository. In Proc. SUITE,
pages 1–4, 2009
[7] E. A. Barbosa, A. Garcia, and M. Mezini. Heuristic Strategies for Recommendation of Exception Handling
Code. In Proc. SBES, pages 171–180, 2012
[8] R. Holmes and G. C. Murphy. Using Structural Context to Recommend Source Code Examples. In Proc.
ICSE, pages 117–125, 2005
[9] W. Takuya and H. Masuhara. A Spontaneous Code Recommendation Tool Based on Associative Search. In
Proc. SUITE, pages 17–20, 2011.
[10] M. M. Rahman, S. Yeasmin, and C. K. Roy. Towards a Context-Aware IDEBased Meta Search Engine for
Recommendation about Programming Errors and Exceptions. In Proc. CSMR-WCRE, pages 194–203, 2014
[11] M. M. Rahman and C.K. Roy. On the Use of Context in Recommending Exception Handling Code
Examples. In Proc. SCAM, 10 pp., 2014 (to appear)
[12] M. M. Rahman and C.K. Roy. SurfClipse: Context-Aware Meta Search in the IDE. In Proc. ICSME, 4 pp., 2014
(to appear)
[13] Slide 13, Meta Search Engine, http://en.wikipedia.org/wiki/Metasearch_engine 52
SoftwareResearchLab,UofS
52. MAPPING BETWEEN PROBLEM SOLVING
STEPS & PROPOSED APPROACHES
54
Step II: Associated with
QueryClipse
Step I: Associated with SurfClipse
Step III: Associated with
ContentSuggest
Step IV: Associated with
SurfExample
Proposed
Approach
53. THREATS TO VALIDITY (SURFCLIPSE)
Search not real time yet, generally takes about 20-
25 seconds per search. Multithreading used,
extensive parallel processing needed.
Search engines constantly evolving, same results
may not be produced at later time.
Experimented with common exceptions, which are
widely discussed and available in the web.
55
SoftwareResearchLab,UofS
54. TRADITIONAL SEARCH QUERIES
Popular queries by search engines may not be
relevant all the time
Preparing a suitable query is non-trivial
Trial and error approach in query formulation
Ad-hoc (e.g., error message) queries may not
reflect the context of the exception
56
SoftwareResearchLab,UofS
56. PROPOSED METRICS (3)
Trace Token Rank (TTR)
Trace graph developed based on implied relationship
Calculated using Graph-based term-weighting
An adaptation from Google’s PageRank algorithm
Degree of Interest (DOI)
Heuristic proximity of token to the exception location
Associated with call references in the stack trace
Trace Token Frequency (TTF)
Frequency of trace token in context code
Associated with method call and object instantiation
58
SoftwareResearchLab,UofS
57. EXPERIMENT OVERVIEW
50 exceptions, their technical details and context
code segments collected from our first study.
Recommended queries evaluated by searching with
Google, Bing and Yahoo!
Queries compared with existing approaches
Query ranks validated with experiments
Applicability of query validated using a user study
Performance metrics– precision, recall, % of
exceptions solved
59
SoftwareResearchLab,UofS
58. RESULTS ON DIFFERENT RANKING ASPECTS
60
Rank
Aspects
Metrics Google Bing Yahoo!
Top 10 Top 20 Top 10 Top 20 Top 10 Top 20
{DOI, TTR} MAPK
R
PTCS
36.36%
15.19%
48.00%
36.36%
15.19%
48.00%
49.24%
27.84%
70.00%
49.24%
30.68%
76.00%
51.70%
34.09%
76.00%
51.61%
35.23%
78.00%
{TTR, TTF} MAPK
R
PTCS
38.23%
15.34%
46.00%
38.23%
15.34%
46.00%
50.18%
29.55%
70.00%
50.09%
31.25%
74.00%
45.46%
30.68%
68.00%
44.60%
32.39%
70.00%
{DOI, TTF} MAPK
R
PTCS
37.26%
17.61%
50.00%
37.26%
17.61%
50.00%
49.53%
27.84%
72.00%
48.23%
30.11%
74.00%
53.49%
30.68%
78.00%
51.35%
32.95%
78.00%
{DOI, TTR,
TTF}
MAPK
R
PTCS
34.06%
13.64%
42.00%
34.06%
13.64%
42.00%
51.85%
27.84%
72.00%
50.44%
31.25%
76.00%
55.31%
31.82%
76.00%
53.40%
35.23%
80.00%
[ MP = Mean Average Precision at K, R = Recall,
PTCS= % of Exceptions Solved]
59. COMPARISON WITH EXISTING APPROACHES
61
SoftwareResearchLab,UofS
Approach Metrics Google Bing Yahoo!
Top 10 Top 20 Top 10 Top 20 Top 10 Top 20
Traditional
(Only error
message)
MAPK
R
PTCS
38.97%
19.88%
52.00%
38.97%
19.88%
52.00%
44.11%
24.43%
58.00%
43.82%
26.14%
60.00%
43.18%
25.00%
56.00%
43.18
25.00%
56.00%
Cordeiro et
al.
MAPK
R
PTCS
21.33%
10.80%
36.00%
21.17%
11.93%
38.00%
19.22%
11.93%
34.00%
19.22%
13.07%
36.00%
15.94%
10.80%
32.00%
16.60%
13.06%
40.00%
Ponzanelli et
al.
MAPK
R
PTCS
14.36%
9.09%
24.00%
14.36%
9.09%
24.00%
30.27%
12.50%
38.00%
29.98%
13.07%
38.00%
28.12%
12.50%
38.00%
28.12%
12.50%
38.00%
Proposed
approach
MAPK
R
PTCS
34.06%
13.64%
42.00%
34.06%
13.64%
42.00%
51.85%
27.84%
72.00%
50.44%
31.25%
76.00%
55.31%
31.82%
76.00%
53.40%
35.23%
80.00%
[ MP = Mean Average Precision at K, R = Recall,
PTCS= % of Exceptions Solved]
60. FINDINGS FROM USER STUDY
62
SoftwareResearchLab,UofS
Query No. 1 2 3 4 5 APS MAPS
PS (Rank I) 0.75 0.89 1.00 1.00 1.00 0.93
PS (Rank II) 0.67 0.72 0.93 1.00 0.63 0.79 0.84
PS (Rank III) 0.67 0.72 1.00 0.93 0.63 0.79
[ PS = Pyramid Score, APS = Average Pyramid Score,
MAPS= Mean Average Pyramid Score]
61. THREATS TO VALIDITY
Long query generation due to lengthy error
message in the stack trace.
Less user-friendly query due to complex program
tokens– class name, method name in the trace
information.
63
SoftwareResearchLab,UofS
63. PROPOSED METRICS & SCORES (3)
Content Density (CTD)
Text Density
Density of any textual content within a tag
Link Density
Density of link-based content (i.e., <a>, <input>)
Code Density
Density of code related content (i.e., <code>, <pre>)
Content Relevance (CTR)
Text Relevance
Relevance of any textual content within a tag
Code Relevance
Relevance of code related content within a tag
Content Score (CTS)
Combines both Content Density and Content Relevance 65
SoftwareResearchLab,UofS
64. EXPERIMENT OVERVIEW
500 web pages,150 exceptions and their details
(i.e., stack trace, context code) as dataset
40% of the pages from StackOverflow Q & A site
Evaluated against manually prepared gold sets
Evaluated for both relevant and noise-free content
recommendation
Compared with four existing approaches
Performance metrics– precision, recall, F1-
measure
66
SoftwareResearchLab,UofS
65. RESULT ON DIFFERENT ASPECTS OF PAGE
CONTENT (RELEVANT CONTENT SUGGESTION)
67
SoftwareResearchLab,UofS
Content
Aspect
Metrics SO Pages Non-SO Pages All Pages
Content
Density
(CTD)
MP
MR
MF
50.91%
91.74%
62.32%
49.50%
75.71%
53.76%
50.07%
82.18%
57.22%
Content
Relevance
(CTR)
MP
MR
MF
86.63%
52.17%
61.07%
69.17%
57.66%
55.88%
76.23%
55.44%
57.98%
{CTD, CTR} MP
MR
MF
89.91%
74.90%
80.07%
74.12%
80.76%
73.91%
80.50%
78.39%
76.40%
[ MP = Mean Precision, MR = Mean Recall,
MF= Mean F1-measure]
67. THREATS TO VALIDITY
Changes in the look and feel of the page due to
removal of <style> and <script> tags
Lack of a fully-fledged user study
69
SoftwareResearchLab,UofS
68. PROPOSED METRICS (3)
Structural Relevance (Rstr)
API Object Match (AOM)
Field Access Match (FAM)
Method Invocation Match (MIM)
Data Dependency Match (DDM)
Lexical Relevance (Rlex)
Cosine Similarity
Code Clone Measure
Quality of Exception Handler (Qehc)
Readability (RA)
Average Handler Actions (AHA)
Handler to Code Ratio (HCR)
70
SoftwareResearchLab,UofS
69. EXPERIMENT OVERVIEW
65 exceptions and context code segments
4400 code examples from 700+ repositories of
Eclipse, Apache, Facebook and Twitter
Evaluated against manually prepared gold set
Compared with four existing approaches
Performance metrics– precision, recall, # and % of
exceptions handled
71
SoftwareResearchLab,UofS
70. RESULT ON DIFFERENT RANKING ASPECTS
72
SoftwareResearchLab,UofS
Ranking Aspects Metrics Top 5 Top 10 Top 15
Structure (Rstr) MAPK
R
PEH
38.07%
50.00%
69.23%
33.84%
61.93%
75.38%
32.64%
69.32%
81.54%
Content (Rlex) MAPK
R
PEH
35.00%
45.45%
66.15%
33.85%
63.63%
75.38%
33.08%
70.45%
81.54%
{Structure (Rstr),
Content (Rlex)}
MAPK
R
PEH
43.08%
51.70%
69.23%
38.69%
66.48%
75.38%
37.33%
74.43%
81.54%
{Structure (Rstr),
Content (Rlex),
Quality (Qehc)}
MAPK
R
PEH
41.92%
57.39%
73.85%
39.92%
68.75%
81.54%
38.64%
76.70%
86.15%
[ MAPK = Mean Average Precision at K, R = Recall,
PEH= % of exceptions handled]
71. COMPARISON WITH EXISTING APPROACHES
73
SoftwareResearchLab,UofS
Recommender Metrics Top 5 Top 10 Top 15
Barbosa et al. MAPK
R
PEH
16.15%
16.47%
27.69%
14.69%
25.57%
38.46%
13.72%
31.25%
44.62%
Holmes & Murphy MAPK
R
PEH
4.62%
11.36%
24.62%
2.31%
21.59%
38.46%
2.31%
27.84%
47.69%
Takuya & Masuhara MAPK
R
PEH
21.54%
15.34%
33.85%
20.51%
27.27%
47.69%
19.74%
30.68%
47.69%
Bajracharya et al. MAPK
R
PEH
8.46%
10.80%
18.46%
7.95%
15.91%
27.69%
6.41%
19.32%
30.77%
Proposed approach MAPK
R
PEH
41.92%
57.39%
73.85%
39.92%
68.75%
81.54%
38.64%
76.70%
86.15%
[ MAPK = Mean Average Precision at K, R = Recall,
PEH= % of exceptions handled]
72. THREATS TO VALIDITY
Subjective bias in gold set development
Limited size of dynamic corpus for recommendation
Limited number of exceptions for experiments
74
SoftwareResearchLab,UofS
73. TRADITIONAL WEB SEARCH
No ties between IDE and web
browsers
Does not consider problem-
context
Environment-switching is
distracting & time-consuming
Often not much productive
(trial & error approach)
75
SoftwareResearchLab,UofS
74. TRADITIONAL SUPPORT FOR POST-SEARCH
CONTENT ANALYSIS
Keyword highlighting in title based on search
query
Very limited meta description using keywords or
phrases.
Page URL, Page visit statistics
Little clues for actual content in the web page
Forces one to browse the page
76
SoftwareResearchLab,UofS
75. TRADITIONAL CODE SEARCH ENGINES
No ties with the IDE
Returns hundreds of result
pages
Keyword matching search
No support for search query
formulation
77
SoftwareResearchLab,UofS
77. USER STUDY GROUPING
79
P1,P2,P3,P4,P5,P6
A (P1,P4,P5) B (P2,P3,P6)
EC1, EC2,
EC3, EC4
S1
(EC1,
EC2)
S2
(EC3,
EC4)
T/S1 + E/S2 T/S2 + E/S1
I (P1,P5) II (P4) I (P2) II (P3,P6)
TE ET TE ET
Editor's Notes
Hello everyone, My name is Mohammad Masudur Rahman.
Welcome to my presentation.
In this talk, I would show how the context of the programming errors or exceptions can be exploited in different kinds of recommendation for problem solving within the IDE.
Thus the title of my talk is – Exploiting Context in Dealing with Programming Errors and Exceptions
Programming exception is a very frequent and a common experience for software programmers or developers.
Once an exception is encountered, the developer identifies the target source line that triggers the exception,
and attempts to guess what makes the exception to be thrown.
In order to dig deeper, the developer often debugs the source code.
However, the debugging may not reveal the real cause behind the exception or may fail to provide any useful
explanation which can be used for the solution of the exception.
Thus at some point, the developer needs to look at web for solution or relevant information.
Study shows that developers spend about 19% of their programming time in web surfing.
In order to solve an exception using web search, they generally follow several steps.
The first step is web search using search engines in the web browser.
Here, they face several practical challenges:
The web browser works in a different context and it does not know the detailed context of the encountered exceptions. Thus it returns results which might be inapplicable for the current programming problem and its context. For example, let us consider NullPointerException, this exception can trigger under hundreds of circumstances, and it is nearly impossible to return useful search results without knowing the detailed context of the exception.
This search forces the developer to leave IDE and search in the browser, which is often distracting and time-consuming.
One might be also interested to collect search query from the IDE.
A traditional way for query selection is to copy the exception message, the first line of stack traces, and use it as the search query.
- Sometimes, this query may return relevant results; however, there is a great chance that query would not be enough for most of the exceptions.
- One may also attempt to develop a custom search query which often needs a deeper understanding about the cause or details of the exception, and it is also time-consuming.
Moreover, the developer needs to switch to web browser from the IDE in order to perform the web search.
In this step, the developer manually chooses the result pages one by one from the ranked list, and check the content for relevance. However, it is often found that
They browse through the whole page patiently and at last discover that the page is not much relevant.
Browsing and analyzing a number of pages is a great burden for the developers, and also time-consuming.
Once the exception is solved properly, the next step a developer would take is to add an effective handler for the exception. The idea is to make the application robust enough so that it does not collapse with the occurrence of the exception in the future, and at the same time records collects enough information about the exception for further maintenance. However,
The support from IDE for exception handling is not enough, it just adds an generic handler.
Traditional code search engines also do not support much in finding relevant examples for exception handling.
Thus we observe the following issues with traditional approaches while dealing with programming errors and exceptions.
The support for query formulation is not enough. Search engines suggest popular queries, but they may not be relevant for the current problem at hand.
The search engine is unaware of the detailed context of the exception, and thus the search may not be much effective.
Support for post-search content analysis is extremely limited.
Support for exception handling is not enough both from IDE and the code search engines.
Now, lets see how our proposed approaches in this thesis help the developers in
dealing with the programming errors and exceptions.
In this thesis, we propose and evaluate four recommendation approaches to overcome those four challenges/limitations followed by a task-oriented user study.
In the first study, we introduce SurfClipse: A context-aware meta search engine for programming errors and exceptions.
In the second study, we propose and evaluate QueryClipse: A context-aware query recommender for web search.
In the third study, we introduce ContentSuggest that recommends only the relevant sections from a given web page for manual analysis.
In the fourth study, we propose and evaluate SurfExample: An IDE-based context-aware code example recommender that recommends code examples for exception handling.
While each of the approaches is extensively evaluated and validated, in order to investigate the applicability of our approaches in real life problem solving,
we integrate all four approaches into one plug-in ExcClipse, and conduct a user study with six graduate research students.
In this talk, we would focus mostly on the first study in details, and would provide overview on the rest studies.
One of our important observations about traditional or ad-hoc approaches is that the technical details or the programming context of an encountered exception is not properly used each of which has great potential for problem solving.
In this thesis, we exploit such information extensively, and attempt to help in different search related activities.
This slide shows some examples.
Once an unhandled exception encounters, IDE reports the stack trace.
The first line of stack trace generally describes the technical error message.
The programming code that triggers the exception is context, it is also called the context code.
the technical details and the programming context of the errors and exceptions, and recommend different
SurfClipse: An IDE-based context-aware meta search engine.
The search engine focuses on two important aspects: (1) context-awareness and (2) meta search.
We conduct a simple test with 75 exceptions in order to determine the feasibility of those two ideas.
In the first case, we just perform the regular keyword matching search using the error messages of the exceptions with 3 popular search engines Google, Bing and Yahoo.
In the second case, we carefully choose some context tokens in the search queries and repeat the search.
From the table, we see that context is very important for each of the search engines to return expected results.
More importantly, we note that the search engines return solutions for different sets of exceptions, that means if we use only one search engine solutions for all exceptions cannot be obtained always at once.
This may be due to their search indices or ranking algorithms.
So, depending on a single one, for example, Google may not be a good idea.
So, the key IDEA is - Meta Search engine.
This is our proposed context-aware meta search model for IDE-based recommendation.
It has tow modules:
Client module (Eclipse plugin)
Server module (Web service)
Once the developer selects an exception from Error log or console view, the client module captures the error message, stack trace and the context code likely responsible for exception and sends to the computation (server) module.
Upon getting the search request, the computation module sends the error message to multiple search engines. We use Google, Bing, Yahoo and the StackOverflow API to collect results and use them to develop the corpus.
Once the corpus is developed, we apply our proposed metrics and algorithms to produce a result set that is relevant to the encountered exception.
Then the results are sent to the client in the IDE panel.
The developer can click each result and can browse page within their working environment.
So, basically, we are providing four interesting and essential things in this model.
A complete IDE-based web search solution
Meta search from within the IDE
Context-aware search
SAAS based search solution, that can be leveraged by any IDE of any platform.
The proposed model has two working modes–Proactive and Interactive.
We design two modes to provide different needs of the developers and to meet demands of different problem solving situations.
For example, the proactive mode provides recommendation without any effort. Once an exception occurs, the IDE detects the exception, develops the search query, collect other context details, and initiates the search. Once the results are available, the developer is notified of the results.
In case of interactive mode, the developer can choose an error or exception encountered within the IDE, and can start the search through context-menu command. The IDE is responsible to extract the exception details and collect the results from the server.
The model also provides a keyword-based interface where developer can perform keyword-based search. During keyword-based search, IDE also recommends a set of suitable keywords representing the current problem context (e.g., exception details).
These are the individual metrics we consider to estimate those four aspects of ranking
The metrics are grouped into different aspects for ranking.
All weights are normalized.
We consider the heuristic weights for the metrics through controlled iterations of experiments.
We started with some wise guesses, and then ran iterations to reach the best weights for each ranking aspect.
We conducted experiments with 75 exceptions related to Eclipse plugin development and Java Application development
We collected them from the workspace of the graduate students, and different online sources
The slide shows how different aspects of the ranking control the results.
We see that content-relevance alone is not enough. However, we estimate content-relevance in a more efficient way in our approach where we consider stack trace also, which the search engines cannot do.
Considering context in association with context is always helpful.
Moreover, we considers two more aspects, which provides more positive effects in the results.
Here we show the result comparison against two existing approaches.
The proactive mode of our model is aligned with these two methods by automatic search query generation without developer involvement.
In this mode, client searches with its generated query from the context (e.g., stack trace and context-code), and collects the results.
The first approach by Cordeiro et al which considers only stack traces, and depends on StackOverflow data dump, definitely cannot answer all questions. It provides a recall of 24%, whereas
Our proposed approach performs significantly well.
The second approach Seahawk, considers the context-code, not specialized for exception related search. Still it is relevant to our works. However, we found it suffers from the same lack of the work by Cordeiro et al
The interactive mode of our model is aligned with the working principles of search engines– search query generation with developer involvement.
In this mode, the client recommends suitable keywords from the context, and developer chooses from them for search.
We use the same set of keywords from the recommended list for both our approach and the search engines.
So, basically, the real comparison should be against the search engines which we did, and found these results.
All 3 search engines performed almost the same except with Google’s precision.
Our method provides a precision comparable to Google, but it outperforms all in terms of recall. It provides 90.66% recall, whereas the search
Engines provide 77.33% at best.
Recently, we tested with 150 exceptions, and found 92% recall, whereas the search engines performed nearly 80%
So, it also validates our previous experiment results.
In the first study, we did not focus much on search query generation which is also one of most important supports that the developers need for effective web search.
In the second study, we conduct our study on search query recommendation.
QueryClipse– The context-aware search query recommender.
In this study, what we do is—
We analyze the stack trace and context code of an encountered exception and recommends a ranked list of suitable search queries.
Thus QueryClipse provides four features
Context-aware query: automatically analyzes the technical details and context of the exception and generates queries.
Ranked list: Provides a ranked list queries based on extensive analysis
Custom query: Generates token graph from stack trace based on implied relationships to assist in custom search query development.
Query length customization: Query length can be customized
Now, in the first two studies, we recommended relevant search queries and relevant web pages. However, as we observed the post-search content analysis is completely manual
and the checking of page relevance is a tough job.
This is time-consuming especially when a number of web pages are browsed.
In the third study, we propose
ContentSuggest– a context-aware page content recommender that recommends only the relevant section from the page.
The traditional support for post-search content analysis is quite limited. For example, the search engine only highlights the query tokens, and provides a very little meta description about the page,
Which is not often enough. Thus the developer needs to browse every single page in the list in order to see if it is relevant.
What we do in this study is—
We analyze the technical details and context of the encountered exception, and we also analyze different sections of a chosen web page,
Then we display only the most relevant section so that the developer can check the relevance of the page without consulting the whole page.
For example, this marked section is the most relevant for an exception of interest, and ContentSuggest displays that section for relevance checking of the page.
Thus ContentSuggest provides three distinguishing features
Relevant section suggestion – exploits available information about the exception in the IDE, and returns the relevant section only
Less content, less overhead– The developer needs to consult less information for the same task, which is a great help.
Noise-free version of the page– It also returns a noise-free version of the page that discards advertisements, unexpected widgets and so on.
While the previous three studies are involved with the solution of an encountered exception in the IDE, exceptions should be also properly handled
In order to make the application system robust and secure.
However, existing support from IDE or the code search engines are not enough
In this study, we propose and evaluate SurfExample: An IDE-based context-aware code example recommender that recommends code examples for exception handling.
SurfExample– our code example recommender for exception handling.
SurfExample takes the code under development in the IDE, analyzes it and collects relevant and useful code examples from GitHub code repositories.
For example, the recommended code example is relevant, and at the same time, it also contains a list of carefully developed handlers.
A developer can learn effective handling from different such examples, and improve his/her coding.
Thus SurfExample has three distinguishing features from both technical and user’s perspectives.
Graph-based structural relevance– While previous approaches are mostly based on heuristics, we propose a graph-based structural similarity matching technique.
We introduce the handler quality paradigm, which was previously done through manual analysis that can be biased.
We integrate with GitHub repositories using API endpoints, which provides access of a large number of quality examples within the IDE to the developers.
While each of the studies is extensively evaluated and validated against existing studies from the literature,
In order to further validate our approaches,
we integrate all four approaches into one plug-in ExcClipse, and conduct a user study with six graduate research students.
-- We involve six participants from SRLab
-- Chose four tasks for each of four programming exceptions
-- Used questionnaire and observation checklist for data collection
-- Both participants and problem sets were grouped into different categories
-- Each study was conducted in three sessions – training, execution and evaluation.
We identify these nine features both functional and non-functional for comparing our approach with traditional alternatives.
Each of the participants perform four tasks for each of the problem task
And rated different features.
We note that our approach is rated higher for each of the features.
Each participant rated independently, and we have shown the average measures.
We also conducted Mann-Whitney U test, and found that rating for our approach is significantly higher than that of traditional alternatives.
This is the overall ratings by each of the participants which also show that
Our approach is highly applicable in real life problem solving.
We also checked significance test, found that the difference is significant.
We experienced a few threats to validity with different approaches. For example,
The search engines are constantly evolving, thus the statistics shown may change later.
Sometimes QueryClipse can recommend long query if the error message or the tokens are too lengthy
Code example gold set for SurfExample may have some subjective bias.
The user study conducted a limited number of participants.
Conclusion.
Thus to conclude my talk,
In the first study, we propose SurfClipse which found promising in terms of recommendation accuracy.
In the second study, we propose QueryClipse that recommends more effective search queries compared to traditional queries or queries from existing approaches.
ContentSuggest, the page content recommender has a great potential for problem solving.
SurfExample, is a novel approach for the recommendation of exception handling code examples.
In future, we plan for more granular and focused recommendation.
Such as which particular paragraph in the web page contains solution for the exception
More customized and semantic search query.
Whether an exception should be handled or thrown.
We experienced these 3 threats to validity.
-search not real time yet
-search engines are constantly evolving, so has reproducibility issues
-experimented with widely available exceptions and bound by the limitations of provided results from search engines
Each metric is normalized
Only class name and method name chosen as keywords