The document discusses limitations of current evaluation practices for spreadsheet fault localization tools. It notes that each research group uses its own datasets, which are rarely made public and often tailored to the evaluated approach, making comparison difficult. It advocates developing a common benchmark system using real-world spreadsheets with real faults from diverse domains. The document also calls for more user studies and field research to assess usability and user acceptance in real-life scenarios, rather than just laboratory experiments.
Interactive fault localization leveraging simple user feedback - by Liang GongLiang Gong
Many fault localization methods have been proposed in the literature. These methods take in a set of program execution profiles and output a list of suspicious program elements. The list of program elements ranked by their suspiciousness is then presented to developers for manual inspection. Currently, the suspicious elements are ranked in a batch process where developers' inspection efforts are rarely utilized for ranking. The inaccuracy and static nature of existing fault localization methods prompt us to incorporate user feedback to improve the accuracy of the existing methods. In this paper, we propose an interactive fault localization framework that leverages simple user feedback. Our framework only needs users to label the statements examined as faulty or clean, which does not require additional effort than conventional non-interactive methods. After users label suspicious program elements as faulty or clean, our framework incorporates such information and re-orders the rest of the suspicious program elements, aiming to expose truly faulty elements earlier. We have integrated our solution with three well-known fault localization methods: Ochiai, Tarantula, and Jaccard. The evaluation on five Unix programs and the Siemens test suite shows that our solution achieves significant improvements on fault localization accuracy.
Software engineering practitioners often spend significant amount of time and effort to debug. To help practitioners perform this crucial task, hundreds of papers have proposed various fault localization techniques. Fault localization helps practitioners to find the location of a defect given its symptoms (e.g., program failures). These localization techniques have pinpointed the locations of bugs of various systems of diverse sizes, with varying degrees of success, and for various usage scenarios. Unfortunately, it is unclear whether practitioners appreciate this line of research. To fill this gap, we performed an empirical study by surveying 386 practitioners from more than 30 countries across 5 continents about their expectations of research in fault localization. In particular, we investigated a number of factors that impact practitioners’ willingness to adopt a fault localization technique. We then compared what practitioners need and the current state-of-research by performing a literature review of papers on fault localization techniques published in ICSE, FSE, ESEC-FSE, ISSTA, TSE, and TOSEM in the last 5 years (2011-2015). From this comparison, we highlight the directions where researchers need to put effort to develop fault localization techniques that matter to practitioners.
Locating Crashing Faults based on Crash Stack TracesLiang Gong
Software crashes due to its increasing complexity. Once a crash happens, a crash report could be sent to software developers for investigation upon user permission. Because of the large number of crash reports and limited information, debugging for crashes is often a tedious and labor-intensive task. In this paper, we propose a statistical fault localization framework to help developers locate functions that contain crashing faults. We generate the execution traces for the failing traces based on the crash stack, and the passing traces from normal executions. We form program spectra by combining generated passing and failing trace, and then apply statistical fault localization techniques such as Ochiai to locate the crashing faults. We also propose two heuristics to improve the fault localization performance. We evaluate our approach using the real-world Firefox crash report data. The results show that the performance of our method is promising. Our approach permits developers to locate 63.9% crashing faults by examining only 5% Firefox 3.6 functions in the spectra.
Visualizing the Problem Domain for Spreadsheet Users: A Mental Model PerspectiveBennett Kankuzi
In this paper presentation, we introduce a new spreadsheet visualization tool as well as an empirical evaluation of its usability and of its effects on mental models of users. The tool translates traditional spreadsheet
formulas into problem domain narratives and highlights referenced cells. The tool was found to be easy to learn and helped the participants to locate more errors in spreadsheets. Furthermore, the tool increased the use of the domain mental model in error descriptions and appeared to improve the mapping between the spreadsheet model and the domain model. Full paper can be downloaded at http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6883040 The paper was presented at the 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) which was held between 28th July, 2014 to 1st August, 2014 in Melbourne, Australia.
Interactive fault localization leveraging simple user feedback - by Liang GongLiang Gong
Many fault localization methods have been proposed in the literature. These methods take in a set of program execution profiles and output a list of suspicious program elements. The list of program elements ranked by their suspiciousness is then presented to developers for manual inspection. Currently, the suspicious elements are ranked in a batch process where developers' inspection efforts are rarely utilized for ranking. The inaccuracy and static nature of existing fault localization methods prompt us to incorporate user feedback to improve the accuracy of the existing methods. In this paper, we propose an interactive fault localization framework that leverages simple user feedback. Our framework only needs users to label the statements examined as faulty or clean, which does not require additional effort than conventional non-interactive methods. After users label suspicious program elements as faulty or clean, our framework incorporates such information and re-orders the rest of the suspicious program elements, aiming to expose truly faulty elements earlier. We have integrated our solution with three well-known fault localization methods: Ochiai, Tarantula, and Jaccard. The evaluation on five Unix programs and the Siemens test suite shows that our solution achieves significant improvements on fault localization accuracy.
Software engineering practitioners often spend significant amount of time and effort to debug. To help practitioners perform this crucial task, hundreds of papers have proposed various fault localization techniques. Fault localization helps practitioners to find the location of a defect given its symptoms (e.g., program failures). These localization techniques have pinpointed the locations of bugs of various systems of diverse sizes, with varying degrees of success, and for various usage scenarios. Unfortunately, it is unclear whether practitioners appreciate this line of research. To fill this gap, we performed an empirical study by surveying 386 practitioners from more than 30 countries across 5 continents about their expectations of research in fault localization. In particular, we investigated a number of factors that impact practitioners’ willingness to adopt a fault localization technique. We then compared what practitioners need and the current state-of-research by performing a literature review of papers on fault localization techniques published in ICSE, FSE, ESEC-FSE, ISSTA, TSE, and TOSEM in the last 5 years (2011-2015). From this comparison, we highlight the directions where researchers need to put effort to develop fault localization techniques that matter to practitioners.
Locating Crashing Faults based on Crash Stack TracesLiang Gong
Software crashes due to its increasing complexity. Once a crash happens, a crash report could be sent to software developers for investigation upon user permission. Because of the large number of crash reports and limited information, debugging for crashes is often a tedious and labor-intensive task. In this paper, we propose a statistical fault localization framework to help developers locate functions that contain crashing faults. We generate the execution traces for the failing traces based on the crash stack, and the passing traces from normal executions. We form program spectra by combining generated passing and failing trace, and then apply statistical fault localization techniques such as Ochiai to locate the crashing faults. We also propose two heuristics to improve the fault localization performance. We evaluate our approach using the real-world Firefox crash report data. The results show that the performance of our method is promising. Our approach permits developers to locate 63.9% crashing faults by examining only 5% Firefox 3.6 functions in the spectra.
Visualizing the Problem Domain for Spreadsheet Users: A Mental Model PerspectiveBennett Kankuzi
In this paper presentation, we introduce a new spreadsheet visualization tool as well as an empirical evaluation of its usability and of its effects on mental models of users. The tool translates traditional spreadsheet
formulas into problem domain narratives and highlights referenced cells. The tool was found to be easy to learn and helped the participants to locate more errors in spreadsheets. Furthermore, the tool increased the use of the domain mental model in error descriptions and appeared to improve the mapping between the spreadsheet model and the domain model. Full paper can be downloaded at http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=6883040 The paper was presented at the 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) which was held between 28th July, 2014 to 1st August, 2014 in Melbourne, Australia.
Field of Study and Research Methods for an Effect of Cognitive and Informatio...Yury Solonitsyn
Authors show one of the possible approaches to the experimental research of the cognitive and information load on PC’s users – the mental load or amount of efforts, spent because of the necessity to recognise screen form’s elements and to process the data presented.
Andrei Balkanskii, Artem Smolin, Yury Solonitsyn
ITMO University, Saint Petersburg, Russia
Crediting informatics and data folks in life science teamsCarole Goble
Science Europe LEGS Committee: Career Pathways in Multidisciplinary Research: How to Assess the Contributions of Single Authors in Large Teams, 1-2 Dec 2015, Brussels
The People Behind Research Software crediting from the informatics, technical point of view
Impact your Library UX with Contextual InquiryRachel Vacek
A contextual inquiry is a research study that involves in-depth interviews where users walk through common tasks in the physical environment in which they typically perform them. It can be used to better understand the intents and motivations behind user behavior. In this session, learn what’s needed to conduct a contextual inquiry and how to analyze the ethnographic data once collected. I'll cover how to synthesize and visualize your findings as sequence models and affinity diagrams that directly inform the development of personas and common task flows. Finally, learn how this process can help guide your design and content strategy efforts while constructing a rich picture of the user experience.
201209 An Introduction to Building Affective-Driven Self-Adaptive Software Javier Gonzalez-Sanchez
One important characteristic in modern software systems is self-
adaptation, the capability of monitoring and reacting to changes into the environment. A particular case of self-adaptation is affect-driven self- adaptation. Affect-driven self-adaptation involves using sensing devices to measure physiological signals of human affectivestate’s (emotions) changes, learning about the meaning of those changes, and then reacting (self-adapting) in consequence. Affect-driven self-adaptive systems take advantage of brain-computer interfaces, eye-tracking, face-based emotion recognition, and sensors to measure physiological signals.
Systems such as learning environments, health care systems, and videogames are able to take advantage of affect-driven self-adaptive capabilities. Today these capabilities are brittle, costly to change, difficult to reuse, and limited in scope. A software factory approach has been suggested to make feasible adding affective-driven self-adaptive capabilities either into new or existing systems. Software factories capture knowledge of how to produce applications that share common characteristics and make that knowledge available in the form of assets (patterns, model, framework, and tools) and systematically apply those assets to automate development reducing cost and time while improving product quality.
This talk provides a sneak peek of affective-driven self-adaptive capabilities and explores how a software factory approach is used to build affect-driven self-adaptive software.
Bug tracking system plays major role in identifying bug in various software system. Present paper underlined the importance of bug tracking system in the field of software development and had also identified the various drawbacks of the current bug tracking system by exploring the various literatures and research papers published up to year 2014. Various recommendations and suggestions have also been provided after analyzing the obstacles faced by developers and users due to functioning of current bug tracking system. An extensive survey is presented in this regards and the provided analysis is truly based on what have been analyzed s in context with bug tracking system.
A talk presented to the US Networking and Information Technology Research and Development (NITRD) Program's High End Computing Interagency Working Group, 16 January 2020
Usability Testing in Federal Libraries: A Case Studynullhandle
Presentation given to the Federal Library Information Network (FLICC) Emerging Technologies Working Group on improvised usability testing of a redesigned electronic resources access portal for the U.S. Supreme Court Library.
Field of Study and Research Methods for an Effect of Cognitive and Informatio...Yury Solonitsyn
Authors show one of the possible approaches to the experimental research of the cognitive and information load on PC’s users – the mental load or amount of efforts, spent because of the necessity to recognise screen form’s elements and to process the data presented.
Andrei Balkanskii, Artem Smolin, Yury Solonitsyn
ITMO University, Saint Petersburg, Russia
Crediting informatics and data folks in life science teamsCarole Goble
Science Europe LEGS Committee: Career Pathways in Multidisciplinary Research: How to Assess the Contributions of Single Authors in Large Teams, 1-2 Dec 2015, Brussels
The People Behind Research Software crediting from the informatics, technical point of view
Impact your Library UX with Contextual InquiryRachel Vacek
A contextual inquiry is a research study that involves in-depth interviews where users walk through common tasks in the physical environment in which they typically perform them. It can be used to better understand the intents and motivations behind user behavior. In this session, learn what’s needed to conduct a contextual inquiry and how to analyze the ethnographic data once collected. I'll cover how to synthesize and visualize your findings as sequence models and affinity diagrams that directly inform the development of personas and common task flows. Finally, learn how this process can help guide your design and content strategy efforts while constructing a rich picture of the user experience.
201209 An Introduction to Building Affective-Driven Self-Adaptive Software Javier Gonzalez-Sanchez
One important characteristic in modern software systems is self-
adaptation, the capability of monitoring and reacting to changes into the environment. A particular case of self-adaptation is affect-driven self- adaptation. Affect-driven self-adaptation involves using sensing devices to measure physiological signals of human affectivestate’s (emotions) changes, learning about the meaning of those changes, and then reacting (self-adapting) in consequence. Affect-driven self-adaptive systems take advantage of brain-computer interfaces, eye-tracking, face-based emotion recognition, and sensors to measure physiological signals.
Systems such as learning environments, health care systems, and videogames are able to take advantage of affect-driven self-adaptive capabilities. Today these capabilities are brittle, costly to change, difficult to reuse, and limited in scope. A software factory approach has been suggested to make feasible adding affective-driven self-adaptive capabilities either into new or existing systems. Software factories capture knowledge of how to produce applications that share common characteristics and make that knowledge available in the form of assets (patterns, model, framework, and tools) and systematically apply those assets to automate development reducing cost and time while improving product quality.
This talk provides a sneak peek of affective-driven self-adaptive capabilities and explores how a software factory approach is used to build affect-driven self-adaptive software.
Bug tracking system plays major role in identifying bug in various software system. Present paper underlined the importance of bug tracking system in the field of software development and had also identified the various drawbacks of the current bug tracking system by exploring the various literatures and research papers published up to year 2014. Various recommendations and suggestions have also been provided after analyzing the obstacles faced by developers and users due to functioning of current bug tracking system. An extensive survey is presented in this regards and the provided analysis is truly based on what have been analyzed s in context with bug tracking system.
A talk presented to the US Networking and Information Technology Research and Development (NITRD) Program's High End Computing Interagency Working Group, 16 January 2020
Usability Testing in Federal Libraries: A Case Studynullhandle
Presentation given to the Federal Library Information Network (FLICC) Emerging Technologies Working Group on improvised usability testing of a redesigned electronic resources access portal for the U.S. Supreme Court Library.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
1. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
1
Tool-Supported Fault
Localization in Spreadsheets:
Limitations of Current
Evaluation Practice
Birgit Hofer, Franz Wotawa
Dietmar Jannach, Thomas Schmitz
Kostyantyn Shchekotykhin
2. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
2
An Overview of Limitations
of Current Evaluation Practice
1
2
3
Lack of benchmarks systems
Usability and user acceptance
Field research
3. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
3
Benchmark Systems – Current Situation
There is no public data set for spreadsheet fault
localization
Researcher create own benchmark systems
Take existing corpus (e.g. EUSES [FR05]) or collect
individual spreadsheets
Apply mutation operators, e.g. [AE09] on them or
manually inject faults
[FR05] M. Fisherand G. Rothermel:“The EUSES spreadsheetcorpus:Ashared resource forsupporting
experimentationwith spreadsheetdependability mechanisms.”1stWorkshop on End-User
Software Engineering,2005.
[AE09] R.Abraham and M.Erwig. Mutation Operators forSpreadsheets.IEEE Transactionson Software
Engineering,2009.
1
4. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
4
Some Examples I
Hofer et. al [HRW13]
o “… we are evaluating the … approaches by means of the EUSES
spreadsheet corpus. We skipped around 240 Excel 5.0
spreadsheets that are not compatible with our implementation, …
o we removed all spreadsheets containing less than 5 formulas …
o we automaticallycreated up to five first-order mutants. Amutant of a
spreadsheet is created by randomly choosing a formula cell of the
spreadsheet and applying a mutation operator on it. According to the
classification of spreadsheet mutation operators ofAbraham and
Erwig, we used the following mutation operators …”
Jannach and Schmitz [JS14]
o “For the performance analysis, we selected a number of artificial and
real-world spreadsheetsin which we manuallyinjected faults.”
[HRW13] B. Hofer, A. Riboira, F. Wotawa, and R. Abreu, E. Getzner: “On the Empirical Evaluation of Fault Localization
Techniques for Spreadsheets.” FASE 2013.
[JS14] D. Jannach and T. Schmitz: “Model-based diagnosis of spreadsheet programs - A constraint-based debugging
approach.” Automated Software Engineering, Springer, 2014.
5. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
5
Some examples II
Abraham and Erwig [AE08]
o “… we use spreadsheets that have been used in previous
empirical studies. The spreadsheets have been picked to include as
many different kinds of formulas, and formulas with branching …
o We generate mutant spreadsheets by seeding faults in the original
spreadsheets using the mutation operators given in Table 1. The
mutation operators have been designed to reflect errors reported in
spreadsheet literature …”
Außerlechner et al. [AFW13]
o “Since MINION is not able to deal with Real numbers …, we created a
specific spreadsheet corpus that contains spreadsheets with Integer
values only … Whereas some of the spreadsheets are artificially
created, 21 spreadsheets are real-life programs … “
[AE08] R. Abraham, and M. Erwig: “Test-Driven Goal-Directed Debugging in Spreadsheets.” IEEE Symposioum on
Visual Languages and Human-Centric Computing, 2008.
[AFW13] S. Ausserlechner et al.: “The Right Choice Matters! SMT Solving Substantially Improves Model-Based
Debugging of Spreadsheets.” QSIC 2013.
6. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
6
Current Situation - Consequences
Each research group uses own data set
rarely made publicly available
often made to fit the evaluated approach
comparison of approaches difficult
7. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
7
We need a corpus that contains …
Real world spreadsheets
Large spreadsheets, not toy examples
Spreadsheets with real faults, not only seeded faults
Input-/output relations that reveal the fault
8. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
8
Ways to get there
Laboratory: spreadsheet construction exercises
Excellent starting point: Kooper Corpus [AP10]
Larger spreadsheets
Different domains and exercises
Real life
[AP10] S.Aurigemma,and R.Panko:“The detection of human spreadsheeterrors by humans versus
inspection (auditing)software,”CoRR,2010.
9. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
9
Usability and User Acceptance
Mostly offline experiments
Information from the user required, e.g.
Correctness of values
Expected values
Specification of several test cases
Is a user willing / able to provide these inputs?
User studies are necessary to answer these questions.
2
10. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
10
Field research
Setting
Laboratory experiments vs. everyday use
Participant
Students vs. managers
Scenario
Artificial problem vs. real problem
3
11. B. Hofer, D. Jannach, T. Schmitz, K. Shchekotykhin, and F. Wotawa:
„ Tool-supported fault localization in spreadsheets: Limitations of current evaluation practice“
11
Proposals for future work
Improve comparability and reproducibility
Develop common benchmark system
Focus on usability and user acceptance
Make user studies
Focus on real life scenarios (not only laboratory experiments)
Make field research, questionnaires …