Components - Graph Based Detection of Library API LimitationsICSM 2011
Paper: Graph-based Detection of Library API Imitations
Authors: Chengnian Sun, Siau-Cheng Khoo, Shao Jie Zhang (All from National University of Singapore)
Session: Research Track Session 7: Component
Industry - Estimating software maintenance effort from use cases an indu...ICSM 2011
Paper: Estimating Software Maintenance Effort from Use Cases: an Industrial Case Study
Authors:Yan Ku, Jing Du, Ye Yang, Qing Wang
Session: Industry Tracking 5: Metrics and
Estimation
Faults and Regression testing - Localizing Failure-Inducing Program Edits Bas...ICSM 2011
Paper: Localizing Failure-Inducing Program Edits Based on Spectrum Information.
Authors: Lingming Zhang, Miryung Kim, Sarfraz Khurshid.
Session: Research Track Session 1: Faults and Regression Testing
Reliability and Quality - Predicting post-release defects using pre-release f...ICSM 2011
Paper : Predicting Post-release Defects Using Pre-release Field Testing Results
Authors : Foutse Khomh, Brian Chan, Ying Zou, Anand Sinha and Dave Dietz
Session: Research Track Session 9: Reliability and Quality
Industry - The Evolution of Information Systems. A Case Study on Document Man...ICSM 2011
Paper : The Evolution of Information Systems. A Case Study on Document Management
Authors : Paolo Salvaneschi
Session: Industry Track Session 3: Evolution and migration
Components - Crossing the Boundaries while Analyzing Heterogeneous Component-...ICSM 2011
Paper: "Crossing the Boundaries while Analyzing Heterogeneous Component-Based Software Systems"
Authors: Amir Reza Yazdanshenas, Leon Moonen
Session: Research Track Session 7: Components
Components - Graph Based Detection of Library API LimitationsICSM 2011
Paper: Graph-based Detection of Library API Imitations
Authors: Chengnian Sun, Siau-Cheng Khoo, Shao Jie Zhang (All from National University of Singapore)
Session: Research Track Session 7: Component
Industry - Estimating software maintenance effort from use cases an indu...ICSM 2011
Paper: Estimating Software Maintenance Effort from Use Cases: an Industrial Case Study
Authors:Yan Ku, Jing Du, Ye Yang, Qing Wang
Session: Industry Tracking 5: Metrics and
Estimation
Faults and Regression testing - Localizing Failure-Inducing Program Edits Bas...ICSM 2011
Paper: Localizing Failure-Inducing Program Edits Based on Spectrum Information.
Authors: Lingming Zhang, Miryung Kim, Sarfraz Khurshid.
Session: Research Track Session 1: Faults and Regression Testing
Reliability and Quality - Predicting post-release defects using pre-release f...ICSM 2011
Paper : Predicting Post-release Defects Using Pre-release Field Testing Results
Authors : Foutse Khomh, Brian Chan, Ying Zou, Anand Sinha and Dave Dietz
Session: Research Track Session 9: Reliability and Quality
Industry - The Evolution of Information Systems. A Case Study on Document Man...ICSM 2011
Paper : The Evolution of Information Systems. A Case Study on Document Management
Authors : Paolo Salvaneschi
Session: Industry Track Session 3: Evolution and migration
Components - Crossing the Boundaries while Analyzing Heterogeneous Component-...ICSM 2011
Paper: "Crossing the Boundaries while Analyzing Heterogeneous Component-Based Software Systems"
Authors: Amir Reza Yazdanshenas, Leon Moonen
Session: Research Track Session 7: Components
Abstract:
Though in essence an engineering discipline, software engineering research has always been struggling to demonstrate impact. This is reflected in part by the funding challenges that the discipline faces in many countries, the difficulties we have to attract industrial participants to our conferences, and the scarcity of papers reporting industrial case studies.
There are clear historical reasons for this but we nevertheless need, as a community, to question our research paradigms and peer evaluation processes in order to improve the situation. From a personal standpoint, relevance and impact are concerns that I have been struggling with for a long time, which eventually led me to leave a comfortable academic position and a research chair to work in industry-driven research.
I will use some concrete research project examples to argue why we need more inductive research, that is, research working from specific observations in real settings to broader generalizations and theories. Among other things, the examples will show how a more thorough understanding of practice and closer interactions with practitioners can profoundly influence the definition of research problems, and the development and evaluation of solutions to these problems. Furthermore, these examples will illustrate why, to a large extent, useful research is necessarily multidisciplinary. I will also address issues regarding the implementation of such a research paradigm and show how our own bias as a research community worsens the situation and undermines our very own interests.
On a more humorous note, the title hints at the fact that being a scientist in software engineering and aiming at having impact on practice often entails leading two parallel careers and impersonate different roles to different peers and partners.
Bio:
Lionel Briand is heading the Certus center on software verification and validation at Simula Research Laboratory, where he is leading research projects with industrial partners. He is also a professor at the University of Oslo (Norway). Before that, he was on the faculty of the department of Systems and Computer Engineering, Carleton University, Ottawa, Canada, where he was full professor and held the Canada Research Chair (Tier I) in Software Quality Engineering. He is the coeditor-in-chief of Empirical Software Engineering (Springer) and is a member of the editorial boards of Systems and Software Modeling (Springer) and Software Testing, Verification, and Reliability (Wiley). He was on the board of IEEE Transactions on Software Engineering from 2000 to 2004. Lionel was elevated to the grade of IEEE Fellow for his work on the testing of object-oriented systems. His research interests include: model-driven development, testing and verification, search-based software engineering, and empirical software engineering.
Abstract:
Botnets, which are networks of malware-infected machines that are controlled by an adversary, are the root cause of a large number of security threats on the Internet. A particularly sophisticated and insidious type of bot is Torpig, which is a malware program that is designed to harvest sensitive information (such as bank account and credit card data) from its victims. In this talk, I will report on our efforts to take control of the Torpig botnet for ten days. Over this period, we observed more than 180 thousand infections and recorded more than 70 GB of data that the bots collected.
While botnets have been hijacked before, the Torpig botnet exhibits certain properties that make the analysis of the data particularly interesting. First, it is possible (with reasonable accuracy) to identify unique bot infections and relate that number to the more than 1.2 million IP addresses that contacted our command and control server during the ten day period. This shows that botnet estimates that are based on IP addresses are likely to report inflated numbers. Second, the Torpig botnet is large, targets a variety of applications, and gathers a rich and diverse set of information from the infected victims. This allowed us to perform interesting data analysis that goes well beyond simply counting the number of stolen credit cards. In this talk I will discuss the analysis that we performed on the data collected and the lessons learned from the analysis, as well as from the process of obtaining (and losing) the botnet.
Bio:
Richard A. Kemmerer is the Computer Science Leadership Professor and a past Department Chair of the Department of Computer Science at the University of California, Santa Barbara. Dr. Kemmerer received the B.S. degree in Mathematics from the Pennsylvania State University in 1966, and the M.S. and Ph.D. degrees in Computer Science from the University of California, Los Angeles, in 1976 and 1979, respectively. His research interests include formal specification and verification of systems, computer system security and reliability, programming and specification language design, and software engineering.
Dr. Kemmerer is a Fellow of the IEEE Computer Society, a Fellow of the Association for Computing Machinery, and he is the 2007 recipient of The Applied Security Associates Distinguished Practitioner Award. He is a member of the IFIP Working Group 11.3 on Database Security, and a member of the International Association for Cryptologic Research. He is a past Editor-in-Chief of IEEE Transactions on Software Engineering, and he has served on the editorial boards of the ACM Computing Surveys and IEEE Security and Privacy and on the Board of Governors of the IEEE Computer Society. He served on Microsoft’s Trustworthy Computing Academic Advisory Board (2002-2010) and on the National Science Foundations/CISE Advisory Board (2002-2004).
Paper: SCOTCH: Improving Test-to-Code Traceability using Slicing and Conceptual Coupling
Authors: Abdallah Qusef, Gabriele Bavota, Rocco Oliveto, Andrea De Lucia, David Binkley
Session: Research Track Session 3: Dynamic Analysis
Postdoc symposium - A Logic Meta-Programming Foundation for Example-Driven Pa...ICSM 2011
Paper: A Logic Meta-Programming Foundation for Example-Driven Pattern Detection in Object-Oriented Programs
Author: Coen De Roover
Session: Post-doctoral symposium
ERA - A Comparison of Stemmers on Source Code Identifiers for Software SearchICSM 2011
Paper: A Comparison of Stemmers on Source Code Identifiers for Software
Search
Authors: Andrew Wiese, Valerie Ho, Emily Hill.
Session: ERA1 - Linguistic Analysis of Software Artifacts
ERA - Clustering and Recommending Collections of Code Relevant to TaskICSM 2011
Paper: Clustering and Recommending Collections of Code Relevant to Task
Authors: Seonah Lee and Sungwon Kang
Session: Early Research Achievements Track Session 3: Managing and Supporting Software Maintenance Activities
Industry - Relating Developers' Concepts and Artefact Vocabulary in a Financ...ICSM 2011
Paper: Relating Developers' Concepts and Artefact Vocabulary in a Financial
Software Module
Authors: Tezcan Dilshener and Michel Wermelinger
Session: Industry Track 2 - Reverse Engineering
Industry - Testing & Quality Assurance in Data Migration Projects ICSM 2011
Paper: Testing & Quality Assurance in Data Migration Projects
Authors: Klaus Haller, Florian Matthes, Christopher Schulz
Session: Industry Track Session 3: Evolution and migration
Natural Language Analysis - Mining Java Class Naming ConventionsICSM 2011
Paper: Mining Java Class Naming Conventions
Authors: Simon Butler, Michel Wermelinger, Yijun Yu and Helen Sharp
Session: Research Track 4 - Natural Language Analysis
Industry - Evolution and migration - Incremental and Iterative Reengineering ...ICSM 2011
Paper: Incremental and Iterative Reengineering towards Software Product Line: An Industrial Case Study
Authors: Gang Zhang, Liwei Shen, Xin Peng, Zhenchang Xing and Wenyun Zhao
Session: Industry Track Session 3: Evolution and migration
Natural Language Analysis - Expanding Identifiers to Normalize Source Code Vo...ICSM 2011
Paper: Expanding Identifiers to Normalize Source Code Vocabulary
Authors: Dave Binkley and Dawn Lawrie
Session: Research Track 4: Natural Language Analysis
Paper: Tracking Technical Debt- An Exploratory Case Study
Authors: Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, Clauirton Siebra
Session: Early Research Achievement Track Session 3
ERA - Measuring Maintainability of Spreadsheets in the Wild ICSM 2011
Paper: Measuring Maintainability of Spreadsheets in the Wild
Authors: José Pedro Correia and Miguel Alexandre Ferreira
Session: Early Research Achievements Track Session 2: Software Changes and Maintainability
Faults and Regression Testing - Fault interaction and its repercussionsICSM 2011
Paper: Fault Interaction and its Repercussions
Authors: Nicholas DiGiuseppe and James A. Jones
Seesion: Research Track 1: Faults and Regression Testing
Metrics - You can't control the unfamiliarICSM 2011
Paper: You Can't Control the Unfamiliar: A Study on the Relations Between Aggregation Techniques for Software Metrics
Authors: Bogdan Vasilescu, Alexander Serebrenik and Mark Van Den Brand
Session: Research Track 11 - Metrics
Industry - Precise Detection of Un-Initialized Variables in Large, Real-life ...ICSM 2011
Paper: "Precise Detection of Un-Initialized Variables in Large, Real-life COBOL Programs in Presence of Un-realizable Paths"
Authors: Rahul Jiresal, Adnan Contractor and Ravindra Naik
Session: Industry Track Session 4: Program analysis and Verification
A Qualitative Study on Performance Bugs (MSR 2012)Bram Adams
Software performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experi- enced developers and require changes to more code than non-performance bugs. In order to be able to improve the resolution of performance bugs, a better understanding is needed of the current practice and shortcomings of reporting, reproducing, tracking and fixing performance bugs. This paper qualitatively studies a random sample of 400 performance and non-performance bug reports of Mozilla Firefox and Google Chrome across four dimensions (Impact, Context, Fix and Fix validation). We found that developers and users face problems in reproducing performance bugs and have to spend more time discussing performance bugs than other kinds of bugs. Sometimes performance regressions are tolerated as a trade- off to improve something else.
http://sail.cs.queensu.ca/publications/pubs/MSR2012_Zaman.pdf
Abstract:
Though in essence an engineering discipline, software engineering research has always been struggling to demonstrate impact. This is reflected in part by the funding challenges that the discipline faces in many countries, the difficulties we have to attract industrial participants to our conferences, and the scarcity of papers reporting industrial case studies.
There are clear historical reasons for this but we nevertheless need, as a community, to question our research paradigms and peer evaluation processes in order to improve the situation. From a personal standpoint, relevance and impact are concerns that I have been struggling with for a long time, which eventually led me to leave a comfortable academic position and a research chair to work in industry-driven research.
I will use some concrete research project examples to argue why we need more inductive research, that is, research working from specific observations in real settings to broader generalizations and theories. Among other things, the examples will show how a more thorough understanding of practice and closer interactions with practitioners can profoundly influence the definition of research problems, and the development and evaluation of solutions to these problems. Furthermore, these examples will illustrate why, to a large extent, useful research is necessarily multidisciplinary. I will also address issues regarding the implementation of such a research paradigm and show how our own bias as a research community worsens the situation and undermines our very own interests.
On a more humorous note, the title hints at the fact that being a scientist in software engineering and aiming at having impact on practice often entails leading two parallel careers and impersonate different roles to different peers and partners.
Bio:
Lionel Briand is heading the Certus center on software verification and validation at Simula Research Laboratory, where he is leading research projects with industrial partners. He is also a professor at the University of Oslo (Norway). Before that, he was on the faculty of the department of Systems and Computer Engineering, Carleton University, Ottawa, Canada, where he was full professor and held the Canada Research Chair (Tier I) in Software Quality Engineering. He is the coeditor-in-chief of Empirical Software Engineering (Springer) and is a member of the editorial boards of Systems and Software Modeling (Springer) and Software Testing, Verification, and Reliability (Wiley). He was on the board of IEEE Transactions on Software Engineering from 2000 to 2004. Lionel was elevated to the grade of IEEE Fellow for his work on the testing of object-oriented systems. His research interests include: model-driven development, testing and verification, search-based software engineering, and empirical software engineering.
Abstract:
Botnets, which are networks of malware-infected machines that are controlled by an adversary, are the root cause of a large number of security threats on the Internet. A particularly sophisticated and insidious type of bot is Torpig, which is a malware program that is designed to harvest sensitive information (such as bank account and credit card data) from its victims. In this talk, I will report on our efforts to take control of the Torpig botnet for ten days. Over this period, we observed more than 180 thousand infections and recorded more than 70 GB of data that the bots collected.
While botnets have been hijacked before, the Torpig botnet exhibits certain properties that make the analysis of the data particularly interesting. First, it is possible (with reasonable accuracy) to identify unique bot infections and relate that number to the more than 1.2 million IP addresses that contacted our command and control server during the ten day period. This shows that botnet estimates that are based on IP addresses are likely to report inflated numbers. Second, the Torpig botnet is large, targets a variety of applications, and gathers a rich and diverse set of information from the infected victims. This allowed us to perform interesting data analysis that goes well beyond simply counting the number of stolen credit cards. In this talk I will discuss the analysis that we performed on the data collected and the lessons learned from the analysis, as well as from the process of obtaining (and losing) the botnet.
Bio:
Richard A. Kemmerer is the Computer Science Leadership Professor and a past Department Chair of the Department of Computer Science at the University of California, Santa Barbara. Dr. Kemmerer received the B.S. degree in Mathematics from the Pennsylvania State University in 1966, and the M.S. and Ph.D. degrees in Computer Science from the University of California, Los Angeles, in 1976 and 1979, respectively. His research interests include formal specification and verification of systems, computer system security and reliability, programming and specification language design, and software engineering.
Dr. Kemmerer is a Fellow of the IEEE Computer Society, a Fellow of the Association for Computing Machinery, and he is the 2007 recipient of The Applied Security Associates Distinguished Practitioner Award. He is a member of the IFIP Working Group 11.3 on Database Security, and a member of the International Association for Cryptologic Research. He is a past Editor-in-Chief of IEEE Transactions on Software Engineering, and he has served on the editorial boards of the ACM Computing Surveys and IEEE Security and Privacy and on the Board of Governors of the IEEE Computer Society. He served on Microsoft’s Trustworthy Computing Academic Advisory Board (2002-2010) and on the National Science Foundations/CISE Advisory Board (2002-2004).
Paper: SCOTCH: Improving Test-to-Code Traceability using Slicing and Conceptual Coupling
Authors: Abdallah Qusef, Gabriele Bavota, Rocco Oliveto, Andrea De Lucia, David Binkley
Session: Research Track Session 3: Dynamic Analysis
Postdoc symposium - A Logic Meta-Programming Foundation for Example-Driven Pa...ICSM 2011
Paper: A Logic Meta-Programming Foundation for Example-Driven Pattern Detection in Object-Oriented Programs
Author: Coen De Roover
Session: Post-doctoral symposium
ERA - A Comparison of Stemmers on Source Code Identifiers for Software SearchICSM 2011
Paper: A Comparison of Stemmers on Source Code Identifiers for Software
Search
Authors: Andrew Wiese, Valerie Ho, Emily Hill.
Session: ERA1 - Linguistic Analysis of Software Artifacts
ERA - Clustering and Recommending Collections of Code Relevant to TaskICSM 2011
Paper: Clustering and Recommending Collections of Code Relevant to Task
Authors: Seonah Lee and Sungwon Kang
Session: Early Research Achievements Track Session 3: Managing and Supporting Software Maintenance Activities
Industry - Relating Developers' Concepts and Artefact Vocabulary in a Financ...ICSM 2011
Paper: Relating Developers' Concepts and Artefact Vocabulary in a Financial
Software Module
Authors: Tezcan Dilshener and Michel Wermelinger
Session: Industry Track 2 - Reverse Engineering
Industry - Testing & Quality Assurance in Data Migration Projects ICSM 2011
Paper: Testing & Quality Assurance in Data Migration Projects
Authors: Klaus Haller, Florian Matthes, Christopher Schulz
Session: Industry Track Session 3: Evolution and migration
Natural Language Analysis - Mining Java Class Naming ConventionsICSM 2011
Paper: Mining Java Class Naming Conventions
Authors: Simon Butler, Michel Wermelinger, Yijun Yu and Helen Sharp
Session: Research Track 4 - Natural Language Analysis
Industry - Evolution and migration - Incremental and Iterative Reengineering ...ICSM 2011
Paper: Incremental and Iterative Reengineering towards Software Product Line: An Industrial Case Study
Authors: Gang Zhang, Liwei Shen, Xin Peng, Zhenchang Xing and Wenyun Zhao
Session: Industry Track Session 3: Evolution and migration
Natural Language Analysis - Expanding Identifiers to Normalize Source Code Vo...ICSM 2011
Paper: Expanding Identifiers to Normalize Source Code Vocabulary
Authors: Dave Binkley and Dawn Lawrie
Session: Research Track 4: Natural Language Analysis
Paper: Tracking Technical Debt- An Exploratory Case Study
Authors: Yuepu Guo, Carolyn Seaman, Rebeka Gomes, Antonio Cavalcanti, Graziela Tonin, Fabio Q. B. Da Silva, André L. M. Santos, Clauirton Siebra
Session: Early Research Achievement Track Session 3
ERA - Measuring Maintainability of Spreadsheets in the Wild ICSM 2011
Paper: Measuring Maintainability of Spreadsheets in the Wild
Authors: José Pedro Correia and Miguel Alexandre Ferreira
Session: Early Research Achievements Track Session 2: Software Changes and Maintainability
Faults and Regression Testing - Fault interaction and its repercussionsICSM 2011
Paper: Fault Interaction and its Repercussions
Authors: Nicholas DiGiuseppe and James A. Jones
Seesion: Research Track 1: Faults and Regression Testing
Metrics - You can't control the unfamiliarICSM 2011
Paper: You Can't Control the Unfamiliar: A Study on the Relations Between Aggregation Techniques for Software Metrics
Authors: Bogdan Vasilescu, Alexander Serebrenik and Mark Van Den Brand
Session: Research Track 11 - Metrics
Industry - Precise Detection of Un-Initialized Variables in Large, Real-life ...ICSM 2011
Paper: "Precise Detection of Un-Initialized Variables in Large, Real-life COBOL Programs in Presence of Un-realizable Paths"
Authors: Rahul Jiresal, Adnan Contractor and Ravindra Naik
Session: Industry Track Session 4: Program analysis and Verification
A Qualitative Study on Performance Bugs (MSR 2012)Bram Adams
Software performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experi- enced developers and require changes to more code than non-performance bugs. In order to be able to improve the resolution of performance bugs, a better understanding is needed of the current practice and shortcomings of reporting, reproducing, tracking and fixing performance bugs. This paper qualitatively studies a random sample of 400 performance and non-performance bug reports of Mozilla Firefox and Google Chrome across four dimensions (Impact, Context, Fix and Fix validation). We found that developers and users face problems in reproducing performance bugs and have to spend more time discussing performance bugs than other kinds of bugs. Sometimes performance regressions are tolerated as a trade- off to improve something else.
http://sail.cs.queensu.ca/publications/pubs/MSR2012_Zaman.pdf
Undo for Mobile Phones Does Your Mobile Phone Need an Undo Key? Do You?Marco Loregian
The undo function is not accessory, but still it has not been
introduced to mobile devices in an appropriate way. Undo is still
shaped to fit text editing, and it has changed a little only for
graphical editing so far. In this paper, we report the results of a
survey with which we investigated why and how to add this
functionality to regular mobile phones – not only to smartphones
or high-end handheld devices. Our respondents suggested an undo
model that is linear, sequential, with variable granularity
(according to the context of use) and requiring confirmation
before execution (both to improve awareness and avoid additional
mistakes).
Concentrated dispersion & emulsion stability and instability analysis.
Formulation and quality control of
• emulsions
• suspensions
• foams
The TurbiScan MA 2000 is designed as a formulation and a product
stability control tool. The early stage detection allows to quickly correct
formulations and to shorten ageing tests.
The kinetic analysis visualisation is the only way to document stability studies in an easy to interpret format. Providing information about the mechanisms involved in a destabilisation, the TurbiScan MA 2000 allows to fully understand these physical processes and to apply the proper correction to the formulation.
Further informations there : http://www.formulaction.com/stability-turbiscan-classic.html
Online Detection of Shutdown Periods in Chemical Plants: A Case StudyManuel Martín
In process industry, chemical processes are controlled and monitored by using readings from multiple physical sensors across the plants. Such physical sensors are also supplemented by soft sensors, i.e. adaptive predictive models, which are often used for computing hard-to-measure variables of the process. For soft sensors to work well and adapt to changing operating conditions they need to be provided with relevant data. As production plants are regularly stopped, data instances generated during shutdown periods have to be identified to avoid updating these predictive models with wrong data. We present a case study concerned with a large chemical plant operation over a 2 years period. The task is to robustly and accurately identify the shutdown periods even in case of multiple sensor failures. State-of-the-art methods were evaluated using the first half of the dataset for calibration purposes and the other half for measuring the performance. Results show that shutdowns (i.e. sudden changes) can be quickly detected in any case but the detection delay of startups (i.e. gradual changes) is directly related with the choice of a window size.
Remediation Statistics: What Does Fixing Application Vulnerabilities Cost?Denim Group
For the security industry to mature more data needs to be available about the true cost of security vulnerabilities. Data and statistics are starting to be released, but most of this currently focuses on the prevalence of different types of vulnerabilities and incidents rather than the costs of addressing the underlying issues. This session presents statistics from the remediation of 15 web-based applications in order to provide insight into the actual cost of remediating application-level vulnerabilities.
The presentation begins by setting out a structured model for software security remediation projects so that time spent on tasks can be consistently tracked. It lays out possible sources of bias in the underlying data to allow for better-informed consumption of the final analysis. Also it discusses different approaches to remediating vulnerabilities such as fixing easy vulnerabilities first versus fixing serious vulnerabilities first.
Next, historical data from the fifteen remediation projects is presented. This data consists of the average cost to remediate specific classes of vulnerabilities – cross-site scripting, SQL injection and so on – as well as the overall project composition to demonstrate the percentage of time spent on actual fixes as well as the percentages of time spent on other supporting activities such as environment setup, testing and verification and deployment. The data on the remediation of specific vulnerabilities allows for a comparison of the relative difficulty of remediating different vulnerability types. The data on the overall project composition can be used to determine the relative “efficiency” of different projects.
Finally, analysis of the data is used to create a model for estimating remediation projects so that organizations can create realistic estimates in order to make informed remediate/do not remediate decisions. In addition, characteristics of the analyzed projects are mapped to project composition to demonstrate best practices that can be used to decrease the cost of future remediation efforts.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Tutorial 2 - Practical Combinatorial (t-way) Methods for Detecting Complex Faults in Regression Testing
1. Combinatorial Testing
Rick Kuhn and Raghu Kacker
National Institute of
Standards and Technology
Gaithersburg, MD
ICSM 2011, Williamsburg, VA 26 Sept 2011
2. Enhancements to meet new
requirements can be tricky
Software maintenance
can:
• introduce new faults
• expose untested paths
• trigger unexpected
interactions
• result in loss of
previous functions
⇒ need to find problems no one thought of, perhaps
even more than with original development
⇒ combinatorial methods excel at testing for rare
events
3. Combinatorial Methods in Testing
• Goals – reduce testing cost, improve cost-benefit ratio for
software assurance
• Merge automated test generation with combinatorial methods
• New algorithms to make large-scale combinatorial testing practical
• Accomplishments – huge increase in performance, scalability
+ widespread use in real-world applications: >800 tool users; >11,000
manual/tutorial users
• Joint research with many organizations
4. What is NIST and why are we doing this?
• A US Government agency
• The nation’s measurement and testing
laboratory – 3,000 scientists, engineers,
and support staff including
3 Nobel laureates
Research in physics, chemistry,
materials, manufacturing, computer
science
Analysis of engineering failures,
including buildings, materials, and ...
5. Empirical Data on Software Failures
• We studied software failures in a variety of
fields including 15 years of FDA medical
device recall data
• What causes software failures?
• logic errors?
• calculation errors?
• interaction faults?
• inadequate input checking? Etc.
• What testing and analysis would have prevented failures?
• Would statement coverage, branch coverage, all-values, all-pairs etc.
testing find the errors?
Interaction faults: e.g., failure occurs if
pressure < 10 (1-way interaction <= all-values testing catches)
pressure<10 && volume>300 (2-way interaction <= all-pairs testing catches)
6. Software Failure Internals
• How does an interaction fault manifest itself in code?
Example: pressure < 10 & volume > 300 (2-way interaction)
if (pressure < 10) {
// do something
if (volume > 300) { faulty code! BOOM! }
else { good code, no problem}
} else {
// do something else
}
A test that included pressure = 5 and volume = 400 would
trigger this failure
12. Finally
Network security (Bell, 2006)
(orange)
Curves appear
to be similar
across a
variety of
application
domains.
Why this
distribution?
13. So, how many parameters are
involved in really tricky faults?
• The interaction rule: most failures are triggered by
one or two parameters, and progressively fewer
by three, four, or more parameters, and the maximum
interaction degree is small.
• Maximum interactions for fault triggering was 6
• More empirical work needed
• Reasonable evidence that maximum interaction
strength for fault triggering is relatively small
14. How does this knowledge help?
If all faults are triggered by the interaction of t or
fewer variables, then testing all t-way
combinations can provide strong assurance.
(taking into account: value propagation issues, equivalence
partitioning, timing issues, more complex interactions, . . . )
16. How Many Tests Would It Take?
There are 10 effects, each can be on or off
All combinations is 210 = 1,024 tests
What if our budget is too limited for these
tests?
Instead, let’s look at all 3-way interactions …
17. Now How Many Would It Take?
There are = 120 3-way interactions.
10
3
3 binary variables = 8 values: 000, 001, 010…
This is 120 x 23 = 960 possible combinations
Since we can pack 3 triples into each test, we
need no more than 320 tests.
BUT, each test exercises many triples:
0 1 1 0 0 0 0 1 1 0
We can pack a lot into one test, so what’s the
smallest number of tests we need?
18. Covering Arrays
Ex: all triples in only 13 tests, 10
3 23 = 960 combinations
One column per
One row per test: parameter:
For any 3 parameters, every combination occurs at
least once in the array
19. Cost and Volume of Tests
Number of tests: proportional to vt log n
for v values, n variables, t-way interactions
•
• Thus:
•Tests increase exponentially with interaction strength t : BAD,
but unavoidable
•But logarithmically with the number of parameters : GOOD!
Example: suppose we want all 4-way combinations of n
parameters, 5 values each:
•
5000
4500
4000
3500
3000
2500 Tests
2000
1500
1000
500
0
10 20 30 40 50
Variables
20. Covering Arrays – key points
• Not the same as orthogonal arrays
• OAs – all combinations covered same
number of times
• CAs – all combinations covered at least
once
• Developed 1990s as a theoretical construct
• Extends Design of Experiments concept
• NP hard problem but good algorithms now
More on these issues in Part II of the tutorial
21. Two ways of using combinatorial testing:
inputs or configurations (or both)
1. Inputs
parameter values,
=> discretized, small number of values for each input
account_balance: 0, 999, MAXINT, …
state: “VA”, “NC”, “MD”, “PA”, …
parameter properties,
=> may have multiple properties per parameter
state_sales_tax: 0, 3.0 to 5.0, 5.0 to 6.0, >6.0
state_CONUS: Y or N
state_distance: 0 to 100, 101 to 500, etc.
=> map combinations of properties to input values
=> often require constraints
2. Configurations, such as max file size, number of input buffers,
comm protocol
22. Combinations of inputs
Suppose we have a system with on-off switches, with
switch settings as inputs to control software:
23. How do we test this?
34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests
24. What if we knew no failure involves more than
3 switch settings interacting?
• 34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests
• If only 3-way interactions, need only 33 tests
• For 4-way interactions, need only 85 tests
27. Configurations to Test
Degree of interaction coverage: 2
Number of parameters: 5
Maximum number of values per parameter: 3
Number of configurations: 10
-------------------------------------
Configuration #1:
1 = OS=XP t # Configs % of Exhaustive
2 = Browser=IE
3 = Protocol=IPv4 2 10 14
4 = CPU=Intel
3 18 25
5 = DBMS=MySQL
------------------------------------- 4 36 50
Configuration #2:
1 = OS=XP 5 72 100
2 = Browser=Firefox
3 = Protocol=IPv6
4 = CPU=AMD
5 = DBMS=Sybase
-------------------------------------
Configuration #3:
1 = OS=XP
2 = Browser=IE
3 = Protocol=IPv6
4 = CPU=Intel
5 = DBMS=Oracle
. . . etc.
28. The Test Oracle Problem
• Creating test data is the easy part!
• How do we check that the code worked correctly
on the test input?
• Crash testing server or other code to ensure it does not
crash for any test input (like ‘fuzz testing’)
- Easy but limited value
• Built-in self test with embedded assertions – incorporate
assertions in code to check critical states at different points in
the code, or print out important values during execution
• Full scale model-checking using mathematical model of
system and model checker to generate expected results for
each input - expensive but tractable
29. Available Tools
• Covering array generator – basic tool for test input or
configurations;
• Sequence covering array generator – new concept; applies
combinatorial methods to event sequence testing
• Combinatorial coverage measurement – detailed analysis of
combination coverage; automated generation of supplemental
tests; helpful for integrating c/t with existing test methods
• Domain/application specific tools:
• Access control policy tester
• .NET config file generator
35. Algorithm comparison
• Smaller test sets faster, with a more advanced user interface
• First parallelized covering array algorithm
• More information per test
IPOG ITCH (IBM) Jenny (Open Source) TConfig (U. of Ottawa) TVG (Open Source)
T-Way
Size Time Size Time Size Time Size Time Size Time
2 100 0.8 120 0.73 108 0.001 108 >1 hour 101 2.75
3 400 0.36 2388 1020 413 0.71 472 >12 hour 9158 3.07
4 1363 3.05 1484 5400 1536 3.54 1476 >21 hour 64696 127
>1
5 4226 18s NA day
4580 43.54 NA >1 day 313056 1549
6 10941 65.03 NA >1 day 11625 470 NA >1 day 1070048 12600
Traffic Collision Avoidance System (TCAS): 273241102
Times in seconds
37. Testing Smartphone Configurations
Options: keyboard, screen orientation, screen size, number of
screens, navigation method, and on, and on, and on . . .
What if we built the app
for this:
… and we have to make
sure it works on these
38. Testing Smartphone Configurations
Android configuration options:
int HARDKEYBOARDHIDDEN_NO; int ORIENTATION_LANDSCAPE;
int HARDKEYBOARDHIDDEN_UNDEFINED; int ORIENTATION_PORTRAIT;
int HARDKEYBOARDHIDDEN_YES; int ORIENTATION_SQUARE;
int KEYBOARDHIDDEN_NO; int ORIENTATION_UNDEFINED;
int KEYBOARDHIDDEN_UNDEFINED; int SCREENLAYOUT_LONG_MASK;
int KEYBOARDHIDDEN_YES; int SCREENLAYOUT_LONG_NO;
int KEYBOARD_12KEY; int SCREENLAYOUT_LONG_UNDEFINED;
int KEYBOARD_NOKEYS; int SCREENLAYOUT_LONG_YES;
int KEYBOARD_QWERTY; int SCREENLAYOUT_SIZE_LARGE;
int KEYBOARD_UNDEFINED; int SCREENLAYOUT_SIZE_MASK;
int NAVIGATIONHIDDEN_NO; int SCREENLAYOUT_SIZE_NORMAL;
int NAVIGATIONHIDDEN_UNDEFINED; int SCREENLAYOUT_SIZE_SMALL;
int NAVIGATIONHIDDEN_YES; int SCREENLAYOUT_SIZE_UNDEFINED;
int NAVIGATION_DPAD; int TOUCHSCREEN_FINGER;
int NAVIGATION_NONAV; int TOUCHSCREEN_NOTOUCH;
int NAVIGATION_TRACKBALL; int TOUCHSCREEN_STYLUS;
int NAVIGATION_UNDEFINED; int TOUCHSCREEN_UNDEFINED;
int NAVIGATION_WHEEL;
39. Configuration option values
Parameter Name Values # Values
HARDKEYBOARDHIDDEN NO, UNDEFINED, YES 3
KEYBOARDHIDDEN NO, UNDEFINED, YES 3
KEYBOARD 12KEY, NOKEYS, QWERTY, UNDEFINED 4
NAVIGATIONHIDDEN NO, UNDEFINED, YES 3
NAVIGATION DPAD, NONAV, TRACKBALL, UNDEFINED, 5
WHEEL
ORIENTATION LANDSCAPE, PORTRAIT, SQUARE, UNDEFINED 4
SCREENLAYOUT_LONG MASK, NO, UNDEFINED, YES 4
SCREENLAYOUT_SIZE LARGE, MASK, NORMAL, SMALL, UNDEFINED 5
TOUCHSCREEN FINGER, NOTOUCH, STYLUS, UNDEFINED 4
Total possible configurations:
3 x 3 x 4 x 3 x 5 x 4 x 4 x 5 x 4 = 172,800
40. Number of configurations generated
t # Configs % of Exhaustive
2 29 0.02
3 137 0.08
4 625 0.4
5 2532 1.5
6 9168 5.3
48. Example: Input Testing for Pizza Orders
6x217x217x217x4x3x2x2x5x2
= WAY TOO MUCH TO TEST
Simplified pizza ordering:
6x4x4x4x4x3x2x2x5x2
= 184,320 possibilities
49. Ordering Pizza Combinatorially
Simplified pizza ordering:
6x4x4x4x4x3x2x2x5x2
= 184,320 possibilities
2-way tests: 32
3-way tests: 150
4-way tests: 570
5-way tests: 2,413
6-way tests: 8,330
If all failures involve 5 or fewer
parameters, then we can have
confidence after running all 5-way
tests.
50. Case study 1: Traffic
Collision Avoidance
System (TCAS) module
• Used in previous testing research
• 41 versions seeded with errors
• 12 variables: 7 boolean, two 3-value, one 4-
value, two 10-value
• All flaws found with 5-way coverage
• Thousands of tests - generated by model
checker in a few minutes
52. Results
• Roughly consistent with data on large systems
• But errors harder to detect than real-world examples
Detection Rate for TCAS Seeded Tests per error
Errors
350.0
100% 300.0
80% 250.0
200.0
Tests
60% Tests per error
Detection
150.0
40% rate
100.0
20%
50.0
0% 0.0
2 way 3 way 4 way 5 way 6 way 2 way 3 way 4 way 5 way 6 way
Fault Interaction level Fault Interaction level
Bottom line for model checking based combinatorial testing:
Expensive but can be highly effective
53. Case study 2:
Document Object Model Events
• DOM is a World Wide Web Consortium standard
incorporated into web browsers
• NIST Systems and Software division develops tests for
standards such as DOM
• DOM testing problem:
• large number of events handled by separate
functions
• functions have 3 to 15 parameters
• parameters have many, often continuous, values
• verification requires human interaction (viewing
screen)
• testing takes a long time
55. World Wide Web Consortium
Document Object Model Events
Test Results
% of
t Tests Not
Orig. Pass Fail
Run
2 702 1.92% 202 27 473
3 1342 3.67% 786 27 529
4 1818 4.96% 437 72 1309
5 2742 7.49% 908 72 1762
11.54
6 4227 1803 72 2352
%
All failures found using < 5% of
original exhaustive test set
56. Buffer Overflows
• Empirical data from the National Vulnerability Database
• Investigated > 3,000 denial-of-service vulnerabilities reported in
the NIST NVD for period of 10/06 – 3/07
• Vulnerabilities triggered by:
• Single variable – 94.7%
example: Heap-based buffer overflow in the SFTP protocol
handler for Panic Transmit … allows remote attackers to execute
arbitrary code via a long ftps:// URL.
• 2-way interaction – 4.9%
example: single character search string in conjunction with a single
character replacement string, which causes an "off by one
overflow"
• 3-way interaction – 0.4%
example: Directory traversal vulnerability when register_globals is
enabled and magic_quotes is disabled
and .. (dot dot) in the page parameter
63. Case study 3: Modeling & Simulation
• “Simured” network simulator
• Kernel of ~ 5,000 lines of C++ (not including GUI)
• Objective: detect configurations that can
produce deadlock:
• Prevent connectivity loss when changing network
• Attacks that could lock up network
• Compare effectiveness of random vs.
combinatorial inputs
• Deadlock combinations discovered
• Crashes in >6% of tests w/ valid values (Win32
version only)
64. Simulation Input Parameters
Parameter Values 5x3x4x4x4x4x2x2
1 DIMENSIONS 1,2,4,6,8 x2x4x4x4x4x4
2 NODOSDIM 2,4,6 = 31,457,280
3 NUMVIRT 1,2,3,8 configurations
4 NUMVIRTINJ 1,2,3,8
5 NUMVIRTEJE 1,2,3,8 Are any of them
6 LONBUFFER 1,2,4,6 dangerous?
7 NUMDIR 1,2
8 FORWARDING 0,1 If so, how many?
9 PHYSICAL true, false
Which ones?
10 ROUTING 0,1,2,3
11 DELFIFO 1,2,4,6
12 DELCROSS 1,2,4,6
13 DELCHANNEL 1,2,4,6
14 DELSWITCH 1,2,4,6
66. Network Deadlock Detection
Detected 14 configurations that can cause deadlock:
14/ 31,457,280 = 4.4 x 10-7
Combinatorial testing found more deadlocks than
random, including some that might never have been
found with random testing
Why do this testing? Risks:
• accidental deadlock configuration: low
• deadlock config discovered by attacker: much higher
(because they are looking for it)
70. How to automate checking
correctness of output
• Creating test data is the easy part!
• How do we check that the code worked correctly
on the test input?
• Crash testing server or other code to ensure it does not crash for
any test input (like ‘fuzz testing’)
- Easy but limited value
• Built-in self test with embedded assertions – incorporate
assertions in code to check critical states at different points in the
code, or print out important values during execution
• Full scale model-checking using mathematical model of system
and model checker to generate expected results for each input
- expensive but tractable
71. Crash Testing
• Like “fuzz testing” - send packets or other input
to application, watch for crashes
• Unlike fuzz testing, input is non-random;
cover all t-way combinations
• May be more efficient - random input generation
requires several times as many tests to cover the
t-way combinations in a covering array
Limited utility, but can detect
high-risk problems such as:
- buffer overflows
- server crashes
72. Built-in Self Test
through Embedded Assertions
Simple example:
assert( x != 0); // ensure divisor is not zero
Or pre and post-conditions:
/requires amount >= 0;
/ensures balance == old(balance) - amount &&
result == balance;
73. Built-in Self Test
Assertions check properties of expected result:
ensures balance == old(balance) - amount
&& result == balance;
•Reasonable assurance that code works correctly across
the range of expected inputs
•May identify problems with handling unanticipated inputs
•Example: Smart card testing
• Used Java Modeling Language (JML) assertions
• Detected 80% to 90% of flaws
74. Using model checking to produce tests
The system can never
Yes it can, and
get in this state! here’s how …
Model-checker test
production:
if assertion is not true,
then a counterexample
is generated.
This can be
converted to a test
case.
Black & Ammann, 1999
75. Model checking example
-- specification for a portion of tcas - altitude separation.
-- The corresponding C code is originally from Siemens Corp.
MODULE main
VAR
Cur_Vertical_Sep : { 299, 300, 601 };
High_Confidence : boolean;
...
init(alt_sep) := START_;
next(alt_sep) := case
enabled & (intent_not_known | !tcas_equipped) : case
need_upward_RA & need_downward_RA : UNRESOLVED;
need_upward_RA : UPWARD_RA;
need_downward_RA : DOWNWARD_RA;
1 : UNRESOLVED;
esac;
1 : UNRESOLVED;
esac;
...
SPEC AG ((enabled & (intent_not_known | !tcas_equipped) &
!need_downward_RA & need_upward_RA) -> AX (alt_sep = UPWARD_RA))
-- “FOR ALL executions,
-- IF enabled & (intent_not_known ....
-- THEN in the next state alt_sep = UPWARD_RA”
76. Computation Tree Logic
The usual logic operators,plus temporal:
A φ - All: φ holds on all paths starting from the
current state.
E φ - Exists: φ holds on some paths starting from
the current state.
G φ - Globally: φ has to hold on the entire
subsequent path.
F φ - Finally: φ eventually has to hold
X φ - Next: φ has to hold at the next state
[others not listed]
execution paths
states on the execution paths
SPEC AG ((enabled & (intent_not_known |
!tcas_equipped) & !need_downward_RA & need_upward_RA)
-> AX (alt_sep = UPWARD_RA))
“FOR ALL executions,
IF enabled & (intent_not_known ....
THEN in the next state alt_sep = UPWARD_RA”
77. What is the most effective way to integrate
combinatorial testing with model checking?
• Given AG(P -> AX(R))
“for all paths, in every state,
if P then in the next state, R holds”
• For k-way variable combinations, v1 & v2 & ... & vk
• vi abbreviates “var1 = val1”
• Now combine this constraint with assertion to produce
counterexamples. Some possibilities:
1. AG(v1 & v2 & ... & vk & P -> AX !(R))
2. AG(v1 & v2 & ... & vk -> AX !(1))
3. AG(v1 & v2 & ... & vk -> AX !(R))
78. What happens with these assertions?
1. AG(v1 & v2 & ... & vk & P -> AX !(R))
P may have a negation of one of the vi, so we get
0 -> AX !(R))
always true, so no counterexample, no test.
This is too restrictive!
1. AG(v1 & v2 & ... & vk -> AX !(1))
The model checker makes non-deterministic choices for
variables not in v1..vk, so all R values may not be covered
by a counterexample.
This is too loose!
2. AG(v1 & v2 & ... & vk -> AX !(R))
Forces production of a counterexample for each R.
This is just right!
79. All faults found
with 5-way tests
Detection Rate for TCAS Seeded Tradeoff of up-front
Errors spec creation time
vs. only a few
100% minutes to run
tests
80%
60%
Detection
40% rate
20%
0%
2 way 3 way 4 way 5 way 6 way
Fault Interaction level
81. Combinatorial Coverage Measurement
Tests Variables Variable Variable-value Coverage
pairs combinations
a b c d covered
1 0 0 0 0 ab 00, 01, 10 .75
2 0 1 1 0 ac 00, 01, 10 .75
3 1 0 0 1 ad 00, 01, 11 .75
4 0 1 1 1 bc 00, 11 .50
bd 00, 01, 10, 11 1.0
cd 00, 01, 10, 11 1.0
82. Graphing Coverage Measurement
Bottom line:
100% coverage of 33% of
All combinations
combinations
75% coverage of half of covered to at least
50%
combinations
50% coverage of 16% of
combinations
91. Combinatorial Sequence Testing
•We want to see if a system works correctly regardless of
the order of events. How can this be done efficiently?
• Failure reports often say something like:
'failure occurred when A started if B is not already connected'.
• Can we produce compact tests such that all t-way sequences
covered (possibly with interleaving events)?
Event Description
a connect flow meter
b connect pressure gauge
c connect satellite link
d connect pressure readout
e start comm link
f boot system
92. Sequence Covering Array
• With 6 events, all sequences = 6! = 720 tests
• Only 10 tests needed for all 3-way sequences,
results even better for larger numbers of events
• Example: .*c.*f.*b.* covered. Any such 3-way seq covered.
Test Sequence
1 a b c d e f
2 f e d c b a
3 d e f a b c
4 c b a f e d
5 b f a d c e
6 e c d a f b
7 a e f c b d
8 d b c f e a
9 c e a d b f
10 f b d a e c
93. Sequence Covering Array Properties
• 2-way sequences require only 2 tests
(write events in any order, then reverse)
• For > 2-way, number of tests grows with log n, for n events
• Simple greedy algorithm produces compact test set
300
250
200
2-way
Tests 150
3-way
4-way
100
50
0
5 10 20 30 40 50 60 70 80
Number of events
94. Case study 4: Laptop application
Problem: connect many
peripherals, order of
connection may affect
application
96. Results
• Tested peripheral connection for 3-way
sequences
• Some faults detected that would not have
been found with 2-way sequence testing; may
not have been found with random
• Example:
• If P2-P1-P3 sequence triggers a failure,
then a full 2-way sequence covering array
would not have found it
(because 1-2-3-4-5-6-7 and 7-6-5-4-3-2-1
is a 2-way sequence covering array)
98. Fault location
Given: a set of tests that the SUT fails, which
combinations of variables/values triggered the failure?
variable/value combinations
in passing tests
These are the ones we
want
variable/value combinations in
failing tests
99. Fault location – what's the problem?
If they're in failing set but not in
passing set:
1. which ones triggered the failure?
2. which ones don't matter?
( ) combinations
t n
out of v t
Example:
30 variables, 5 values each
= 445,331,250
5-way combinations
142,506 combinations
in each test
100. Integrating into Testing Program
• Test suite development
• Generate covering arrays for tests
OR
• Measure coverage of existing tests
and supplement
• Training
• Testing textbooks – Mathur,
Ammann & Offutt,
• Combinatorial testing “textbook”
on ACTS site
• User manuals
• Worked examples
101. Industrial Usage Reports
• Work with US Air Force on sequence covering arrays, submitted
for publication
• WWW Consortium DOM Level 3 events conformance test suite
• Cooperative Research & Development Agreement
with Lockheed Martin Aerospace - report to be released 3rd or 4th
quarter 2011
• New projects with NASA IV&V facility
102. Technology Transfer
Tools obtained by 800+ organizations;
NIST “textbook” on combinatorial testing
downloaded 11,000+ times since Oct. 2010
Collaborations: USAF 46th Test Wing, Lockheed
Martin, George Mason Univ., Univ. of Maryland
Baltimore County, Johns Hopkins Univ. Applied
Physics Lab, Carnegie Mellon Univ., NASA IV&V
We are always interested in working with others!
103. Please contact us if you
would like more
information.
Rick Kuhn Raghu Kacker
kuhn@nist.gov raghu.kacker@nist.gov
http://csrc.nist.gov/acts