Invited Talk, ISSTA 2nd International Workshop on End-to-end Test Script Engineering
July 16, 2012, Minneapolis.
Limitations of x-unit testing framework, MTS testing framework that combined test objects with procedural aspects of TTCN.
In this session, Marc Hornbeek of Spirent Communications will explain the "change-driven" method and give an overview of associated tools and actual experiences for automated, intelligent software build verification. Learn how this method enables higher test coverage during continuously varying build schedules and priorities, creating a closed-loop continuous build/test environment that is vastly more efficient than the traditional regression approaches that often lead to bottlenecks in continuous change and Agile environments.
Unit testingandcontinousintegrationfreenest1dot4JAMK
This document discusses unit testing, integration testing, and continuous integration. It provides information on different levels of testing including unit/module/component testing, integration testing, system testing, and acceptance testing. It also describes techniques for unit testing like test-driven development. Continuous integration is introduced as running automated tests and building software with each code commit. Static and dynamic code analysis, code coverage, and using stubs/mocks for integration testing are also summarized.
Quality assurance in distributed continuous deliverymingjin
The document discusses quality assurance for distributed continuous delivery. It addresses challenges like testing across legacy and distributed systems. It promotes a shared responsibility between QA and developers for quality. It advocates building quality in through techniques like automated testing, functional testing at boundaries, and validating deployments. It also stresses the importance of communication, collaboration, and co-education between QA and developers.
- Author: Kuan-Yu Liao, Chia-Yuan Chang, and James Chien-Mo Li, National Taiwan University
- Publication: IEEE Transaction on Computer-Aided Design of Integrated Circuits and Systems, 2011
The document provides an introduction and overview of mutation testing. It explains that mutation testing involves deliberately introducing small errors or mutations into code to test if tests can detect those errors. It discusses how mutations are made by changing operators or statements, running tests on the mutated code, and analyzing whether tests fail as they should to kill the mutations. The document uses an example of mutating a subtraction method to demonstrate how mutation testing works and what it reveals about test coverage and adequacy.
The document describes the Test Authoring Tool (TAO) process extension, which provides a way to organize and manage the collaborative activities involved in large-scale computer-based assessment projects beyond just test execution and content creation. TAO extends these activities by defining workflows and integrating customizable web services. It allows users to design processes by adding activities, services, and logic to control workflow transitions. This provides a structured approach for organizing all the components of an assessment project.
A Survey on Automatic Test Generation and Crash ReproductionSung Kim
The document provides an outline for a PhD qualifying exam on automatic test generation and crash reproduction. It discusses various approaches to automatic test generation including feedback-directed random generation, captured-object based generation, genetic evolution based generation, symbolic execution, and dynamic symbolic execution. It also covers approaches to automatic crash reproduction including record-and-replay and static analysis based approaches. The document identifies common challenges faced by these techniques such as the object creation problem in automatic test generation.
The document discusses test driven development (TDD) and the TDD rhythm of adding a test, running the test, making the test pass through coding, and refactoring code. It provides diagrams showing the TDD rhythm and process. It also discusses acceptance criteria and tests, different types of tests, test automation, and the benefits of TDD such as more testable and loosely coupled code, increased confidence, and easier maintenance through refactoring.
In this session, Marc Hornbeek of Spirent Communications will explain the "change-driven" method and give an overview of associated tools and actual experiences for automated, intelligent software build verification. Learn how this method enables higher test coverage during continuously varying build schedules and priorities, creating a closed-loop continuous build/test environment that is vastly more efficient than the traditional regression approaches that often lead to bottlenecks in continuous change and Agile environments.
Unit testingandcontinousintegrationfreenest1dot4JAMK
This document discusses unit testing, integration testing, and continuous integration. It provides information on different levels of testing including unit/module/component testing, integration testing, system testing, and acceptance testing. It also describes techniques for unit testing like test-driven development. Continuous integration is introduced as running automated tests and building software with each code commit. Static and dynamic code analysis, code coverage, and using stubs/mocks for integration testing are also summarized.
Quality assurance in distributed continuous deliverymingjin
The document discusses quality assurance for distributed continuous delivery. It addresses challenges like testing across legacy and distributed systems. It promotes a shared responsibility between QA and developers for quality. It advocates building quality in through techniques like automated testing, functional testing at boundaries, and validating deployments. It also stresses the importance of communication, collaboration, and co-education between QA and developers.
- Author: Kuan-Yu Liao, Chia-Yuan Chang, and James Chien-Mo Li, National Taiwan University
- Publication: IEEE Transaction on Computer-Aided Design of Integrated Circuits and Systems, 2011
The document provides an introduction and overview of mutation testing. It explains that mutation testing involves deliberately introducing small errors or mutations into code to test if tests can detect those errors. It discusses how mutations are made by changing operators or statements, running tests on the mutated code, and analyzing whether tests fail as they should to kill the mutations. The document uses an example of mutating a subtraction method to demonstrate how mutation testing works and what it reveals about test coverage and adequacy.
The document describes the Test Authoring Tool (TAO) process extension, which provides a way to organize and manage the collaborative activities involved in large-scale computer-based assessment projects beyond just test execution and content creation. TAO extends these activities by defining workflows and integrating customizable web services. It allows users to design processes by adding activities, services, and logic to control workflow transitions. This provides a structured approach for organizing all the components of an assessment project.
A Survey on Automatic Test Generation and Crash ReproductionSung Kim
The document provides an outline for a PhD qualifying exam on automatic test generation and crash reproduction. It discusses various approaches to automatic test generation including feedback-directed random generation, captured-object based generation, genetic evolution based generation, symbolic execution, and dynamic symbolic execution. It also covers approaches to automatic crash reproduction including record-and-replay and static analysis based approaches. The document identifies common challenges faced by these techniques such as the object creation problem in automatic test generation.
The document discusses test driven development (TDD) and the TDD rhythm of adding a test, running the test, making the test pass through coding, and refactoring code. It provides diagrams showing the TDD rhythm and process. It also discusses acceptance criteria and tests, different types of tests, test automation, and the benefits of TDD such as more testable and loosely coupled code, increased confidence, and easier maintenance through refactoring.
Lessons learned validating 60,000 pages of api documentationBob Binder
The document discusses lessons learned from validating over 60,000 pages of API documentation at Microsoft. It provides an overview of Microsoft's protocol quality assurance process, which included developing model-based test suites to validate technical documentation against actual Windows services. Key aspects of the process included requirements engineering to derive testable requirements from documentation statements, modeling protocol behavior, and using the Spec Explorer tool to automatically generate and execute test cases from the models. The process uncovered over 50,000 issues in the documentation, most before test execution, and helped close an antitrust case regarding Microsoft's interoperability documentation.
Popular Delusions, Crowds, and the Coming Deluge: end of the Oracle?Bob Binder
Invited Talk at the 20th CREST Open Workshop, The Oracle Problem for Automated Software Testing. University College of London. May 21, 2012
Pragmatic Innovations for test oracles, a new Oracle Taxonomy, Characterization of test oracles, Challenges.
Keynote, ETSI Model-Based Testing User Conference. Tallinn, Estonia September 27, 2012.
High-level discussion of model-based testing and trends driving software/system reliability. Explains how emergent behavior in complex systems ("dragon kings") causes catastrophic failures. My Multi-dimensional testing strategy can reveal this hard to find bugs/failure modes, but this requires a better approach to model-based testing. Overview: Is software eating the world? Bugs, Black Swans, Dragon Kings. Multi-dimensional Testing. Challenges.
How to Release Rock-solid RESTful APIs and Ice the Testing BackBlobBob Binder
REST APIs are a key enabling technology for the cloud. Mobile applications, service-oriented architecture, and the Internet of Things depend on reliable and usable REST APIs. Unlike browser, native, and mobile apps, REST APIs can only be tested with software that drives the APIs. Unlike developer-centric hand-coded unit testing, adequate testing of REST APIs is truly well-suited to advanced automated testing.
As most web service applications are developed following an Agile process, effective testing must also avoid the "testing backblob," in which work to maintain hand-coded BDD-style test suites exceeds available time after a few iterations.
This talk presents a methodology for developing and testing REST APIs using model-based automation that has the beneficial side-effect of shrinking the testing backblob.
Software Test Patterns: Successes and ChallengesBob Binder
This document discusses the successes and challenges of using test patterns over the past 10 years. It describes how test patterns were useful for articulating testing insights and practices, but have not been widely adopted. Reasons for limited adoption include the proliferation of templates, confusion between different pattern types, and the perception that using patterns requires too much additional modeling effort. The document also suggests that while innovators create new patterns, those seeking existing patterns may be less influential. It argues that test patterns will remain important for building a conceptual framework for testing and efficiently sharing solutions, especially as software systems increase in complexity.
Model-based Testing: Taking BDD/ATDD to the Next LevelBob Binder
Slides from presentation at the Chicago Quality Assurance Association, February 25, 2014.
Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) are well-established Agile practices that rely on the knowledge and intuition of testers, product owners, and developers to identify and then translate statements into test suites. But the resulting test suites often cover only a small slice of happy-path behavior. And, as a BDD specification and its associated test code base grows over time, work to maintain it either crowds out new development and testing or, typically, is simply ignored. Either is high-risk. That’s how Agile teams get eaten by the testing BackBlob.Model Based Testing is a tool-based approach to automate the creation of test cases. This presentation will outline the techniques and benefits of MBT, and show how model-based testing can address both problems. A detailed demo of Spec Explorer, a free model-based testing tool shows how a model is constructed and used to create and maintain a test suite.
Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.
However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.
Design Patterns play a big role in helping achieve this goal of building a good and robust framework.
In this talk, we will talk about, and see examples of various types of patterns you can use for:
Build your Test Automation Framework
Test Data Management
Locators / IDs (for finding / interacting with elements in the browser / app)
Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!
The document discusses mVerify's Test Objects framework for automated software testing. It was presented at the 2006 Google Test Automation Conference. The framework was influenced by TTCN-3 and XUnit testing frameworks and aims to generate test objects from models, support distributed testing across platforms, and make testing intuitive through one-click repetition and smart progress bars. A demo was presented to illustrate these capabilities.
Oplægget blev holdt ved et seminar i InfinIT-interessegruppen Softwaretest den 28. september 2010.
Læs mere om interessegruppen på http://www.infinit.dk/dk/interessegrupper/softwaretest/softwaretest.htm
The document provides an overview of QuickTest Professional (QTP), including:
1. QTP is a tool used for functional and regression testing of applications. It supports testing of various software using built-in and external add-ins.
2. The object repository in QTP stores object identification properties that are used to recognize and interact with objects during testing.
3. Tests in QTP can include multiple actions, which can be arranged hierarchically. Data tables are used to support data-driven testing by storing test and action iteration data.
TMPA-2017: Regression Testing with Semiautomatic Test Selection for Auditing ...Iosif Itkin
TMPA-2017: Tools and Methods of Program Analysis
3-4 March, 2017, Hotel Holiday Inn Moscow Vinogradovo, Moscow
Regression Testing with Semiautomatic Test Selection for Auditing of IMS Database
Alexey Ruchay, Ivan Kliavin, Tatiana Kotova, Julia Ivanova, Applied Technologies
Would like to know more?
Visit our website:
www.tmpaconf.org
www.exactprosystems.com/events/tmpa
Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro
San Jose Selenium Meet-up PushToTest TestMaker PresentationClever Moe
With the Selenium project team's focus on making the WebDriver APIs a W3C standard, this is a good time to talk about where Selenium is going and the support it is getting from commercial open source companies. Frank Cohen, CEO at PushToTest, will discuss Selenium tools - record/playback utilities, deploying Selenium scripts to the Cloud, results analysis tools to surface functional issues and performance bottlenecks, and operational test database repositories - needed to be productive and successful with Selenium.
The document provides an overview of mVerify Corporation and its mobile testing solution called MTS. MTS aims to address the challenges of testing mobile applications by providing a platform that can simulate millions of users and configurations to thoroughly test apps. The solution slashes testing time and costs while improving reliability and performance. mVerify has seen early traction and seeks funding to further develop the platform and expand its customer base and sales.
JMeter - Performance testing your webappAmit Solanki
JMeter is an open-source tool for performance and load testing web applications. It can test applications by simulating heavy loads to determine stability and identify performance bottlenecks. JMeter simulates multiple users accessing web services concurrently to model expected usage during peak periods. It provides instant visual feedback and allows tests to be run interactively or in batch mode for later analysis. Tests are composed of thread groups, samplers, timers, listeners and other elements to control test flow and capture response data. JMeter also supports distributed testing across multiple servers to simulate very large loads.
The document provides an overview of automation testing using Quick Test Professional (QTP). It discusses the need for automation testing, rules of automation, and the different automation tools available. It then covers the life cycle of automation testing using QTP, including analyzing the application, selecting a tool, designing and recording test scripts, running tests, and finding and reporting defects. The document also describes key QTP concepts like object identification, the object repository, recording modes, parameters, reusable actions, and checkpoints.
The document describes a proposed smart debugger framework. It would help students debug programs by finding bugs and managing test cases. It would also help programmers manage code and test automation. Additionally, the framework would aid researchers by visualizing experiment results to generate new hypotheses. The framework uses data flows to collect coverage information from running test cases on programs to compute metrics and rank covered entities for students, programmers or researchers.
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on Agile Test Design And Automation of a Life-Critical Medical Device by Thomas Kauders. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
Gatling is Open Source Stress testing tool.
Why Gatling:
- High Performance.
- Multi Threading vs (Akka) Actor Model.
- Synchronous Blocking IOs vs asynchronous Non-blocking IOs Netty.
Bibhas Kumar Senapati has over 6 years of experience in IT and has worked as a test engineer, junior test analyst, senior test analyst, and automation test lead. He has expertise in technologies like Windows, Unix, Linux, MySQL, Oracle, SQL Server, QTP, WinRunner, Test Director, Selenium, and LoadRunner. He holds an engineering degree and certifications in software testing and domains like forex, trade finance, and investment banking. His experience includes automation testing roles for projects with Morgan Stanley, Standard Bank of South Africa, UBS, Goldman Sachs, and Citi.
Lessons learned validating 60,000 pages of api documentationBob Binder
The document discusses lessons learned from validating over 60,000 pages of API documentation at Microsoft. It provides an overview of Microsoft's protocol quality assurance process, which included developing model-based test suites to validate technical documentation against actual Windows services. Key aspects of the process included requirements engineering to derive testable requirements from documentation statements, modeling protocol behavior, and using the Spec Explorer tool to automatically generate and execute test cases from the models. The process uncovered over 50,000 issues in the documentation, most before test execution, and helped close an antitrust case regarding Microsoft's interoperability documentation.
Popular Delusions, Crowds, and the Coming Deluge: end of the Oracle?Bob Binder
Invited Talk at the 20th CREST Open Workshop, The Oracle Problem for Automated Software Testing. University College of London. May 21, 2012
Pragmatic Innovations for test oracles, a new Oracle Taxonomy, Characterization of test oracles, Challenges.
Keynote, ETSI Model-Based Testing User Conference. Tallinn, Estonia September 27, 2012.
High-level discussion of model-based testing and trends driving software/system reliability. Explains how emergent behavior in complex systems ("dragon kings") causes catastrophic failures. My Multi-dimensional testing strategy can reveal this hard to find bugs/failure modes, but this requires a better approach to model-based testing. Overview: Is software eating the world? Bugs, Black Swans, Dragon Kings. Multi-dimensional Testing. Challenges.
How to Release Rock-solid RESTful APIs and Ice the Testing BackBlobBob Binder
REST APIs are a key enabling technology for the cloud. Mobile applications, service-oriented architecture, and the Internet of Things depend on reliable and usable REST APIs. Unlike browser, native, and mobile apps, REST APIs can only be tested with software that drives the APIs. Unlike developer-centric hand-coded unit testing, adequate testing of REST APIs is truly well-suited to advanced automated testing.
As most web service applications are developed following an Agile process, effective testing must also avoid the "testing backblob," in which work to maintain hand-coded BDD-style test suites exceeds available time after a few iterations.
This talk presents a methodology for developing and testing REST APIs using model-based automation that has the beneficial side-effect of shrinking the testing backblob.
Software Test Patterns: Successes and ChallengesBob Binder
This document discusses the successes and challenges of using test patterns over the past 10 years. It describes how test patterns were useful for articulating testing insights and practices, but have not been widely adopted. Reasons for limited adoption include the proliferation of templates, confusion between different pattern types, and the perception that using patterns requires too much additional modeling effort. The document also suggests that while innovators create new patterns, those seeking existing patterns may be less influential. It argues that test patterns will remain important for building a conceptual framework for testing and efficiently sharing solutions, especially as software systems increase in complexity.
Model-based Testing: Taking BDD/ATDD to the Next LevelBob Binder
Slides from presentation at the Chicago Quality Assurance Association, February 25, 2014.
Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) are well-established Agile practices that rely on the knowledge and intuition of testers, product owners, and developers to identify and then translate statements into test suites. But the resulting test suites often cover only a small slice of happy-path behavior. And, as a BDD specification and its associated test code base grows over time, work to maintain it either crowds out new development and testing or, typically, is simply ignored. Either is high-risk. That’s how Agile teams get eaten by the testing BackBlob.Model Based Testing is a tool-based approach to automate the creation of test cases. This presentation will outline the techniques and benefits of MBT, and show how model-based testing can address both problems. A detailed demo of Spec Explorer, a free model-based testing tool shows how a model is constructed and used to create and maintain a test suite.
Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.
However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.
Design Patterns play a big role in helping achieve this goal of building a good and robust framework.
In this talk, we will talk about, and see examples of various types of patterns you can use for:
Build your Test Automation Framework
Test Data Management
Locators / IDs (for finding / interacting with elements in the browser / app)
Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!
The document discusses mVerify's Test Objects framework for automated software testing. It was presented at the 2006 Google Test Automation Conference. The framework was influenced by TTCN-3 and XUnit testing frameworks and aims to generate test objects from models, support distributed testing across platforms, and make testing intuitive through one-click repetition and smart progress bars. A demo was presented to illustrate these capabilities.
Oplægget blev holdt ved et seminar i InfinIT-interessegruppen Softwaretest den 28. september 2010.
Læs mere om interessegruppen på http://www.infinit.dk/dk/interessegrupper/softwaretest/softwaretest.htm
The document provides an overview of QuickTest Professional (QTP), including:
1. QTP is a tool used for functional and regression testing of applications. It supports testing of various software using built-in and external add-ins.
2. The object repository in QTP stores object identification properties that are used to recognize and interact with objects during testing.
3. Tests in QTP can include multiple actions, which can be arranged hierarchically. Data tables are used to support data-driven testing by storing test and action iteration data.
TMPA-2017: Regression Testing with Semiautomatic Test Selection for Auditing ...Iosif Itkin
TMPA-2017: Tools and Methods of Program Analysis
3-4 March, 2017, Hotel Holiday Inn Moscow Vinogradovo, Moscow
Regression Testing with Semiautomatic Test Selection for Auditing of IMS Database
Alexey Ruchay, Ivan Kliavin, Tatiana Kotova, Julia Ivanova, Applied Technologies
Would like to know more?
Visit our website:
www.tmpaconf.org
www.exactprosystems.com/events/tmpa
Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro
San Jose Selenium Meet-up PushToTest TestMaker PresentationClever Moe
With the Selenium project team's focus on making the WebDriver APIs a W3C standard, this is a good time to talk about where Selenium is going and the support it is getting from commercial open source companies. Frank Cohen, CEO at PushToTest, will discuss Selenium tools - record/playback utilities, deploying Selenium scripts to the Cloud, results analysis tools to surface functional issues and performance bottlenecks, and operational test database repositories - needed to be productive and successful with Selenium.
The document provides an overview of mVerify Corporation and its mobile testing solution called MTS. MTS aims to address the challenges of testing mobile applications by providing a platform that can simulate millions of users and configurations to thoroughly test apps. The solution slashes testing time and costs while improving reliability and performance. mVerify has seen early traction and seeks funding to further develop the platform and expand its customer base and sales.
JMeter - Performance testing your webappAmit Solanki
JMeter is an open-source tool for performance and load testing web applications. It can test applications by simulating heavy loads to determine stability and identify performance bottlenecks. JMeter simulates multiple users accessing web services concurrently to model expected usage during peak periods. It provides instant visual feedback and allows tests to be run interactively or in batch mode for later analysis. Tests are composed of thread groups, samplers, timers, listeners and other elements to control test flow and capture response data. JMeter also supports distributed testing across multiple servers to simulate very large loads.
The document provides an overview of automation testing using Quick Test Professional (QTP). It discusses the need for automation testing, rules of automation, and the different automation tools available. It then covers the life cycle of automation testing using QTP, including analyzing the application, selecting a tool, designing and recording test scripts, running tests, and finding and reporting defects. The document also describes key QTP concepts like object identification, the object repository, recording modes, parameters, reusable actions, and checkpoints.
The document describes a proposed smart debugger framework. It would help students debug programs by finding bugs and managing test cases. It would also help programmers manage code and test automation. Additionally, the framework would aid researchers by visualizing experiment results to generate new hypotheses. The framework uses data flows to collect coverage information from running test cases on programs to compute metrics and rank covered entities for students, programmers or researchers.
Thomas Kauders - Agile Test Design And Automation of a Life-Critical Medical ...TEST Huddle
EuroSTAR Software Testing Conference 2010 presentation on Agile Test Design And Automation of a Life-Critical Medical Device by Thomas Kauders. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/
Gatling is Open Source Stress testing tool.
Why Gatling:
- High Performance.
- Multi Threading vs (Akka) Actor Model.
- Synchronous Blocking IOs vs asynchronous Non-blocking IOs Netty.
Bibhas Kumar Senapati has over 6 years of experience in IT and has worked as a test engineer, junior test analyst, senior test analyst, and automation test lead. He has expertise in technologies like Windows, Unix, Linux, MySQL, Oracle, SQL Server, QTP, WinRunner, Test Director, Selenium, and LoadRunner. He holds an engineering degree and certifications in software testing and domains like forex, trade finance, and investment banking. His experience includes automation testing roles for projects with Morgan Stanley, Standard Bank of South Africa, UBS, Goldman Sachs, and Citi.
Open Source Test Workshop for CIOs, CTOs, ManagersClever Moe
This Open Source Test Workshop is for senior IT and business executives needing visibility and management tools and methodology into all the demand for IT. Shows how to bring Open Source Testing into your organization.
The document discusses unit testing in Java. It introduces the JUnit testing framework and describes unit testing methodology, including how to design testable code, write maintainable tests, and handle indirect inputs and outputs through stubs and mocks. The document provides guidance on test-driven development and testing legacy code. It contrasts unit testing with functional testing and discusses best practices for test granularity, assertions, and fixtures.
Visual Studio 2010 Testing & Lab Management ToolsAyman El-Hattab
The document discusses Microsoft's tools for test management in software development projects. It describes how Visual Studio and Team Foundation Server can be used to manage requirements, tasks, bugs, and test cases throughout the development process. It also outlines how lab management and virtual machines help enable automated testing.
Database Unit Testing Made Easy with VSTSSanil Mhatre
This document discusses database unit testing using Visual Studio Team System (VSTS). It begins with an overview of software testing basics and unit testing principles. It then covers database unit testing terminology, principles of isolation and independence, and testable interfaces of stored procedures. The document outlines different levels of unit testing and factors to consider. It demonstrates implementing database unit testing in VSTS 2010 and new features in VSTS 2012. The goal is to show how VSTS can be used to test database code and improve quality.
Agile Open Source Performance Testing Workshop for Business ManagersClever Moe
The document discusses open source performance testing and the Agile Performance Test Methodology and Tools. It promotes the PushToTest open source testing platform, which provides functionality for authoring, executing, and analyzing tests across multiple environments. The presentation covers the benefits of the PushToTest approach for both individuals and organizations implementing agile development and performing load and performance testing.
The document discusses how the transition to wafer-level chip scale package (WLCSP) devices impacts testing costs. It notes that traditional test flows for packaged chips involve multiple discrete steps and machines, while WLCSP requires redefining the test flow to a single integrated platform to accommodate the new package type and simplify returns processing. Adopting a combined tester and inspection platform that can handle multiple input and output options could significantly reduce capital equipment, floor space needs, staffing requirements and inventory costs compared to the traditional multi-machine approach.
Achieving Very High Reliability for Ubiquitous Information Technology Bob Binder
1) The document discusses achieving very high reliability for ubiquitous information technology through full test automation.
2) It outlines the new IT reality of growing usage, mobility, and need for high reliability of "five nines" or 99.999% uptime.
3) The strategy proposed is taking a full end-to-end testing approach through automated test generation and execution to achieve the reliability needed for ubiquitous IT to scale to millions of users.
The Tester’s Dashboard: Release Decision SupportBob Binder
The document discusses metrics for supporting release decisions based on model-based testing. It describes using an operational profile to generate test cases, calculating model coverage metrics, using a reliability demonstration chart to assess risk, and measuring relative proximity to compare expected and actual failure rates. A case study applies these methods to a word processing app and missile defense system. Key observations are that model coverage ensures sufficient testing, reliability demonstration charts assume flat profiles which may be optimistic, and relative proximity indicates when failure intensities match expectations.
Performance Testing Mobile and Multi-Tier ApplicationsBob Binder
Invited Talk, Chicago Quality Assurance Association, Chicago, June 26, 2007. Overview of performance testing strategy for handheld devices and multi-tier systems.
The document discusses lessons learned from testing object-oriented systems. It covers the state of the art in object-oriented test design, automation, and representation. It also examines the state of the practice, finding that the best organizations implement systematic testing at multiple scopes from classes to subsystems. With rigorous testing following design patterns, world-class quality below 0.025 defects per function point is achievable.
Juniper Networks Ignite! Testing Conference. Sunnyvale California, November 9, 2011.
Overview of model-based testing. Two case studies. Thumbnail introduction to fee and free MBT tools.
Keynote, ISSRE-13, St. Malo, France, November 4, 2004.
Outline: 21st Century IT Trends, Mobile Technology Crisis, Test Effectiveness Levels, Level 4 Case Study, Reliability Arithmetic, Test Performance Envelope.
This document discusses factors that affect testability and strategies for improving testability. It defines testability as the ability to produce tests to verify complex systems. Higher testability allows for more effective testing with the same resources. The document identifies controllability and observability as the main factors that determine a system's testability. It provides examples of how characteristics like complexity, non-determinism, and lack of visibility into state diminish testability. Techniques for improving testability include adding points of control and observation, using state test helpers, building tests into the system, and designing for well-structured and deterministic code.
WTS is a mobile systems verification tool that allows for end-to-end automated testing of wireless applications on thousands of simulated personal digital assistants (PDAs) from a single computer. It generates test cases for up to one million virtual users, simulating real-world user behavior, mobility patterns, and wireless conditions. Current versions support testing on 15 actual PDA models, with more in development. WTS' value proposition is that it can productize proven automated testing techniques to deliver high-fidelity, scalable testing of any wireless application on any device from a single system.
Invited Talk: C-SPIN, the Chicago Software Process Improvement Network. January 7, 2009, Schaumburg, Illinois. Overview of themes and concepts from ISSRE 2008.
Invited Talk, Chicago Quality Assurance Association. January, 27 2010, Chicago.
Overview of cloud computing - new challenges for achieving high reliability
This document provides an overview of software testing models, patterns, and tools. It discusses test design patterns at various scopes, from the method level to the system level. It also covers test automation patterns, levels of testing from manual to fully automated, and test oracles. The speaker will provide a case study and take questions.
The Tester’s Dashboard: Release Decision SupportBob Binder
Industry track, ISSRE-18. San Jose November 2, 2010.
Shows how to the Reliability Demonstration Chart, state transition coverage, and new metric "Relative Proximity" can be used to make better release decisions.
Keynote, Google Test Automation Conference. Hyderabad, India, October 28, 2010.
Overview of software testability and implications for managers, developers, and testers. Discusses six aspects: process, requirements, design, white-box, black-box, and test tooling. Shows that testers are not typically in control of these aspects, which leads to sub-optimal software development outcomes.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
6. Testing in Flatland
TestCases TestSuites • Run A
A pqrstv
p s p q B C
• Run B
q t B r s t u
r s C
r u • Run C
C t u
t u
July 16, 2012 MTS - Controllable Test Objects 6
8. xUnit’s One Track Strategy
• Define and nest groups • Obstacle to any rich
• Run any sequence of strategy
groups – Select, repeat
• No support for group or – Skip, Halt
member – Time
– Repeat – Broadcast
– Select – Synchronize
July 16, 2012 MTS - Controllable Test Objects 8
9. Composition Isn’t Enough
• atom == test case • How to do ( p | q )
• ( ) group
• * quantification • How to do (t ( p | q ))*
• | select (or)
xUnit is like reg exp without * ? + |
A formal language without alternation and Kleene star
July 16, 2012 MTS - Controllable Test Objects 9
10. Composition Isn’t Enough
Why Repeat? Why Select?
• Iterate over collection • Respond to environment
• Iterate over generation configuration/conditions
strategy • Skip
• Performance test over – Depends on earlier
interval – Don’t halt long complex test
run
• Capacity test to limits
– Count failures for reliability
estimate
July 16, 2012 MTS - Controllable Test Objects 10
14. MTS: Any App, Any Platform
Console Host Agent Host
MTS Console
MTS Test
TEST RUN
Agent
REPORTS
Agent Host
MTS Remote
Agent
MTS Test
Agent Client
Under Test
Client Host Under Test
MTS Remote
Agent
MTS Remote
Client
Agent
Under Test
Host Under Test may be
Server
Client Host Under Test Cell Phone Under Test
PDA
Desktop Server Host Under Test
MTS Remote
Agent
Server
Embedded Processor
Server Network Equipment
Under Test Access Point
Base Station
Server Host Under Test
July 16, 2012 MTS - Controllable Test Objects 14
15. MTS Approach
• Test harness for distributed, end to end
testing
– xUnit: composable test objects
– TTCN: adapters, concurrency, …
– Tcl: defacto standard for test script automation
July 16, 2012 MTS - Controllable Test Objects 15
16. About TTCN-3
• Procedural DSL similar • Well-suited to model-based
to CORBA IDL testing
– Generate TTCN from MSCs,
FSMs, etc.
• Point of Observation – Generate impl source from
TTCN
and Control (adapters) – Compile & run impl
• Modular, but not • Same test suite can be used
necessarily composable on many implementations
July 16, 2012 MTS - Controllable Test Objects 16
17. Tcl
• Open source iut.set {1}
interpreter
iut.set {$x}
• LISP-ish evaluation iut.set {[genX{$y}]}
iut.set {[eval $codeForX]}
July 16, 2012 MTS - Controllable Test Objects 17
18. Tcl
• Open source
interpreter
• LISP-ish evaluation
• Object-oriented
[incr Tcl]
July 16, 2012 MTS - Controllable Test Objects 18
20. TestObject methods
Skip rest of test object when its preconditions aren't met.
require
Automatically called when a TestObject is run.
Create starting conditions for a test; allocate necessary resources. setup
Run tests. The user's implementation is expected to execute test
test
actions leading to one or more test verdicts.
Restore pretest conditions; release resources the user test object
cleanup
has claimed.
Skip or halt rest of a test run when postconditions aren't met.
ensure
Automatically called after cleanup completes.
July 16, 2012 MTS - Controllable Test Objects 20
21. Using require
::itcl::class aTestCase{
inherit TestObject
::itcl::body aTestCase::require {} {
if {[getRemoteHostStatus] eq "Connected"} {
set remoteHostconnection true
} else {
userLogEntry "Remote Host not ready, skipping"
set remoteHostconnection false
}
return $remoteHostconnection
}
::itcl::body aTestCase::test {} { … }
July 16, 2012 MTS - Controllable Test Objects 21
22. Completion Actions
return Current Test object Caller Test object Test objects run stack Counters
keyword
OK No effect, Destruct No effect. No effect. No
(ensure) change.
SKIP Destruct Continue with the No effect. No
next test object. change.
HALT Destruct Immediately exit the Terminate the test run. No
method that called Percolate the HALT to the change.
this test object. run stack.
EXCP Capture current Tcl Immediately exit the Terminate the test run. Increment
errorInfo. Destruct the method that called Percolate the EXCP to all error
current test object. this test object. test objects in the run counter.
July 16, 2012
stack.
MTS - Controllable Test Objects 22
23. Using ensure and Completion Actions
::itcl::class checkDir {inherit TestObject
::itcl::body checkDir::require {} { … }
::itcl::body checkDir::setup {} { … }
::itcl::body checkDir::test {} { … }
::itcl::body checkDir::cleanup {} { … }
::itcl::body checkDir::ensure {
set currentDir [pwd]
if { ! [regexp {mydir} $currentDir]} {
return SKIP
} else {
return OK
}
}
July 16, 2012 MTS - Controllable Test Objects 23
24. MTS run command
run <target> repeat • Hardcoded target
<times> <arg list> run myTestObject
• “Sources in” target file • Target name in a variable
TestObject subclass run $anyTestObject
• Runs this TestObject
• Returns control to the • Function computes target
run [eval whichTestObject]
caller
July 16, 2012 MTS - Controllable Test Objects 24
25. Selection and Repetition in MTS
How to do ( p | q ) How to do (t ( p | q ))*
::itcl::body pq::test{} { ::itcl::body mySuite::test{reps} {
if {[rhStatus]} { run strategy1 -repeat $reps
run remoteTest }
} else {
run localTest ::itcl::body strategy1::test{} {
} run t
} run pq
}
July 16, 2012 MTS - Controllable Test Objects 25
26. TestObject helpers
Start other test objects or create a test suite run <target>
Evaluate and log pass/fail conditions check <expression>
Time any action during a test run Timer <method>
Object interface for test run as a whole testRun <method>
Monitor remote host resource utilization rpm <method>
Check the remote host getRHostStatus myHost
Perform data-driven tests submit <args>
Annotate test run reports userLogEntry <args>
July 16, 2012 MTS - Controllable Test Objects 26
27. IMPLEMENTING
MODAL CLASS TEST PATTERN
July 16, 2012 MTS - Controllable Test Objects 27
28. Modal Class Test PatternTwoPlayerGame
T wo Play erG am e ( )
α
TwoPlayerGame p1 _S tart ( ) / p2 _S tart ( ) / ThreePlayerGame( ) /TwoPlayerGame( )
+TwoPlayerGame() s im u lat eV olle y( )
G am e S tarte d
s im u lat eV olle y( ) α
+p1_Start( )
+p1_WinsVolley( ) p1_Start( ) / p3_Start( )/
-p1_AddPoint( ) p1 _W ins V olle y ( ) p2 _W ins V olle y ( ) simulateVolley( ) simulateVolley( )
[th is .p 1_ Sc ore ( ) < 20 ] / [th is .p 2_ Sc ore ( ) < 20 ] / Game Started
+p1_IsWinner( ) th is .p 1A ddP oin t( ) th is .p 2A ddP oin t( )
+p1_IsServer( ) s im u lat eV olle y( ) p1 _W ins V olle y ( ) / s im u lat eV olle y( )
+p1_Points( ) s im u lat eV olle y( ) p2_Start( ) /
+p2_Start( ) P la ye r 1 P la ye r 2 simulateVolley( )
+p2_WinsVolley( ) S erv ed S erv ed p1_WinsVolley( ) /
-p2_AddPoint( ) p2 _W ins V olle y ( ) / simulateVolley( )
s im u lat eV olle y( )
+p2_IsWinner( ) p1 _W ins V olle y ( ) p2 _W ins V olle y ( )
+p2_IsServer( ) [th is .p 1_ Sc ore ( ) = = 20] / [th is .p 2_ Sc ore ( ) = = 20] /
+p2_Points( ) th is .p 1A ddP oin t( ) th is .p 1A ddP oin t( ) p2_WinsVolley( )
+~( ) p1_WinsVolley( ) [this.p2_Score( ) < 20] / p3_WinsVolley( )
P la ye r 1 P la ye r 2
[this.p1_Score( ) < 20] / this.p2AddPoint( ) [this.p3_Score( ) < 20] /
p1 _Is W in ner( ) / Won Won p2 _Is W in ner( ) / this.p1AddPoint( ) simulateVolley( ) this.p3AddPoint( )
retu rn T R UE ; retu rn T R UE ;
simulateVolley( ) simulateVolley( )
~( ) ~( ) p1_WinsVolley( ) / p2_WinsVolley( ) /
ω
simulateVolley( ) simulateVolley( )
Player 1 Player 2 Player 3
Served Served Served
p2_WinsVolley( ) / p3_WinsVolley( ) /
simulateVolley( ) simulateVolley( )
ThreePlayerGame p1_WinsVolley( ) p3_WinsVolley( )
[this.p1_Score( ) == 20] / [this.p3_Score( ) == 20] /
T h ree P la y erG a m e ( ) / T wo P la y erG am e ( )
α this.p1AddPoint( ) this.p3AddPoint( )
p 3_ S tart ( ) / p3_WinsVolley( ) /
ThreePlayerGame G a m e S ta rt ed
s im ulat eV o lley ( ) simulateVolley( )
p 3_ W ins V o lle y( ) / p2_WinsVolley( )
+ThreePlayerGame() s im ulat eV o lley ( ) [this.p2_Score( ) == 20] /
+p3_Start( ) p 3_ W ins V o lle y( )
this.p1AddPoint( )
[t his .p 3_ S co re ( ) < 2 0] /
+p3_WinsVolley( ) th is . p3 A dd P oint ( )
T w oP lay erG am e ( )
-p3_AddPoint( ) s im ulat eV o lley ( )
p 1_ W ins V o lle y( ) /
+p3_IsWinner( ) s im ulat eV o lley ( )
+p3_IsServer( ) P la y er 3
+p3_Points( ) S erv e d
p1_IsWinner( ) / p2_IsWinner( ) / p3_IsWinner( ) /
+~( ) p 2_ W ins V o lle y( ) / return TRUE; Player 1 return TRUE; Player 2 Player 3 return TRUE;
s im ulat eV o lley ( )
Won Won Won
p 3_ W ins V o lle y( )
[t his .p 3_ S co re ( ) = = 2 0] /
th is . p3 A dd P oint ( ) ~( )
~( ) ~( )
ω
P la y er 3
W on p 3_ Is W in ne r( ) /
ret urn T R UE ;
~( )
ω
July 16, 2012 MTS - Controllable Test Objects 28
29. Modal Class Test Design
1 ThreePlayerGame( )
2 p1_Start( )
8 P layer 2 Served
3 p2_Start( )
4 p3_Start( )
11 P layer 3 Served
5 p1_WinsVolley( )
6 p1_WinsVolley( )[this.p1_Score( ) < 20] P layer 1 Served 17 om ega
*7
7 p1_WinsVolley( ) [this.p1_Score( ) == 20] P layer 1 W on
14
8 p2_WinsVolley( ) P layer 1 W on
9 p2_WinsVolley( ) [this.p2_Score( ) < 20]
*6 P layer 1 Served
10 p2_WinsVolley( ) [this.p2_Score( ) == 20]
2 Sneak path tests
*9 P layer 2 Served
not shown
11 P layer 3 Served
1 3
alpha Gam eS tarted P layer 2 Served 17 om ega
* 10
P layer 2 W on
15
P layer 2 W on
Note iterations
5 P layer 1 Served
4
* 12 P layer 3 Served
17 om ega
11 p3_WinsVolley( )
* 13 P layer 3 W on
16
12 p3_WinsVolley( ) [this.p3_Score( ) < 20]
P layer 3 Served P layer 3 W on
13 p3_WinsVolley( ) [this.p3_Score( ) == 20] 8
P layer 2 Served
14 p1_IsWinner( )
15 p2_IsWinner( )
16 p3_IsWinner( ) 5 P layer 1 Served
17 ~( )
July 16, 2012 MTS - Controllable Test Objects 29
30. Implementing Modal Class Design
xUnit MTS
• TestCase for each transition (20) • TestObject for each transition (20)
• Hand hack TestSuite for each • Top level TestObject uses run to
root to leaf path (17) and one to
run them all call transition test objects
– addTest – Iteration trival
– addTestSuite • Parameterized strategies
• How to iterate transitions? – N+
• Embed strategy in single large test
– Markov
object?
– Very smelly – Etc.
• Develop code generator that writes • Parameterized state spec
test suites?
July 16, 2012 MTS - Controllable Test Objects 30
31. 1 ThreePlayerGame( )
2 p1_Start( )
8 P layer 2 Served
3 p2_Start( )
4 p3_Start( )
11 P layer 3 Served
5 p1_WinsVolley( )
6 p1_WinsVolley( )[this.p1_Score( ) < 20] P layer 1 Served 17 om ega
*7
7 p1_WinsVolley( ) [this.p1_Score( ) == 20] P layer 1 W on
14
Dude, where’s my object?
8 p2_WinsVolley( ) P layer 1 W on
9 p2_WinsVolley( ) [this.p2_Score( ) < 20]
*6 P layer 1 Served
10 p2_WinsVolley( ) [this.p2_Score( ) == 20]
2
*9 P layer 2 Served
11 P layer 3 Served
1 3
alpha Gam eS tarted P layer 2 Served 17 om ega
* 10
P layer 2 W on
15
P layer 2 W on
5 P layer 1 Served
4
* 12 P layer 3 Served
xUnit MTS
17 om ega
11 p3_WinsVolley( )
* 13 P layer 3 W on
16
12 p3_WinsVolley( ) [this.p3_Score( ) < 20]
P layer 3 Served P layer 3 W on
13 p3_WinsVolley( ) [this.p3_Score( ) == 20] 8
P layer 2 Served
14 p1_IsWinner( )
15 p2_IsWinner( )
16 p3_IsWinner( ) 5 P layer 1 Served
17 ~( )
1 8 15 1 6 11 1 8 15
2 9 16 2 7 12 2 9 16
3 10 17 8 3 10 17
3 13
11 21
4 11 18 4 18
4 9 14
5 12 19 5 12 19
5 10 15
6 13 20 6 13 20
16
7 14 7 14
TestCase objects TestSuite objects TestObject objects
July 16, 2012 MTS - Controllable Test Objects 31
32. Controllable Test Objects
Design Goal TTCN xUnit MTS
Select, Iterate, any level Yes No Yes
Generate with MBT strategies Yes Yes Yes
Logical/Physical separation Yes No Yes
Platform agnostic/robust Yes ? Yes
Channel agnostic/robust Yes ? Yes
Distributed control Yes No Yes
Minimize Test Agent resources NA No Yes
Minimize brittleness Yes No Yes
Intuitive interaction No Yes Yes
Composable No Yes Yes
July 16, 2012 MTS - Controllable Test Objects 32
33. MTS hard problem queue
• Protocol to drive remote target host IUT from test host
– Bi-directional, lightweight (BEEP ?)
– Platform and stack agnostic
– Target environment control (remote command line)
– Distributed adapter RPC
• Forced reflection to generate adapters
• Logical/physical mapping in generative models
• Scalable asynchronous remote agents
– Control strategy
– Provisioning
– Protocol
July 16, 2012 MTS - Controllable Test Objects 33
34. Question Time
Bob Binder
rvbinder@gmail.com
www.robertvbinder.com
July 16, 2012 MTS - Controllable Test Objects 34