Model Based Testing Tools discusses model-based testing where a model of the system is used to automatically generate test cases. It outlines how MBT works by deriving an abstract test suite from the model, then mapping those tests to concrete executable tests. The benefits of MBT include increased testing effectiveness and potential cost savings. Several MBT tools used in industry are also described, including Conformiq, Reactis Tester, and Spec Explorer.

1. Model Based Testing Tools International Islamic University Department of Computer Science Presenters: Muhammad Husnain & Muhammad Waseem

2. Outline Introduction What is MBT How mostly MBTT Works Benefits of MBTT Problems of MBTT Current State Tools in Industry Software Tools for MBT Conclusion References

3. Model Based Testing One way to generate test cases automatically is “model-based testing” where a model of the system is used for test case generation.

4. Model Based Testing “ Model-based testing is a testing technique where the runtime behavior of an implementation under test is checked against predictions made by a formal specification, or model .” - Colin Campbell, Microsoft Research In other words: A model describes how a system should behave in response to an action. Supply the action and see if the system responds as you expect.

5. Models Simply, a model of software is a depiction of its behavior. Behavior can be described in terms of the input sequences accepted by the system, the actions, conditions, output logic, or the flow of data through the application’s modules and routines. B y U sing M odeling a shareable, reusable, precise description of the system is acquired.

6. Models İn SW Testing Finite State Machines Statecharts UML Markov Chains Grammars

7. How Mostly MBTT Works?

8. How MBT Works? The model is usually an abstract, partial presentation of the system under test's desired behavior. The test cases derived from this model are functional tests on the same level of abstraction as the model. These test cases are collectively known as the abstract test suite

9. How MBT Works? Contd. The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. This is done by mapping the abstract test cases to concrete test cases suitable for execution.

10. How MBT Works? Cont Model Implementation Test Oracle Test Cases Run Provides actual results for Provides expected results for Generates Pass Fail

11. Problems of MBT

12. Problems of MBT

13. Limitations of Executable UML methodology: relative immaturity limited availability of tools lack of a concrete standard syntax for action languages. We believe that these shortcomings will be solved as the technology matures.

14. Motivation behind MBTT Testing is expensive 30-50% of development costs Poor testing is VERY expensive Down time Maintenance costs Rework Law suits Model Based Testing promises Increased effectiveness of testing Similar or decreased costs Reuse of design artifacts

15. Benefits of MBT

16. Benefits of MBT Contd. Cost A typical test engineer Earns €41 500 ($50 000 USD) per year Works 40 to 50 hours per week A typical test machine Costs €250 ($300 USD) per year to buy and operate Can work 100 hours per week

17. Benefits of MBT Shorter schedules, lower cost, and better quality Enhanced communication between developers and testers Early exposure of ambiguities in specification and design Capability to automatically generate many non-repetitive and useful tests Test harness to automatically run generated tests Eases the updating of test suites for changed requirements Capability to evaluate regression test suites Capability to assess software quality These benefits all require an initial investment in tools and training.

18. Current State of MBT in Industry

19. Current State of MBT in Industry IBM Research Most projects were done by PhD Holders Very Little was successful (Just in the Lab, never actually went into market) Too Complicated to use and analyze Microsoft Research The Spec# programming system is a new attempt at a more cost effective way to develop and maintain high-quality software

20. Current State of MBT in Industry Spec# is a computer language which includes and extends C# Adds pre/post conditions, contracts Adds high-level data types with convenient notations Adds logical quantifiers like FORALL and EXISTS A Spec# model is just a program! You can run it like a C# program It can call framework code You can explore it (run all of its possible behavior) which is how we do MBT with Spec#

21. Current State of MBT in Industry (Spec Explorer) A model exploration and testing tool for .NET: Authoring of models in Word Model exploration (generating FSM) FSM visualization FSM traversal and test suite generation Automatic implementation binding Online test-suite execution and offline test-suite code generation

22. Software Tools for MBT

23. Tools we are going to discuss: Conformiq Test Generator Leirios Test Generator Reactis Tester TorX AsmL MaTeLo

24. SpecTest TOSTER Bridge Point iUMLite Prosasim PathMATE Tools we are going to discuss:

25. Software Tools for MBT Conformiq Test Generator is a model-based testing tool using UML state charts that represent testing strategies

26. Leirios Test Generator Leirios Test Generator is a model-based testing tool that generates tests automatically from deterministic system specifications

27. Reactis Tester Reactis Tester is another model-based testing tool that focuses on control systems Works with MATLAB Tool downloaded AND available

28. AsmL AsmL Test Tool can generate tests directly from an AsmL model

29. AsmL is the Abstract State Machine Language. It is an executable specification language based on the theory of Abstract State Machines . The current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. It is fully interoperable with other .NET languages. AsmL generates .NET assemblies which can either be executed from the command line, linked with other .NET assemblies, or packaged as COM components. AsmL

30. TOSTER TOSTER ( T he O bject-oriented S oftware T esting Environment ) a test generation and execution system produced by the Warsaw University of Technology. It incorporates technology for mapping the information in UML diagrams to the source code of an application. It also generates and runs test cases based on expected results derived from the UML state diagrams. There appear to be two test generation algorithms, but no explicit testing directives.

31. SpecTest SpecTest is an automatic test case generator from George Mason University. It accepts models written in SCR or UML, and generates test cases based on a choice of two coverage criteria, with a further two planned for implementation.

32. iUMLite www.kc.com iUML Modeler that providing users with the ability to build complete and consistent executable UML models. iUML Simulator that providing users with the ability to execute, test and debug these executable UML models.

33. Limitations of iUMLite free version: Capacity restrictions are placed on the models that can be constructed and executed, specifically a maximum of: --) 3 Versions per Domain --) 15 Classes per Domain version cont……..

34. Documentations can only be produced in HTML format The import/export facilities for moving models and components between databases are not available. Only one user at a time may access a given database. The code generator provided can only be used for the purposes of executing the models in the iUML Simulator. It cannot be used to deliver target software. Similarly, iUMLite does not come with the iCCG Configurable Code Generation capability. The freely available copy of their tool is not capable of any code generation though. Limitations of iUMLite free version:

35. How to get License It is not greatly easy to acquire an evaluation version of this tool. In order to evaluate the software, a form has to be filled from the company’s web-site. After filling and submitting the form, the company will again enter in contact with the prospective evaluator in order to ask further questions. The license key given is tied to the serial number of the hard disk, therefore it can’t be transferred to another computer.

36. TestMaster TestMaster is an automated test program generator that includes a GUI development environment that is ideal for building models at both abstract and implementation dependent levels. http://www.teradyne.com/prods/sst/product_center/sstprod.html http://www.dnaent.com

37. MaTeLo Usage Model Editor is a first MaTeLo module which let you create a project, design and edit a usage model of the system under test. Testor is the second MaTeLo module which let you automatically generate test cases by browsing the usage model and following one path or another by randomly selecting transitions according to their occurring probability. Testor finally can help you manage the test campaign and get some information about it such as coverage, system reliability or Mean Time To Failure.

38. What is a usage model? A usage model represents the interactions between a system and its users. . A MaTeLo usage model is an extended Markov chain for test and validation purposes. It’s a state diagram where a transition from a state to another is related to a system input and its activation frequency in the current state. A chain is made of 2 specific states: Invoke and Terminate. Those states usually represent the system start and stop. Chains can be nested, in order to get a hierarchical model.

39. USAGE MODEL STRUCTURE When to perform the MaTeLo model? Ideally, the best moment to design your usage model is when you’ll have written about 2/3 of your system’s specifications. Eventually, you can update your model just before the test campaign to ensure that your model consistency with the specification at this time.

40. Cont… What’s the usage model use? The usage model lets you generate test sequences made of stimuli to perform on the system as well as the results to expect from the system. A model design stands for a test plan design

41. How to model? The most useful entry documents for MaTeLo modeling are the system’s functional specifications. Ideally, this specification will be made of Use Cases, which will dramatically reduce the system understanding effort. modeling should consider the main validation aims, in order to better focus on those aims. Such aims could be: reliability, performance, robustness … A chain should not be too sequential (wrong model) and avoid redundancy.

42. Elementary Modeling Rules: There can only be one transition going out the A state and entering the B state. (except for conditional transitions) No transition should enter the Invoke state, and no transition should go out from the Terminate state. A sub-chain respects the same design rules as the main chain.

43. Profiles Management It is possible to use one or more usage profiles . Profiles make it possible to model various behaviors of SUT and thus obtain different tests according to the usage frequencies allocated to the transitions according to the selected profile.

44. Testor Testor is the second module of the MaTeLo system dedicated to the automatic generation of the Test Plans and to the Test Campaign Analysis.

45. Typical use of Testor 1.Use Testor to open a test project created by UM Editor. 2. Use the first panel to do the Model Checking and request a MC reporting, to analyze the parameters (stationary distribution, test case length and visiting probability).

46. Cont… 3. Use the second panel to generate a test suite. It’s possible to generate: Most probable test suite Random test suite Limit test suite Minimum test suite (only if your model does not include asynchronous or conditional elements)

47. Cont.. 4. Use the third panel to enter manually or import the test execution for each system version of the SUT. Debugging test: not described by a test suite, debugging test is a simple test like “launching the application” Most probable test: one test for the most probable usage of the SUT Random Test (xN campaign) Limit Test Minimum test Adding test: additional test, no generate by MaTeLo

48. Conformiq Conformiq Qtronic is a tool for both online testing of systems and for offline generation of test scripts It can use the model to directly test a running system or to generate test scripts that can be independently executed afterwards. Generated test scripts can be stored in version control system tests can be sent to colleagues or to execute them independently

49. Are You Feeling Like This OK…. Let us Finish……….

50. Conclusion

51. Conclusion Modeling is not easy What details do you leave out? How do you check the model? There are no recipes for success What needs to happen Easier notation Incremental approach Pilots and examples

52. Questions Does tool cover stress testing? Does Tool cover how user friendly the system is or it only covers functionality? How do we guarantee that the generated expected outputs and inputs are enough and cover all the cases? How to make for the errors and exceptions that may occur due to running the program on different environments and different OSs? How is automated generation of inputs and outputs work when an actual human interaction is required as part of the system testing?

53. References

54. References http://www.goldpractices.com/index.php http://en.wikipedia.org/wiki/Model-based_testing http://blogs.msdn.com/nihitk/articles/144664.aspx http://research.microsoft.com/specsharp/ http://react.cs.uni-sb.de/mbt2006/talks/mbt4 masses .pdf http://react.cs.uni-sb.de/mbt2006/talks/ModelBasedTestingSoberEvaluation.pdf http://aetgweb.argreenhouse.com/papers/1999-icse.pdf www.agedis.de Colin Campbell, Wolfgang Grieskamp, Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes “Spec Explorer: An Integrated Environment for Model-Based Testing.” Foundations of Software Engineering Microsoft Research, Redmond

55. References EXECUTABLE UML by Selo Sulistyo and Warsun Najib “ Towards Executable Aspect-Oriented UML models” By “Lidia Fuentes and Pablo Sánchez” ,March 12-13, 2007. http://www.eclipse.org/uml2/ http://www.magicdraw.com “ Model Driven Testing “ By Brian Berger, Majdi Abuelbassal, and Mohammad Hossain, 1997 VP-UML User’s Guide (Part 1)

56. Thanks