Presentation Of Mbt Tools


Published on

this presentation is about Model Base Testing Tools.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • An example of a model-based testing workflow. IXIT refers to "implementation extra information" and denotes here the total package of information that is needed when the abstract test suite is converted into an executable one. Typically it includes information about test harness, data mappings and SUT configuration.
  • Presentation Of Mbt Tools

    1. 1. Model Based Testing Tools International Islamic University Department of Computer Science Presenters: Muhammad Husnain & Muhammad Waseem
    2. 2. Outline <ul><li>Introduction </li></ul><ul><ul><li>What is MBT </li></ul></ul><ul><li>How mostly MBTT Works </li></ul><ul><li>Benefits of MBTT </li></ul><ul><li>Problems of MBTT </li></ul><ul><li>Current State Tools in Industry </li></ul><ul><li>Software Tools for MBT </li></ul><ul><li>Conclusion </li></ul><ul><li>References </li></ul>
    3. 3. Model Based Testing <ul><li>One way to generate test cases automatically is “model-based testing” where a model of the system is used for test case generation. </li></ul>
    4. 4. Model Based Testing <ul><li>“ Model-based testing is a testing technique where the runtime behavior of an implementation under test is checked against predictions made by a formal specification, or model .” - Colin Campbell, Microsoft Research </li></ul><ul><li>In other words: </li></ul><ul><ul><li>A model describes how a system should behave in response to an action. </li></ul></ul><ul><ul><li>Supply the action and see if the system responds as you expect. </li></ul></ul>
    5. 5. Models <ul><li>Simply, a model of software is a depiction of its behavior. Behavior can be described in terms of the </li></ul><ul><ul><li>input sequences accepted by the system, </li></ul></ul><ul><ul><li>the actions, </li></ul></ul><ul><ul><li>conditions, </li></ul></ul><ul><ul><li>output logic, or the flow of data through the application’s modules and routines. </li></ul></ul><ul><li>B y U sing M odeling a shareable, reusable, precise description of the system is acquired. </li></ul>
    6. 6. Models İn SW Testing <ul><li>Finite State Machines </li></ul><ul><li>Statecharts </li></ul><ul><li>UML </li></ul><ul><li>Markov Chains </li></ul><ul><li>Grammars </li></ul>
    7. 7. How Mostly MBTT Works?
    8. 8. How MBT Works? <ul><li>The model is usually an abstract, partial presentation of the system under test's desired behavior. </li></ul><ul><li>The test cases derived from this model are functional tests on the same level of abstraction as the model. </li></ul><ul><li>These test cases are collectively known as the abstract test suite </li></ul>
    9. 9. How MBT Works? Contd. <ul><li>The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. </li></ul><ul><li>Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. </li></ul><ul><li>This is done by mapping the abstract test cases to concrete test cases suitable for execution. </li></ul>
    10. 10. How MBT Works? Cont Model Implementation Test Oracle Test Cases Run Provides actual results for Provides expected results for Generates Pass Fail
    11. 11. Problems of MBT
    12. 12. Problems of MBT
    13. 13. Limitations of Executable UML methodology: <ul><li>relative immaturity </li></ul><ul><li>limited availability of tools </li></ul><ul><li>lack of a concrete standard syntax for action languages. </li></ul>We believe that these shortcomings will be solved as the technology matures.
    14. 14. Motivation behind MBTT <ul><li>Testing is expensive </li></ul><ul><ul><li>30-50% of development costs </li></ul></ul><ul><li>Poor testing is VERY expensive </li></ul><ul><ul><li>Down time </li></ul></ul><ul><ul><li>Maintenance costs </li></ul></ul><ul><ul><li>Rework </li></ul></ul><ul><ul><li>Law suits </li></ul></ul><ul><li>Model Based Testing promises </li></ul><ul><ul><li>Increased effectiveness of testing </li></ul></ul><ul><ul><li>Similar or decreased costs </li></ul></ul><ul><ul><li>Reuse of design artifacts </li></ul></ul>
    15. 15. Benefits of MBT
    16. 16. Benefits of MBT Contd. <ul><li>Cost </li></ul><ul><ul><li>A typical test engineer </li></ul></ul><ul><ul><ul><li>Earns €41 500 ($50 000 USD) per year </li></ul></ul></ul><ul><ul><ul><li>Works 40 to 50 hours per week </li></ul></ul></ul><ul><ul><li>A typical test machine </li></ul></ul><ul><ul><ul><li>Costs €250 ($300 USD) per year to buy and operate </li></ul></ul></ul><ul><ul><ul><li>Can work 100 hours per week </li></ul></ul></ul>
    17. 17. Benefits of MBT <ul><li>Shorter schedules, lower cost, and better quality </li></ul><ul><li>Enhanced communication between developers and testers </li></ul><ul><li>Early exposure of ambiguities in specification and design </li></ul><ul><li>Capability to automatically generate many non-repetitive and useful tests </li></ul><ul><li>Test harness to automatically run generated tests </li></ul><ul><li>Eases the updating of test suites for changed requirements </li></ul><ul><li>Capability to evaluate regression test suites </li></ul><ul><li>Capability to assess software quality </li></ul><ul><li>  </li></ul><ul><li>These benefits all require an initial investment in tools and training. </li></ul>
    18. 18. Current State of MBT in Industry
    19. 19. Current State of MBT in Industry <ul><li>IBM Research </li></ul><ul><ul><li>Most projects were done by PhD Holders </li></ul></ul><ul><ul><li>Very Little was successful (Just in the Lab, never actually went into market) </li></ul></ul><ul><ul><li>Too Complicated to use and analyze </li></ul></ul><ul><li>Microsoft Research </li></ul><ul><ul><li>The Spec# programming system is a new attempt at a more cost effective way to develop and maintain high-quality software </li></ul></ul>
    20. 20. Current State of MBT in Industry <ul><li>Spec# is a computer language which includes and extends C# </li></ul><ul><ul><li>Adds pre/post conditions, contracts </li></ul></ul><ul><ul><li>Adds high-level data types with convenient notations </li></ul></ul><ul><ul><li>Adds logical quantifiers like FORALL and EXISTS </li></ul></ul><ul><li>A Spec# model is just a program! </li></ul><ul><ul><li>You can run it like a C# program </li></ul></ul><ul><ul><li>It can call framework code </li></ul></ul><ul><ul><li>You can explore it (run all of its possible behavior) which is how we do MBT with Spec# </li></ul></ul>
    21. 21. Current State of MBT in Industry (Spec Explorer) <ul><li>A model exploration and testing tool for .NET: </li></ul><ul><ul><li>Authoring of models in Word </li></ul></ul><ul><ul><li>Model exploration (generating FSM) </li></ul></ul><ul><ul><li>FSM visualization </li></ul></ul><ul><ul><li>FSM traversal and test suite generation </li></ul></ul><ul><ul><li>Automatic implementation binding </li></ul></ul><ul><ul><li>Online test-suite execution and offline test-suite code generation </li></ul></ul>
    22. 22. Software Tools for MBT
    23. 23. Tools we are going to discuss: <ul><li>Conformiq Test Generator </li></ul><ul><li>Leirios Test Generator </li></ul><ul><li>Reactis Tester </li></ul><ul><li>TorX </li></ul><ul><li>AsmL </li></ul><ul><li>MaTeLo </li></ul>
    24. 24. <ul><li>SpecTest </li></ul><ul><li>TOSTER </li></ul><ul><li>Bridge Point </li></ul><ul><li>iUMLite </li></ul><ul><li>Prosasim </li></ul><ul><li>PathMATE </li></ul>Tools we are going to discuss:
    25. 25. Software Tools for MBT <ul><li>Conformiq Test Generator is a model-based testing tool using UML state charts that represent testing strategies </li></ul>
    26. 26. Leirios Test Generator <ul><li>Leirios Test Generator is a model-based testing tool that generates tests automatically from deterministic system specifications </li></ul>
    27. 27. Reactis Tester <ul><li>Reactis Tester is another model-based testing tool that focuses on control systems </li></ul><ul><li>Works with MATLAB </li></ul><ul><li>Tool downloaded AND available </li></ul>
    28. 28. AsmL <ul><li>AsmL Test Tool can generate tests directly from an AsmL model </li></ul>
    29. 29. <ul><li>AsmL is the Abstract State Machine Language. It is an executable specification language based on the theory of Abstract State Machines . The current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. </li></ul><ul><li>It is fully interoperable with other .NET languages. AsmL generates .NET assemblies which can either be executed from the command line, linked with other .NET assemblies, or packaged as COM components. </li></ul>AsmL
    30. 30. TOSTER <ul><li>TOSTER ( T he O bject-oriented S oftware T esting Environment ) </li></ul><ul><li>a test generation and execution system produced by the Warsaw University of Technology. It incorporates technology for mapping the information in UML diagrams to the source code of an application. It also generates and runs test cases based on expected results derived from the UML state diagrams. There appear to be two test generation algorithms, but no explicit testing directives. </li></ul>
    31. 31. SpecTest <ul><li>SpecTest is an automatic test case generator from George Mason University. It accepts models written in SCR or UML, and generates test cases based on a choice of two coverage criteria, with a further two planned for implementation. </li></ul>
    32. 32. iUMLite <ul><li> </li></ul><ul><li>iUML Modeler that providing users with the ability to build complete and consistent executable UML models. </li></ul><ul><li>iUML Simulator that providing users with the ability to execute, test and debug these executable UML models. </li></ul>
    33. 33. Limitations of iUMLite free version: <ul><li>Capacity restrictions are placed on the models that can be constructed and </li></ul><ul><li>executed, specifically a maximum of: </li></ul><ul><li>--) 3 Versions per Domain </li></ul><ul><li>--) 15 Classes per Domain version </li></ul><ul><li>cont…….. </li></ul>
    34. 34. <ul><li>Documentations can only be produced in HTML format </li></ul><ul><li>The import/export facilities for moving models and components between databases are not available. </li></ul><ul><li>Only one user at a time may access a given database. </li></ul><ul><li>The code generator provided can only be used for the purposes of executing the models in the iUML Simulator. It cannot be used to deliver target software. </li></ul><ul><li>Similarly, iUMLite does not come with the iCCG Configurable Code Generation capability. </li></ul><ul><li>The freely available copy of their tool is not capable of any code generation though. </li></ul>Limitations of iUMLite free version:
    35. 35. How to get License <ul><li>It is not greatly easy to acquire an evaluation version of this tool. In order to evaluate the software, a form has to be filled from the company’s web-site. After filling and submitting the form, the company will again enter in contact with the prospective evaluator in order to ask further questions. The license key given is tied to the serial number of the hard disk, therefore it can’t be transferred to another computer. </li></ul>
    36. 36. TestMaster <ul><li>TestMaster is an automated test program generator that includes a GUI development environment that is ideal for building models at both abstract and implementation dependent levels. </li></ul><ul><li> </li></ul><ul><li> </li></ul>
    37. 37. MaTeLo <ul><li>Usage Model Editor is a first MaTeLo module which let you create a project, design and edit a usage model of the system under test. </li></ul><ul><li>Testor is the second MaTeLo module which let you automatically generate test cases by browsing the usage model and following one path or another by randomly selecting transitions according to their occurring probability. Testor finally can help you manage the test campaign and get some information about it such as coverage, system reliability or Mean Time To Failure. </li></ul>
    38. 38. What is a usage model? <ul><li>A usage model represents the interactions between a system and its users. . </li></ul><ul><li>A MaTeLo usage model is an extended Markov chain for test and validation purposes. It’s a state diagram where a transition from a state to another is related to a system input and its activation frequency in the current state. </li></ul><ul><li>A chain is made of 2 specific states: Invoke and Terminate. Those states usually represent the system start and stop. </li></ul><ul><li>Chains can be nested, in order to get a hierarchical model. </li></ul>
    39. 39. USAGE MODEL STRUCTURE <ul><li>When to perform the MaTeLo model? </li></ul><ul><li>Ideally, the best moment to design your usage model is when you’ll have written about 2/3 of your system’s specifications. </li></ul><ul><li>Eventually, you can update your model just before the test campaign to ensure that your model consistency with the specification at this time. </li></ul>
    40. 40. Cont… <ul><li>What’s the usage model use? </li></ul><ul><li>The usage model lets you generate test sequences made of stimuli to perform on the system as well as the results to expect from the system. A model design stands for a test plan design </li></ul>
    41. 41. How to model? <ul><li>The most useful entry documents for MaTeLo modeling are the system’s functional specifications. Ideally, this specification will be made of Use Cases, which will dramatically reduce the system understanding effort. </li></ul><ul><li>modeling should consider the main validation aims, in order to better focus on those aims. Such aims could be: reliability, performance, robustness … </li></ul><ul><li>A chain should not be too sequential (wrong model) and avoid redundancy. </li></ul>
    42. 42. Elementary Modeling Rules: <ul><li>There can only be one transition going out the A state and entering the B state. (except for conditional transitions) </li></ul><ul><li>No transition should enter the Invoke state, and no transition should go out from the Terminate state. </li></ul><ul><li>A sub-chain respects the same design rules as the main chain. </li></ul>
    43. 43. Profiles Management <ul><li>It is possible to use one or more usage profiles . Profiles make it possible to model various behaviors of SUT and thus obtain different tests according to the usage frequencies allocated to the transitions according to the selected profile. </li></ul>
    44. 44. Testor <ul><li>Testor is the second module of the MaTeLo system dedicated to the automatic generation of the Test Plans and to the Test Campaign Analysis. </li></ul>
    45. 45. Typical use of Testor <ul><li>1.Use Testor to open a test project created by UM Editor. </li></ul><ul><li>2. Use the first panel to do the Model Checking and request a MC reporting, to analyze the parameters (stationary distribution, test case length and visiting probability). </li></ul>
    46. 46. Cont… <ul><li>3. Use the second panel to generate a test suite. It’s possible to generate: </li></ul><ul><li>Most probable test suite </li></ul><ul><li>Random test suite </li></ul><ul><li>Limit test suite </li></ul><ul><li>Minimum test suite (only if your model does not include asynchronous or conditional elements) </li></ul>
    47. 47. Cont.. <ul><li>4. Use the third panel to enter manually or import the test execution for each system version of the SUT. </li></ul><ul><li>Debugging test: not described by a test suite, debugging test is a simple test like “launching the application” </li></ul><ul><li>Most probable test: one test for the most probable usage of the SUT </li></ul><ul><li>Random Test (xN campaign) </li></ul><ul><li>Limit Test </li></ul><ul><li>Minimum test </li></ul><ul><li>Adding test: additional test, no generate by MaTeLo </li></ul>
    48. 48. Conformiq <ul><li>Conformiq Qtronic is a tool for both online testing of systems and for offline generation of test scripts </li></ul><ul><li>It can use the model to directly test a running system or to generate test scripts that can be independently executed afterwards. </li></ul><ul><li>Generated test scripts can be stored in version control system </li></ul><ul><li>tests can be sent to colleagues or to execute them independently </li></ul>
    49. 49. Are You Feeling Like This OK…. Let us Finish……….
    50. 50. Conclusion
    51. 51. Conclusion <ul><li>Modeling is not easy </li></ul><ul><ul><li>What details do you leave out? </li></ul></ul><ul><ul><li>How do you check the model? </li></ul></ul><ul><ul><li>There are no recipes for success </li></ul></ul><ul><li>What needs to happen </li></ul><ul><ul><li>Easier notation </li></ul></ul><ul><ul><li>Incremental approach </li></ul></ul><ul><ul><li>Pilots and examples </li></ul></ul>
    52. 52. Questions <ul><li>Does tool cover stress testing? </li></ul><ul><li>Does Tool cover how user friendly the system is or it only covers functionality? </li></ul><ul><li>How do we guarantee that the generated expected outputs and inputs are enough and cover all the cases? </li></ul><ul><li>How to make for the errors and exceptions that may occur due to running the program on different environments and different OSs? </li></ul><ul><li>How is automated generation of inputs and outputs work when an actual human interaction is required as part of the system testing? </li></ul>
    53. 53. References
    54. 54. References <ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> masses .pdf </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li>Colin Campbell, Wolfgang Grieskamp, Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes “Spec Explorer: An Integrated Environment for Model-Based Testing.” Foundations of Software Engineering Microsoft Research, Redmond </li></ul>
    55. 55. References <ul><li>EXECUTABLE UML by Selo Sulistyo and Warsun Najib </li></ul><ul><li>“ Towards Executable Aspect-Oriented UML models” By “Lidia Fuentes and Pablo Sánchez” ,March 12-13, 2007. </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li>“ Model Driven Testing “ By Brian Berger, Majdi Abuelbassal, and Mohammad Hossain, 1997 </li></ul><ul><li>VP-UML User’s Guide (Part 1) </li></ul>
    56. 56. Thanks