1. Esin Karaman
SWE550
Boğaziçi University
1

2. Agenda
Software testing process evalution
What is model-based testing
The process of model-based testing
MBT (Model-based testing) Tools
Prerequisites for MBT
Benefits of MBT
Limitations of MBT
2

3. «Life is too short for manual testing.»
Harry Robinson, test architect for Microsoft’s Engineering Excellence Group and a driving force behind Microsoft’s model-based testing initiative, 1999-2004
3

4. Software testing process evalution
4
Manual Testing
Capture/Replay
Script Base Testing
Keyword-Driven Automated Testing
Model-Based

5. What is model-based testing?
Model-based testing is a variant of testing for automatic generation of test cases using models describing some aspects of the software under test (SUT).
It can automate the complete test design process, given a suitable model, and produces complete test sequences that can be transformed into executable tests cripts.
Model-based testing is the automation of the design of black-box tests
5

6. What is model-based testing?
The difference from the usual black-box testing is that rather than manually writing tests based on the requirements documentation, we instead create a model of the expected SUT behaviour, which captures some of the requirements. Then the model based testing tools are used to automatically generate tests from that model.
6

7. The process of model-based testing
Create a model detailed enough to describe the characteristics that we want to test
Automatically generate abstract test cases from the model by using a model-based testing tool
Transform abstract test cases into into executable test scripts
Execute test scripts by a test execution tool
7

8. 8
Model-Based Testing Process

9. Modelling
A model of the SUT is built on the grounds of requirements
Includes expected behaviour
Must be concise and precise
9

10. Modelling (2)
Model-based testing uses models of the SUT and its environment.
Abstraction
- Function
- Data
M1-M3 is suitable
10

11. Modelling (3)
Modelling notations
State based (or Pre/Post) Notations
Transition-based Notations
Functional Notations
Operational Notations
Stochastic Notations
Data-Flow Notations
11

12. Test requirements selection
Used to control the generation of tests Steps:
Test selection criteria are defined
Test selection criteria are then transformed into test case specifications
12

13. Test selection criteria
Defining «best» test selection criteria is not possible. Test engineer chooses adequate test selection criteria. Criteria types:
Structural model coverage criteria
Data coverage criteria
Requirements-based coverage criteria
Ad-Hoc test case specifications
13

14. Test generation
Once the model and the test case specification are defined, an abstract test suite is generated.
Technology:
Random
Graph-search algorithms
Model-checking
Symbolic execution
Theorem proving
14

15. Test concretization(transformation)
Concretize abstract test suite to executable level (test script).
This can be done by a MBT Tool using some translation tables supplied by the test engineer.
The resulting executable tests may be Junit in Java, or in a dynamic language such as Tcl or Python, or in a dedicated test scripting language.
15

16. Test execution
Test scripts are executed and outcomes are compared to expected ones.
Tests are marked as pass, fail or inconclusive
Two approaches:
Online
Offline
16

17. Test execution(2)
Online Testing
Tests are executed as they are generated
MBT tool is tightly coupled to the SUT
Slow
Offline Testing
Tests are generated first, executed afterwards
Decouples the generation and execution phases
It can be used repeatedly for regression purposes
17

18. MBT Tools
TorX
LTG (LEIRIOS Test Generator)
Matlab Simulink V&V
JUMBL
AETG
18

19. Prerequisites for MBT
Someone enthusiastic
Test execution maturity
Modelling skills
Access to training, tools, and experts
The right kind of application
19

20. Benefits of MBT
Fault detection : MBT founds roughly the same number of errors as manually designed test suits
Reduced testing cost and time
Improved test quality
Requirements defect detection
Requirements evolution
20

21. Limitations of MBT
The model designers must be expert in the application area. This may require some training costs.
Usually used only for functional testing, little experience for other kinds of testing.
Sophisticated approach to testing, so it requires a reosonable level of test maturity. It is unwise to start using MBT without some experience with automated test execution.
21

22. References
Mark Utting, Alexander Pretschner and Bruno Legeard , A Taxonomy of Model-Based Testing, 2006
Bruno Legeard, Mark Utting, Practical Model-Based Testing A Tools Approach, 2007
A. Pretschner, W. Prenninger, S. Wagner, C. Kuhnel, M. Baumgartner, B. Sostawa, One Evaluation of Model-Based Testing and its Automation, 2005
22