This document summarizes a survey of model-based testing tools for test case generation. It outlines the survey criteria used to evaluate the tools, including how they support model specification, test generation, test description, and overall testing process. Fourteen different tools are then compared based on these criteria. The document concludes with suggestions for future directions in model-based testing, such as improved support for online testing of real-time systems.

1. MAKE YOUR GLOBAL
MARKET EASY
www.eglobalmark.com
A SURVEY ON MODEL-BASED TESTING
TOOLS FOR TEST CASE GENERATION
Wenbin Li, Easy Global Market, France

2. OUTLINE
TMPA 2017, Wenbin Li (Easy Global Market) Page 2
Easy Global Market (EGM)
˃ Activities
˃ Projects
Model-Based Testing (MBT)
˃ Concept and Workflow
˃ Aspects
Survey Criteria
˃ Model Specification
˃ Test Generation
˃ Test Description
˃ Overall Support
Survey MBT Tools
Future Directions

3. EASY GLOBAL MARKET: ACTIVITIES
TMPA 2017, Wenbin Li (Easy Global Market) Page 3

4. EU H2020: PHANTOM
TMPA 2017, Wenbin Li (Easy Global Market) Page 4
•Multi-Objective Programming for Parallel Computing System
Model-based Testing for

5. MODEL BASED TESTING WORKFLOW
TMPA 2017, Wenbin Li (Easy Global Market) Page 5
(1) Creation of MBT Models
(2) Generation of Test Cases
(3) Concretization of Test Cases
(4) Execution of Test Cases
(5) Results Analysis

6. MODEL BASED TESTING ASPECTS
TMPA 2017, Wenbin Li (Easy Global Market) Page 6
Three Aspects
˃ Test Case Generation
˃ Test Data Generation
˃ Test Script Generation
Survey:
˃ Overall Support
˃ Special Focus on Test Case Gen.
Why Test Cases Generation?
˃ Complicated Selection Criteria
˃ Significant Benefits

7. SURVEY CRITERIA
TMPA 2017, Wenbin Li (Easy Global Market) Page 7
“A taxonomy of model-based testing approaches”
by Utting, M., Pretschner, A., Legeard, B.
Extension
˃ Test Description
˃ Overall Support

8. SURVEY CRITERIA
TMPA 2017, Wenbin Li (Easy Global Market) Page 8

9. SURVEY CRITERIA: MODEL SPECIFICATION
TMPA 2017, Wenbin Li (Easy Global Market) Page 9
Model
Specification
Scope
Characteristics
Paradigm
Input(I)/Input&Ouput(IO)
Untimed(U)/Timed(T)
Deterministic(D)/Non-Deterministic(N)/Both(B)
Discrete(D)/Continuous(C)/Hybrid(H)
Pre-Post Domains
Transition Based
History Based
Functional
Operational
Stochastic
Data Flow

10. SURVEY CRITERIA: TEST GENERATION
TMPA 2017, Wenbin Li (Easy Global Market) Page 10
Test Generation
Test Selection
Criteria
Technology
Structural Coverage
Data Coverage
Requirement Coverage
Test Case Specification
Random&Stochastic
Fault Based
Random Generation
Search Based
Model Checking
Symbolic Execution
Theorem Proving
Constraint Solving
Process Online/Offline/Both

11. SURVEY CRITERIA: TEST DESCRIPTION
TMPA 2017, Wenbin Li (Easy Global Market) Page 11
Test
Description
Test Traceability
Test Oracle
Format
Yes/No
Yes/No
Testing Languages: TTCN-3, Junit, etc.
Programing Languages: Java, C++, etc.
Mark Up Languages: XML, HTML, etc.
Test Frameworks: Selenium, Robot, etc.
Other Formats

12. SURVEY CRITERIA: OVERALL SUPPORT
TMPA 2017, Wenbin Li (Easy Global Market) Page 12
Overall
Support
Step
Coverage
Tool Type
Open Source(O)
Academic(A)
Commercial(C)
Step 1, 2, 3, 4, 5
Aspect
Coverage
1. Test Case Generation
2. Test Data Generation
3. Test Script Generation

13. SURVEY ON MBT TOOLS: MODELS AND TEST
TMPA 2017, Wenbin Li (Easy Global Market) Page 13
Tools
Model Specification Test Description
Scope Characteristics Paradigm Traceability Oracles Formats
DIVERSITY IO U/B/D Transition Based No Yes TTCN-3
FMBT IO U/D/D Pre-Post Domains No No AAL/Python
Modbat IO U/B/D Transition based Yes Yes Other formats
TCG IO U/B/D Transition based No No Other formats
Tcases I U/D/D Pre-Post Domains No No XML, JUnit
MISTA IO U/B/D
Operational,
Transition Based
No No
Python, HTML,
Selenium, etc.
MoMuT IO T/D/H Transition Based Yes No Other formats
HTG IO T/D/H
Transition Based,
Functional No No C/C++
Lurette IO T/B/H Functional Yes Yes Lutin
VERA IO U/D/D Transition based No Yes XML
CompleteTest IO T/D/H Data Flow Yes Yes C
CertifyIT IO U/D/D
Pre-Post Domains,
Transition Based
Yes Yes
JUnit, HTML, Perl,
HP quality center, etc.
PragmaDev IO T/B/H
Transition-based,
History Based
Yes Yes TTCN-3, C/C++

14. SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT
TMPA 2017, Wenbin Li (Easy Global Market) Page 14
Tools
Test Generation Overall Support
Test Selection Technology Process Step Aspect Type
DIVERSITY
Structural Coverage,
Random&Stochastic,
Test Case Specification
Symbolic Execution Offline 1-2 1,2 O
FMBT
Structural Coverage,
Test Case Specification
Search Based Both 1-5 1,3 O
Modbat Random&Stochastic,
Random Generation,
Search Based
Both 1-5 1 O
TCG
Structural Coverage,
Random&Stochastic
Search Based,
Random Generation
Offline 1-2 1,2 O
Tcases Data Coverage Constraint Solving Offline 1-3 1 O
MISTA
Structural Coverage,
Random&Stochastic
Search Based,
Random Generation
Both 1-4 1,3 O
MoMuT Fault Based Search Based Offline 1-2 1 A
HTG Data Coverage
Random Generation,
Search Based
Offline 1-3 1 A
Lurette Random&Stochastic Random Generation Online 1-3 1,2 A
VERA Structural Coverage Search based Both 1-5 1,3 A
CompleteTest Structural Coverage
Model Checking,
Search Based
Offline 1-3 1 A
CertifyIT
Structural Coverage,
Test Case Specification
Search Based,
Model Checking Offline 1-3 1,3 C
PragmaDev
Structural Coverage,
Random&Stochastic
Symbolic Execution Offline 1-5 1,2 C

15. DIVERSITY
TMPA 2017, Wenbin Li (Easy Global Market) Page 15

16. SURVEY ON MBT TOOLS: MODELS AND TEST
TMPA 2017, Wenbin Li (Easy Global Market) Page 16
Tools
Model Specification Test Description
Scope Characteristics Paradigm Traceability Oracles Formats
DIVERSITY IO U/B/D Transition Based No Yes TTCN-3
FMBT IO U/D/D Pre-Post Domains No No AAL/Python
Modbat IO U/B/D Transition based Yes Yes Other formats
TCG IO U/B/D Transition based No No Other formats
Tcases I U/D/D Pre-Post Domains No No XML, JUnit
MISTA IO U/B/D
Operational,
Transition Based
No No
Python, HTML,
Selenium, etc.
MoMuT IO T/D/H Transition Based Yes No Other formats
HTG IO T/D/H
Transition Based,
Functional No No C/C++
Lurette IO T/B/H Functional Yes Yes Lutin
VERA IO U/D/D Transition based No Yes XML
CompleteTest IO T/D/H Data Flow Yes Yes C
CertifyIT IO U/D/D
Pre-Post Domains,
Transition Based
Yes Yes
JUnit, HTML, Perl,
HP quality center, etc.
PragmaDev IO T/B/H
Transition-based,
History Based
Yes Yes TTCN-3, C/C++

17. SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT
TMPA 2017, Wenbin Li (Easy Global Market) Page 17
Tools
Test Generation Overall Support
Test Selection Technology Process Step Aspect Type
DIVERSITY
Structural Coverage,
Random&Stochastic,
Test Case Specification
Symbolic Execution Offline 1-2 1,2 O
FMBT
Structural Coverage,
Test Case Specification
Search Based Both 1-5 1,3 O
Modbat Random&Stochastic,
Random Generation,
Search Based
Both 1-5 1 O
TCG
Structural Coverage,
Random&Stochastic
Search Based,
Random Generation
Offline 1-2 1,2 O
Tcases Data Coverage Constraint Solving Offline 1-3 1 O
MISTA
Structural Coverage,
Random&Stochastic
Search Based,
Random Generation
Both 1-4 1,3 O
MoMuT Fault Based Search Based Offline 1-2 1 A
HTG Data Coverage
Random Generation,
Search Based
Offline 1-3 1 A
Lurette Random&Stochastic Random Generation Online 1-3 1,2 A
VERA Structural Coverage Search based Both 1-5 1,3 A
CompleteTest Structural Coverage
Model Checking,
Search Based
Offline 1-3 1 A
CertifyIT
Structural Coverage,
Test Case Specification
Search Based,
Model Checking Offline 1-3 1,3 C
PragmaDev
Structural Coverage,
Random&Stochastic
Symbolic Execution Offline 1-5 1,2 C

18. FUTURE DIRECTIONS
TMPA 2017, Wenbin Li (Easy Global Market) Page 18
Support of online testing for real time systems
Generation of test cases in standard testing language
Improvement of MBT tool interfaces
Record of traceability
Automatic generation of test oracle

19. THANK YOU
Tel:
E.mail:www.eglobalmark.com
•Dr. Wenbin Li
•Research Engineer
•+33-(0)9 82 33 06 72
•wenbin.li@eglobalmark.com