This document discusses model based testing using Spec Explorer. It begins with an introduction to model based testing and its benefits over traditional testing. It then demonstrates modeling a simple "Say Hello" application in Spec Explorer, including generating test cases from the model and executing them. Key benefits of model based testing include avoiding integration issues and post-release defects by more thoroughly testing interactions between units.

1. MODEL BASED TESTING WITH
SPEC EXPLORER
By Rachid Kherrazi, Kyra Hameleers and Adrian Iankov
Radboud University Nijmegen 04-12-2015

2. MODEL BASED TESTING WITH
SPEC EXPLORER AND ASDSPEC
• Part 1 – Introduction to Model Based Testing
– What is MBT
– Why MBT
• Part 2 – MBT with Spec Explorer
– Modeling, test case generation and execution
– Say Hallo example
Rachid Kherrazi

3. IntegrationDesign
Coding
Requirements System test
Test case specification
(design of logical test case)
(reduction of number of test cases by application of test techniques
Test execution
(reporting)
Test case generation
(design of physical test case)
(selection of input values and calculation of expected results)
TEST PROCESS
3 main steps in test process
Module
Subsystem
System
3Rachid Kherrazi

4. MBT IS THE AUTOMATION OF TEST CASE
GENERATION
Manual Automatic
Test case specification
(design of logical test cases)
Test case execution
(and reporting)
Test case generation
(design of physical test case)
(Actions & expected results)
Modelbasedtesting
Traditionalmanualtesting
Traditionalautomatictesting
Manual C# Scripting
Testing skills + Scripting skills + Modeling skills
Modeling
Lightweight
Modeling languages
Generate
Scripting
Rachid Kherrazi

5. MBT PROCESS IN A NUTSHELL
Requirements
SUT
Expected outputs
(Test Oracle)
Inputs
(Test sequences)
Verdict Failed
Actual result # expected
result
Test suite
generate
author
feedback
feedback
feedback
Feedback
Model based analysis
(via e.g. model exploration, Model checking)
control
Model
Actual outputs
Implementation Bug
Modeling error
Specification error
Design error
Conformance failures
Rachid Kherrazi

6. TYPICAL ISSUES WITH TRADITIONAL TESTING
• Many problems discovered at a late stage of
development process, often costly to resolve.
• In addition, they affect the planning of development
processes in that they must be resolved 'In between'
Several pre and post release defects not found with
traditional testing, which results in
Several delays in planning
With related quality/cost impact
Typical what we
hear from our
customers
Rachid Kherrazi

7. ROOT CAUSE
• Unit testing is insufficient
– Single unit may work properly in isolation
– Incorrect interaction between units may cause serious
reliability failures
• System-level testing
– Requires model of system behavior
– Behavior is often reactive/nondeterministic
• Implementation is multi-threaded or distributed
• Thread scheduling hard to control
– State space is typically infinite
– Traditional FSM-based testing does not scale
Rachid Kherrazi

8. MBT HELPS TO AVOID INTEGRATION
ISSUES AND POST RELEASE DEFECTS
Review
Static testing
Unit testing
Manual testing
happy flows
Manual testing
alternative flows
Manual testing
bad weather flows
Typical uncovered
Post release defects
Typical uncovered
Integration issues
This is typical domain
where Model Based
Testing helps
You can try to avoid
integration issues and
post release defects by
adding manual test case,
but it will be complex
will take lot of effort and
finally costly
Increased complexity and effort of testing
Manual exhaustive
exploratory testing
Rachid Kherrazi

9. MODEL BASED ANALYSIS AND TESTING
Unit testing
Integration testing
System testing
Acceptance testing
Coding
Design
Specification
Requirements
Static analysis Dynamic Analysis
Model based SW Analysis
Visualization, simulation, safety
analysis (design errors)
Model based SW testing
Execution of test suite against
implementation
Some MBT tools (like NModel, PyModel, Spex Explorer, ModelJUnit) allow also Model Based Analysis
e.g. safety violations (where the program reaches forbidden states), deadlocks (where the program seems to stop running
and stops responding to events) livelocks (where the program cycles endlessly but can’t make progress)
Simulation= A model-based analysis technique, where runs of a model program are observed
Rachid Kherrazi

10. OTHER DRIVERS FOR MBT
Our world evolves….and becomes complexer
– Cyber-Physical Systems/EoT
– Machine2Machine
– Digital Transformation
Increased need to manage complexity
 Increased need to technology enablers
Innovative company’s need innovative tools and approaches to develop
innovative products and services
 Model Based System Engineering
 Model Driven Engineering
 Model Based Testing
Rachid Kherrazi

11. MBT TOOLS AND SPEC EXPLORER FAMILY
• Explosion of tools (most from academia, little products)
• Trend
– from graphical to textual modeling
– Platform independent /open sources
• The story of the Spec Explorer “family”
ASML (Microsoft research)  NModel
PyModel
ModelJUinit
RubyModel
…..
Open sources
Textual modeling
Spec Explorer
Add-in MS VS
(Product, support,
documentation)
History
Platform independent
Rachid Kherrazi

12. MODELING WITH SPEC EXPLORER: LOOK AND FEEL
Explore &
Analyze
Remodel
Generate
C# Model
(or other .Net
Language)
Model Graph
Test Suite
Execute
Nunit
http://www.nunit.com/
Rachid Kherrazi

13. Environment
System Under Test
E
S
Abstraction
SE
Model based
testing
S: includes all details of SUT but nothing about expected environment
E: full info about environment but nothing about SUT behavior
SE: most extreme situation, too match detail, unpractical
Abstraction is essential  M is typical for model based testing
Abstraction: data, function, communication abstraction…
M
MODEL ABSTRACTION
Rachid Kherrazi

14. EXAMPLE: HELLO BEST
TESTERS
A typical use case:
# Action Expected result
1 Start application App. started
2 Press Say Hello button Text box = “Hello Best Testers”
3 Press Clear button Text box = “ “
4 Exit application App. closed
Rachid Kherrazi

15. MODEL: A SIMPLE FSM
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
States, Variables, Actions, pre and post condition
Rachid Kherrazi

16. Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
Test selection criteria: Random walk with length N=10
TEST SUITE GENERATION
Rachid Kherrazi

17. TEST SUITE GENERATION
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
Test selection criteria: Visit all state
Rachid Kherrazi

18. TEST SUITE GENERATION
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
Test selection criteria: Chinese Postman (visit all transitions a.s.a.p) (shortest path)
Rachid Kherrazi

19. C# MODEL
namespace DemoModel
{
public enum State
{
NotRunning,
Running
}
static class ModelProgram
{
static State appStatus = State.NotRunning;
static string TextBox=" ";
[Probe]
public static string Satatus()
{
return string.Format("appStatus:{0},TextBox:{1}", appStatus, TextBox);
}
[AcceptingStateCondition]
static bool InitialStatus()
{
return appStatus == State.NotRunning ;//&& TextBox == " ";
}
States declaration
Initialization
Optional, just for visualization in graph
When to stop in case of test case
Rachid Kherrazi

20. C# MODEL
[Rule(Action = "StartApp()/result")]
static Implementation.State StartApp()
{
Condition.IsTrue(appStatus == State.NotRunning);
Requirement.Capture("requirement1.1:StartApp");
appStatus = State.Running;
return (Implementation.State)appStatus;
}
[Rule(Action = "StopApp()/result")]
static Implementation.State StopApp()
{
Condition.IsTrue(appStatus == State.Running);
Requirement.Capture("requirement1.2:StopApp");
appStatus = State.NotRunning;
TextBox = " ";
return (Implementation.State)appStatus;
}
}
}
Rachid Kherrazi

21. C# MODEL
[Rule(Action = "ClearText()/result")]
static string ClearText()
{
Condition.IsTrue(appStatus == State.Running);
TextBox = "";
Requirement.Capture("requirement1.3:ClearText()/result");
Console.WriteLine("Clear text");
return TextBox;
}
[Rule(Action = "SayHallo(A)/result")]
static string SayHallo(string A)
{
Condition.IsTrue(appStatus == State.Running);
Requirement.Capture("requirement1.4:SayHallo");
TextBox = A;
return TextBox;
}
Action declaration
Requirement capture
Rachid Kherrazi

22. Offline
Tests are generated and stored in files
Test execution is separate (different environment)
Can generate ones, but executed many times
e.g.VS or Nunit, QulaityCenter
MBT Tool
Offline testing

23. Online
Tests are executed in parallel with generation
MBT tool play the game with system under test
Good for testing non-deterministic systems
MBT Tool
Online testing

24. MBT is…..
Strategies, tools and artifacts
Manage complexity
Reduces the need of manual or human involvement of interaction
Avoids spending time in unskilled repetitive error prone or redundant tasks
Provides bandwidth to Innovate!!