Software Development Automation conference Amsterdam

1. Rachid Kherrazi
8-10-2014 Amsterdam
TESTING ASD INTERFACE COMPLIANCE WITH
ASDSPEC
“A TOOL COMBINING THE BEST OF TWO TECHNIQUES”

2. PEOPLE INVOLVED
• Rachid Kherrazi
– Sr. Consultant-Project Leader @ Nspyre
– Domain
• Process and Product Improvement
• Model Based Testing and Model Driven Engineering
• iSQI Accredited Certified Model Based Tester Trainer
• Also : Arjan van Der Meer & Marc Hamilton
2

3. ABOUT NSPYRE
MARKET SEGMENTS
• High Tech
• Traffic & Infra
• Industry
AREA’S OF EXCELLENCE
(Model Based) Systems Engineering /Model Driven
Engineering / Model Based Testing /Industrial Automation /
Simulation / Big Data / Mobile Solutions
3

4. CONTENTS
1) Introduction to Model Based Testing
2) Generation of test cases form ASD interface model using
ASDSpec tool
3) Case Study and Results
4) ASDSpec Technology
4

5. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
Model Based System
Engineering
System Acceptance
Some of related technology
5

6. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
Model Driven
Engineering
System Acceptance
Some of related technology
6

7. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
System Acceptance
Model Based Testing
Some of related technology
7

8. IntegrationDesign
Coding
Requirements System test
Test case specification
(design of logical test case)
(reduction of number of test cases by application of test techniques
Test execution
(reporting)
Test case generation
(design of physical test case)
(selection of input values and calculation of expected results)
PAGE 8
TEST PROCESS
3 main steps in test process
Module
Subsystem
System
8

9. MBT IS THE AUTOMATION OF TEST CASE
GENERATION
Manual Automatic
Test specification
Test execution
Test generation
Modelbasedtesting
Traditionalmanualtesting
Traditionalautomatictesting
Manual
Scripting
Testing skills + Scripting skills + Modeling skills
Manual
Modeling
9

10. MBT PROCESS IN A NUTSHELL
Requirements
SUT
Expected outputs
(Test Oracle)
Inputs
(Test sequences)
Verdict Failed
Actual result # expected
result
Test suite
generate
author
feedback
feedback
feedback
Feedback
(via review, model
exploration, Model
checking)
control
Test Model
Actual outputs
Implementation Bug
Modeling error
Specification error
Design error
Conformance failures
10

11. EXAMPLE:
MODELING WITH SPEC EXPLORER LOOK AND FEEL
Explore &
Analyze
Remodel
Generate
C# Model
(or other .Net
Language)
Model Graph
Test Suite
11
Execute
Nunit
Note: this is not implementation
code, this is the textual Test model
(describing the interface behavior)

12. System Under Test
WHY ARE MODELS NEEDED FOR
TESTING?
12
Component A
Component C
Component B
Unit A1
Unit A2
Unit B1 Unit B2
Unit C1 Unit C2
Unit testing is insufficient
- Single unit may work properly
in isolation
- Incorrect interaction between
units may cause serious
security/reliability failures
System-level testing:
- Requires model of externally visible system behavior
- Traditional FSM-based testing does not scale
- Behavior is often reactive/nondeterministic
- Implementation is multi-threaded or distributed
- Thread scheduling hard to control
- State space is typically infinite

13. SOME BENEFITS OF MODEL BASED
• Increased productivity (increased automation)
• Better test script maintenance
• Improved product reliability (early defect detection)
• Agility ( Easily react to new feature changes, Reusability of test
semantics, Early test engagement, Drive quality upstream)
• Increased employee satisfaction (challenging, new horizon, fun)
13

14. MBT:
CHALLENGES AND OPPORTUNITIES
Challenges:
MBT is not easy: Different mind shift from traditional testing (Modeling skills)
and steep learning curve and high ramp up cost (tooling)
Opportunities:
Modeling skills specialized training and certification
 e.g. iSQI Certified MBT training (CMBT)
 Nspyre is a iSQI training provider
Tooling More (Open Source) supporting integration of modeling, M2M
transformation,…
 e.g. Nspyre ASDspec
14

15. SUPPORTING TOOLING
Manual Automatic
Test specification
Test execution
Test generation
Modelbasedtesting
Traditionalmanualtesting
15
Testing skills
Manual
Modeling
MBTnext
Semi-automatic
Modeling
Manual
Scripting
+ Scripting skills Modeling skills
Traditionalautomatictesting

16. IDEA: REUSE EXISTING MODELS
16
ASD Interface Model
ASDSpec
SpecExplorer

17. VERUM ASD:SUITE
17
Design model
Interface model
Requirements
Features
Use cases
Prototype
Decomposition
Code generation
Simulation & verification

18. NSPYRE ASDSPEC
18
Design model
Interface model
Requirements
Features
Use cases
Prototype
Decomposition
Test model
Test cases
ASDspec
Code generation
Simulation & verification
Testing
Note: reuse only the
Interface model! Design
model is not used

19. WHAT IS VERIFIED BY ASD
19
System Under Test
Component A
(ASD genereted)
Component C
(ASD genereted)
Component B
(hand written)
Unit A1
Unit A2
Unit B1 Unit B2
Unit C1 Unit C2
ASD verification doesn’t cover:
• Handwritten /legacy code
• Interaction between generated/handwritten component
• Data interaction
 With MBT (Spec Explorer) we can cover all the above aspects!

20. ASDSPEC: WORKFLOW
Spec
Explorer
Model
Cord
file
Spec
Explorer
Test
Cases
20
tester /modeler
Cord file:
• Test Selection /Slicing /Scenario Selection
• Data Combination
• Model Composition
Model:
• Interface behavior
Spec Explorer (MBT tool):
• Generate automatically test cases
MBT

21. ASDSPEC: WORKFLOW
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
Script
Spec
Explorer
Test
Cases
Reuse existing ASD interface models to generate Spec
Explorer MBT models automatically
21
MBT

22. ASDSPEC: DATA AND MODEL COMPOSITION
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
file
Spec
Explorer
Test
Cases
Generated model can be extended with any Spec Explorer feature
in cord file (config file)
22
Cord file:
• Test Selection /Slicing /Scenario Selection
• Data Combination
• Model Composition
MBT

23. USE CASE: CONTAINER TERMINAL
• Multiple components
• Components need to interact to function
• Controller needed to coordinate interaction
23

24. RESULTS:
ASD-CONTAINER TERMINAL
24
Higher overall productivity compared to traditional development.
Main remaining problems:
- Testing interaction and complete system (not fully supported)
- Data interaction not fully supported
- Debugging of third party library (some bugs found in manual written
code, legacy code)  Testing still necessary
Productivity

25. Method
OVERVIEW OF PROJECT STAGES
25
Use case
/SUT
Goal
Technique
Tool
Results
Stage 1
Container Terminal (CT)
Development of control
software for the CT
Model Driven Engineering +
hand written sw
ASD:Suite
Create design Models +
interface models in ASD
Generate code
Productivity (code
generation) c.t. trad. dev
-
Testing complete system,
Interaction, data, external
code
Stage 2
Container Terminal (CT)
Verification of the
developed control software
of the CT (generated +
Hand written)
Model based Testing
Spec Explorer
Create Test Model in Spec
Explorer, generate test
suite
Test Productivity
Modeling skills,
complexity costs
Stage 3
Container Terminal (CT)
Verification of the developed control
software of the CT (generated + Hand
written)
Model based Testing
ASDspec & Spec Explorer
Generate Test Model from existing ASD
interface Model , complete Test Model ,
generate test suite
+ Productivity (code generation, partial test
generation)
Testing complete system, Interaction, data,
external code
(benefits only in case of existing of ASD
Models)

26. RESULTS:
ASDSPEC- CONTAINER TERMINAL
26
Spec Explorer ASD + ASDSpec
Approach Generate test suite Generate test model
Techniques MBT MBT/MDE (M2M)
Effort/complexity Medium Low
Test cases 89 93
Perceived effectiveness Medium High
Bugs Some bugs in hand written code
/HAL
All known bus found
• Low effort model creation
- Easy reuse of existing work
• Testing of complete system
- Legacy code, data interaction
• High Quality, Cost Reduction

27. GENERAL MDSD ARCHITECTURE
Model
Reverse
Engineering
Model
Repository
Model
Code,
Doc, etc.
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
27

28. ASDSPEC ARCHITECTURE
ASDSpec
makes use of
existing MDSD
architecture
Model
Reverse
Engineering
Model
Repository
Spec
Explorer
Model
SpecExplorer
Cord
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
ASD
Interface
models
28

29. ASDSPEC: TECHNOLOGIES
QVTO
ASD
XML
SE
model
Cord
script
Acceleo
Acceleo
ASD
EMF
SE
EMF
Cord
EMF
QVTO
QVTO
ASDSpec
Model transformations bridge gap between ASD
and Spec Explorer
29

30. TOOL STATUS
• Tool as an standalone application.
• (Pending) tool will be available as an open source
Eclipse plugin for the Eclipse community.
• For now available on request
• Future steps
– Support data handling/configuration aspects in
ASDSpec, instead of relying on manual additions in
Spec Explorer,
– Supporting other MBT tools.
30

31. ASDSPEC:
USING BEST OF BOTH WORLDS
• Automatic generation of Spec Explorer test model from existing ASD
interface model
Benefits
• Less effort for model creation
– reuse of existing work
• Testing of complete system including
– Legacy code
– External components
– Data combination testing
– Interaction testing
• Results: High Quality, Reduced cost
Statistical Dynamic
ASDspec
31

32. CONCLUSIONS
• MDE and MBT technologies have matured a lot in the
latest years
• It is a matter of time….. Evolution…..
32

33. Using Formal Specifications to
Support Model Based Testing
ASDSpec: A Tool Combining the
Best of Two Techniques
ASD
Static Dynamic
ASDspec
Static Dynamic
ASDspec

34. 34
Source:
On the Use of Verification and
Quantitative
Analysis in Industry
Jozef Hooman
Radboud University Nijmegen
&
Embedded Systems Institute (ESI),
Eindhoven
26 October 2012
TESTING STILL NECESSARY

35. TESTING STILL NECESSARY
35
Source:
MDD and its impact on
testing...
A nanotech case study
ASD case
Bryan Bakker
EuroSTAR
May 2012

36. TESTING STILL NECESSARY
36
Source:
Ultrafast development of
an Ultra Fast Scanner
Model-Driven Development Day
2011-04-29
Hans Spitshuis