The document outlines a scalable testing process based on formal models, utilizing a multilevel application model to automate test case generation from behavioral requirements. It details the use of Use Case Maps (UCM) for requirements modeling and the VRS verifier for consistency checks, highlighting the challenges of data mapping and test case execution. The proposed approach improves testing efficiency by automating the mapping of tests and leveraging parallel execution to handle large test suites.

1. Pavel Drobintsev, Vsevolod Kotlyarov, Ivan Selin, Alexey Tolstoles, Nikita
Voinov
Peter the Great Saint Petersburg Polytechnic University
Saint Petersburg, Russia
Scalable testing process
based on formal models

2. Model Oriented Approach
• Multilevel model of application during design process
• Behavioral model is built from requirements
• Model is iteratively specified
• Test cases are generated from this model in an automated way
03.03.2017 2

3. VRS/TAT
• Software verification and testing toolset
• Requirements are formalized in UCM language
• VRS verifier runs reachability and consistency checks
• Iterative process of model modification until all checks are passed
• Generation of symbolic traces
• Concretization (symbolic traces – traces with specific values)
• Adapting tests to run them on real system
• Testing
03.03.2017 3

4. Use Case Maps (UCM)
• High-level graphical language
for requirements modelling
• ITU Z.151
• Can be used for creating
structured behavioral diagrams
of interacting agents
• UCM notation doesn’t provide
enough information to
generate traces
03.03.2017 4

5. UCM Metadata
• Auxiliary text field linked to a
UCM responsibility
• Stores information about
variables and signals
• Signal is a way of message
transfer between agents
03.03.2017 5

6. VRS Verifier
• During the initial model development:
• Reachability and consistency checks
• During testing process:
• Generation of symbolic traces based on UCM model control flow
• Concretization (substitution of left, right and middle values from tolerance
range)
03.03.2017 6

7. VRS/TAT Levels of Abstraction
03.03.2017 7
Automated
Automated
Not automated

8. Challenges
• Data mapping problem:
• Abstract models which need to be extended in order to run against real
system
• Extension may require additional information
• New information must comply with what’s already in the model
• Amount of generated test cases may be too big to run using common
methods
03.03.2017 8

9. Data Structures Conversion Problem
• Structure of concrete test signals does not correspond with real
system signals
• Manual creation of detailed test scenarios takes too much time and
resources
03.03.2017 9

10. Data Structures Conversion. Approach
• Process called “Lowering”
• The name comes from descending on lower
levels of abstraction
• In general, “Lowering” can be described as
creating processing rules for each signal and
application of these rules to concrete
scenarios
• An editor was made to restrict user from
making incorrect structures
03.03.2017 10

11. Data Structures Conversion. Restrictions
• There are several restrictions to ensure the correctness of test
scenarios:
• If you separate concrete value into several independent parts, it is prohibited
to change them in a way, when joining them back together will give another
result than it was before separation
• Only structures similar to SUT interfaces could be used
• Only constant values from concretization and templates values are allowed
03.03.2017 11

12. Data Structures Conversion. Results
• Before introduction of “lowering”, test mapping was made by hand
after each tests generation
• After automating the tests mapping there are only 2 manual steps left
in VRS/TAT process:
• UCM model development
• Specifying “lowering” rules
• Both of them only needed to be done once
03.03.2017 12

13. UPD: VRS/TAT Levels of Abstraction
03.03.2017 13
Automated
Automated
Automated
Huge amount

14. Execution problem
• Huge test amount
• A lot of time needed for execution
• Costs increase
• Run tools in parallel
03.03.2017 14

15. VRS Guided Search
• VRS has an option to perform guided search
• Guide is a marked sequence on a model, which must be traversed
• All traces that don’t apply the criteria to travel through guide are cut
off by VRS during traces generation
• The idea is to set guides on UCM model and launch several VRS
instances with corresponding guides
03.03.2017 15

16. VRS Guide
03.03.2017 16

17. Execution process
• User sets guides in the most “thick” places of UCM model
• Several VRS instances run, each on a different compute node with a
unique set of guides
• As a result, whole test suite is divided into several parts
• Each part can be processed independently in an isolated container
(node, VM, etc.)
03.03.2017 17

18. Scalability
• Theoretically, scalability is linear
• In real life, it is not achievable because of irregular test distribution
and different execution complexity of test cases
03.03.2017 18

19. Conclusion
• Tests are generated from the verified model and can be ran against
real systems
• Presented approach allows to speed up the testing process
• It speeds up both tests generation and tests execution
• Scalable process, performance increase is limited by the model
and/or available computational resources
03.03.2017 19

20. Thanks for your attention!
03.03.2017 20

21. Backup 1. New features in lowering. TDL
• There is an ability to use SUT interfaces files (*.tdl) in Lowering Editor
to obtain parameters signal structure
03.03.2017 21

22. Backup 2. Few words on VRS
• VRS works with inner model
in basic protocols (Hoare
triples)
• Each one has:
• Pre-condition
• Process
• Post-condition
03.03.2017 22

23. Backup 2. Few words on VRS
• UCM model is converted into basic
protocols model using UCM
elements location and UCM
metadata (and inverse)
• VRS matches pre and post
conditions of different elements
and checks tolerance ranges to find
all possible traces
• Guided search builds traces from
guide forward and backward
03.03.2017 23