This document discusses modern software testing and formal verification techniques. It covers various testing levels including unit testing, integration testing, and manual testing. It also discusses test-driven development (TDD), behavior-driven development (BDD), and model-based testing (MBT). TDD involves writing tests before code, BDD uses natural language specifications to guide testing, and MBT generates tests from behavioral models. The document provides examples of testing a login system using Spec Explorer to demonstrate MBT.

1. Modern Software Testing and Formal Verification
Techniques
Day 1
Sergey Staroletov
Polzunov Altai State Technical University,
Lenin avenue 46, Barnaul, 656038, Russia
Email: serg soft@mail.ru
June 25, 2019
1 / 65

2. QA in the modern software world
Quality Assurance – is a process to satisfy of a desired level of the
quality for system under implementation or maintenance
Testing – is a process to check the behavior of the system with
respect to given specifications (assumptions, tests)
Formal Verification – is a process to proof of the correspondence
of a model of the system to given formal requirements
NB: Level can be too high or tool low, specifications can be incorrect or
incomplete, models can be wrong.
So: QA is an area where no one does not guarantee nothing.
We are able to improve but are not able to guarantee.
2 / 65

3. Unsafe, uncomfortable, erroneous...but can be used
3 / 65

4. A few words about testing...
The main goal of testing – show possible inconsistence between
expected and actual behaviors.
In other words: to find bugs
Do not prove that we have no bugs
Just to find a counterexample = a bug.
4 / 65

5. General Testing Problem
The problem of testing is answering the question: whether it is
possible for any program to determine does it execute a given
algorithm or has errors and does not execute a given algorithm?
Assume that the program works in a simple single-threading
environment, has a set of control states, and after each step of
transition to a next state, the values of the variables are printed. If
(after the completion of the program) the output is exactly equals
to the expected, the program is recognized as a correct one. The
problem of testing is whether it is possible to do such a test for any
program according only to its description.
5 / 65

6. General Testing Problem is undecidable over current
CPU architectures
A program can be counted as a G¨odel number
The GTP is equals to a formula translation problem
Test(N) =⇒ Result
It is equals to the Turing Machine Halting Problem
So it is undecidable.
The Principle of Methodological Diversity: To satisfy given level or
quality we should propose as much testing methods as possible;
every particular method can have a chance to find a whole nother
bug.
6 / 65

7. V-Model of Developing and Testing
7 / 65

8. MDD: Model-Driven Development
8 / 65

9. MDD in Double-V-Model of Developing and Testing
9 / 65

10. Double-V-Model of Developing and Testing
10 / 65

11. Triple-V-Model of Developing and Testing and
Verification
11 / 65

12. Triple-V-Model of Developing and Testing and
Verification and Tools
12 / 65

13. Testing levels
The level of code =⇒ Static analysers
Unit level =⇒ Unit testing
The level of Functionality =⇒ Automated scripts, Manual testing
by testers according to plans and specifications
13 / 65

14. Manual Testing / Work of a tester / QA-engineer
14 / 65

15. Static analysers
No models to write by the users!
Additional warnings in an IDE
Check common smells and security weaknesses
Integration into Continuous Integration / Continuous Delivery
processes
SonarQube
PVS Studio
cppcheck
...
15 / 65

16. SonarQube
16 / 65

17. PVS Studio
17 / 65

18. Manual Testing / Work of a tester / QA-engineer
18 / 65

19. Unit testing
The approach was created by introducing the JUnit framework
Implemented by Erich Gamma (one of the GoF)
It was coded during a flight from the USA to Europe
We have an expected value
We run a method a get an actual value
We check the correspondence between the actual and the
expected values.
19 / 65

20. Unit testing – usual case
20 / 65

21. Unit testing in a glance
Tests are written by the programmers
They create tests to checks methods without running the whole
app
One Assert per test! Test writer should create a lot of tests for the
different usage cases of a method, to satisfy the coverage of
internal ifs, cycles, work with different values of parameters
Tests are descriptable: they specify how the class should be used
Unit testing can be used at the first stages of learning new
languages, new libraries, new frameworks
The best way to create tests not for their own code, but for the
other teammember’s code.
21 / 65

22. Normal Unit-Testing
22 / 65

23. Why not doing like this?
23 / 65

24. TDD
Test
Driven
Development
Proposed by Kent Beck
24 / 65

25. TDD
Write a test
Write a MINIMAL code to pass the test (or refactor existing)
COMMIT the changes
Think about a new test
At the each study of the developing we have a working code that
corresponds to the tests
Maximum test coverage
Developers are lazy and do not think about complex
implementations
The process is tracked by VCS
Software can be demonstrated to a customer or business by a
request.
25 / 65

26. TDD demo
26 / 65

27. TDD demo : calculator
27 / 65

28. TDD demo : calculator
28 / 65

29. TDD demo : calculator
29 / 65

30. TDD demo : calculator
30 / 65

31. TDD demo : calculator
31 / 65

32. TDD demo : calculator
32 / 65

33. TDD demo : calculator
33 / 65

34. TDD demo : calculator
34 / 65

35. TDD demo : calculator
35 / 65

36. TDD demo : calculator
36 / 65

37. TDD demo : calculator
37 / 65

38. TDD demo : calculator
38 / 65

39. TDD problems
Developer thinks too much about the code
Customer is not involved into the process
Managers think that the process is costly
Small amount of companies use it because lack of experience
how to setup this process
39 / 65

40. BDD
Behavior
Driven
Development
Firstly it was implemented in Cucumber framework
40 / 65

41. BDD process
1 Write specification in almost natural language (it could be done by
the customer!)
2 Link the specification with the code
3 Write code of testing calls, asserts
4 Execute the specification
5 If it fails, write / refactor code as in TDD
41 / 65

42. BDD demo : calculator
Specification in Gherkin language
42 / 65

43. BDD demo : calculator
43 / 65

44. BDD demo : calculator
44 / 65

45. BDD demo : calculator
45 / 65

46. BDD demo : calculator
46 / 65

47. BDD demo : calculator
47 / 65

48. BDD demo : calculator
48 / 65

49. BDD demo : calculator
49 / 65

50. BDD demo : calculator
50 / 65

51. BDD demo : calculator
51 / 65

52. BDD demo : calculator
52 / 65

53. My research on TDD and BDD
Building a process of trustworthy software developing based on
BDD and ontology approaches with further formal verification.
PSSV 2018 Proceedings
An Application of Test-Driven Development Methodology into the
Process of Hardware Creation (a View from a Software
Perspective). PSSV 2019 Proceedings
An IDE (IDEA) plugin to create list of tasks and steps and
generate tests and code. Bachelor’s diploma work of V.V.
Vorobyeva.
53 / 65

54. Introduction to MBT
And if we do not want to create unit tests by hand?
We do not want to support the tests : code is changed too fast
Our system is too complex
Behavior can be described as an automaton
So it is goal to use Model-Based Testing approach
54 / 65

55. Introduction to MBT
Microsoft Spec Explorer
Integrates into MS VS 2012
Based on Abstract State Machines
Cord Script to specify a behavior
Code in C# for implementations under test
55 / 65

56. Login demo
56 / 65

57. Login demo
57 / 65

58. Login demo
58 / 65

59. Login demo
59 / 65

60. Login demo
60 / 65

61. Login demo
61 / 65

62. Login demo
62 / 65

63. Login demo
63 / 65

64. Login demo
64 / 65

65. Learn more about all we have touched today
Want to think about testing and verification in a warm atmosphere?
65 / 65