A tutorial presented in the User Conference on Advanced Automated Testing (UCAAT) October 19-21, 2021 By: Vahid Garousi Bahar Software Engineering Consulting Corporation Queen’s University Belfast, UK www.vgarousi.com @vgarousi Alper Buğra Keleş, Yunus Balaman, Zeynep Özdemir Güler Testinium A.Ş., Istanbul, Turkey

1. #UCAAT
Model-based Testing (MBT) of web and mobile
applications: A practical tutorial
Vahid Garousi, Alper Buğra Keleş, Yunus Balaman, Zeynep Özdemir Güler
Oct. 19, 2021

2. #UCAAT
• What is Model-based testing (MBT)?
• Using MBT in our projects
• MBT testing of our own large web application (Testinium)
• MBT of the web application of one of our clients (BTCTurk)
• MBT of two of UK’s COVID contract-tracing mobile apps
• Benefits of MBT, and when it is a good idea to use
MBT
• Pointers to more resources on MBT
Outline of this tutorial (duration: 1 hour)
2
V

3. #UCAAT
• Model-based testing is
the application of model-
based design for
systematic and
automated test-case
design and, optionally,
also for automated
execution of test cases.
• An example:
What is Model-based testing (MBT)?
3
Valid login
• Nodes, or vertices: “States”
(pages) of the system to verify
• Edges or transitions: Events
such as valid login, click
V

4. #UCAAT
• Of course, to apply any test technique, we need “good” tools…
• There are 100’s of MBT tools (you can see many of them via
Google search)
• Our choice: an open-source popular tool, named GraphWalker
(graphwalker.github.io). Reasons: in one of our articles (mentioned at the end)
To apply MBT, we need (good) tools!
4
V

5. #UCAAT
• What is Model-based testing (MBT)?
• Using MBT in our projects
• Testing our own large web application (Testinium)
• Testing the web application of one of our clients (BTCTurk)
• Testing two of UK’s COVID mobile apps
• Benefits of MBT, and when it is a good idea to use
MBT
• Pointer to more resources on MBT
Outline of this tutorial (duration: 1 hour)
5
V

6. #UCAAT
• System Under Test (SUT): A test automation
management tool: Testinium (testinium.com)
Testing our own large web application (Testinium)
6
V

7. #UCAAT
The entire “login” test model for Testinium
7
V

8. #UCAAT
MBT: Development of nodes/edges’ behavior in Java
using the Selenium framework
8
public void e_valid_login() {
WebElement userNameElement =
methodsPage.findElement(By.id("username"));
userNameElement.clear();
userNameElement.sendKeys(email);
WebElement passwordElement =
methodsPage.findElement(By.id("password"));
passwordElement.clear();
passwordElement.sendKeys(password);
methodsPage.findElement(By.cssSelector(
"input[class$="login-page__submit-btn"][value="Sign In"]")).click();
}
V

9. #UCAAT
Time for a live DEMO: MBT of Testinium
(2-3 minutes). Full execution of MBT suite would take more than 6 hours!
9
A
A video is also available online:
www.youtube.com/watch?v=RizUbMhBTho

10. #UCAAT
• Many online videos from our MBT projects are available on YouTube:
bit.ly/MBTVideosTestinium
• In a nutshell, in MBT, we developed a sophisticated “test” software, via
modeling, to test the given sophisticated Software Under Test (SUT)
• The entire MBT test-suite with all their “glue” code (Selenium) is open-source:
github.com/vgarousi/MBTofTestinium
MBT of Testinium
10
V

11. #UCAAT
• Size measures (metrics) from the MBT project:
• # of test models = 18 (login model is only one of them)
• Total # of nodes=170
• Total # of edges=260
• Size of the Selenium Java code for nodes/edges’ behavior = 4,699 LOC
• Each test run takes about: 6 hours, going through ~28,000 test steps!
(nodes and edges, stopping condition=%100 edge coverage)
MBT of Testinium
11
18 test models (two of them below)
…
V

12. #UCAAT
• Testing more with less effort!
• Covering more test paths with only designing the test models, which are
intuitive and easy to design
• We only design: 170 nodes, and 260 edges, but MBT executed more
than 28,000 test steps!
• Increased power (test effectiveness) in detection of real faults
• In the SUT (Testinium), MBT has detected so far 11 defects
• Many more benefits (we will discuss them at the end)
12
…
MBT of Testinium: Observations and benefits
V

13. #UCAAT
• What is Model-based testing (MBT)?
• Using MBT in our projects
• Testing our own large web application (Testinium)
• Testing the web application of one of our clients (BTCTurk)
• Testing two of UK’s COVID mobile apps
• Benefits of MBT, and when it is a good idea to use
MBT
• Pointer to more resources on MBT
Outline of this tutorial (duration: 1 hour)
13
V

14. #UCAAT
• BtcTurk: Founded in 2013, it
is Turkey's first and the
world’s fourth Bitcoin and
cryptocurrency exchange
platform.
MBT of the web application of one of our clients (BTCTurk)
14
A

15. #UCAAT
• Our strategy: we designed
two MBT test suites:
• An end-to-end test suite (like
a smoke test): only the major
paths through the app
• A complete regression test
suite: exercising all the pages
and all paths (five main
screen, many tabs, and
“many” possible paths)
MBT of BtcTurk mobile app
15
A

16. #UCAAT
• End-to-end test suite (like a smoke test; mainly “Happy” paths):
MBT of BtcTurk mobile app
16
Login
All currency
pairs
A pair’s
details page
My wallet
Buy
Sell
Buy Successful
Sell Successful
View latest
transactions
Buy/sell page
My wallet
My
transactions
Limit tab
Pair Command
Buy Command
Successful
Cancel
Command
My wallet
Sell Command
Successful
Start

17. #UCAAT
• The complete test suite: exercising all the pages
and paths):
• 8 test models
• 303 nodes
• 567 edges (transitions)
MBT of BtcTurk mobile app
V

18. #UCAAT
Demo video: MBT of BtcTurk mobile app
(2-3 minutes)
www.youtube.com/watch?v=dOgHQDnxnKw
V

19. #UCAAT
• What is Model-based testing (MBT)?
• Using MBT in our projects
• MBT testing of our own large web application (Testinium)
• MBT of the web application of one of our clients (BTCTurk)
• MBT of two of UK’s COVID mobile apps
• Benefits of MBT, and when it is a good idea to use
MBT
• Pointer to more resources on MBT
Outline of this tutorial (duration: 1 hour)
19
V

20. #UCAAT
• Vahid was hired as a consultant to assess the quality of
these apps:
• StopCOVID NI app
• Protect Scotland app
• NHS Covid-19 app
MBT of UK’s COVID contact-tracing mobile apps
20
V

21. #UCAAT
MBT of two of UK’s COVID mobile apps
21
• We developed MBT
test suites for two of
these apps
• NHS Covid-19 app
• Protect Scotland
• UI flow diagram of
the Protect Scotland
app: V

22. #UCAAT
MBT of NHS COVID mobile app
22
• 12 test models
• 144 nodes
• 497 edges (transitions)
• Joint work with:
Mark Lee
• Video demo:
(2-3 minutes)
All test models and code at:
github.com/vgarousi/MBTofCOVIDapps
youtube.com/watch?v=fqBqYCP4vyA
V

23. #UCAAT
• What is Model-based testing (MBT)?
• Using MBT in our projects
• Testing our own large web application (Testinium)
• Testing the web application of one of our clients (BTCTurk)
• Testing one of UK’s COVID mobile applications
• Benefits of MBT, and when it is a good idea to use
MBT
• Pointer to more resources on MBT
Outline of this tutorial (duration: 1 hour)
23
V

24. #UCAAT
• Covering more test paths (higher “path” coverage) with
less effort  Saving time and money
• Thus, leading to increased test effectiveness in
detection of faults
• Intangible but important benefits
• Making the work of test engineers more “interesting” and
more organized (according to feedback from our team
members)
• Ability to see the “big picture” of test-case design, and the
models being directly executable
• And many more…
Benefits of MBT
24
V (and voice by A)

25. #UCAAT
• MBT is a proven approach for testing mobile and web
applications
• It is almost always a good idea to use MBT, specially
when the SUT is complex and has many pages and UI
flows
When it is a good idea to use MBT
25
V

26. #UCAAT
• What is Model-based testing (MBT)?
• Using MBT in our projects
• Testing our own large web application (Testinium)
• Testing the web application of one of our clients (BTCTurk)
• Testing one of UK’s COVID mobile applications
• Benefits of MBT, and when it is a good idea to use
MBT
• Pointers to more resources on MBT
Outline of this tutorial (duration: 1 hour)
26
V

27. #UCAAT
• A nice book:
• Our articles:
• A detailed 22-page article
• doi.org/10.1016/j.jss.2021.111032
• And many good other online materials
Pointers to more resources on MBT
27
V

28. #UCAAT
Questions?
Contact us:
v.garousi@qub.ac.uk
alper.keles@testinium.com
yunus.balaman@testinium.com
zeynep.ozdemir@testinium.com
28