Invited talk for the NatWest Bank Group (National Westminster Bank)
October 20, 2021
By:
Vahid Garousi
Bahar Software Engineering Consulting Corporation
Queen’s University Belfast, UK
www.vgarousi.com
@vgarousi
Model Call Girl in Narela Delhi reach out to us at 🔝8264348440🔝
Automated Testing Best Practices in Finance
1. 1/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Experience and best practices from
automated software testing in the
finance domain
Vahid Garousi, PhD, PEng
Affiliations:
Associate Professor of Software Engineering (Senior Lecturer)
Queen’s University Belfast, Northern Ireland, UK
Director and Managing Consultant
Bahar Software Engineering Consulting Limited, UK
vahid@vgarousi.com
www.vgarousi.com
@vgarousi
vgarousi
Invited talk for the NatWest QCOE (NWG)
Testing Conference
Oct. 19, 2021
1:
2:
2. 2/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Outline
My career background
Past/ongoing software-testing projects …
Experience and best practices from automated testing
in two finance projects (two of our clients/partners):
Testing the web and mobile applications of BtcTurk A.Ş.:
Turkey's first and the world’s fourth Bitcoin and
cryptocurrency exchange platform. (Project running since
2020-)
Testing the mobile applications of KuveytTürk Bank A.Ş.: a
large Turkish bank. (Project ran between 2018-2020)
Note: The information, presented in this talk, have either
already been published as technical articles – and OK’ed by
the clients - thus are “unclassified” and there is permission
to share them
Observations / retrospective
… of best practices in automated testing in the finance
domain
Q/A
3. 3/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
My background (international experience)
Education:
PhD in Software Engineering, Carleton University, Ottawa, Canada, 2006
MSc in Computer Engineering, University of Waterloo, Canada, 2003
BSc in Software Engineering, Sharif University of Technology, Tehran, Iran, 2000
Work experience:
Associate Professor, Queen’s University Belfast, UK, 2019-
Managing Consultant, Bahar Software Engineering Consulting, since 2001-
Past:
Associate Professor, Wageningen University, Netherlands, 2017-2019
Associate Professor, Hacettepe University, Ankara, Turkey, 2015-2017
Associate Professor, University of Calgary, Canada, 2006-2014
Ethnic background: Turkish / Canadian
2013
2019
2015
2009
4. 4/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Consulting projects (a partial list)
Improving capability and maturity of test automation projects
Client: Testinium is a large test-services company in Turkey
Project running since 2019-
Test automation of web and mobile applications
Client: BtcTurk A.Ş., Turkey's first and the world’s fourth Bitcoin and cryptocurrency exchange
platform
Project running since 2020-
Test automation of mobile applications
Client: KuveytTürk Bank A.Ş., a large Turkish bank
Duration: 2018-2020
Improving test automation practices
Client: HAVELSAN (Hava Elektronik Sanayi, Aviation Electronics Industries)
Duration: 2015-2017
Automated testing of law-management software suite
Client: Innova IT Solutions, Ankara, Turkey
Duration: 2016-2017
Development and assessment of effective test automation infrastructure
Client: MR Control Systems International, Calgary, Canada
Duration: 2008-2011
Decision-support for deciding “when to automate testing” and “what (test cases) to
automate”
Client: Pason Systems Corporation, Calgary, Canada
Duration: 2010-2012
And many more…
5. 5/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Outline
My career background
Past/ongoing software-testing projects …
Experience and best practices from
automated testing in two finance projects
(two of our clients/partners):
Testing the web and mobile applications of BtcTurk
A.Ş.: Turkey's first and the world’s fourth largest
Bitcoin and cryptocurrency exchange platform.
(Project running since 2020-)
Testing the mobile applications of KuveytTürk Bank
A.Ş.: a large Turkish bank. (Project ran between 2018-
2020)
Observations / retrospective
… of best practices in automated testing in the finance
domain
Q/A
6. 6/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Systems Under Test (SUT)
Mobile app:
BtcTurk (BtcTurk.com) serves more
than 3.8 million registered users
7. 7/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Systems Under Test (SUT)
Web apps:
A simple version,
and a pro version
8. 8/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Setup of our Test Automation Team
Test Automation
Engineers
Test
Automation
Consultant
Client
Test Automation
Team-lead
Product owner,
and Software
Engineers
…
Systems Under
Test (SUT)
Test
Develop / update
Co-developing the test
automation strategy and
monitoring the entire test
process
PARTNERSHIP
SINCE 2019
9. 9/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Test automation strategy
GUI
API
Business Logic
Database
Calls
Calls
Returns data
Returns data
Class
(Unit)
Class
(Unit)
…
System Under Test (SUT)
and its architecture
GUI testing
API / service
testing
Unit tests
Developed by the
Software Engineers
at BtcTurk (not us)
Using the BDD
tool: Gauge
Test
Test
Test
Two approaches:
1- Model-based Testing
2-BDD (tool: Gauge)
Test automation
suites:
Some
example
test
artifacts
next…
10. 10/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Model-based Testing (MBT): Basics
Let’s first see: What is
Model-based testing
(MBT)?
Model-based testing is
the application of model-
based design for
systematic and
automated test-case
design and, optionally,
also for automated
execution of test cases.
An example:
Valid login
• Nodes, or vertices: “States”
(pages) of the system to verify
• Edges or transitions: Events
such as valid login, click
Let’s hear from you: Have you ever used MBT in your projects? If yes, how have you
used it? Join at slido.com #777
11. 11/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Model-based Testing (MBT): Basics
Developing the “behavior” of nodes/edges in Java using
the Selenium framework
public void e_valid_login() {
WebElement userNameElement =
methodsPage.findElement(By.id("username"));
userNameElement.clear();
userNameElement.sendKeys(email);
WebElement passwordElement =
methodsPage.findElement(By.id("password"));
passwordElement.clear();
passwordElement.sendKeys(password);
methodsPage.findElement(By.cssSelector(
"input[class$="login-page__submit-btn"][value="Sign In"]")).click();
}
12. 12/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Strategy for Model-based Testing (MBT)
Our strategy: we designed two MBT test suites:
An end-to-end test suite (like a smoke test): only the major paths
through the app
A complete regression test suite: exercising all the pages and all
paths (five main screen, many tabs, and “many” possible paths)
Details next…
MBT test tool:
An open-source popular tool, named GraphWalker
(graphwalker.github.io)
It is in wide use, e.g., in companies such as Spotify, and testing the
information kiosks software of New York’s transportation authority
(bit.ly/MBTGuidingTestingDecisions)
13. 13/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT of BtcTurk mobile app
End-to-end test suite (like a smoke test; mainly “Happy” paths):
Login
All currency
pairs
A pair’s
details page
My wallet
Buy
Sell
Buy Successful
Sell Successful
View latest
transactions
Buy/sell page
My wallet
My
transactions
Limit tab
Pair Command
Buy Command
Successful
Cancel
Command
My wallet
Sell Command
Successful
Start
Since team members
were all Turkish, Turkish
language is used for
naming ;) Also, UI has
two languages
English
translations
14. 14/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT of BtcTurk mobile app
The complete test suite: exercising all the
pages and paths):
8 test models, 303 nodes, 567 edges (transitions)
Login LoginPro
Currency
pairs
My
wallet
My
Account
Deposit/
Transfer
(Funding)
Trade
Panel
Move
among
Tabs
15. 15/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT of BtcTurk mobile app
The complete test suite: how the 8 test models are connected
One model for each app “page” (tab)
Login, regular
version of app
Login Pro version
5
3
4
2
16. 16/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT of BtcTurk mobile app
The complete test suite: exercising all the pages and
paths): Full MBT run takes about 2 hours…
www.youtube.com/watch?v=dOgHQDnxnKw
17. 17/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT: What are its benefits?
Let me ask you first…
Tangible benefits:
Covering more test paths (higher “path” coverage) with less effort
Saving time and money
Thus, leading to increased test effectiveness in detection of faults
Intangible, but important, benefits:
Ability to see the “big picture” of test-case design, and the models
being directly executable
Making the work of test engineers more “interesting” / fun and more
organized (according to feedback from our team members)
And much more…
One of our meetings in
Istanbul, 2019
(before COVID)
Join at slido.com #777
18. 18/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT: Challenges that we faced and how we addressed them
Challenges How we addressed them
Challenge 1- Choosing the “right” MBT tool
from amongst the very large pool of available
MBT tools
Tools Criteria
1-Matching test requirements- Essential 2a-Right level of model abstraction- Essential
TestModeller - Exhaustive activity diagram, resulting in repetition of nodes
(youtu.be/nctAQHsmjpI)
Failed the essential criterion 2
TestOptimal Supports web/mobile apps, but not specific for them Web page UI flow diagram
Tricentis Tosca Seems like a test-data management tool (youtu.be/f6aBpa95kLc). While the introduction on its
website mentions MBT, support for MBT is very limited. Not possible to design cycles and
complex flow/edge structures
Failed the essential criterion 1
-
SpecExplorer Support for MBT of web/mobile apps seems very limited. Most of focus is on API and unit
testing.
Failed the essential criterion 1
-
GraphWalker Specific for web/mobile apps Web page UI flow diagram
NModel - Test model is in a programmatic format, instead of visual
diagrams (doi.org/10.1007/978-3-642-05031-2_14)
Failed the essential criterion 2
TCases Support for MBT of web/mobile apps seems very limited. Most of focus is on test-case design
for input space exploration.
Failed the essential criterion 1
-
ModBat Focus is on API testing. No support for MBT of web/mobile apps. (fmv.jku.at/modbat)
Failed the essential criterion 1
-
MoMuT Focus is on embedded system testing. No support for MBT of web/mobile apps. (momut.org)
Failed the essential criterion 1
-
CrawlJax It produces as output a state-flow graph of the dynamic DOM states and the event-based
transitions between them. Focus is not on GUI testing of web apps
Failed the essential criterion 1
-
(1) the tool matching the test requirements, e.g., type of SUT (for us, this was web/mobile apps)
(2) tool being fit to the operating environment, e.g., “right level” of model abstraction, team’s expertise
(3) tool’s usability
(4) availability of support for the tool
(5) tool’s cost
We did a research on all knowledge and evidence shared
on the internet by experts (53 blogs and white papers),
about this topic. We synthesized the criteria for choosing
the right tool under into five categories:
19. 19/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
MBT: Challenges that we faced and how we addressed them
Challenges How we addressed them
Challenge 2- We observed a general
shortage of knowledge and resources on
best practice and “design patterns” for
designing MBT models
Challenge 3-
By sharing the best practices, we have
identified a number of design patterns,
e.g., one model for each web/mobile app
page, clear separation of events (edges)
and assertions (nodes)
Four graph traversal algorithms (test-
path “generators”) are supported by
GraphWalker: Random, Weighted
random, Quick random, A* (A-star).
Which one shall we use?
We are investigating the different options
and assessing their strength, e.g.. Fault
detection effectives versus efficiency,
(time taken for test execution)
20. 20/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Let’s see another test approach in our BtcTurk project
GUI
API
Business Logic
Database
Calls
Calls
Returns data
Returns data
Class
(Unit)
Class
(Unit)
…
System Under Test (SUT)
and its architecture
GUI testing
API / service
testing
Unit tests
Developed by the
Software Engineers
at BtcTurk (not us)
Using the BDD
tool: Gauge
Test
Test
Test
Two approaches:
1- Model-based Testing
2-BDD (tool= Gauge)
Test automation
suites:
Reminder: Our test automation strategy:
21. 21/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing of BtcTurk apps, using Gauge tool
Develops
Gauge test-suite
Web
browser
Test
step
Test specification
file Test-step
implementation
Test
scenario
1...*
Test concept
1...*
1
1
In Java or
another
language System Under
Test (SUT)
Call
Call Return
Gauge
Call
Return
Test
outputs
Return
Anlayzes
Test enigneer
Call
A specification file:
(can have many test scenarios)
A scenario
A concept:
A step
Step implementations:
Examples from: docs.gauge.org/overview.html
22. 22/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD tools: Cucumber versus Gauge
Behaviour-Driven Development (BDD) tools such as Cucumber are
powerful tools for automated testing
But tools such as Cucumber have certain limitations:
They often enforce strict syntax for test cases, like the “Given-When-
Then” format, which may not always be easy to write for all test cases.
The more recent test-automation framework Gauge addresses that
limitation since it does not prescribe the BDD testing process with a
strict syntax.
In Gauge, writing a test case is as easy as writing down the flow of test
cases in several itemized sentences in a natural language, like English.
Thus, Gauge could be a better tool than Cucumber
23. 23/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing of BtcTurk apps, using Gauge tool
We started our test-suite development… But soon we faced a number
of challenges
Challenge 1: See below… A tool is NOT enough. We need “systematic” test-case design
Terms are mostly in Turkish, but I am
sure you can see some issues …
Issues here:
Duplications in test code
Test logic: is, sometimes, hard to
understand (low understandability)
Test code: Hard to read and hard to
maintain
There should be a “better” way
of developing test code (scripts)…
Inside element “X”, write value of “Y”
24. 24/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing of BtcTurk apps, using Gauge tool
These challenges have also been observed by many other test engineers
Challenge 1: Having a test tool is NOT enough. Do you agree?
25. 25/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing of BtcTurk apps, using Gauge tool
For this purpose, we devised a systematic test-case “design” approach, using test
models which we designed manually (essentially flow diagram of the web/mobile app)
We derive four types of test cases from the model:
1. Transition from the current node (page) to its next neighbouring nodes (pages)
2. Node to itself (often, error checking, like: Invalid Login)
3. Input UI tests (in single node/page/unit level)
4. End-to-end testing: testing a path of nodes
https://www.slideshare.net/vahidgarousi/test-
automation-with-the-gauge-framework-experience-
and-best-practices-seetest-2020-conference
More details in…
26. 26/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing of BtcTurk apps, using Gauge tool
Example Gauge test suites:
27. 27/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing of BtcTurk apps, using Gauge tool
Another challenge:
The test-script LOC of our Gauge test-suites soon expanded and reached more than 50K
test LOC
So it was clear that, just like application (production) code-base, test code-base also
needs refactoring, maintenance and quality improvement (e.g., better test-code design /
architecture / modularity)
Solution:
We had developed an in-house tool for this need in our past projects, so we used it…
Our GaugeDepend tool
… is a dependency analysis
tool for visualizing and
exploring dependency
graphs for Gauge test suites.
Such as visualization will
allow quality assessment of
test-automation scripts.
Is open source:
github.com/vgarousi/
GaugeDepend
Demo video:
youtube.com/watch?v=oehq
DgX8R8E
Test
specification
files:
Test
scenario: Test steps: Test concepts
(Gauge terminology):
Let’s zoom in…
28. 28/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Outline
My career background
Past/ongoing software-testing projects …
Experience and best practices from
automated testing in two finance projects
(two of our clients/partners):
Testing the web and mobile applications of BtcTurk
A.Ş.: Turkey's first and the world’s fourth Bitcoin and
cryptocurrency exchange platform. (Project running
since 2020-)
Testing the mobile applications of KuveytTürk Bank
A.Ş.: a large Turkish bank. (Project ran between 2018-
2020)
Observations / retrospective
… of best practices in automated testing in the finance
domain
Q/A
29. 29/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Automated testing of the mobile applications of KuveytTürk Bank
App Under Test
(AUT):
The app had 73
GUI screens
(pages)
30. 30/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Automated testing of the mobile applications of KuveytTürk Bank
We used Appium to develop a large GUI test suite:
In the first phase: 28 test classes in Java using Appium
Testing 28 of the app’s total 73 GUI screens (pages) – the
most critical features (risk-based testing)
In total, 16,351 test LOC (TLOC)
In the 2nd phase: we tested all the GUI pages
31. 31/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Automated testing of the mobile applications of KuveytTürk Bank
Some example test code (Java Appium framework):
32. 32/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Automated testing of KuveytTürk apps: Challenges observed and best
practices
We observed a number of “classical” challenges,
e.g., the need to better design and develop the
test scripts
Best practices that we applied:
Use of test (design) patterns (xunitpatterns.com) such as: the
Four-Phase Test pattern (setup of test “fixture”, exercise / call
the SUT, result verification, and fixture teardown)
33. 33/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Automated testing of KuveytTürk apps: Challenges observed and best
practices
Another best practice that we applied:
Decoupling of test execution parameters from test scripts (an
example is below)
34. 34/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Testing the mobile applications of KuveytTürk Bank A.Ş.
Our mobile device farm:
More details in:
35. 35/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Outline
My career background
Past/ongoing software-testing projects …
Experience and best practices from
automated testing in two finance projects
(two of our clients/partners):
Testing the web and mobile applications of BtcTurk
A.Ş.: Turkey's first and the world’s fourth Bitcoin and
cryptocurrency exchange platform. (Project running
since 2020-)
Testing the mobile applications of KuveytTürk Bank
A.Ş.: a large Turkish bank. (Project ran between 2018-
2020)
Observations / retrospective
… of best practices in automated testing in the finance
domain
Q/A
36. 36/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Experience-based observations: Automated testing in finance
We all know that quality of web and
mobile apps in finance are extremely
business-critical
Automated testing of these apps are
crucial and critical to business success
At the end of the day, we need a
combination of automated and manual
(exploratory) testing
Each would find different types of
defects, and have pro’s and con’s
With experience in these topics, we can
certainly plan and execute a successful
test approach
Azimo: Global
Money
Transfers
Australia and
New Zealand
Banking Group
37. 37/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Outline
My career background
Past/ongoing software-testing projects …
Experience and best practices from
automated testing in two finance projects
(two of our clients/partners):
Testing the web and mobile applications of BtcTurk
A.Ş.: Turkey's first and the world’s fourth Bitcoin and
cryptocurrency exchange platform. (Project running
since 2020-)
Testing the mobile applications of KuveytTürk Bank
A.Ş.: a large Turkish bank. (Project ran between 2018-
2020)
Observations / retrospective
… of best practices in automated testing in the finance
domain
Q/A
39. 39/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
R&D Projects (in the UK, EU and Turkey)
Two recent projects:
XANDAR: X-by-Construction Design framework for Engineering
Autonomous & Distributed Real-time Embedded Software Systems
(2021-2024)
Eight (8) partners from across Europe, including five industry and
three academic partners
Budget: €5 million Euros
I “lead” the Work Package (WP) on Verification and Validation (V&V)
TESTOMAT (The Next Level of Test Automation)
Project funded by the EU
34 partners from across Europe
Budget: €21 million Euros
www.testomatproject.eu (2018-2020)
And 12 other major projects…
40. 40/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
BDD testing using Gauge tool: Challenges and solutions
Opinions of other test engineers:
“If you feel that Cucumber is a pain when used in large projects due to its
Given-When-Then syntax becoming messy in long run, Gauge can take out
that pain to some extent.
Even here, the flexibility of free text can be a demerit as they don’t have a
defined format.
But tests may end up looking like a messy text document unless imposed strict
restrictions”.
https://medium.com/@divi6.1990/why-i-chose-gauge-over-cucumber-5b3de478d889
“Gauge follows Markdown Syntax for scripting the tests in an unstructured
way - which means more liberty to make the test like a specific document and
also more risk of ending up with scripted tests in many different styles.
Having some in-team standards will help to prevent any chaos.” [8]
www.linkedin.com/pulse/things-consider-when-choosing-best-test-automation-framework-akbari