This document summarizes Dr. Vahid Garousi's presentation on deciding what and when to automate testing based on his experience from two test automation projects. The presentation discusses test automation strategies for two systems - a legal software system in Turkey and control software for oil rigs in Canada. It outlines criteria used to determine what test cases to automate, such as focusing on important features and risk-based testing. Statistics on the test suites developed for each system are also provided.

1. 1Dr. Vahid Garousi
Deciding what and when to automate in testing:
Experience from multiple projects
Vahid Garousi
Associate Professor of Software Engineering
Information Technology Group (INF)
Wageningen University, Netherlands
www.vgarousi.com
www.wur.nl/inf
@vgarousi
A talk for:
June 21, 2018

2. 2Dr. Vahid Garousi
My background
Education:
PhD in Software Engineering, Carleton University, Ottawa,
Canada, 2006
(International) Work experience:
Associate Professor, Wageningen University, Netherlands,
Since 2017
Associate Professor, Ankara, Turkey, 2014-2017
Associate Professor, University of Calgary, and Maral
Software Consulting Corporation, Canada, 2006-2014

3. 3Dr. Vahid Garousi
What do I do (as a researcher and consultant)
 Do professors only teach? ;)
 We collaborate with and help software engineers develop
and test software in an effective / efficient manner
SE
researcher
Software
Software companies
and engineers
Clients (customers)
using software
systems
Help engineer
(develop, test, ...)
software in an
effective and
efficient manner
Engineer and
release
software
Software to
be used by
Characterize /
formalize software
engineering
practices,
challenges, etc.
Solicit
requirements
/ needs
Publish
papers

4. 4Dr. Vahid Garousi
Topic of this talk
 Test automation is a good practice
 But it may provide disappointing outcomes if not applied
in the right time, and with the right strategy
 Just to say that “we automate” is not enough: What is
the most suitable test automation “strategy”?
 Two of the questions for test automation “strategy”:
 What (which test cases) to automate? We cannot (and
should not) automate all the tests! We need to find out
which test cases to automate and which ones should be
executed manually?
 When should we start automating tests (during the
software development life cycle, SDLC)?

5. 5Dr. Vahid Garousi
Let’s explore what others say
 Many people have challenges w.r.t. these questions and
are talking about them...

6. 6Dr. Vahid Garousi
Outline of the talk
 Background
 Experience on answering “What and when to automate”
in two test-automation projects
 “Wisdom of the crowd” on these questions:
 A systematic review of online sources and technical papers on this
topic

7. 7Dr. Vahid Garousi
Outline of the talk
 Background
 Experience on answering “What and when to automate”
in two test-automation projects
 Project 1:
 Business domain: Finance
 SUT: A family of software systems (web and cloud applications) to support
law and legal activities of companies and government
 Location of involvement: Turkey
 Project 2:
 Business domain: Automation and control systems
 SUT: Control (embedded) software for oil rig reservoirs
 Location of involvement: Canada
 “Wisdom of the crowd” on these questions:
 A systematic review of online sources and technical papers on this
topic

8. 8Dr. Vahid Garousi
Project 1: Context
 Collaborator company:
 Staff members > 1,000 people
 Clients in 37+ countries
 Head-quarter: Ankara, Turkey

9. 9Dr. Vahid Garousi
Project 1: Systems Under Test (SUTs)
 A family of software systems (web and cloud applications) to support
legal activities and operations of a large number of clients, including
the national court authority
 The two products are named:
 (1) Lega (~414 KLOC, and ~1,500 GUI screens)
 (2) LegaLite: similar features to Lega, but newer and lighter (~105 KLOC)

10. 10Dr. Vahid Garousi
Project 1: Need analysis
 Lega:
 Been in the market for 8+ years
 Was not developed with TDD  Thus had no automated tests
 Continually undergone all four types of software maintenance
activities: adaptive, perfective, corrective and preventive
maintenance  Thus the need for automated tests has been felt
 Had a large number of manual test suites which had 100%
requirements (use-case) coverage  need to “transition” them to
automated tests (also called “testing migration” *)
 LegaLite:
 A newer and lighted product compared to Lega
 Based on lessons learnt from Lega, this product was developed
using the Behavior-Driven Development (BDD) practices, and
automated tests
 * https://blog.fugue.co/2016-03-21-testing-migration-shifting-from-manual-to-automatic.html

11. 11Dr. Vahid Garousi
Project 1: Test automation strategy
 Automated testing levels...
 Test automation “pyramid”
 For both SUTs, we implemented test automation in service and GUI levels
 Reason:
 Lega’s production code was not developed with automated unit testing in mind
 Thus its unit-level testability was quite low.
 Unit-test automation for it would have required major refactoring
 We were asked by the management to focus on end-user (business) relevance...
Unit tests
GUI tests
Service/API tests
Number of tests
Increase in:
- Costs to develop and maintain
- Execution time
- Possibility of false negatives
But also in:
- Coverage
- End-user (business) relevance

12. 12Dr. Vahid Garousi
Automated GUI testing: Architecture
Execution tool
(IntelliJ IDE)
Test
framework
(Spock)
Geb testing
adapter
Geb
Selenium
WebDriver
Web browser SUT
Writes test scripts
and executes
Test engineer
Calls
Test scripts
• Spock is a suitable
framework for BDD: the
given-when-then test
pattern
• Setup (given), exercise
(when), verify (then) and
teardown
• A.k.a.: Arrange, Act, Assert

13. 13Dr. Vahid Garousi
Project 1: Deciding which test cases to automate
 “Whether you waste time debugging because you don’t have enough tests or
you waste time maintaining tests that don’t need to be there, at the end of the
day both of those things amount to waste”.
 What others are saying?

14. 14Dr. Vahid Garousi
Project 1: Deciding which test cases to automate
 Our criteria for decision making:
 Only automating the test cases that would return benefits
and positive Return On Investment (ROI)
 Automating tests for straightforward, happy paths and the
most important and commonly-used features, e.g., Use-
case: “Make a payment for a fine”
 Choosing tests for automation based on expected risk of
failures related to features (risk-based testing), e.g., Use-
case: “Send email reminders about late fines”
 Details on each criteria next...

15. 15Dr. Vahid Garousi
Project 1: Deciding which test cases to automate
 Our criteria for decision making:
 Only automating the test cases that would return benefits
and positive Return On Investment (ROI).
Upfront cost of
automation
(development
of automated
test suite, etc).
Payoff point
(“sweat spot”!)
Cost saving: Only if the
decision to automate (and
how much of it) has been
made properly, then we will
see this cost saving

16. 16Dr. Vahid Garousi
Project 1: Deciding which test cases to automate
 Our criteria for decision making:
 Only automating the test cases that would return benefits
and positive Return On Investment (ROI).
 The “Ugly side” of test automation. To rush into
automation without a proper test strategy…
Upfront cost of automation
(development of automated
test suite, etc.)
No payoff point
(“sweat spot”) ??
Costs to maintain the test
suite is growing, since an
IMPROPER test automation
strategy was followed
Project manager: Let’s
abandon test automation
altogether and not use
automated test suites 

17. 17Dr. Vahid Garousi
Project 1: Deciding which test cases to automate
 Our criteria for decision making:
 Automating tests for straightforward, happy paths and the
most important and commonly-used features, e.g., Use-
case: “Make a payment for a fine” and “Login”. Details...
 Important features should work. First priority!
 Automated “smoke testing”, also known as “Build
verification testing”: a non-exhaustive set of tests that aim at
ensuring that the most important functions work

18. 18Dr. Vahid Garousi
Project 1: Deciding which test cases to automate
 Our criteria for decision making:
 Choosing tests for automation based on expected risk of
failures related to features (risk-based testing), e.g., Use-
case: “Send automated email reminders about late fines”.
Details...
 Defects in such features (use cases) will incur the most risk
(negative impact)
 For the above example, if the automated email reminder is
NOT sent properly to clients, there will be financial $ and
even legal implications...

19. 19Dr. Vahid Garousi
Project 1: Statistics for the test suites of the two
SUTs
Metrics SUT
Legal LegaLite
Total LOC of SUT code 414 KLOC 105 KLOC
Number of manual test
cases (suites), both the
service test suites and GUI
test suites (100% req. cov.)
1,590 (210) 1,485 (185)
Number of automated test
cases (suites)
605 (80) 1,364 (170)
Ratio of automation (from
the above)
38% 91%
Total LOC of test scripts LOC of service tests: 8 KLOC
LOC of GUI tests: 12 KLOC
Total: ~20KLOC
LOC of service tests: 22 KLOC
LOC of GUI tests: 18 KLOC
Total: ~40KLOC
Ratio of test code to
production code
5% 38%
Development cost of
automated test suites
2 persons x 3 months=6
person-months (during 2015)
5 person-days per month for
12 months=60 days=2 person-
months (during 2016)
Maintenance cost of
automated test suites
1 person x 2 days/month 1 person x 3 days/month

20. 20Dr. Vahid Garousi
Project 1: Impact
 Benefit for the partner: The solution helped the industry partner
make the “right” decisions w.r.t. the above and reduce testing
costs and increase their software quality using test automation
 Academic impact: The following paper:

21. 21Dr. Vahid Garousi
Outline of the talk
 Background
 Experience on answering “What and when to automate”
in two test-automation projects
 Project 1:
 Business domain: Finance
 SUT: A family of software systems (web and cloud applications) to support
law and legal activities of companies and government
 Location of involvement: Turkey
 Project 2:
 Business domain: Automation and control systems
 SUT: Control (embedded) software for oil rig reservoirs
 Location of involvement: Canada
 “Wisdom of the crowd” on these questions:
 A systematic review of online sources and technical papers on this
topic

22. 22Dr. Vahid Garousi
Project 2: Context
 Collaborator company: Pason Corporation, Calgary, Canada. Developing
software and hardware for oil rig reservoirs
 Systems under test (SUT’s): embedded software written in C/C++ running on
Real-Time Unix OS

23. 23Dr. Vahid Garousi
Project 2: Offered solution
 Solution: Decision support for balancing manual software
testing versus test automation
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
“Exercise”
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
AM
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
Computer
(Automated)
Human
(Manual)
M
OR/
AND
AM
System Under Test
(SUT)
AM
AM
Activity
Data/
Entity
Legend
Test-
Result
Reporting

24. 24Dr. Vahid Garousi
Project 2: Offered solution
 Solution: Decision support for balancing manual software
testing versus test automation
 A genetic algorithm (GA) was developed for this purpose
UC: use-case
Goal: maximize
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
“Exercise”
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
AM
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
Computer
(Automated)
Human
(Manual)
M
OR/
AND
AM
System Under Test
(SUT)
AM
AM
Activity
Data/
Entity
Legend
Test-
Result
Reporting

25. 25Dr. Vahid Garousi
Project 2: Offered solution
 Solution: Decision support for balancing manual software
testing versus test automation

26. 26Dr. Vahid Garousi
Project 2: Impact
 Benefit for the partner: The solution helped the industry partner
make the “right” decisions w.r.t. the above and reduce testing
costs and increase their software quality using test automation
 Academic impact: The following papers:
Invited paper

27. 27Dr. Vahid Garousi
Outline of the talk
 Background
 Experience on answering “What and when to
automate” in two test-automation projects
 “Wisdom of the crowd” on these questions:
 A systematic review of online sources and technical
papers on this topic

28. 28Dr. Vahid Garousi
“Wisdom of the crowd” on “What and when
to automate”
• 52 from grey literature (blog posts,
white papers, YouTube videos, etc.)
• 26 from research literature
(papers)
Data from 78 sources:
Data synthesis...

29. 29Dr. Vahid Garousi
Wisdom of the crowd on “What and when to automate”
You can read the full
PDF of the paper:
www.goo.gl/gY4zcQ
Main content

30. 30Dr. Vahid Garousi
END of the talk
 Background
 Experience on answering “What and when to
automate” in two test-automation projects
 “Wisdom of the crowd” on these questions:
 A systematic review of online sources and technical
papers on this topic
Discussions: Questions / Answers

31. 31Dr. Vahid Garousi
An example test script written in the Groovy language