Invited talk given at: The Dutch Testing Day event (de Nederlandse Testdag, www.testdag.nl), Nov. 6, 2018, Delft, the Netherlands

1. 1Dr. Vahid Garousi
Industry-academia collaborations in
software testing
Success stories from Canada and Turkey
Dr. Vahid Garousi
Associate Professor of Software Engineering
Information Technology Group
Wageningen University, Netherlands
www.vgarousi.com
www.wur.nl/en/Persons/Vahid-V-Vahid-Garousi.htm
@vgarousi
Nov. 6, 2018
Delft, the Netherlands

2. 2Dr. Vahid Garousi
Outline
 Background
 Of the Principal Investigator (PI), and his research teams in the past
 A review of research projects (industry-academia
collaborations):
 Research projects in Turkey
 Research projects in Canada

3. 3Dr. Vahid Garousi
Background
Education:
PhD, Carleton University, Ottawa, Canada, 2006
MSc, University of Waterloo, Canada, 2003
BSc, Sharif University of Technology, Tehran, Iran, 2000
Work experience:
Associate Prof., Wageningen University, Netherlands, 2017-
Associate Prof., Hacettepe University, Ankara, Turkey, 2015-
2017
Assistant Prof., University of Calgary, Canada, 2006-2014
Managing partner, Maral Software Engineering Consulting,
Canada, 2008-2015

4. 4Dr. Vahid Garousi
Research in University of Calgary, Canada
2006-2014
• Established and led the Software Quality Engineering
Research Group (SoftQual)
• www.softqual.ucalgary.ca
• Alumni
 1 Post-doctoral
fellow
 3 PhD students
 9 MSc students
 30+ undergraduate
students
• More than 10 industrial
partners such as IBM
• More than $1.3 Million
CND in funding

5. 5Dr. Vahid Garousi
Hacettepe University Software Engineering (HUSE) group
 3 faculty members
 20+ graduate students
 10+ industry partners

6. 6Dr. Vahid Garousi
Goal of software engineering research (includes: SW testing)
 What we, SE researchers, usually do:
SE
researcher
Software
Software companies
and engineers
Clients (customers)
using software
systems
Develop
approaches to help
engineers develop,
and test software in
an effective and
efficient manner
Engineer and
release
software
Software to
be used by
Characterize /
formalize software
engineering
practices,
challenges, etc.
Solicit
requirements
/ needs
Publish
papers

7. 7Dr. Vahid Garousi
Reviewing the state of the SE community
Software industry
(n = ~23 million people)
SE academia
(n = ~4,000 people)
... ...
Researchers
in another
institution
...
Software
engineers of a
company
...
Software engineers
working in another
company
Sharing knowledge in a
conference, or online
sources
SE researchers in an
institution
Sharing knowledge in a
conference, by reviewing
each others’ papers, joint
work, etc.
... ...
......
...
......
...
... ...
Conducting
industry-academia
collaborations
www.evansdata.com/reports/viewRelease.php?reportID=9

8. 8Dr. Vahid Garousi
Outline
 Background
 Of the Principal Investigator (PI), and his research teams in the past
 A review of research projects (industry-academia
collaborations):
 Research projects in Turkey
 Research projects in Canada

9. 9Dr. Vahid Garousi
Research projects in Turkey
 (Anonymous) company in the aviation industry
 Need: How to optimize regression testing with multiple
cost and benefit factors (such as: test execution time,
and number of faults detected)
 Industry partner: HAVELSAN (Aviation
Electronics Industries), 2015-2017
1. Test automation for several test groups
2. Assessing and improving an in-house test automation
framework for test groups
3. ...
 Industry partner: Innova IT Solutions
 Research need: How to develop a high-quality
automated test strategy for a family of law-management
software?
 And several other partners...

10. 10Dr. Vahid Garousi
Research approach

11. 11Dr. Vahid Garousi
Industrial context and the software under test
 The subject project, SUT, was a safety-critical industrial
controller software in the aviation domain.
 The functionality of the SUT is defined by 935 system-
level requirements.
 To verify those system-level requirements, the test team
had developed 54 different test suites, each containing a
large number of test cases (test steps).
 In total, there were 30,834 test cases (test steps) for the
entire system.

12. 12Dr. Vahid Garousi
Need analysis and motivations for the study
 Each time when a software fix was implemented, the
customer test team used to first evaluate the list of affected
system requirements and the regression test-set proposed
by the contractor test team.
 The customer test team would then either accept the
proposed regression test-set or would request adding
more test suites to it.
 The entire approach was ad-hoc, subjective and purely
based on expert opinion.
 There was an urgent need for a systematic, more effective
and efficient regression test-selection approach.

13. 13Dr. Vahid Garousi
Regression testing with multiple cost and benefit factors
 The stakeholders reported multiple cost and benefit factors
(objectives) for regression testing
 Two example cost objectives (5 in total):
 (1) Test execution time: time required for executing a test suite
 (2) Third party cost: the cost of services procured from third party
organizations for tests
 Two example value (benefit) objectives (4 in total):
 (1) Number of detected faults: experience has shown that as the
software is developed and periodically upgraded, the phenomenon of
re-emergence of similar defects is quite common
 (2) Faults severity: it is not only the number of faults detected by a test
suite that is important, but also the severity of faults. Test suites which
detect more severe faults are better to be prioritized in regression
testing.

14. 14Dr. Vahid Garousi
Development of a Genetic algorithm (GA)
 To solve the regression testing problem at hand, with multiple cost and
benefit factors, we developed a GA (named MORTOGA)
1 ...0 1 1
Number of test suites=54
Gene (corresponding to a
test suite)
...1 1 0
0 ...0 0 1
...
GA
population
size
Chromosome (corresponding
to a “candidate” regression
test set)
GA population (pool)
0
• “1” means
existence of the
given test suite in
the solution
(regression test-set)
• “0” means its
absence
𝐹𝑖𝑡𝑛𝑒𝑠𝑠 𝑐
=
∀g ∈ Genes(c) ∀o ∈ Objectives
𝐜𝐨𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭 𝑜 × 𝑂𝑏𝑗𝑉𝑎𝑙𝑢𝑒(𝑔, 𝑜)

15. 15Dr. Vahid Garousi
Comparing the three approaches w.r.t. aggregate
cost and benefit values
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
Version 1 Version 2 Version 3 Version 4 Version 5
Sumofcostandbenefitvalues
Manual method: Cost
Manual method: Benefit
MORTOGA: Cost
MORTOGA: Benefit
Selective method: Cost
Selective method: Benefit

16. 16Dr. Vahid Garousi
Making a “difference” in practice
 Since the proposed approach (MORTOGA) has
been beneficial in the industrial context in
saving the costs of regression testing, it is
currently in active use in the company and
project under study.

17. 17Dr. Vahid Garousi
Research projects in Turkey
 (Anonymous) company in the aviation industry
 Need: How to optimize regression testing with multiple
cost and benefit factors (such as: test execution time,
and number of faults detected)
 Industry partner: HAVELSAN (Aviation
Electronics Industries), 2015-2017
1. Test automation for several test groups
2. Assessing and improving an in-house test automation
framework for test groups
3. ...
 Industry partner: Innova IT Solutions
 Research need: How to develop a high-quality
automated test strategy for a family of law-management
software?
 And several other partners...

18. 18Dr. Vahid Garousi
Effective automated testing of helicopter simulation software
 An example product: Helicopter simulators
 Sophisticated software and embedded systems
 The system was tested manually in the past
 But manual testing was tedious /
expensive, and not very effective in
finding faults
 Need for “effective” automated testing
Automated test
software
(to be developed)
Tests
Software Under Test (SUT)

19. 19Dr. Vahid Garousi
Effective automated testing of helicopter simulation software
 Architecture of the Software Under Test (SUT):
Weather Server
Debriefing System
Instructor
Console
Terrain Server
Mission & Flight
System
Sound System
Image Generator
System
Distributed Mission System Auxillary System
Visual Database
controls
Cockpit (Hardware)
System
Instructor
Student pilot
interacts
interacts
Message injection and
monitoring tool DIM
intersects

20. 20Dr. Vahid Garousi
Automated testing of helicopter simulation software
 A test-automation software (framework) was developed
 New “test patterns” were developed and used
 Test automation best-practices were applied
 Such as: when and what (test cases) to automate in testing
 An example test case: testing an important tactical variable (fuel tank level of
the helicopter) for a duration of 1,000 seconds in the interval of one second
when flying between two points on a given flight path (many possibilities)
 Test objective: to ensure fuel is consumed in the “right amount” during the trip
Flight path: different possibilities
Point A Point B

21. 21Dr. Vahid Garousi
Automated testing of helicopter simulation software
 Benefits: Thanks to test automation, test engineers could now reproduce
and repeat many test cases without the need for a “human” tester (pilot)
 Many faults were detected and fixed
Automated test software
Tests
Software Under Test (SUT)

22. 22Dr. Vahid Garousi
Collaborations with HAVELSAN
 Several papers have been published so far…

23. 23Dr. Vahid Garousi
Outline
 Background
 Of the Principal Investigator (PI), and his research teams in the past
 A review of research projects (industry-academia
collaborations):
 Research projects in Turkey
 Research projects in Canada

24. 24Dr. Vahid Garousi
Research projects in Canada
 Project 1
 Topic: Model-based testing and analysis of distributed, embedded and real-time
systems
 Years: 2007-2012
 Funding: NSERC Discovery grant
 No collaboration with industry in this project
 Project 2
 Topic: Deciding “when” to automate testing and “what (which test cases)” to automate.
A typical problem for many companies
 Years: 2009-2018
 Industry partners: Pason Systems Corporation, Calgary, Canada and a few others in
Turkey
 Funding: NSERC ENGAGE grant
 And several more...
Discussed next...

25. 25Dr. Vahid Garousi
Deciding “when” to automate testing and “what (which test cases)” to
automate
 A typical problem for many companies
 One of the industry partners: Pason Corporation, Calgary, Canada.
Developing software and hardware for oil rig reservoirs
 Systems under test (SUT’s):

26. 26Dr. Vahid Garousi
Decision support for balancing manual software testing
versus test automation
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
“Exercise”
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
AM
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
Computer
(Automated)
Human
(Manual)
M
OR/
AND
AM
System Under Test
(SUT)
AM
AM
Activity
Data/
Entity
Legend
Test-
Result
Reporting

27. 27Dr. Vahid Garousi
 A genetic algorithm (GA) was developed for this purpose
UC: use-case
Goal: maximize
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
“Exercise”
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
AM
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
Computer
(Automated)
Human
(Manual)
M
OR/
AND
AM
System Under Test
(SUT)
AM
AM
Activity
Data/
Entity
Legend
Test-
Result
Reporting
Decision support for balancing manual software testing
versus test automation

28. 28Dr. Vahid Garousi
Decision support for balancing manual software testing
versus test automation

29. 29Dr. Vahid Garousi
Decision support for balancing manual software testing versus test
automation
 Output/impact:
 Industry impact: The solution helped the industry partner make the “right”
decisions w.r.t. the above and reduce testing costs and increase their software
quality using test automation
 Academic impact: The following papers:
Invited paper

30. 30Dr. Vahid Garousi
My research approach / vision
 To conduct rigorous and relevant software-engineering research
 To have academic contributions / impact:
 Papers
 Training of young researchers in Software Engineering (many of my past
students are in academia or the software industry)
 To have industrial impact (“relevant” research):
 Knowledge exchange with the software industry (technology transfer)
 Collaborate with the software industry to help engineers develop, and test
software in an effective and efficient manner
 Via effective industry-academia collaborations (“win-win”): we have
had good success so far ;)

31. 31Dr. Vahid Garousi
End of the talk
 Background
 Of the Principal Investigator (PI), and his research teams in the past
 A review of research projects (industry-academia
collaborations):
 Research projects in Turkey
 Research projects in Canada