Experience in conducting 19 secondary (survey) studies in Software Engineering By: Dr. Vahid Garousi Invited (online) talk for the University of Adelaide, Australia May 8, 2020

1. 1Vahid Garousi
Experience in conducting 19 secondary
(survey) studies in Software Engineering
Dr. Vahid Garousi
Associate Professor (Senior Lecturer) of Software Engineering
Queen’s University Belfast, UK
v.garousi@qub.ac.uk
www.vgarousi.com
@vgarousi
Invited (online) talk for the
University of Adelaide, Australia
May 8, 2020
Based on studies conducted between 2009-2020:
• 11 Systematic Literature Mapping (SLM) studies
• 4 Systematic Literature Reviews (SLR)
• 3 Multivocal Literature Reviews (MLR)
• 1 Grey Literature Review (GLR)

2. 2Vahid Garousi
About me
Education:
PhD, Carleton University, Ottawa, Canada, 2006
MSc, University of Waterloo, Canada, 2003
BSc, Sharif University of Technology, Tehran, Iran, 2000
Work experience:
Associate Professor, Queen’s University Belfast, UK, 2019-
Associate Professor, Wageningen University, Netherlands, 2017-2019
Associate Professor, Hacettepe University, Ankara, Turkey, 2015-2017
Associate Professor, University of Calgary, Canada, 2006-2014
Senior Consultant, Bahar Software Engineering Consulting, since 2008-
(Working closely with software companies since 2001)
Software Engineer, Corsha Software Corp., Montreal, Canada, 1997-2001

3. 3Vahid Garousi
A big “thank you” to my co-authors (since 2003)…
 I have been fortunate to have the chance to collaborate with these
great researchers / practitioners since 2003

4. 4Vahid Garousi
Outline: Experience in conducting secondary (survey)
studies in Software Engineering
 What is a secondary study?
 What are the different types of secondary studies?
 Systematic Literature Mapping (SLM) studies
 Systematic Literature Reviews (SLR)
 Multivocal Literature Reviews (MLR)
 Grey Literature Reviews (GLR)
 How do they differ?
 Why should (do) we conduct secondary studies?
 What values do they provide? Their academic impact and industry impact
 It all depends on their RQs (should be as insightful as possible)
 How do we conduct secondary studies
 General process for conducing secondary studies
 Efforts spent on in conducting secondary studies
 Experience-based guidelines for effective and efficient data extraction
In each discussion, we will
review a few working
examples (papers)

5. 5Vahid Garousi
Terminology: What is a Secondary study?
 Secondary (survey) study: A study of (primary research) studies, or
using data from primary studies

6. 6Vahid Garousi
Terminology: What is a Review Study?

7. 7Vahid Garousi
Secondary studies in other fields
 … where they were adopted from, to software engineering
Adopted to

8. 8Vahid Garousi
SLRs in Social Sciences
 There are many books…

9. 9Vahid Garousi
Outline: Experience in conducting secondary (survey)
studies in Software Engineering
 Secondary (survey) studies and their history
 What are the different types of secondary studies?
 Systematic Literature Mapping (SLM) studies
 Systematic Literature Reviews (SLR)
 Multivocal Literature Reviews (MLR)
 Grey Literature Reviews (GLR)
 …
 How do they differ?
 Why should (do) we conduct secondary studies?
 What values do they provide? Their academic impact and industry impact
 It all depends on their RQs (should be as insightful as possible)
 How do we conduct secondary studies
 General process for conducing secondary studies
 Efforts spent on in conducting secondary studies
 Experience-based guidelines for effective and efficient data extraction

10. 10Vahid Garousi
Different types of secondary studies
 "A picture is worth a thousand words"
is conducted by
SLM (SM)
SLR
MLRMLM
Papers in formal
(academic) literature
of
includes
of
Mapping
is conducted by
Synthesis of
evidence
GLM
GLR
Sources in grey
literature
of
includes
is conducted by
of
includes
includes
includes
includes
SM/SLM: Systematic (literature)
mapping (classification)
SLR: Systematic literature review
GLM: Grey literature mapping
GLR: Grey literature review
MLM: Multivocal literature mapping
MLR: Multivocal literature review
Method of
analysis
Sources under
study
(Primary
studies)
Secondary
studies
Technical
online videos
White
papers
Blog posts
Types of secondary studies
Tertiary
studies STRSTM
<<abstract>>
Tertiary study
<<abstract>>
Secondary
study
Types of tertiary studies
Mapping Synthesis
ofof
is conducted by
is a study of a set of is a study of a set of
Systematic
Tertiary Review
Systematic
Tertiary Mapping
Collected data
or artifacts
under analysis
Source code Software
processes
Software models
Software engineers, teams
and organizations
Other data /
artifacts
study discuss / share experience about
<<abstract>>
Data or artifacts under
analysis
Software tools
Looks
overwhelming…
Let’s look at it, piece
by piece
Hint: Right click, then “Z”, then
pan around

11. 11Vahid Garousi
How do SLRs and SLMs differ?
 A Systematic Literature Review (SLR) :
 Synthesizes research evidence published in a given area
 A Systematic Literature Mapping (SLM /SM):
 Provides a classification (mapping) of papers published in a given area
 A SLM study often requires less effort compared to SLRs, while providing a more
coarse-grained overview
 A SLM is usually a first step towards a more in-depth SLR (first classify the papers,
then synthesize their findings together)
 All systematic literature studies should be: unbiased and repeatable
 Summary:
 SLR = Synthesis of research evidence in a given area
 SLM (SM) = Classification (mapping) of papers in a given area

12. 12Vahid Garousi
How do SLRs and SLMs differ? Examples…

13. 13Vahid Garousi
How do SLRs and SLMs differ? By their RQs
 Different focus. SLRs synthesize
evidence and have more depth. SLM
SLR

14. 14Vahid Garousi
MLRs and GLRs: If we want to (and we should!) synthesize
knowledge /evidence from industry in our SE research
 Multivocal Literature Reviews (MLR): both industrial (grey) and
academic literature
 Grey Literature Reviews (GLR): reviewing only industrial (grey)
literature

15. 15Vahid Garousi
Outline: Experience in conducting secondary (survey)
studies in Software Engineering
 Secondary (survey) studies and their history
 What are the different types of secondary studies?
 Systematic Literature Mapping (SLM) studies
 Systematic Literature Reviews (SLR)
 Multivocal Literature Reviews (MLR)
 Grey Literature Reviews (GLR)
 …
 How do they differ?
 Why should (do) we conduct secondary studies?
 What values do they provide? Their academic impact and industry impact
 The raised RQs will determine the value and impact of the studies (RQs
should be as insightful as possible)
 How do we conduct secondary studies
 General process for conducing secondary studies
 Efforts spent on in conducting secondary studies
 Experience-based guidelines for effective and efficient data extraction

16. 16Vahid Garousi
Values provided by secondary studies: Academic impacts/benefit
 Helping the researchers (authors) themselves for
determining what research topic to work on …
 E.g., a few my PhD students in the past

17. 17Vahid Garousi
Values provided by secondary studies: Academic impact
 Helping other researchers in their works. Can be measured by citations
 Some of my top cited papers are secondary studies:
Sorted

18. 18Vahid Garousi
Values provided by secondary studies: Industrial impact

19. 19Vahid Garousi
Importance of raising insightful Research Questions
 The raised RQs will determine the value and impact of the studies
 RQs should be as “insightful” as possible
 Not just “convenient” RQs (for data which is “easy” to extract! But may not
provide much real value)
Real industry needs led to RQs

20. 20Vahid Garousi
Outline: Experience in conducting secondary (survey)
studies in Software Engineering
 Secondary (survey) studies and their history
 What are the different types of secondary studies?
 Systematic Literature Mapping (SLM) studies
 Systematic Literature Reviews (SLR)
 Multivocal Literature Reviews (MLR)
 Grey Literature Reviews (GLR)
 …
 How do they differ?
 Why should (do) we conduct secondary studies?
 What values do they provide? Their academic impact and industry impact
 It all depends on their RQs (should be as insightful as possible)
 How to conduct secondary studies
 General process for conducing secondary studies
 Efforts spent on in conducting secondary studies
 Experience-based guidelines for effective and efficient data extraction

21. 21Vahid Garousi
How to conduct secondary studies
 Three guideline papers
have been published:
 For conducting SLRs, 2004
 For conducting SLMs, 2015
 For conducting MLRs, 2019
Each present certain processes (with similarities, but also differences), e.g., for
SLMs:

22. 22Vahid Garousi
Outline: Experience in conducting secondary (survey)
studies in Software Engineering
 Secondary (survey) studies and their history
 What are the different types of secondary studies?
 Systematic Literature Mapping (SLM) studies
 Systematic Literature Reviews (SLR)
 Multivocal Literature Reviews (MLR)
 Grey Literature Reviews (GLR)
 …
 How do they differ?
 Why should (do) we conduct secondary studies?
 What values do they provide? Their academic impact and industry impact
 It all depends on their RQs (should be as insightful as possible)
 How to conduct secondary studies
 General process for conducing secondary studies
 Efforts spent on in conducting secondary studies
 Experience-based guidelines for effective and efficient data extraction

23. 23Vahid Garousi
Conducting secondary studies is VERY effort-intensive, and
error-prone
0
10
20
30
40
50
60
70
1 6 11 16 21 26 31 36 41 46 51 56 61 66
Dataextractiontime(inminutes)
Papers' extraction order in each researcher's assigned pool (time horizon)
PhD student 1
PhD student 2
Senior Researcher 1
Senior Researcher 2
 See the empirical data below for data extraction efforts of one SLR
 We thus need to find ways to make the work effective and efficient
PhD student 1 PhD student 2 Senior Researcher 1 Senior Researcher 2 Total
Number of papers
assigned to extract
108 91 58 55 312
Total extraction time (in
hours)
54.0 33.7 7.8 16.5 112.0
Average extraction time
per paper (minutes)
30.0 22.2 8.1 18.0 21.5

24. 24Vahid Garousi
Guidelines for effective and efficient data extraction
 Even a paper on that
topic!
Process for efficient data extraction

25. 25Vahid Garousi
Outline: Experience in conducting secondary (survey)
studies in Software Engineering
 Secondary (survey) studies and their history
 What are the different types of secondary studies?
 Systematic Literature Mapping (SLM) studies
 Systematic Literature Reviews (SLR)
 Multivocal Literature Reviews (MLR)
 Grey Literature Reviews (GLR)
 How do they differ?
 Why should (do) we conduct secondary studies?
 What values do they provide? Their academic impact and industry impact
 It all depends on their RQs (should be as insightful as possible)
 How do we conduct secondary studies
 General process for conducing secondary studies
 Efforts spent on in conducting secondary studies
 Experience-based guidelines for effective and efficient data extraction
Questions /
Answers