This document discusses systematic literature reviews and the process for conducting them. It describes the key steps as: planning the review by identifying the need and forming research questions; developing a review protocol; executing a search strategy; selecting primary studies; assessing study quality; extracting and synthesizing data; and reporting results. Cohen's Kappa statistic is presented as a way to measure inter-rater reliability during study selection. Examples are provided for each step of the process.

1. Empirical Software Engineering (MSE 507 )
Dr. Assad Abbas
Department of Computer Science
COMSATS Institute of Information Technology, Islamabad
assadabbas@comsats.edu.pk

2. 2
October 14, 2022
Systematic Literature Reviews
 Review of existing literature is an essential step before beginning
any new research.
 Systematic reviews (SRs) synthesize the existing research work in
a manner that can be analyzed, assessed, and interpreted to draw
meaningful conclusions.
 The aim of conducting an SR is to gather and interpret empirical
evidence from the available research with respect to formed
research questions.
 The benefit of conducting an SR is to summarize the existing trends
in the available research, identify gaps in the current research, and
provide future guidelines for conducting new research.
 Also provide empirical evidence in support or opposition of a given
hypothesis.
SRs are better planned, more rigorous, and thoroughly analyzed as
compared to surveys or literature reviews

3. 3
October 14, 2022
Surveys vs. Systematic Reviews
Literature survey is the process of summarizing,
organizing, and documenting the existing research to
understand the research carried out in the field.
An SR is the process of systematically and critically
analyzing the information extracted from the existing
research to answer the established research
questions.
The literature survey only provides the summary of
the results of existing literature, whereas an SR
opens avenues for new research as it provides future
directions for researchers based on thorough
analysis of existing literature

4. 4
October 14, 2022
Surveys vs. Systematic Reviews
Kitchenham (2007) defined SR as:
A systematic literature review (often referred to as a
systematic review) is a means of identifying,
evaluating and interpreting all available research
relevant to a particular research question, or topic
area, or phenomenon of interest.
The studies included in the review are known as
primary studies and the SRs themselves are known
as secondary studies

5. 5
October 14, 2022
Surveys vs. Systematic Reviews

6. 6
October 14, 2022
Importance of Systematic Reviews
 They gather important empirical evidence on the technique or
method being focused in the SR. On the basis of the empirical
evidence, the strengths and weaknesses of the technique
may be summarized.
 They identify the gaps in the current research.
 They report the commonalities and the differences in the
primary studies.
 They provide future guidelines and framework to researchers
and practitioners to perform new research.

7. 7
October 14, 2022
Empirical Study Process
 The steps in process are:
Planning the review
Conducting the review
Reporting the review
results

8. 8
October 14, 2022
Empirical Study Process
Planning the Review
Identify the Need for SR
 To justify the importance of the SR, this step involves the
review of all the existing SRs conducted in the domain, thus
recognizing the existing works and identifying the areas that
need to be addressed in the new SR.
How many primary studies are available in the software
engineering context?
What are the strength and weaknesses of the existing SR
(if any) in the software engineering context?
What is the practical relevance of the proposed SR?
How will the proposed SR guide practitioners and
researchers?
How can the quality of the proposed SR be evaluated?

9. 9
October 14, 2022
Empirical Study Process
Planning the Review
Identify the Need for SR

10. 10
October 14, 2022
Empirical Study Process
Planning the Review
Formation of Research Questions
 The area of the systematic review and the specific
research questions set the focus for the identification of
the primary studies, the extraction of data from the
studies and the analysis.
 Hence, the research questions must be well thought
through and phrased

11. 11
October 14, 2022
Empirical Study Process
Planning the Review
Formation of Research Questions
 The process involves identification of relevant issues that need to
be answered by the SR
 The data for answering the identified research questions is
collected from the primary studies.
 Determine the target audience, the tools and techniques to be
evaluated, outcomes of the study, and the environment in which
the study is conducted (academic or industry)
Which areas have already been explored in the existing reviews (if
any)?
Which areas are relevant and need to be explored/answered during
the proposed SR?
Are the questions important to the researchers and software
practitioners?
Will the questions assess any similarities in the trends or identify any
deviation from the existing trends?

12. 12
October 14, 2022
Empirical Study Process
 Planning the Review
Formation of Research Questions


13. 13
October 14, 2022
Empirical Study Process
 Planning the Review
Developing the Review Protocol
 The review protocol defines the procedures for the
systematic literature review. It also acts as a log for
conducting the review
Background and rationale
Research questions
Search strategy for primary studies
Study selection criteria
Study selection procedures
Study quality assessment checklists and procedures
Data extraction strategy
Synthesis of the extracted data
Dissemination strategy
Project timetable

14. 14
October 14, 2022
Empirical Study Process
 Planning the Review
Developing the Review Protocol
 Identification of data synthesis tools and techniques to summarize
and interpret the information obtained from the primary studies
 Tables and charts may be used to highlight the similarities and
differences between the primary studies
 Statistical measures, such as mean, median, standard deviation
 Box plot, scatter plot, funnel plot, line charts
 Example of detailed description
To summarize the number of ML techniques used in primary
studies the SRML case study will use a visualization technique,
that is, a line graph to depict the number of studies pertaining to
the ML techniques in each year, and presented a classification
taxonomy of various ML techniques with their major categories
and subcategories.

15. 15
October 14, 2022
Empirical Study Process
Planning the Review
Evaluate the review protocol
 Should be performed by independent reviewers
 Requires confirmation of the following:
Development of appropriate search strings that are
derived from research questions
Adequacy of inclusion and exclusion criteria
Completeness of quality assessment questionnaire
Design of data extraction forms that address various
research questions
Appropriateness of data analysis procedures

16. 16
October 14, 2022
Empirical Study Process
Conducting the Review
Search Strategy Execution

17. 17
October 14, 2022
Empirical Study Process
Conducting the Review
Search Strategy Execution
 Specifying search strings on different databases
 Manual searching for journal / conference articles
 Sending direct queries to authors
 Snowballing—proceeding through reference list
 Should use multiple databases but it results in
duplication
 Also you can consider using different search operators.
 Boolean expressions AND, OR
 A quick list of operators can be found on this link.

18. 18
October 14, 2022
Empirical Study Process
 Conducting the Review
Search Strategy Execution
 Example
Software AND (fault OR defect OR error) AND (proneness
OR prone OR prediction OR probability) AND (regression OR
ML OR soft computing OR data mining OR classification OR
Bayesian network OR neural network [NN] OR decision tree
OR support vector machine OR genetic algorithms OR
random forest [RF]
 Electronic digital libraries
IEEE Xplore
ScienceDirect
ACM Digital Library
Wiley Online Library
Google Scholar
SpringerLink
Web of Science

19. 19
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
 Inclusion and exclusion criteria

20. 20
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
 Sometimes not clear if a certain study is related or not
 Can use Cohen Kappa statistic
 Cohen’s kappa statistic measures
interrater reliability (sometimes
called interobserver agreement)
Po = the relative observed agreement among raters.
Pe = the hypothetical probability of chance agreement
The Kappa statistic varies from 0 to 1,
where.
•0 = agreement equivalent to chance.
•0.1 – 0.20 = slight agreement.
•0.21 – 0.40 = fair agreement.
•0.41 – 0.60 = moderate agreement.
•0.61 – 0.80 = substantial agreement.
•0.81 – 0.99 = near perfect agreement
•1 = perfect agreement.

21. 21
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Example:
 Two researchers A and B rate
50 studies
20 studies were rated Yes by both.
15 studies were rated No by both.
Overall, rater A said Yes to 25 studies and No to 25.
Overall, Rater B said Yes to 30 studies and No to 20.
Step 1:
Calculate po (the observed proportional agreement):
20 studies were rated Yes by both.
15 studies were rated No by both.
So,
Po = number in agreement / total = (20 + 15) / 50 = 0.70.
The Kappa statistic varies from 0 to 1,
where.
•0 = agreement equivalent to chance.
•0.1 – 0.20 = slight agreement.
•0.21 – 0.40 = fair agreement.
•0.41 – 0.60 = moderate agreement.
•0.61 – 0.80 = substantial agreement.
•0.81 – 0.99 = near perfect agreement
•1 = perfect agreement.

22. 22
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Step 2: Find the probability that the raters would randomly both say Yes.
Rater A said Yes to 25/50 studies, or 50%(0.5).
Rater B said Yes to 30/50 studies, or 60%(0.6).
The total probability of the raters both saying Yes randomly is:
0.5 * 0.6 = 0.30.
Step 3: Calculate the probability that the raters would randomly both say No.
Rater A said No to 25/50 studies, or 50%(0.5).
Rater B said No to 20/50 studies, or 40%(0.4).
The total probability of the raters both saying No randomly is:
0.5 * 0.4 = 0.20.
Step 4: Calculate Pe. Add your answers from Step 2 and Step 3 to get the overall probability that the raters
would randomly agree. Pe = 0.30 + 0.20 = 0.50.
Step 5: Insert your calculations into the formula and solve:
k = (Po – pe) / (1 – pe )= (0.70 – 0.50) / (1 – 0.50) = 0.40.
k = 0.40, which indicates fair agreement.
The Kappa statistic varies from 0 to 1,
where.
•0 = agreement equivalent to chance.
•0.1 – 0.20 = slight agreement.
•0.21 – 0.40 = fair agreement.
•0.41 – 0.60 = moderate agreement.
•0.61 – 0.80 = substantial agreement.
•0.81 – 0.99 = near perfect agreement
•1 = perfect agreement.

23. 23
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
 Alternatively, other quality assessment method based
on questionnaire
Yes=1, Partly= 0.5, No=0

24. 24
October 14, 2022
Empirical Study Process
Conducting the Review
Study Quality Assessment
 Assessment based on quality
assessment questions
Data Extraction
 After the selection of primary
studies, the information from the
primary studies is collected in the
data extraction forms designed
during the planning phase and is
based on the research questions

25. 25
October 14, 2022
Empirical Study Process
Conducting the Review
Data Synthesis
 Qualitative results are summarized in tabular format
 Quantitative results are presented in the form of tables and
plots etc.

26. 26
October 14, 2022
Empirical Study Process
Reporting the Review
The results of the
review can be
reported in the form of
 Journal or
conferences
 Technical report
 Thesis

27. 27
October 14, 2022
Other variants
Mapping study
Scoping review
Meta analysis

28. 28
October 14, 2022
Source
Malhotra, Ruchika. Empirical research in software
engineering: concepts, analysis, and applications.
CRC press, 2016.