Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Lecture 3.pptx
1. Empirical Software Engineering (MSE 507 )
Dr. Assad Abbas
Department of Computer Science
COMSATS Institute of Information Technology, Islamabad
assadabbas@comsats.edu.pk
2. 2
October 14, 2022
Systematic Literature Reviews
Review of existing literature is an essential step before beginning
any new research.
Systematic reviews (SRs) synthesize the existing research work in
a manner that can be analyzed, assessed, and interpreted to draw
meaningful conclusions.
The aim of conducting an SR is to gather and interpret empirical
evidence from the available research with respect to formed
research questions.
The benefit of conducting an SR is to summarize the existing trends
in the available research, identify gaps in the current research, and
provide future guidelines for conducting new research.
Also provide empirical evidence in support or opposition of a given
hypothesis.
SRs are better planned, more rigorous, and thoroughly analyzed as
compared to surveys or literature reviews
3. 3
October 14, 2022
Surveys vs. Systematic Reviews
Literature survey is the process of summarizing,
organizing, and documenting the existing research to
understand the research carried out in the field.
An SR is the process of systematically and critically
analyzing the information extracted from the existing
research to answer the established research
questions.
The literature survey only provides the summary of
the results of existing literature, whereas an SR
opens avenues for new research as it provides future
directions for researchers based on thorough
analysis of existing literature
4. 4
October 14, 2022
Surveys vs. Systematic Reviews
Kitchenham (2007) defined SR as:
A systematic literature review (often referred to as a
systematic review) is a means of identifying,
evaluating and interpreting all available research
relevant to a particular research question, or topic
area, or phenomenon of interest.
The studies included in the review are known as
primary studies and the SRs themselves are known
as secondary studies
6. 6
October 14, 2022
Importance of Systematic Reviews
They gather important empirical evidence on the technique or
method being focused in the SR. On the basis of the empirical
evidence, the strengths and weaknesses of the technique
may be summarized.
They identify the gaps in the current research.
They report the commonalities and the differences in the
primary studies.
They provide future guidelines and framework to researchers
and practitioners to perform new research.
7. 7
October 14, 2022
Empirical Study Process
The steps in process are:
Planning the review
Conducting the review
Reporting the review
results
8. 8
October 14, 2022
Empirical Study Process
Planning the Review
Identify the Need for SR
To justify the importance of the SR, this step involves the
review of all the existing SRs conducted in the domain, thus
recognizing the existing works and identifying the areas that
need to be addressed in the new SR.
How many primary studies are available in the software
engineering context?
What are the strength and weaknesses of the existing SR
(if any) in the software engineering context?
What is the practical relevance of the proposed SR?
How will the proposed SR guide practitioners and
researchers?
How can the quality of the proposed SR be evaluated?
10. 10
October 14, 2022
Empirical Study Process
Planning the Review
Formation of Research Questions
The area of the systematic review and the specific
research questions set the focus for the identification of
the primary studies, the extraction of data from the
studies and the analysis.
Hence, the research questions must be well thought
through and phrased
11. 11
October 14, 2022
Empirical Study Process
Planning the Review
Formation of Research Questions
The process involves identification of relevant issues that need to
be answered by the SR
The data for answering the identified research questions is
collected from the primary studies.
Determine the target audience, the tools and techniques to be
evaluated, outcomes of the study, and the environment in which
the study is conducted (academic or industry)
Which areas have already been explored in the existing reviews (if
any)?
Which areas are relevant and need to be explored/answered during
the proposed SR?
Are the questions important to the researchers and software
practitioners?
Will the questions assess any similarities in the trends or identify any
deviation from the existing trends?
13. 13
October 14, 2022
Empirical Study Process
Planning the Review
Developing the Review Protocol
The review protocol defines the procedures for the
systematic literature review. It also acts as a log for
conducting the review
Background and rationale
Research questions
Search strategy for primary studies
Study selection criteria
Study selection procedures
Study quality assessment checklists and procedures
Data extraction strategy
Synthesis of the extracted data
Dissemination strategy
Project timetable
14. 14
October 14, 2022
Empirical Study Process
Planning the Review
Developing the Review Protocol
Identification of data synthesis tools and techniques to summarize
and interpret the information obtained from the primary studies
Tables and charts may be used to highlight the similarities and
differences between the primary studies
Statistical measures, such as mean, median, standard deviation
Box plot, scatter plot, funnel plot, line charts
Example of detailed description
To summarize the number of ML techniques used in primary
studies the SRML case study will use a visualization technique,
that is, a line graph to depict the number of studies pertaining to
the ML techniques in each year, and presented a classification
taxonomy of various ML techniques with their major categories
and subcategories.
15. 15
October 14, 2022
Empirical Study Process
Planning the Review
Evaluate the review protocol
Should be performed by independent reviewers
Requires confirmation of the following:
Development of appropriate search strings that are
derived from research questions
Adequacy of inclusion and exclusion criteria
Completeness of quality assessment questionnaire
Design of data extraction forms that address various
research questions
Appropriateness of data analysis procedures
17. 17
October 14, 2022
Empirical Study Process
Conducting the Review
Search Strategy Execution
Specifying search strings on different databases
Manual searching for journal / conference articles
Sending direct queries to authors
Snowballing—proceeding through reference list
Should use multiple databases but it results in
duplication
Also you can consider using different search operators.
Boolean expressions AND, OR
A quick list of operators can be found on this link.
18. 18
October 14, 2022
Empirical Study Process
Conducting the Review
Search Strategy Execution
Example
Software AND (fault OR defect OR error) AND (proneness
OR prone OR prediction OR probability) AND (regression OR
ML OR soft computing OR data mining OR classification OR
Bayesian network OR neural network [NN] OR decision tree
OR support vector machine OR genetic algorithms OR
random forest [RF]
Electronic digital libraries
IEEE Xplore
ScienceDirect
ACM Digital Library
Wiley Online Library
Google Scholar
SpringerLink
Web of Science
19. 19
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Inclusion and exclusion criteria
20. 20
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Sometimes not clear if a certain study is related or not
Can use Cohen Kappa statistic
Cohen’s kappa statistic measures
interrater reliability (sometimes
called interobserver agreement)
Po = the relative observed agreement among raters.
Pe = the hypothetical probability of chance agreement
The Kappa statistic varies from 0 to 1,
where.
•0 = agreement equivalent to chance.
•0.1 – 0.20 = slight agreement.
•0.21 – 0.40 = fair agreement.
•0.41 – 0.60 = moderate agreement.
•0.61 – 0.80 = substantial agreement.
•0.81 – 0.99 = near perfect agreement
•1 = perfect agreement.
21. 21
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Example:
Two researchers A and B rate
50 studies
20 studies were rated Yes by both.
15 studies were rated No by both.
Overall, rater A said Yes to 25 studies and No to 25.
Overall, Rater B said Yes to 30 studies and No to 20.
Step 1:
Calculate po (the observed proportional agreement):
20 studies were rated Yes by both.
15 studies were rated No by both.
So,
Po = number in agreement / total = (20 + 15) / 50 = 0.70.
The Kappa statistic varies from 0 to 1,
where.
•0 = agreement equivalent to chance.
•0.1 – 0.20 = slight agreement.
•0.21 – 0.40 = fair agreement.
•0.41 – 0.60 = moderate agreement.
•0.61 – 0.80 = substantial agreement.
•0.81 – 0.99 = near perfect agreement
•1 = perfect agreement.
22. 22
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Step 2: Find the probability that the raters would randomly both say Yes.
Rater A said Yes to 25/50 studies, or 50%(0.5).
Rater B said Yes to 30/50 studies, or 60%(0.6).
The total probability of the raters both saying Yes randomly is:
0.5 * 0.6 = 0.30.
Step 3: Calculate the probability that the raters would randomly both say No.
Rater A said No to 25/50 studies, or 50%(0.5).
Rater B said No to 20/50 studies, or 40%(0.4).
The total probability of the raters both saying No randomly is:
0.5 * 0.4 = 0.20.
Step 4: Calculate Pe. Add your answers from Step 2 and Step 3 to get the overall probability that the raters
would randomly agree. Pe = 0.30 + 0.20 = 0.50.
Step 5: Insert your calculations into the formula and solve:
k = (Po – pe) / (1 – pe )= (0.70 – 0.50) / (1 – 0.50) = 0.40.
k = 0.40, which indicates fair agreement.
The Kappa statistic varies from 0 to 1,
where.
•0 = agreement equivalent to chance.
•0.1 – 0.20 = slight agreement.
•0.21 – 0.40 = fair agreement.
•0.41 – 0.60 = moderate agreement.
•0.61 – 0.80 = substantial agreement.
•0.81 – 0.99 = near perfect agreement
•1 = perfect agreement.
23. 23
October 14, 2022
Empirical Study Process
Conducting the Review
Selection of Primary Studies
Alternatively, other quality assessment method based
on questionnaire
Yes=1, Partly= 0.5, No=0
24. 24
October 14, 2022
Empirical Study Process
Conducting the Review
Study Quality Assessment
Assessment based on quality
assessment questions
Data Extraction
After the selection of primary
studies, the information from the
primary studies is collected in the
data extraction forms designed
during the planning phase and is
based on the research questions
25. 25
October 14, 2022
Empirical Study Process
Conducting the Review
Data Synthesis
Qualitative results are summarized in tabular format
Quantitative results are presented in the form of tables and
plots etc.
26. 26
October 14, 2022
Empirical Study Process
Reporting the Review
The results of the
review can be
reported in the form of
Journal or
conferences
Technical report
Thesis