The authors introduce a systematic review of articles in the area of software engineering security challenges on the cloud. The review examines articles that were published between 2014 and 2019. The procedure for article qualification relied on the elucidation of Preferred Reporting Items for Systematic Reviews and Meta-Analyses premises. Meta-analysis checklist was employed to explore the analytical quality of the reviewed papers. This paper presents the results of an SLR on research approaches and tools for the deployment and orchestration of IoT systems. From thousands of relevant publications, 17 primary studies were systematically identified and reviewed for data extraction and synthesis to answer the predefined research questions. Focusing on an urban-scale, this study systematically reviews 70 journal articles, published

1. Systematic literature review
All academic projects should include a review of current literature in order to provide the Reader with sufficient context to understand the detail of the
project itself. The purpose of a systematic literature review (SLR) is very different, however. Instead of providing background information, an SLR treats
a body of written material as a data source in its own right which it exploits to answer one or more defined research questions. Like any other research
methods, an SLR needs to be carefully planned and executed so that the results of the research are credible.
When to use
SLR is the appropriate method when a research question can be answered by interrogating the literature. For example, if the objective is to pin down an
objective definition of a developing technological area, an SLR could be used to identify its core characteristics by identifying trends or correlations in
published articles. Alternatively, an examination of the literature could help to identify the most effective technical approach to solving a particular class
of problem. In this type of research, each published article constitutes a single data point and the review amounts to a statistical examination of the
whole dataset.
Kitchenham and Charters (2007) give three reasons for doing and SLR:
 To summarise the existing evidence concerning a treatment or technology e.g. to summarise the empirical evidence of the benefits and
limitations of a specific agile method.
 To identify any gaps in current research in order to suggest areas for further investigation.
 To provide a framework/background in order to appropriately position new research activities.
Process
An SLR is characterised by a rigorous approach which may not be present in an ordinary literature review. Without a defined and controlled process, the
review cannot claim to be systematic and the results and conclusions cannot be trusted. A project that centres on an SLR should take care to identify
and apply recognised structures and techniques as it would with any other research method.
Kitchenham and Charters (2007) divide the process of conducting an SLR into three main stages which are captured diagrammatically in Fig. 1 (Xiao
and Watson, 2019):

2. Figure 1: SLR process
Kitchenham and Charters provide guidelines on all the steps mentioned here as do Xiao and Watson. The main thing to bear in mind is that any
academic study should be repeatable and therefore the methods used should be explicitly described. Okoli (2017) emphasises this point using a slightly
different SLR process.
One of the key parts of an SLR is the definition of a research protocol, and one reason for that is to demonstrate that the process is free from bias.
Having a set of rules for selecting literature, and providing evidence of sticking to those rules is important if your Reader is to believe your conclusions.
For discussion and guidelines on developing a research protocol, see Drucker et al. (2016) and Keenan (2018).
The British Medical Association defines a 27-step process for conducting an SRL which is summarised in Fig. 2.
Figure 2: BMA SLR process
Examples
As well as articles such as those cited above which provide explicit guidance on conducting an SLR, looking at examples can be a great source of
inspiration in terms of configuring the process to suit your particular project and in terms of presenting the results. Very often, SLRs are easy to find in
Google Scholar because their titles end with '...: a systematic review' but this is not a hard and fast rule. Here are a few to start with:

Security Challenges in Software Engineering for the Cloud: A Systematic Review


3. The authors introduce a systematic review of articles in the area of
software engineering security challenges on the cloud. The review examines
articles that were published between 2014 and 2019. The procedure for
article qualification relied on the elucidation of Preferred Reporting Items
for Systematic Reviews and Meta-Analyses premises. Meta-analysis checklist
was employed to explore the analytical quality of the reviewed papers.


Advances in Deployment and Orchestration Approaches for IoT - A Systematic Review

This paper presents the results of an SLR on research approaches and tools
for the deployment and orchestration of IoT systems. From thousands of
relevant publications, 17 primary studies were systematically identified and
reviewed for data extraction and synthesis to answer the predefined research
questions.


Machine learning applications in urban building energy performance forecasting: A systematic review

Focusing on an urban-scale, this study systematically reviews 70 journal
articles, published in the field of building energy performance forecasting
between 2015 and 2018. The recent literature have been categorized according
to five criteria: 1. Learning Method, 2. Building Type, 3. Energy Type, 4.
Input Data, and 5. Time-scale.
What is a systematic review?
A systematic review is a literature review that gathers together all available research within a delimited research area and
according to a specific methodology. Systematic reviews usually rate at the top of evidence hierarchy since they analyze and
evaluate results from all available, original research articles that answers a specific research question.
The entire process of conducting a systematic review, from formulating a research question to establishing a protocol, searching the
literature and finally collating, examining and analysing the results, needs to proceed according to a carefully planned methodology.
All stages of the process should be documented. Systematic reviews sometimes contain so-called meta-analyses, wherein the
collected data is combined using statistical methods.
There is a great deal of literature on method, and textbooks on the topic of conducting a systematic review; these describe the
process as well as the steps usually involved in conducting a systematic review. There are also international guidelines that detail
how systematic reviews should be reported, the Prisma Guidelines.

4. Creative Commons BY SA
Illustration created by Karolinska Institutet University Library.
Tips for literature on methodology
 Cochrane Handbook for Systematic Reviews of Interventions - Cochrane Collaboration
 JBI Manual for Evidence Synthesis - JBI, University of Adelaide
Before you start, consider the following:
What kind of review are you writing? There are many different kinds of reviews, and the kind that will suit you best will depend on
a variety of factors, for instance, how much time you have, as well as what kind of research question you’re working with. Cornell
University Library has created a decision tree that can help you determine what kind of review might best suit your goals and
constraints.
How much time do you have? It can take between 6 months and 2 years to conduct a systematic review. If you have less time than
that available, you may want to conduct a rapid review instead.
Does your team have the expertise required? Apart from field-specific knowledge and awareness of research methodology, you may
need people in your team who are experts in statistical analysis and literature searches. To reduce risk for bias, it is also
recommended that at least two people, independent of one another, review all found articles and choose which studies to include in
the review.
Do you have access to the necessary tools? Apart from suitable databases in which to conduct your search, you will also want access
to a program that can help you manage your references, such as EndNote. You may also want to use a program that can help you
screen your references, such as Rayyan. Find additional examples of useful tools at Systematic Review Toolbox.
Is a similar review already in the works? Conduct a search in PROSPERO, for example, to ensure that there are no ongoing projects
that resemble yours.
Protocol - Plan on writing a protocol. The protocol can be registered, in PROSPERO for example, or in another open repository,
such as Open Science Framework or Figshare.
Start with a clear research question

5. A key starting point for a systematic review is a clear, carefully delimited research question. There are several different frameworks
available that might be able to help you structure and delimit your research question. In clinical research, it is common to use the
PICO-structure: Population, Intervention, Control and Outcome.
Create search blocks
To ensure that your research question is “searchable,” identify its key elements. You will
use these to create search blocks that will then form the basis for the search strings
used in the different databases.
Let’s take the following research question as an example:
• Does routine use of inhaled oxygen in acute myocardial infarction improve patient-centered outcomes, in
particular pain and death?
In the example above, we’ve marked some of the key elements in the research question that could potentially create useful search
blocks. Note that you would only rarely use all parts of the PICO-question in a search; most of the time, you will focus on
population and intervention. In the example above, the search blocks would most likely consist of Population = patients with heart
infarction, and Intervention = inhaled oxygen.
A general principle is that a systematic search should consist of only a few blocks. The more search blocks you have, the narrower
your search will be; the narrower your search, the greater your risk of excluding relevant articles.
Develop a search strategy
An important aspect of a systematic review is an exhaustive literature search. The search should have high sensitivity, that is, it
should be created in such a way that the search will retrieve all relevant articles within a research area, or at least discover as
many as possible. This is also called conducting a wide search. A wide search means that not all search results will be relevant. In
systematic literature searching, a precision of two-three percent is common. In contrast to a wide search, a narrow (or precise)
search will result in a larger share of relevant hits, but you also run the risk of missing relevant studies. In the picture below you can
see how the different search strategies compare to one another.
Wide search
+ Finds a significant part of the relevant literature
- Will most likely produce a large amount of irrelevant hits
Narrow search
+ Does often have a greater accuracy
- Loss of relevant literature
A systematic search strategy is often constructed using several different search blocks, in which every search block contains both
subject headings and free-text terms. The search strategy will usually be translated to several different databases. Sometimes grey

6. literature, such as dissertations or clinical trials, is included in the search as well. The database search can also be supplemented
with other search strategies, for instance handsearching in selected journals, reviewing reference lists or undertaking a citation
analysis.
It is recommended that your strategy be reviewed by at least one other person before the final literature searches are conducted.
The following checklist may be helpful during this review: Peer Review of Electronic Search Strategies.
Get started with your literature search
It’s often a good idea to start with a slightly less structured search in the various databases, a so-called test search. When you test
search, you simultaneously discover what terminology is common in the field, and thus find more search terms.
At this point, it’s a good idea to see if there are any systematic reviews within your research area that have already been published.
Sometimes published reviews will include their search strategy in the appendix.
You should also identify a few key articles, that is, the most significant studies within your research field. These should ideally be
articles that correspond to your research question as closely as possible. You can use these key articles to both construct your own
search strategy and to test it: if your search does not retrieve your key articles, then your strategy needs to be modified.
You may also want to see if there are any validated search filters you can use. Search filters are sets of search terms chosen to
restrict a search to a selection of references, such as articles based on method or study type.
Websites with validated search filters
Cochranes search filter for RCT-studies can be found in Technical Supplement to Chapter 4: Searching for and selecting studies 3.6
Search filters (p. 58-63)
InterTASC Information Specialists' Sub-group Search Filters Resource is a collaborative resource where you can find many search
filters, both published and unpublished.
Some search filters are also integrated in databases such asPubMed/MEDLINE, PsycInfo och CINAHL. In PubMed, for instance, you
can take a look at Clinical Queries. Here, you can use search filters to delimit your search to clinical trials, for instance, or studies
about Covid-19.
Finding search terms
Determining what the relevant search terms are is an important part of the systematic search. It’s often a good idea to start with
a test search where you use the subject headings and synonyms you’re already familiar with. By scanning titles, abstracts and
subject headings, you may find additional, useful search terms.
To ensure that your search retrieves as many relevant studies as possible, you should use both subject headings and free-text
terms in your search.
Subject headings are used in databases to tag all articles on a subject. When you include the subject heading in your search you will
find articles about a subject even if the authors of the article have chosen other, adjacent words to describe the article themselves.

7. It may take a while before articles in a database are tagged with subject headings, so in order to locate newer articles you also need
to include free-text terms in your search. Some databases, for instance Web of Science, do not have a list of subject headings at all;
to conduct a search there, you have to use free-text terms.
How do I find and use controlled vocabulary search terms?
How do I find and use free-text terms?
How do I combine search terms?
How can I test whether my search strategy works?
Other ways to conduct a search
For some topics and research questions, it might be appropriate to supplement your database search with other kinds of search
strategies. This might include handsearching selected journals, or doing a citation analysis.
If you want to include grey literature in your systematic review, there are particular databases you will want to search. Often you
will need to use a modified and simplified search strategy to search for grey material. To find government reports or similar texts,
you may even have to go to the organization’s website and search directly among their publication lists.
In combination with systematic searches, it is also recommended that you do a forwards and backwards citation searching on the
studies included in your review. That is, that you review the citations and references for the included publications. A variety of tools
and databases allow you to search for citation data, including Web of Science, Google Scholar and SpiderCite.
How do I search for citations and references?
Search examples from different databases
When conducting a search for a systematic review, it is recommended that you search in several different databases. In biomedicine
and health, the most commonly used databases are PubMed/MEDLINE, Embase and Cochrane Library. Depending on the topic, you
may want to supplement your search with a multi-disciplinary database like Web of Science, or subject-specific databases, such as
CINAHL, PsycInfo or ERIC.
Always use identical search blocks and free-text search terms in all the databases. Subject headings, however, will need to be
adjusted to each database’s controlled vocabulary list. You will also need to adapt field tags and other symbols.
In many databases, the default setting is that the search is conducted in all fields or in a combination of several fields. However, in
a systematic literature search, it is recommended that you specify the search field manually instead. This will give you more control
over the search and will also make your search strategy more transparent.
Apart from the tips provided below, KIB’s search team has also developed a database sheet as support while conducting a
systematic review. Other tools can help you translate the search syntax used in different databases, for example, Systematic Review
Accelerator Polyglot.

8. How do I search in PubMed?
How do I search in MEDLINE Ovid?
How do I search in Embase
How do I search in Web of Science?
How do I search in CINAHL?
Documentation
PRISMA
As with any type of research, the review process in a systematic review should be transparently documented in all parts, clearly
reported in the final publication, and reproducible.
To help you do this, follow established guidelines, such as those found in the PRISMA Guidelines (Preferred Reporting Items
for Systematic Reviews and Meta-Analyses).
The guidelines describe how systematic reviews should be reported. PRISMA 2020 consists of a checklist with 27 points and
several flowcharts.
According to PRISMA’s checklist, all databases, registries and other sources used to find articles should be reported, in addition to
the date when each source was last searched. The search strategy should be documented in full for each database, and can be
published in an appendix to the published review article.
There is a particular checklist for how the search itself should be reported: PRISMA-S
Update your search before publication
According to the Cochrane Handbook, a search should be updated before publication if more than twelve months (ideally, six
months) have passed since the original search was conducted. Journals also often demand that a search be updated if several
months have passed since the article was submitted.
A quick way to find new articles is to repeat the search and delimit it to a certain interval of publishing dates. If you do so,
however, you risk missing material that may have been added retrospectively to the database. A safer way is to delimit your search
to the dates when articles were last added to the database, or use EndNote to deduplicate your new references.
Find out more:
 Bramer, W., & Bain, P. (2017). Updating search strategies for systematic reviews using EndNote. Journal of the
Medical Library Association : JMLA, 105(3), 285–289

 Updating a search. University of South Australia.

9. Reference management
Systematic literature searches tend to generate a great deal of references. Since you search many databases, you will also likely end
up with duplicate references.
To handle this amount of references, we recommend learning how to use a reference management software such as EndNote. In
EndNote, you can both organize your references and get rid of duplicates.
How do I delete duplicates in EndNote?
Screening
During the screening process you may want to get help from specific programs developed to review article abstracts as well as full-
text articles. They are often collaborative and tag, for instance, inclusion- and exclusion criteria. At the moment, KI does not have a
license for these kinds of programs. A few examples:
 Rayyan (free of charge)
 EPPI-Reviewer 4
 Covidence
 DistillerSR
At the library we offer workshops on how to get started using Rayyan. McGill Library also has a helpful guide to Rayyan.
Contact us
Systematic Literature Reviews (SLR) originated in medicine and were developed in an effort to remove bias from the review process.
In order to minimise bias, the review is done by a team, using a strict protocol or methodology. This includes:
 Clearly defined objectives or a focused research question, which doesn’t change
 An agreed (stated) inclusion/exclusion criteria
 An agreed comprehensive search strategy to collect information (which must include specialist sources)
 A clear method for screening material
Unlike other types of reviews SLRs can be:
 Repeated
 Updated
SLRs are useful for:
 Summarising data
 Synthesising research in a particular field
 Interpreting findings (risk assessment)
 Identifying future research
 Answering specific questions
 Problem solving

10. 1.
5. Systematic Literature Review (SLR) 5.1 Pengantar SLR 5.2 Tahapan Planning 5.3 Tahapan Conducting 5.4 Tahapan Reporting
2.
3.
5.1 Pengantar SLR
4.
5.
Literature Review • Literature Review is a critical and in depth evaluation of previous research (Shuttleworth, 2009)(https://explorable.com/what-
is-a-literature-review) • A summary and synopsis of a particular area of research, allowing anybody reading the paper to establish the reasons for
pursuing a particular research • A good Literature Review evaluates quality and findings of previous research
6.
7.
ManfaatMereviewLiteratur • Memperdalampengetahuantentangbidang yang diteliti (Textbooks) • Mengetahuihasilpenelitian yangberhubungan dan
yang sudah pernahdilaksanakan (Related Research) (Paper) • Mengetahuiperkembanganilmupadabidang yang kitapilih (state-of-the-art) (Paper) •
Memperjelasmasalahpenelitian (Paper)
8.
9.

11. Literature Review Methods • Typesand Methods of Literature Review: • Traditional Review • Systematic Literature Review or Systematic Review •
Systematic Mapping Study (Scoping Study) • Tertiary Study • SLR is now well established review method in the field of software engineering
(Kitchenham & Charters, Guidelines in performing Systematic Literature Reviews in Software Engineering, EBSE Technical Report version 2.3,
2007)
10.
11.
1. Traditional Review • Provides an overview of the research findings on particular topics • Advantages: produce insightful, valid syntheses of the
research literature if conducted by the expert • Disadvantages: vulnerable to unintentional and intentional bias in the selection, interpretation and
organization of content • Examples: • Liao et al., Intrusion Detection System: A Comprehensive Review, Journal of Network and Computer
Applications, 36(2013) • Galar et al., A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches,
IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol. 42, No. 4, July 2012 • CagatayCatal, Software fault
prediction: A literature review and current trends, Expert Systems with Applications 38 (2011)
12.
13.
2. Systematic Mapping Study • Suitable for a very broad topic • Identify clusters of evidence (making classification) • Direct the focus of future SLRs
• To identify areas for future primary studies • Examples: • Neto et al., A systematic mapping study of software product lines testing, Information
and Software Technology Vol. 53, Issue 5, May 2011 • Elberzhager et al., Reducing test effort: A systematic mapping study on existing approaches,
Information and Software Technology 54 (2012)
14.
15.
3. Systematic Literature Review (SLR) • The purpose of a systematic literature reviews is to provide as complete a list as possible of all the published
studies relating to a particular subject area • A process of identifying, assessing, and interpreting all available research evidence, to provide answers
for a particular research question • A form of secondary study that uses a well-defined methodology • SLRs are well established in other disciplines,
particularly medicine. They integrate an individual clinical expertise and facilitate access to the outcomes of the research (Kitchenham & Charters,
Guidelines in performing Systematic Literature Reviews in Software Engineering, EBSE Technical Report version 2.3, 2007)
16.
17.
3. Systematic Literature Review (SLR) Examples of SLR: • Hall et al., A Systematic Literature Review on Fault Prediction Performance in Software
Engineering, IEEE Transaction on Software Engineering, Vol. 38, No. 6, 2012 • Romi Satria Wahono, A Systematic Literature Review of Software
Defect Prediction: Research Trends, Datasets, Methods and Frameworks, Journal of Software Engineering, Vol. 1, No. 1, April 2015 • Matthias
Galster, Danny Weyns, Dan Tofan, BartoszMichalik, and Paris Avgeriou, Variability in Software Systems: A Systematic Literature Review, IEEE
Transactions on Software Engineering, Vol 40, No 3, 2014
18.
19.
4. Tertiary study • Is a SLR of SLRs • To answer a more wider question • Uses the same method as in SLR • Potentially less resource intensive •
Examples: • Kitchenham et al., Systematic literature reviews in software engineering – A tertiary study, Information and Software Technology 52
(2010) • Cruzes et al., Research synthesis in software engineering: A tertiary study, Information and Software Technology 53 (2011)
20.
21.
Tahapan SLR 1. Formulate the Review’sResearchQuestion 2. Develop the Review’sProtocol 5.1 PLANNING 1. Identify the RelevantLiterature 2.
Perform Selection of PrimaryStudies 3. Perform DataExtraction 4. Assess Studies’ Quality 5. Conduct Synthesis of Evidence 5.2 CONDUCTING 1.
Write Up the SLR Paper 2. Choose the Right Journal 5.3 REPORTING

12. 22.
23.
5.1Tahapan Planning Formulate the Review’s Research Question Develop the Review’s Protocol
24.
25.
Formulate the Review’s Research Question • Features of good question: • The RQ is meaningful and important to practitioners and researchers. • The
RQ will lead to changes in current software engineering practice or to increase confidence in the value of current practice • The RQ will identify
discrepancies between commonly held beliefs and the reality • RQ can be derived primarily based on researcher’s interest • An SLR for PhD thesis
should identify existing basis for the research work and where it fits in the current body of knowledge
26.
27.
The Research Question (RQ) • Is the most important part in any SLR • Is not necessarily the same as questions addressed in your research • Is used
to guide the search process • Is used to guide the extraction process • Data analysis (synthesis of evidence) is expected to answer your SLR’s RQ
28.
29.
RQ and PICOC The formulation of RQs about effectiveness of a treatment should focus on 5 elements known as PICOC: • Population (P)- the target
group for the investigation (e.g. people, software etc.) • Intervention (I) - specifies the investigation aspects or issues of interest to the researchers •
Comparison (C)– aspect of the investigation with which the intervention is being compared to • Outcomes (O)– the effect of the intervention •
Context (C)– the setting or environment of the investigation (Petticrew et al., Systematic Reviews in the Social Sciences: A Practical Guide,
Blackwell Publishing, 2006)
30.
31.
Example of PICOC (Kitchenham et al., 2007) Kitchenham et al., A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies, IEEE
Transactions on Software Engineering, 33 (5), 2007
32.
33.
Example of PICOC (Wahono, 2015) Romi Satria Wahono, A Systematic Literature Review of Software Defect Prediction: Research Trends, Datasets,
Methods and Frameworks, Journal of Software Engineering, Vol. 1, No. 1, pp. 1-16, April 2015
34.
35.
Example of RQs (Kitchenham, 2007) Kitchenham et al., A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies, IEEE
Transactions on Software Engineering, 33 (5), 2007 • RQ1: What evidence is there that cross-company estimation models are not significantly
different from within-company estimation models for predicting effort for software/Web projects? • RQ2: What characteristics of the study data
sets and the data analysis methods used in the study affect the outcome of within- and cross-company effort estimation accuracy studies? • RQ3:
Which experimental procedure is most appropriate for studies comparing within- and cross-company estimation models?
36.
37.
Example of RQs (Davis et al., 2006) Davis et al., Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic
Review, 14th IEEE Requirements Engineering Conference, 2006 • RQ: What elicitation technique is most efficient in a particular setting?

13. 38.
39.
Example of RQs (Radjenovic et al., 2013) Radjenovic et al., Software fault prediction metrics: A systematic literature review, Information and
Software Technology, Vol. 8, No. 55, pp. 1397-1418, 2013 • RQ1: Which software metrics for fault prediction exist in literature? • RQ2: What
data sets are used for evaluating metrics?
40.
41.
Example of RQ (Wahono, 2015) Romi Satria Wahono, A Systematic Literature Review of Software Defect Prediction: Research Trends, Datasets,
Methods and Frameworks, Journal of Software Engineering, Vol. 1, No. 1, pp. 1-16, April 2015
42.
43.
2. Develop the Review’s Protocol • A plan that specifies the basic review procedures (method) • Components of a protocol: • Background • Research
Questions • Search terms • Selection criteria • Quality checklist and procedures • Data extraction strategy • Data synthesis strategy
44.
45.
5.2Tahapan Conducting Identify the Relevant Literature Perform Selection of Primary Studies Perform Data Extraction Assess Studies’ Quality
Conduct Synthesis of Evidence
46.
47.
1. Identifying Relevant Literature • Involves a comprehensive and exhaustive searching of studies to be included in the review • Define a search
strategy • Search strategies are usually iterative and benefit from: • Preliminary searches (to identify existing review and volume of studies) • Trial
searches (combination of terms from RQ) • Check the search results against list of known studies • Consult the experts in the field
48.
49.
Approach to Construct Search String • Derive major terms used in the review questions based on the PICOC • List the keywords mentioned in the
article • Search for synonyms and alternative words • Use the boolean OR to incorporate alternative synonyms • Use the boolean AND to link major
terms
50.
51.
Example of Search String (Kitchenham et al., 2007) • Kitchenham et al. (2007) used their structured questions to construct search strings for use
with electronic databases: • Population: software OR application OR product OR Web OR WWW OR Internet OR World-Wide Web OR project OR
development • Intervention: cross company OR cross organisation OR cross organization OR multiple-organizational OR multiple-organisational
model OR modeling OR modelling effort OR cost OR resource estimation OR prediction OR assessment • Contrast: within-organisation OR within-
organization OR within-organizational OR within-organisational OR single company OR single organisation • Outcome: Accuracy OR Mean
Magnitude Relative Error • The search strings were constructed by linking the four OR lists using the Boolean AND
52.
53.
Example of Search String (Wahono, 2015) Search String: (software OR applicati* OR systems ) AND (fault* OR defect* OR quality OR error-prone)
AND (predict* OR prone* OR probability OR assess* OR detect* OR estimat* OR classificat*) Romi Satria Wahono, A Systematic Literature Review

14. of Software Defect Prediction: Research Trends, Datasets, Methods and Frameworks, Journal of Software Engineering, Vol. 1, No. 1, pp. 1-16,
April 2015
54.
55.
Example of Search String (Salleh et al., 2011) • The complete search term initially used : (student* OR undergraduate*) AND (pair programming
OR pair-programming) AND ((experiment* OR measurement OR evaluation OR assessment) AND (effective* OR efficient OR successful) • A very
limited number of results retrieved when using the complete string, thus a much simpler string was derived. • Subject librarian suggested to revise
the search string: “pair programming” OR “pair-programming”
56.
57.
Sources of Evidence • Digital libraries • Reference lists from relevant primary studies and review articles • Journals (including company journals such
as the IBM Journal of Research and Development), grey literature (i.e. technical reports, work in progress) • Conference proceedings • Research
registers • The Internet (google) • Direct contact specific researcher(s)
58.
59.
Studies SelectionStrategy(Wahono, 2015) • Publication Year: • 2000-2013 • Publication Type: • Journal • Conference Proceedings • Search String:
softwareAND(fault* OR defect* OR quality OR error-prone) AND(predict* OR prone* OR probability OR assess* OR detect* ORestimat*
ORclassificat*) • Selected Studies: • 71
60.
61.
Sources of Evidence (Kitchenham et al., 2007) • The search strings were used on 6 digital libraries: • INSPEC , El Compendex, Science Direct, Web
of Science, IEEExplore, ACM Digital library • Search specific journals and conf. proceedings: • Empirical Software Engineering (J) • Information and
Software Technology (J) • Software Process Improvement and Practice (J) • Management Science (J) • International Software Metrics Symposium
(C) • International Conference on Software Engineering (C) • Manual search: • Evaluation and Assessment in Software Engineering (C) • Check
references of each relevant article • Contact researchers
62.
63.
Managing Bibliography • Use relevant Bibliographic package to manage large number of references • E.g. Mendeley, EndNote, Zotero, JabRef
Reference Manager etc.
64.
65.
Documenting the Search • The process of conducting SLR must be transparent and replicable • The review should be documented in sufficient detail
• The search should be documented and changes noted • Unfiltered search results should be saved for possible reanalysis
66.
67.
2. Selection of Studies • Primary studies need to be assessed for their actual relevance • Set the criteria for including or excluding studies (decided
earlier during protocol development, can be refined later) • Inclusion & exclusion criteria should be based on RQ • Selection process should be piloted
• Study selection is a multistage process
68.

15. 69.
Selection of Studies(Kitchenham et al., 2007) • Kitchenham et al. (2007) used the following inclusion criteria: • Any study that compared
predictions of cross-company models with within-company models based on analysis of single company project data. • They used the following
exclusion criteria: • Studies where projects were only collected from a small number of different sources (e.g. 2 or 3 companies) • Studies where
models derived from a within-company data set were compared with predictions from a general cost estimation model.
70.
71.
Selection of Studies (Wahono, 2015)
72.
73.
Selection of Studies (Salleh et al., 2011) • Inclusion criteria: • to include any empirical studies of PP that involved highereducation students as the
population of interest. • Exclusion criteria: • Papers presenting unsubstantiated claims made by the author(s), for which no evidence was available. •
Papers about Agile/XP describing development practices other than PP, such as test-first programming, refactoring etc. • Papers that only
described tools (software or hardware) that could support the PP practice. • Papers not written in English. • Papers involving students but outside
higher education
74.
75.
3. Assessing Studies’ Quality • To provide more detailed Inclusion/Exclusion criteria • To check whether quality differences provide an explanation for
differences in study results • As a means of weighting the importance of individual studies when results are being synthesized • To guide the
interpretation of findings and determine the strength of inferences • To guide recommendations for further research
76.
77.
Assessing Studies’ Quality • Quality relates to the extent to which the study minimizes bias and maximizes internal and external validity(Khan et al.
2001) • Quality Concepts Definition (Kitchenham & Charter, 2007)
78.
79.
Assessing Studies’ Quality • Assessing quality of studies: • Methodology or design of the study • Analysis of studies’ findings • Quality checklist or
instrument need to be designed to facilitate quality assessment • Most quality checklists include questions aimed at assessing the extent to which
articles have addressed bias and validity
80.
81.
Study Quality Assessment (Salleh et al., 2011)
82.
83.
Study Quality Assessment(Kitchenham et al., 2007) Kitchenham et al. (2007) constructed a quality questionnaire based on 5 issues affecting the
quality of the study: • Is the data analysis process appropriate? • Did studies carry out a sensitivity or residual analysis? • Were accuracy statistics
based on the raw data scale? • How good was the study comparison method? • The size of the within-company data set(e.g < 10 projects
considered poor quality)
84.

16. 85.
4. Data Extraction • Involve reading the full text article • Data extracted from primary studies should be recorded using data extraction form • The
form should be designed and piloted when the protocol is defined • Collect all the information that can be used to answer the RQ and the study’s
quality criteria • Both quality checklist and review data can be included in the same form • In case of duplicates publications (reporting the same
data), refer the most complete one • For validation, a set of papers should be reviewed by 2 or more researchers. Compare results and resolve any
conflicts
86.
87.
5. Synthesis of Evidence • Involves collating and summarizing the results of the included primary studies • Key objectives of data synthesis(Cruzes &
Dyba, 2011): • to analyze and evaluate multiple studies • to select appropriate methods for integrating or providing new interpretive explanations
about them • Synthesis can be: • Descriptive (narrative/non-quantitative) • Quantitative (e.g. meta-analysis) (Cruzes et al., Research Synthesis in
Software Engineering: A tertiary study, Information and Software Technology, 53(5), 2011)
88.
89.
Descriptive Synthesis (Narrative) “An approach to the synthesis of findings from multiple studies that relies primarily on the use of words and text
to summarize and explain the findings of the synthesis. It adopts a textual approach to the process of synthesis to ‘tell the story’ of the findings
from the included studies.” (Popay et al. 2006) • Use tables to tabulate information extracted from included studies (e.g. population, number of
included studies, study quality etc.) • Tables should be structured to highlight similarity or differences of study outcomes • Were the findings
consistent (homogeneous) or inconsistent?
90.
91.
Quantitative Synthesis (Meta-Analysis) • Meta-analysis can be used to aggregate results or to pool data from different studies • The outcome of a
meta-analysis is an average effect size with an indication of how variable that effect size is between studies • Meta-analysis involves three main
steps: 1. Decide which studies to be included in the meta-analysis 2. Estimate an effect size for each individual study 3. Combine the effect sizes
from the individual studies to estimate and test the combined effect • Results of the meta-analysis can be presented in a forest plot
92.
SYSTEMATIC LITERATURE REVIEW
A systematic literature review (SLR) identifies, selects and critically appraises research in order to answer a clearly formulated question (Dewey, A. & Drahota, A. 2016). The
systematic review should follow a clearly defined protocol or plan where the criteria is clearly stated before the review is conducted. It is a comprehensive, transparent search
conducted over multiple databases and grey literature that can be replicated and reproduced by other researchers. It involves planning a well thought out search strategy
which has a specific focus or answers a defined question. The review identifies the type of information searched, critiqued and reported within known timeframes. The search
terms, search strategies (including database names, platforms, dates of search) and limits all need to be included in the review.
Pittway (2008) outlines seven key principles behind systematic literature reviews
 Transparency
 Clarity
 Integration
 Focus
 Equality
 Accessibility
 Coverage
Systematic literature reviews originated in medicine and are linked to evidence based practice. According to Grant & Booth (p 91, 2009) "the expansion in evidence-based
practice has lead to an increasing variety of review types". They compare and contrast 14 review types, listing the strengths and weaknesses of each review.
Tranfield et al (2003) discusses the origins of the evidence-based approach to undertaking a literature review and its application to other disciplines including management
and science.
References and additional resources
Dewey, A. & Drahota, A. (2016) Introduction to systematic reviews: online learning module Cochrane Training https://training.cochrane.org/interactivelearning/module-1-
introduction-conducting-systematic-reviews

17. Gough, David A., David Gough, Sandy Oliver, and James Thomas. An Introduction to Systematic Reviews. Systematic Reviews. London: SAGE, 2012.
Grant, M. J. & Booth, A. (2009) A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal 26(2), 91-108
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing
between a systematic or scoping review approach. BMC Med Res Methodol, 18(1), 143. https://doi.org/10.1186/s12874-018-0611-x
Pittway, L. (2008) Systematic literature reviews. In Thorpe, R. & Holt, R. The SAGE dictionary of qualitative management research. SAGE Publications Ltd
doi:10.4135/9780857020109
Tranfield, D., Denyer, D & Smart, P. (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal
of Management 14(3), 207-222
According to the Cochrane Handbook for Systematic Reviews of Interventions, a systematic review attempts to gather all empirical evidence that fits pre-specified criteria in order to
answer a specific research question. A systematic review has: 1) a clearly stated set of objectives with pre-defined eligibility criteria for studies; 2) an explicit, reproducible
methodology; 3) a thorough, objective and reproducible search of a range of sources to identify as many relevant studies as possible; 4) an assessment of the validity of the
findings for the included studies; 5) a systematic presentation and synthesis of the characteristics and findings of the studies.
Source: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from
www.training.cochrane.org/handbook.
The Systematic Review Process
1. Define your research question.
2. Determine that there are no existing systematic reviews or systematic review protocols that address your question.
3. Assemble your research team. The team should ideally include subject area specialists, a specialist versed in systematic review methods and a
librarian/information specialist who has had training in systematic review methods.
4. Develop your protocol, which is a detailed description of the objectives and methods of the review. It should include the rationale and objectives of the review, the
inclusion/exclusion of the criteria, methods for locating studies, quality assessment methods, data extraction methods, data synthesis methods,etc.
5. Register your protocol.
6. Review the literature to search for studies.
7. Screen titles and abstracts to identify potentially relevant studies.
8. Review full-text and apply inclusion and exclusion criteria.
9. Assess quality of eligible studies.
10. Depending on the type of review, extract data from individual studies.
11. Analyze data and synthesize if appropriate.
12. Report findings.
Articles
 EPPI-Centre (2019). What is a systematic review? UCL Institute of Education, University College London.
 Henderson, Lorna K (09/2010). How to write a Cochrane systematic review. Nephrology (Carlton, Vic.) (1320-5358), 15(6), p. 617.
 National Collaborating Centre for Methods and Tools. (2014). Anatomy of a Systematic Review [fact sheet].
 Riesenberg, Lee Ann (04/2014). Conducting a successful systematic review of the literature, part 1. Nursing (Jenkintown, Pa.) (0360-4039), 44 (4), p. 13.
 Riesenberg, Lee Ann (06/2014). Conducting a successful systematic review of the literature, part 2. Nursing (Jenkintown, Pa.) (0360-4039), 44 (6), p. 23.
 Umscheid, Craig A (09/2013). A Primer on Performing Systematic Reviews and Meta-analyses. Clinical infectious diseases (1058-4838), 57 (5), p. 725.
Steps in the Literature Review Process
1. Define the research question (for more)
1. You may need to some exploratory searching of the literature to get a sense of scope, to determine whether you need to narrow or broaden your focus
2. Identify databases that provide the most relevant sources, and identify relevant terms (controlled vocabularies) to add to your search strategy
3. Finalize your research question

18. 2. Determine inclusion/exclusion criteria
1. Think about relevant dates, geographies (and languages), methods, and conflicting points of view
3. Choose databases and conduct the search
1. Conduct searches in the published literature via the identified databases
2. Check to see if this topic has been covered in other discipline's databases
3. Examine the citations of on-point articles for keywords, authors, and previous research (via references) and cited reference searching.
4. Review your results
1. Save your search results in a citation management tool (such as Zotero, Mendeley or EndNote)
2. De-duplicate your search results
3. Make sure that you've found the seminal pieces -- they have been cited many times, and their work is considered foundational
4. Check with your professor or a librarian to make sure your search has been comprehensive
5. Synthesize the information gathered
1. Evaluate the strengths and weaknesses of individual sources and evaluate for bias, methodologies, and thoroughness
2. Group your results in to an organizational structure that will support why your research needs to be done, or that provides the answer to your research
question
3. Develop your conclusions
6. Analyze the information gathered
1. Are there gaps in the literature?
2. Where has significant research taken place, and who has done it?
3. Is there consensus or debate on this topic?
4. Which methodological approaches work best?
7. Write the literature review
1. Pick an organizational structure, i.e., themes, approaches, concepts, methodologies.
1. For example: Background, Current Practices, Critics and Proponents, Where/How this study will fit in
2. Organize your citations and focus on your research question and pertinent studies
3. Compile your bibliography
Note: The first four steps are the best points at which to contact a librarian. Your librarian can help you determine the best databases to use for your topic, assess scope, and
formulate a search strategy.