What is a Systematic Literature Review and how to do it.

1. SystematicLiteratureReview

2. Whatisasystematicreview?

4. Typesofreviews
• Systematic reviews:
Use explicit and rigorous methods to identify, critically appraise, and synthesize
relevant studies
• Narrative reviews:
Summaries of research that lack explicit descriptions of systematic methods
• Meta-analysis:
Quantitative review using statistical analyses
• Meta-ethnography:
Systematic review of qualitative data

5. Whyweneedsystematicreviews

6. Whynottraditionalreviews

7. Majorsourcesofbiasinsystematicreviews
1) Publication bias
Negative
fi
ndings may not get published
Positive
fi
ndings may get published more than once
2) Selection bias
Inclusion and exclusion criteria can create bias
3) Language bias (i.e. English Language)
Authors might be more likely to report positive
fi
ndings in an international
English language journal and negative
fi
ndings in a local journal

8. Disadvantagesofsystematicreviews
๏ Oversimpli
fi
cation with regard to inclusion/exclusion criteria
๏ Over generalisation
๏ Di
ff
erent conclusions depending on the precise form of the "review question"
๏ Lots of e
ff
orts
at least two researchers
๏ Not compatible with short papers

9. KeyStagesinaSystematicReview
De
f
ine research/review question
Identify relevant studies
Assess eligibility
Data extraction /checking
Synthesis
Develop review protocol
Knowledge translation
Study assessment/appraisal

10. De
f
ineresearch/reviewquestion

11. Questionformulation

12. ProtocolDevelopment

13. SearchingforInformation

14. Booleanoperators

15. SelectionofStudies
• Year

16. Data Extraction/Quality
Appraisal

18. DataExtraction
• How much to extract
• Data extraction tools
• Consistency/Standardisation
• E
ffi
cient data extraction
•

19. QualityAssessment&CriticalAppraisal

20. CASPappraisalchecklist

21. DataSynthesis

22. Narrativesynthesis

23. Narrativesynthesis
• How
• Tools
•

24. Meta-analysis/forestPlot

25. Referencemanagement

26. Versioncontrol

27. Thewriteup
• Title
• Introduction
• Method
• Results
• Discussion
• Conclusion
• References

28. Introduction
• Introduce the
fi
eld of research
• State the question to be addressed
• Make reference to the previous reviews of the
fi
eld
• Make clear why further review is need (i.e. number of new papers)
• Introduce any competing models/theories that the review aims to appraise

29. Method:design&procedures
Describe:
• Search criteria and method
• Selection criteria, including explicit justi
fi
cation for study exclusion
• The scales or checklist by which studies are appraised
• Any techniques of synthesis or analysis by which they are presented

30. Method:design&procedures
Tabulate:
• inclusion criteria
• sample size
• baseline cases characteristics
• withdrawal rate
• results of primary and secondary end points (for longitudinal studies)

31. Method
• De
fi
ne participants
• De
fi
ne interventions
• De
fi
ne comparisons
• De
fi
ne outcomes

32. Results
• Normally the largest section of the write up
• Aim for a systematic grouping of the studies

33. Discussion&Conclusions
• Summaries the main results
• Appraise the results (power, generasibility)
• Field signi
fi
cance
• Appraise in relation to the models/theories of phenomenon
• Discuss limitations and biases
• Give guidance to the future research
• Conclude with a succinct summary of main points

34. References
• Fink, A. (1998). Conducting literature research reviews: from paper to the internet.
Thousand Oaks, CA: Sage
• Neiva, F. W., David, J. M. N., Braga, R., & Campos, F. (2016). Towards pragmatic
interoperability to support collaboration: A systematic review and mapping of the
literature. Information and Software Technology, 72, 137-150.
• Kitchenham, B. A., Budgen, D., & Brereton, O. P. (2011). Using mapping studies as the
basis for further research–a participant-observer case study. Information and
Software Technology, 53(6), 638-651.
• Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele
University, 33(2004), 1-26.

35. References
• B.A. Kitchenham, S. Charters, Guidelines For Performing Systematic Litera ture Reviews in
Software Engineering, EBSE Technical Report EBSE-2007-01, 2007.URL http://www.rbsv.eu/
courses/rmtw/mtrl/SLR.pdf.
• Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide.
John Wiley & Sons.
• Cruz-Benito, J., Therón, R., & García-Peñalvo, F. J. (2016, July). Software architectures supporting
human-computer interaction analysis: A literature review. In International Conference on
Learning and Collaboration Technologies (pp. 125-136). Springer International Publishing.
• Neiva, F. W., David, J. M. N., Braga, R., & Campos, F. (2016). Towards pragmatic interoperability
to support collaboration: A systematic review and mapping of the literature. Information and
Software Technology, 72, 137-150.

36. References
• Radant, O., Colomo-Palacios, R., & Stantchev, V. (2014). Analysis of Reasons, Implications and
Consequences of Demographic Change for IT Departments in Times of Scarcity of Talent: A
Systematic Review. International Journal of Knowledge Management (IJKM), 10(4), 1-15.
• Chilton, R., Pearson, M., & Anderson, R. (2015). Health promotion in schools: a scoping review
of systematic reviews. Health Education, 115(3/4), 357-376.
• Sepúlveda, S., Cravero, A., & Cachero, C. (2016). Requirements modeling languages for
software product lines: A systematic literature review. Information and Software Technology,
69, 16-36.
• Chei-Chang Chiou (2009) E
ff
ects of concept mapping strategy on learning performance in
business and economics statistics. Teaching in Higher Education. Feb2009, Vol. 14 Issue 1,
p55-69.

37. References
• Performing systematic reviews in resource-limited settings, Sarah Young and Erin
Eldemere, May 19, 2018 MLA Annual Conference, Atlanta GA
• Kitchenham, B.A., Brereton, P., & Budgen, D. (2015). Evidence-Based Software
Engineering and Systematic Reviews (1st ed.). Chapman and Hall/CRC. https://doi.org/
10.1201/b19467
• Borrego, M., Foster, M.J. and Froyd, J.E. (2014), Systematic Literature Reviews in
Engineering Education and Other Developing Interdisciplinary Fields. J. Eng. Educ.,
103: 45-76. https://doi.org/10.1002/jee.20038
• Systematic Literature Review & Mapping, Juan Cruz-Benito, Education in the
Knowledge Society PhD programme. University of Salamanca 7/11/2016

38. References
• Pearce, Joshua M. (2018) "How to Perform a Literature Review with Free and Open Source
Software," Practical Assessment, Research, and Evaluation: Vol. 23 , Article 8. DOI: https://
doi.org/10.7275/jjhz-sz75, Available at: https://scholarworks.umass.edu/pare/vol23/iss1/8
• Claes Wohlin, Emilia Mendes, Katia Romero Felizardo, Marcos Kalinowski, Guidelines for
the search strategy to update systematic literature reviews in software engineering,
Information and Software Technology, Volume 127, 2020, 106366, ISSN 0950-5849, https://
doi.org/10.1016/j.infsof.2020.106366.
• Nauman bin Ali, Muhammad Usman, A critical appraisal tool for systematic literature
reviews in software engineering, Information and Software Technology, Volume 112, 2019,
Pages 48-50, ISSN 0950-5849, https://doi.org/10.1016/j.infsof.2019.04.006.

39. References
• Biolchini, J. et al. Systematic Review in Software Engineering. PESC (Programa de Engenhatia
de Sistemas e Computação), [s. l.], p. 1-31, 2005
• Barbara Kitchenham, Pearl Brereton, A systematic review of systematic review process research
in software engineering, Information and Software Technology, Volume 55, Issue 12, 2013,
Pages 2049-2075, ISSN 0950-5849, https://doi.org/10.1016/j.infsof.2013.07.010.
• Lame, G. (2019) ‘Systematic Literature Reviews: An Introduction’, in Proceedings of the 22nd
International Conference on Engineering Design (ICED19), Delft, The Netherlands, 5-8 August
2019. DOI:10.1017/ dsi.2019.169
• Emilia Mendes, Claes Wohlin, Katia Felizardo, Marcos Kalinowski, When to update systematic
literature reviews in software engineering, Journal of Systems and Software, Volume 167, 2020,
110607, ISSN 0164-1212, https://doi.org/10.1016/j.jss.2020.110607.

40. References
• Page, M.J., McKenzie, J.E., Bossuyt, P.M. et al. The PRISMA 2020 statement: an updated
guideline for reporting systematic reviews. Syst Rev 10, 89 (2021). https://doi.org/10.1186/
s13643-021-01626-4
• P. V. Torres-Carrión, C. S. González-González, S. Aciar and G. Rodríguez-Morales,
"Methodology for systematic literature review applied to engineering and education," 2018
IEEE Global Engineering Education Conference (EDUCON), 2018, pp. 1364-1373, doi:
10.1109/EDUCON.2018.8363388.
• Lanxin Yang, He Zhang, Haifeng Shen, Xin Huang, Xin Zhou, Guoping Rong, Dong Shao,
Quality Assessment in Systematic Literature Reviews: A Software Engineering Perspective,
Information and Software Technology, Volume 130, 2021, 106397, ISSN 0950-5849, https://
doi.org/10.1016/j.infsof.2020.106397.

41. References
• Types of Literature Reviews. (n.d.). Purdue Libraries and School of Information
Studies. https://guides.lib.purdue.edu/c.php?g=867328&p=6230058
•