1 - Systematic Literature Reviews: introduction and methods

1. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 1 / 78 1 - Systematic literature reviews: introduction and methods Vittorio Scarano Dipartimento di Informatica, Universit`a di Salerno (Italy) vitsca@unisa.it PhD in Computer Science, University of Salerno (Italy) January 14th, 2020

2. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 2 / 78 LESSON PLAN 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments

3. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 4 / 78 MAP 1 Preamble to the two seminars 2 Why Systematic Literature Reviews? 3 Systematic Literature Reviews 4 The entire process 5 Phase 1 - Planning 6 Phase 2 - Conducting 7 Phase 3 - Reporting 8 An example 9 Conclusions and comments

4. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 5 / 78 A CIRCLE THAT CONTAINS HUMAN KNOWLEDGE. . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

5. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 6 / 78 BY END OF ELEMENTARY SCHOOL . . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

6. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 7 / 78 BY END OF HIGH SCHOOL From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

7. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 8 / 78 A BACHELOR: A SPECIALTY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

8. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 9 / 78 A MASTER DEEPENS THE SPECIALTY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

9. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 10 / 78 READING PAPER TAKES TO THE EDGE From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

10. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 11 / 78 THEN, YOU FOCUS . . . From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

11. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 12 / 78 . . . PUSHING THE BOUNDARY From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

12. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 13 / 78 . . . UNTIL THE BOUNDARY GIVES AWAY! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

13. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 14 / 78 HOW THE WORLD LOOKS NOW! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

14. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 15 / 78 BUT, PLEASE, DON’T FORGET THE BIGGER PICTURE! From “The illustrated guide to a Ph.D.” by Matt Might, CS Dept. of Univ. of Utah

15. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 16 / 78 THE MOTIVATIONS TO THESE 2 SEMINARS • Doing literature review is not optional (obviously! ) • It must follow some guidelines, rules (a process). . . • . . . and use advanced tools and techniques • Objectives: completeness, soundness, usefulness, reusability

16. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 17 / 78 THE ORGANIZATION OF THE SEMINARS • Two parts • First seminar on Systematic Literature Review: • principles • the process • some examples • Second seminar on Tools and Instruments to help on bibliographic search and SLRs

18. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 20 / 78 FIRST, WHAT IS A SYSTEMATIC LITERATURE REVIEW Deﬁnition of Systematic Literature Review (SLR) A Systematic Literature Review is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest. • Original guidelines proposed by Barbara Kitchenham (2004) and, then from B.Kitchenham et. al (2004) for “Evidence-based Software Engineering”. • An evolution of experiences and process used in biomedical and healthcare context • Further evolved and used in several contexts (among them, whole CS, but also Social Science, Business and Economics)

19. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 21 / 78 LITERATURE REVIEW/SURVEYS • What we really love to ﬁnd: the most recent the better! • A study of all the available research work and results in a given subject • Possibly, with some critical approach to distinguish between well established results and still open issues • If it includes also ongoing work and future challenges, bingo! • Often called “secondary sources” as opposed to the primary, i.e. the original papers, where experiments/proofs are reported. • Producing a literature review is part of graduate/post-graduate work: a chapter of your PhD thesis will be a Literature Review of the topics you are dealing with • ⇒ a foundation to build novelty upon!

20. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 22 / 78 REASONS FOR A SYSTEMATIC LITERATURE REVIEW • A standard path: from informal reviews and survey, moving on systematic searches of the literature • Important to summarize evidence concerning a methodology/technology • Helpful to identify gaps in current research (hint on further investigation needed!) • Crucial to provide a framework/background to position new research activities

21. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 23 / 78 NOT ONLY FOR PHD STUDENTS DEVOTED TO ACADEMIA! • The ability to study and gain knowledge on current, state-of-the-art, technologies and methodologies is crucial for ICT • Not only in academia, a long path of continuous knowledge update is waiting you out there! • “New topics in CS are going to pop up like popcorn in the pan” (A.Omicini) • Crucial to be able to ensure that 1 you move quickly to the result (well documented process) 2 nothing signiﬁcant was left out 3 you can easily share knowledge within the organisation

23. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 26 / 78 THE BIRTH OF SYSTEMATIC LITERATURE REVIEW • It is first developed in the healthcare domain • The need for a well-founded methodological approach to literature results emerges in medical research in the 90s • It gets popular in the 2000, and then formalised in terms of the so called “Cochrane Reviews” • Barbara Kitchenham, first, proposes it as a tool for the “Evidence-based Software Engineering” in 2004 (with a TR) and later adopted by SE and, by generalization, to many subfields of Computer Science

24. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 27 / 78 IMPORTANCE OF A SYSTEMATIC LITERATURE REVIEW • Literature review must be thorough • Literature review must be fair • A predeﬁned search strategy (protocol) help the assessment of the completeness • Every effort to identify research that does not support their preferred search hypothesis

25. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 28 / 78 PLUS AND MINUS OF SYSTEMATIC LITERATURE REVIEW • Requires considerably more effort • Information about effects of a phenomenon across a wide range of methods • The phenomenon can be assessed as robust and transferable • Combines qualitative with quantitative methods (on metadata) • Agreed-upon, sound (⇒ publishable) result

26. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 29 / 78 FEATURES OF A SLR • Definition of a review protocol that specifies the research question being addressed and the methods that will be used to perform the review. • Definition of a documented, search strategy that aims to detect as much of the relevant literature as possible. • Documentation is critical to assess its rigour and completeness. • It require explicit inclusion and exclusion criteria to assess each potential primary study. • Definition of the information to be analyzed from each primary study, including quality criteria • A systematic review is a prerequisite for quantitative meta-analysis

28. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 32 / 78 THE REVIEW PROCESS • A systematic review involves several discrete activities. • The process is meant as a standardisation of several existing guidelines for systematic reviews • The advantage of a standard, well-deﬁned, process to be followed • Three main phases (subdivided in stages): 1 Planning the Review, 2 Conducting the Review, 3 Reporting the Review. • Although apparently sequential, many of the stages involve iteration. • Activities initiated in the Planning (i.e., the protocol development stage) can be later reﬁned when the review takes place.

29. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 33 / 78 THE STAGES • The stages associated with Planning the review are: 1 Identiﬁcation of the need for a review 2 Development of a review protocol. • The stages associated with Conducting the review are: 1 Identiﬁcation of research 2 Selection of primary studies 3 Study quality assessment 4 Data extraction & monitoring 5 Data synthesis. • Reporting the review is a single stage activity

31. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 36 / 78 THE STAGES OF THE PLANNING PHASE • The stages associated with Planning the review are: 1 Identiﬁcation of the need for a review 2 Development of a review protocol.

32. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 37 / 78 1.1 THE NEED FOR A SYSTEMATIC REVIEW • It arises from the requirement of researchers to summarise all existing information about some phenomenon in a thorough and unbiased manner. • The purpose can be to draw more general conclusion about some phenomenon than is possible from individual studies, • Act as a prelude to further research activities • Researchers should ensure that a systematic review is necessary. • identify and review any existing systematic reviews of the phenomenon of interest against appropriate evaluation criteria.

33. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 38 / 78 HOW TO CHECK EXISTING REVIEWS • What are the review’s objectives? • What sources were searched to identify primary studies? Were there any restrictions? Missing sources? • What were the inclusion/exclusion criteria and how were they applied? • What criteria were used to assess the quality of primary studies and how were they applied? • How were the data extracted from the primary studies? • How were the data synthesised? How were differences between studies investigated? How were the data combined? Was it reasonable to combine the studies? Do the conclusions ﬂow from the evidence?

34. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 39 / 78 1.2 THE DEVELOPMENT OF A REVIEW PROTOCOL • A review protocol specifies the methods that will be used to undertake a specific systematic review. • A pre-defined protocol is necessary to reduce the possibility researcher bias: fairness. • Without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations. • Critical point: in some disciplines (i.e. medicine) review protocols are usually submitted to peer review.

35. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 40 / 78 COMPONENTS OF A PROTOCOL • Background: rationale for the survey. • Research questions that the review is intended answer. • The search strategy for primary studies: search terms and resources to be searched • Study selection (include/exclude) criteria and procedures. • Study quality assessment checklists and procedures. • Data extraction strategy to deﬁne how the information required from each primary study would be obtained • in case of required manipulation, the protocol should make it explicit • Synthesis of the extracted data • Project timetable

36. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 41 / 78 RESEARCH QUESTIONS • A specific problem (P) is tackled using some specific constraints, methods and/or approaches (C) to develop a system, application or algorithm (S). • Goal: what existing solutions are available, how they compare, what the strength of the evidence is and what implications these solutions have. • RQ1: What are the existing solutions to P? • RQ2: How does the different solutions found by addressing RQ1 compare to each other with respect to C? • RQ3: What is the strength of the evidence in support of the different solutions? • RQ4: What implications will these findings have when creating S?

38. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 44 / 78 THE STAGES OF THE CONDUCTING PHASE • The stages associated with Conducting the review are: 1 2.1 Identiﬁcation of research 2 2.2 Selection of primary studies 3 2.3 Study quality assessment 4 2.4 Data extraction & monitoring 5 2.5 Data synthesis.

39. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 45 / 78 2.1 IDENTIFICATION OF RESEARCH • The aim of a systematic review is to ﬁnd as many primary studies relating to the research question as possible using an unbiased search strategy. • Determine and follow a search strategy (in collaboration with librarian) • 1. Preliminary searches aimed at both identifying existing systematic reviews and assessing the volume of potentially relevant studies. • 2. Trial searchers using various combinations of search terms derived from the research question (from many sources) • 3. Reviews of research results • 4. Consultations with experts in the ﬁeld

40. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 46 / 78 SEARCH TERMS • Break down the question into individual facets • Draw up a list of synonyms, abbreviations, and alternative spellings. • Pick other terms can be obtained by considering subject headings used in journals and data bases. • Sophisticated search strings can then be constructed using Boolean AND’s and OR’s.

41. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 47 / 78 INFORMATION SOURCES • ACM digital library, • IEEE Xplore, • ISI web of knowledge, • ScienceDirect, • CiteSeer, • SpringerLink • DBLP • Wiley Inter Science • . . . • For domain speciﬁc sources, experts in the domain will normally know which are the best.

42. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 48 / 78 SOME ADDITIONAL FEATURES OF THIS STAGE • Avoid publication bias: • positive results are more likely to be published than negative results • Recent results: • scan repository of unpublished-preprint results (arXiv, Google Scholar, . . . ) • Document the search (also with the search query): • it must be possible to assess the thoroughness of the search • Useful to be helped with bibliographic management tools

43. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 49 / 78 2.2 STUDY SELECTION • Choose the study selection criteria to identify those primary studies that provide direct evidence about the research questions. • Som generally used criteria for removal: duplicates, same study in different sources (conference ⇒ journal), after/before a certain date • Avoid language bias • Avoid “clique” bias (well-known institutions, labs, researchers)

44. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 50 / 78 2.3 STUDY QUALITY ASSESSMENT • Choose the study selection criteria to identify those primary studies that provide direct evidence about the research questions. • Som generally used criteria for removal: duplicates, same study in different sources (conference ⇒ journal), after/before a certain date • The protocol should deﬁne exactly which inclusion and quality criteria are employed • Avoid language bias • Avoid “clique” bias (well-known institutions, labs, researchers) • Example of inclusion criteria: “The study’s main concern is P”, “The study focuses on C”,“ The study describes an S”

45. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 51 / 78 THREE-STEPS SCREENING • Title and Abstract: apply the inclusion criteria • Introduction and conclusions: apply the inclusion criteria • Full text: apply the inclusion criteria and the quality criteria

46. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 52 / 78 QUALITY CRITERIA • They are designed to answer “What is the strength of the evidence in support of the different solutions?” • Examples are: • QC 1: Is there is a clear statement of the aim of the research? • QC 2: Is the study is put into context of other studies and research? • QC 3: Are system or algorithmic design decisions justified? • QC 4: Is the test data set reproducible? • QC 5: Is the study algorithm reproducible? • QC 6: Is the experimental procedure throughly explained and reproducible? • QC 7: Is it clearly stated in the study which other algorithms the study’s algorithm(s) have been compared with? • QC 8: Are the performance metrics used in the study explained and justified? • QC 9: Are the test results thoroughly analysed? • QC 10: Does the test evidence support the findings presented? • Example: score each QC in 0, 1/2, 1 and add them up for global rank

47. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 53 / 78 2.4 DATA EXTRACTION • Objective: to design data extraction forms to accurately record the information researchers obtain from the primary studies • It will deﬁne a set of numerical values that should be extracted for each study (e.g. number of subjects, performance parameters, . . . ) • To ensure consistency among multiple researcher, some papers should be reviewed by all researchers so that inter-researcher consistency can be assessed.

48. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 54 / 78 2.5 DATA SYNTHESIS • Data synthesis involves collating and summarising the results of the included primary studies. • Synthesis can be descriptive (non-quantitative) • It is sometimes possible to complement a descriptive synthesis with a quantitative summary • It is important to identify whether results from studies are consistent one with another (i.e. homogeneous) or inconsistent (e.g. heterogeneous) in a tabulated/graphical form

50. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 57 / 78 STRUCTURE OF A REPORT OF SLR - 1 • Title • The title should be short but informative. It should be based on the question being asked. In journal papers, it should indicate that the study is a systematic review. • Abstract • Context/Background: the importance of the research questions addressed • Objectives: The questions addressed • Methods: Data Sources, Study selection, Quality Assessment and Data extraction • Results: Main ﬁnding • Conclusions: Implications for practice and future research

51. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 58 / 78 STRUCTURE OF A REPORT OF SLR - 2 • Background: Justiﬁcation of the need for the review. Summary of previous reviews • Review questions: Each review question should be speciﬁed • Review Methods: • Data sources and search strategy • Study selection • Study quality assessment • Data extraction • Data synthesis • Included and excluded studies • Results: Findings (description of primary studies and results of any quantitative summaries)

52. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 59 / 78 STRUCTURE OF A REPORT OF SLR - 3 • Discussion: • Principal findings • Strengths and Weaknesses • Meaning of findings (applicability/generalisability) • Conclusions: recommendations (practical implementations) and future research • Results: Findings (description of primary studies and results of any quantitative summaries) • Acknowledgments • Conflict of Interest • References • Appendices

54. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 62 / 78 A PAPER AS AN EXAMPLE • Artem Voronkov, Leonardo Horn Iwaya, Leonardo A. Martucci, and Stefan Lindskog. 2017. “Systematic Literature Review on Usability of Firewall Conﬁguration”. ACM Comput. Surv. 50, 6, Article 87 (December 2017) 35 pages. • A nice survey on Firewall usability • Published on one of the major sources for Surveys

55. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 63 / 78 THE ABSTRACT

56. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 64 / 78 THE ABSTRACT WITH ANNOTATIONS

57. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 65 / 78 SOURCES

58. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 66 / 78 RESEARCH QUESTIONS

59. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 67 / 78 INCLUSION CRITERIA

60. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 68 / 78 INDEX TERMS: TWO QUERIES

61. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 69 / 78 THE RESULTING QUERIES

62. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 70 / 78 QUALITY CRITERIA

64. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 73 / 78 COMPARISON OF SLR VS LITERATURE SURVEY • A SLR defines a review protocol that specifies the (1) research question being addressed and (2) the methods that will be used to perform the review • A SLR defines the search strategy to help readers to determine its thoroughness and fairness (and also repeatability, up to a certain extent) • searches on digital libraries cannot be replicated • A SLR specifies quality criteria to evaluate each primary study.

65. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 74 / 78 PERSONAL CONSIDERATIONS • Important to follow a process when documenting literature surveys • Secondary sources are important: citations and publication venues are available out there! • Useful tools and instruments can facilitate the effort • If not publishable, SLRs make, at least, a great chapter of a PhD Thesis: thoroughness, fairness and replicability

66. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 76 / 78 READING MATERIAL AND CREDITS • Material and references available at www.isislab.it under “Research” • Slides are available on Slideshare under Open licence Common Creative - Attribution 4.0 International (CC BY 4.0)

67. Preamble Why SLRs? SLRs The process 1 - Plan 2 - Conduct 3 - Report Example Conclusions 77 / 78 QUESTIONS? COMMENTS? ANYTHING?

1 - Systematic Literature Reviews: introduction and methods

You are reading a preview.

1 - Systematic Literature Reviews: introduction and methods

More Related Content

Similar to 1 - Systematic Literature Reviews: introduction and methods

More from Vittorio Scarano

Featured

Related Books

Related Audiobooks