Juan Cruz-Benito
GRIAL Research Group, Department of Computers and Automatics
University of Salamanca, Salamanca, Spain.
Education in the Knowledge Society PhD programme.
University of Salamanca 7/11/2016
How to conduct systematic literature reviewKashif Hussain
The slides show how to conduct systematic literature review (SLR) in any field of research. It is highly important that any SLR should ultimately highlight potential future directions and research gaps so that prospect researchers may focus on those particular areas.
How to write (and publish) a literature reviewMarcel Bogers
How to write (and publish) your literature review? This presentations distinguishes between three types and purposes of "review": (1) a literature review, as part of an empirical study; (2) a stand-alone review article; and (3) a conceptual or theoretical (non-empirical) article. For each of theses types, it gives an overview of considerations for getting done and published (or rejected).
Juan Cruz-Benito
GRIAL Research Group, Department of Computers and Automatics
University of Salamanca, Salamanca, Spain.
Education in the Knowledge Society PhD programme.
University of Salamanca 7/11/2016
How to conduct systematic literature reviewKashif Hussain
The slides show how to conduct systematic literature review (SLR) in any field of research. It is highly important that any SLR should ultimately highlight potential future directions and research gaps so that prospect researchers may focus on those particular areas.
How to write (and publish) a literature reviewMarcel Bogers
How to write (and publish) your literature review? This presentations distinguishes between three types and purposes of "review": (1) a literature review, as part of an empirical study; (2) a stand-alone review article; and (3) a conceptual or theoretical (non-empirical) article. For each of theses types, it gives an overview of considerations for getting done and published (or rejected).
This workshop is meant to be an introduction to the systematic review process. Further information about systematic reviews was available through a research guide. http://libguides.ucalgary.ca/content.php?pid=593664
A firm grasp of scientific method and ability to write clearly and convincingly is a great assert to any professional in sciences.
Conducting research and publishing peer reviewed papers train professionals in both scientific method and writing. Moreover, having research papers in your resume is considered a huge plus in both industry and academia. However, conducting research and getting them published requires professionals to approach the problem and present their solutions form a unique angle. The talk will address research in general and writing research papers. Specifically, the talk will cover peer review process, what is a contribution?, and basic composition of
a research paper, describing potential pitfalls.
A literature review is a critical summary of all the published works on a particular topic. Most research papers include a section on literature review as part of the introduction. However, a literature review can also be published as a standalone article. These slides will help you grasp the basics of writing a literature review.
An introduction to conducting a systematic literature review for social scien...rosie.dunne
An introduction to conducting a systematic literature review for social scientists and health researchers presented by Luke van Rhoon Health Behaviour Change Research Group, School of Psychology, NUI Galway November 2020
Presenting a published paper:
"Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review
approach"
The literature review is not merely a simple compilation or a list of every item and resource with any possible relation to your topic. A good literature review is a critical appraisal of narrowly focused, selected and truly relevant work that provides the current status (perspective) of the topic. This presentation basically is a brief guide on the process of doing and writing a literature review for a thesis, research proposal, research paper, etc.
Bibliometric visualization using VOSviewerLudo Waltman
Presentation at the workshop Research Output & Impact – New Tools and Concepts, organized at Technical University Denmark. Lyngby, Denmark, September 14, 2017.
This workshop is meant to be an introduction to the systematic review process. Further information about systematic reviews was available through a research guide. http://libguides.ucalgary.ca/content.php?pid=593664
A firm grasp of scientific method and ability to write clearly and convincingly is a great assert to any professional in sciences.
Conducting research and publishing peer reviewed papers train professionals in both scientific method and writing. Moreover, having research papers in your resume is considered a huge plus in both industry and academia. However, conducting research and getting them published requires professionals to approach the problem and present their solutions form a unique angle. The talk will address research in general and writing research papers. Specifically, the talk will cover peer review process, what is a contribution?, and basic composition of
a research paper, describing potential pitfalls.
A literature review is a critical summary of all the published works on a particular topic. Most research papers include a section on literature review as part of the introduction. However, a literature review can also be published as a standalone article. These slides will help you grasp the basics of writing a literature review.
An introduction to conducting a systematic literature review for social scien...rosie.dunne
An introduction to conducting a systematic literature review for social scientists and health researchers presented by Luke van Rhoon Health Behaviour Change Research Group, School of Psychology, NUI Galway November 2020
Presenting a published paper:
"Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review
approach"
The literature review is not merely a simple compilation or a list of every item and resource with any possible relation to your topic. A good literature review is a critical appraisal of narrowly focused, selected and truly relevant work that provides the current status (perspective) of the topic. This presentation basically is a brief guide on the process of doing and writing a literature review for a thesis, research proposal, research paper, etc.
Bibliometric visualization using VOSviewerLudo Waltman
Presentation at the workshop Research Output & Impact – New Tools and Concepts, organized at Technical University Denmark. Lyngby, Denmark, September 14, 2017.
Framework for Program Development and EvaluationReference.docxhanneloremccaffery
Framework for Program Development and Evaluation
Reference: Comeau, J. (2011). Framework for program development and evaluation.Unpublished, Capella University, Minneapolis, MN.
L i c e n s e d u n d e r a C r e a t i v e C o m m o n s A t t r i b u t i o n 3 . 0 L i c e n s e .
1. Understand and analyze qualitative program evaluation design.
2. Compare and contrast experimental and quasi-experimental designs.
3. Analyze pretest-posttest designs.
4. Communicate through writing that is concise, balanced, and logically organized.
Unit 3 - Program Evaluation: Qualitative Research Design
INTRODUCTION
This unit focuses on qualitative evaluation design, data collection methods, and evaluating program
effectiveness. Additionally, you will apply this knowledge to a real-world program evaluation.
OBJECTIVES
To successfully complete this learning unit, you will be expected to:
U03S1] Studies - Multimedia and Readings (Complete the following):
• Framework for Program Development and Evaluation view the flow chart/transcript
• Writing an Action Research Dissertation: Part One view the media/transcript
• Writing an Action Research Dissertation: Part Two view the media/transcript
The Writing an Action Research Dissertation media pieces will help you to understand the
academic writing standards for your doctoral program. You are expected to be proficient in this
type of writing by the end of your program. By using the advice and guidance of the media, you can
refine your academic writing and improve your success in this course and throughout your
program.
• Read Chapter 5 - Program Evaluation and Performance Measurement text
o Pay attention to question 7 on page 221. The content this question addresses will be
releant for the first discussion in this unit.
• Read Moore and Tananis's 2009 article, "Measuring Change in a Short-Term
Educational Program Using a Retrospective Pretest Design," from American Journal of
Evaluation, volume 30, issue 2, pages 189–202.
o Pay attention to the research design and data collection methods in this study. You
will be analyzing them for two upcoming assignments, one in this unit and the
other in Unit 5.
Constance
Highlight
Constance
Highlight
Constance
Highlight
Constance
Highlight
[U03A1] Unit 3 Assignment 1 - Program Evaluation: Analysis of Study Design
Using what you have learned through the readings and discussions up to this point in the course, read and analyze the 2009
journal article "Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design" by Moore
and Tananis. After you have finished your reading of the article, formalize your analysis by addressing the following:
• Identify the research design that was employed in the Moore and Tananis study.
• Explain whether the research design is experimental or quasi-experimental. Support your explanation by
comparing and contrasting characteristics between the two types of designs.
◦ Make sure ...
Classification of Researcher's Collaboration Patterns Towards Research Perfor...Nur Hazimah Khalid
A VIVA presentation slide for Master of Computer Science on 24th May 2016 at Faculty of Computing, Universiti Teknologi Malaysia by Nur Hazimah Khalid. Thank you.
Learning Analytics Dashboards for Advisors – A Systematic Literature ReviewIJCI JOURNAL
Learning Analytics Dashboard for Advisors is designed to provide data-driven insights and visualizations to support advisors in their decision-making regarding student academic progress, engagement, targeted support, and overall success. This study explores the current state of the art in learning analytics dashboards, focusing on specific requirements for advisors. By examining existing literature and case studies, this research investigates the key features and functionalities essential for an effective learning analytics dashboard tailored to advisor needs. This study also aims to provide a comprehensive understanding of the landscape of learning analytics dashboards for advisors, offering insights into the advancements, opportunities, and challenges in their development by synthesizing the current trends from a total of 21 research papers used for analysis. The findings will contribute to the design and implementation of new features in learning analytics dashboards that empower advisors to provide proactive and individualized support, ultimately fostering student retention and academic success.
Systematic review international conference slidesvijay kumar
This PowerPoint is about systematic review. The talk was delivered at an International Conference. All videos related to research conferences can be viewed at
https://www.youtube.com/channel/UCNEUKBUIaQG3wr05Sj38oDA/featured
Systematic review article and Meta-analysis: Main steps for Successful writin...Pubrica
A review article is a piece of writing that gives a complete and systematic summary of results available in a certain field while also allowing the reader to perceive the subject from a different viewpoint.
Continue Reading: https://bit.ly/3m7OTqC
For our services: https://pubrica.com/services/research-services/systematic-review/
Why Pubrica:
When you order our services, We promise you the following – Plagiarism free | always on Time | 24*7 customer support | Written to international Standard | Unlimited Revisions support | Medical writing Expert | Publication Support | Biostatistical experts | High-quality Subject Matter Experts.
Contact us:
Web: https://pubrica.com/
Blog: https://pubrica.com/academy/
Email: sales@pubrica.com
WhatsApp : +91 9884350006
United Kingdom: +44-1618186353
A task-based scientific paper recommender system for literature review and ma...Aravind Sesagiri Raamkumar
My PhD oral defense presentation (as of Oct 3rd 2017)
The dissertation can be requested at this link https://www.researchgate.net/publication/323308750_A_task-based_scientific_paper_recommender_system_for_literature_review_and_manuscript_preparation
Systematic Literature Reviews and Systematic Mapping Studiesalessio_ferrari
Lecture slides on Systematic Literature Reviews and Systematic Mapping Studies in software engineering. It describes the different steps, discusses differences between the two methods, and gives guidelines on how to conduct these types of study.
Bridging The Research-Practice Gap Through Evidence-Based Management And Systematic Review.
David Denyer and Rob Briner
Academy of Management Annual Meeting 2014, Philadelphia
Presentation made on December 7th 2016 during ICADL'16
Full text can be found at http://link.springer.com/chapter/10.1007/978-3-319-49304-6_12
Extended version can be found at https://arxiv.org/abs/1609.01415
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
This presentation includes basic of PCOS their pathology and treatment and also Ayurveda correlation of PCOS and Ayurvedic line of treatment mentioned in classics.
A workshop hosted by the South African Journal of Science aimed at postgraduate students and early career researchers with little or no experience in writing and publishing journal articles.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
A review of the growth of the Israel Genealogy Research Association Database Collection for the last 12 months. Our collection is now passed the 3 million mark and still growing. See which archives have contributed the most. See the different types of records we have, and which years have had records added. You can also see what we have for the future.
Executive Directors Chat Leveraging AI for Diversity, Equity, and InclusionTechSoup
Let’s explore the intersection of technology and equity in the final session of our DEI series. Discover how AI tools, like ChatGPT, can be used to support and enhance your nonprofit's DEI initiatives. Participants will gain insights into practical AI applications and get tips for leveraging technology to advance their DEI goals.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Dr. Vinod Kumar Kanvaria
Exploiting Artificial Intelligence for Empowering Researchers and Faculty,
International FDP on Fundamentals of Research in Social Sciences
at Integral University, Lucknow, 06.06.2024
By Dr. Vinod Kumar Kanvaria
2. 6/19/2014
2
What is a literature review (LR)?
• LR is a critical and in depth evaluation of previous research
(Shuttleworth, 2009).
• A summary and synopsis of a particular area of research,
allowing anybody reading the paper to establish the
reasons for pursuing a particular research.
• “An information analysis and synthesis, focusing on
findings and not simply bibliographic citations,
summarizing the substance of a literature and drawing
conclusions from it” (Educational Resources Information
Center, 1982)
• A good LR evaluates quality & findings of previous
research.
3
Why doing LR?
• To establish connection or relationship between
“existing knowledge” and “the problem to be
solved”.
• To refine the problem
• To identify significance of research
• To define research question(s)
4
3. 6/19/2014
3
Types of Literature Review
• Traditional Review (narrative)
• Systematic Literature Review or Systematic
Review
• Systematic Mapping Study (Scoping Study)
• Meta-Analysis
• Tertiary Study
5
“Traditional” literature review
• Provides an overview of the research findings
on particular topics.
• Pros: produce insightful, valid syntheses of the
research literature if conducted by the expert .
• Cons: vulnerable to unintentional and
intentional bias in the selection, interpretation
and organization of content.
6
4. 6/19/2014
4
Systematic Literature Review (SLR)
What is a SLR?
◦ A process of identifying, assessing, and interpreting all
available research evidence, to provide answers for a
particular research question.
◦ A form of secondary study that uses a well-defined
methodology (Kitchenham et al., 2007).
SLRs are well established in other disciplines, particularly
medicine. They integrate an individual clinical expertise and
facilitate access to the outcomes of the research (Budgen et
al, 2006).
7
“Traditional” Review Vs SLR
“the purpose of a SLR is to provide as complete a list as
possible of all the published and unpublished studies
relating to a particular subject area. While traditional
reviews attempt to summarize results of a number of
studies, systematic reviews use explicit and rigorous
criteria to identify, critically evaluate and synthesize
all the literature on a particular topic”.
(Cronin et al., 2008)
8
5. 6/19/2014
5
Systematic Mapping study
• Suitable for a very broad topic
• Identify clusters of evidence (making
classification)
• Direct the focus of future SLRs
• To identify areas for future primary studies
9
Tertiary study
• Is a SLR of SLRs
• To answer a more wider question
• Uses the same method as in SLR
• Potentially less resource intensive
6. 6/19/2014
6
Hierarchy of Review
11
Primary study#2
Secondary study
#2
Tertiary study
e.g. SLR of pair
programming studies in
academic context
e.g. a SLR of SLR in SE
Primary study#1 Primary study#3
Secondary study
#1
Motivations for using SLR
To systematically accumulate, organize, evaluate, and
synthesize all existing research evidence of your research area.
To present fair evaluation of a research topic by using a
trustworthy, rigorous, and auditable methodology.
To produce reliable and unbiased results.
To identify gaps in the existing research that will lead to topics
for further investigation.
To provide as a background to position new research activities.
To support Evidence-based research.
12
7. 6/19/2014
7
Steps in conducting SLR
13
Planning
Reporting
Conducting
• Formulate the review’s research question
• Develop the review’s protocol
• Search the relevant literature
• Perform selection of primary studies
• Perform data extraction
• Assess studies’ quality
• Conduct synthesis of evidence
Write up the SLR report/paper
Guidelines for SLR
• Kitchenham, B. & Charters, S. (2007). Procedures for
performing systematic literature reviews in software
engineering. UK: Keele University and University of
Durham.
• Petticrew, M., & Roberts, H. (2006). Systematic Reviews
in the Social Sciences: A Practical Guide: Blackwell
Publishing.
• Systematic Review. CRD’s guidance for undertaking
reviews in health care (2008).
http://www.york.ac.uk/inst/crd/pdf/Systematic_Revie
ws.pdf
8. 6/19/2014
8
PLANNING PHASE
1) Formulation of RQ
2) Protocol development
The Research Question (RQ)
• Is the most important part in any SLR
• Is not necessarily the same as question(s)
addressed in your research.
• Is used to guide the search process
• Is used to guide the extraction process
• Data analysis (synthesis of evidence) is expected
to answer your SLR’s RQ.
16
9. 6/19/2014
9
Formulation of RQ
• Features of good question:
– The RQ is meaningful and important to practitioners and
researchers.
– The RQ will lead to changes in current practice or to
increase confidence in the value of current practice
– The RQ will identify discrepancies between commonly
held beliefs and the reality.
• RQ can be derived primarily based on researcher’s
interest. E.g. an SLR for PhD thesis should identify
existing basis for the research work and where it
fits in the current Body of knowledge.
17
Formulation of RQ
Petticrew & Robert (2006) suggest that the formulation of RQs
about effectiveness of a treatment should focus on 5 elements
known as PICOC:
Population (P) - the target group for the investigation (e.g.
people, software etc.).
Intervention (I) - specifies the investigation aspects or issues of
interest to the researcher(s).
Comparison (C)– aspect of the investigation with which the
intervention is being compared to.
Outcomes (O)– the effect of the intervention
Context (C)– the setting or environment of the investigation
18
10. 6/19/2014
10
Example of PICOC (Kitchenham et al.,
2007)
19
Population: Software or web project
Intervention: Cross-company project effort estimation model
Comparison: Single-company project effort estimation model
Outcomes: Prediction or estimate accuracy
Context: None
Title: “Cross verses Within-Company Estimation Cost
Estimation Studies: A Systematic Review”
Example of PICOC (Salleh et al., 2011)
Population: CS/SE students in higher education
Intervention: Pair programming
Comparison: N/A
Outcomes: PP’s effectiveness
Context: Review(s) of all empirical studies of PP within the domain of
CS/SE in higher education.
Title: “Empirical Studies of Pair Programming for
CS/SE Teaching in Higher Education: A Systematic
Literature Review”
11. 6/19/2014
11
Example of RQs
Kitchenham et al (2007):
Q1: What evidence is there that cross-company estimation
models are not significantly different from within-company
estimation models for predicting effort for software/Web
projects?
Q2: What characteristics of the study data sets and the data
analysis methods used in the study affect the outcome of within-
and cross-company effort estimation accuracy studies?
Q3: Which experimental procedure is most appropriate for
studies comparing within- and cross-company estimation
models?
21
Example of RQs (cont.)
Major et al. (2011) Systematic Literature Review: Teaching Novices
Programming Using Robots
RQ1: What computer languages are being taught in introductory
programming courses that make use of robots as teaching tools?
RQ2: Are the robots that are being used simulated or physical (real-life)?
RQ3: What are the characteristics (i.e. what is the age, level of education
etc.) of the novices being taught?
RQ4: What types of studies are being performed by researchers that
investigate the teaching of introductory programming concepts using robots?
RQ5: What is the scale (e.g. number of participants) of studies that are being
performed by researchers?
RQ6: Do collected studies suggest that using robotics to teach introductory
programming is effective? 22
12. 6/19/2014
12
Examples of RQ (cont.)
Salleh et al. (2011)
Primary RQ:
What evidence is there of PP studies conducted in higher education
settings that investigated PP’s effectiveness and/or pair compatibility in
CS/SE education?
Sub-RQs:
Q1: What evidence is there regarding pair compatibility factors that affect
the effectiveness of PP as a CS/SE pedagogical tool and which pairing
configurations are considered as most effective?
Q2: How was PP’s effectiveness measured in PP studies and how effective
has PP been when used within higher education settings?
Q3: How was quality measured in the studies that used software quality as
a measure of PP’s effectiveness?
23
Examples of RQ (cont.)
Davis et al. (2006)
• Primary RQ: “What elicitation technique is most
efficient in a particular setting?”
24
13. 6/19/2014
13
SLR Protocol
• A plan that specifies the basic review
procedures
• Components of a protocol:
– Background, RQ, search terms, selection criteria,
quality checklist and procedures, data extraction
strategy, data synthesis strategy, project schedule.
• See example of SLR protocol.
25
CONDUCTING REVIEW
1) Search relevant studies
2) Select relevant studies
3) Assess studies’ quality
4) Perform data extraction
5) Perform synthesis of evidence
14. 6/19/2014
14
1) Searching Relevant Studies
• Involves a comprehensive and exhaustive searching of studies
to be included in the review.
• Define a search strategy
• Consult the subject librarian(s)
• Search strategies are usually iterative and benefit from:
– Preliminary searches (to identify existing review and
volume of studies)
– Trial searches (combination of terms from RQ)
– Check the search results against list of known studies
– Consult the experts in the field
27
1) Searching Relevant Studies (Zhang et
al. (2011)
1. Which approach to be used in search process
(e.g., manual or automated search)?
2. Where (venues or databases) to search, and
which part of article (field) should be searched?
3. What (subject, evidence type) to be searched,
and what are queries (search strings) fed into
search engines?
4. When is the search carried out, and what time
span to be searched?
15. 6/19/2014
15
Common approach to construct search string
Derive major terms used in the review questions
based on the PICOC.
List the keywords mentioned in the article.
Search for synonyms and alternative words
Use the boolean OR to incorporate alternative
synonyms.
Use the boolean AND to link major terms
29
E.g. Search String
Salleh et al. (2011)
• The complete search term initially used :
(student* OR undergraduate*) AND (pair programming OR pair-
programming) AND ((experiment* OR measurement OR evaluation OR
assessment) AND (effective* OR efficient OR successful)
• A very limited number of results retrieved when using the
complete string, thus a much simpler string was derived.
• Subject librarian suggested to revise the search string:
“pair programming” OR “pair-programming”
30
16. 6/19/2014
16
E.g. Search String
Kitchenham et al. (2007) used their structured questions to
construct search strings for use with electronic databases:
Population: software OR application OR product OR Web OR WWW
OR Internet OR World-Wide Web OR project OR development
Intervention: cross company OR cross organisation OR cross
organization OR multiple-organizational OR multiple-organisational
model OR modeling OR modelling effort OR cost OR resource
estimation OR prediction OR assessment
Contrast: within-organisation OR within-organization OR within-
organizational OR within-organisational OR single company OR single
organisation
Outcome: Accuracy OR Mean Magnitude Relative Error
The search strings were constructed by linking the four OR lists using
the Boolean AND. 31
E.g. Search String
• In Kitchenham et al. (2007), the search string produced
was extremely long
• They attempted to produce a search string that was
very specific to their RQ but ended up getting a large
number of false positives.
• “Simpler search string might have been just as
effective”.
32
17. 6/19/2014
17
Sources of Evidence
Digital libraries
Reference lists from relevant primary studies and review
articles
Journals (including company journals such as the IBM
Journal of Research and Development), grey literature (i.e.
technical reports, work in progress)
Conference proceedings
Research registers
The Internet (google)
Direct contact specific researcher(s)
33
E.g. Sources of Evidence
Salleh et al. (2011)
Online databases used: IEEExplore, ACM Digital Library, Current
Contents, EBSCOhost, ISI Web of Science, INSPEC, ProQuest, Sage Full
text Collection, ScienceDirect, SpringerLink, Scopus
Other search engines used: Google scholar, Citeseer, Agile Alliance.
Some databases were selected based on previous studies we were
aware of.
34
18. 6/19/2014
18
E.g. Sources of Evidence
Kitchenham et al. (2007)
The search strings were used on 6 digital libraries:
INSPEC , El Compendex, Science Direct, Web of Science, IEEExplore,
ACM Digital library
Search specific journals and conf. proceedings:
Empirical Software Engineering (J), Information and Software
Technology (J), Software Process Improvement and Practice (J),
Management Science (J), International Software Metrics Symposium
(C), International Conference on Software Engineering (C),
Manual search:
Evaluation and Assessment in Software Engineering (C)
Check references of each relevant article
Contact researcher(s)
35
Using Snowballing technique as a
complement to database searches
• Jalali & Wohlin (2012) recommend applying both backward
& forward snowballing in locating articles to be included in
the SLR.
• Snowballing approach requires a starting set of papers,
which should be based on identifying a set of papers from
leading journals in the area.
• Forward snowballing – i.e. identify articles that have
cited the articles found in the search
• Backward snowballing – i.e. identify articles from the
reference lists.
19. 6/19/2014
19
Snowballing technique
• References
Set of studies found from database search
Backward
Snowball
Forward
Snowball
Issue of publication bias
What is “publication bias”?
Refers to the problem that positive results are more likely to
publish than negative results (Kitchenham & Charter, 2007).
E.g. formal experiments that was failed to reject null
hypothesis are considered less interesting.
Publication bias can lead to systematic bias in SLR
Strategies to address publication bias:
Scanning grey literature
Scanning conference proceedings
Contacting experts working in the area
38
20. 6/19/2014
20
Managing Bibliography
• Use relevant Bibliographic package to manage
large number of references
• E.g. EndNote, Zotero, JabRef Reference
Manager etc.
39
Documenting the Search
• The process of conducting SLR must be transparent and replicable.
• The review should be documented in sufficient detail
• The search should be documented and changes noted.
• Unfiltered search results should be saved for possible reanalysis
Data Source Documentation
Digital Library Name of Database, Search strategy, Date of search, years
covered by search
Journal Hand Searches Name of journal, Years searched
Conference
proceedings
Title of proceedings/Name of conference, Journal name (if
published as part of a journal)
21. 6/19/2014
21
2) Selection of Studies
• Primary studies need to be assessed for their
actual relevance
• Set the criteria for including or excluding studies
(decided earlier during protocol development,
can be refined later)
• Inclusion & exclusion criteria should be based
on RQ
• Selection process should be piloted.
• Study selection is a multistage process.
41
E.g. Selection of Studies
Kitchenham et al. (2007) used the following inclusion criteria:
any study that compared predictions of cross-company
models with within-company models based on analysis of
single company project data.
Exclusion criteria:
studies where projects were only collected from a small
number of different sources (e.g. 2 or 3 companies),
studies where models derived from a within-company data set
were compared with predictions from a general cost
estimation model.
42
22. 6/19/2014
22
E.g. Selection of Studies
Salleh et al. (2011)
• Inclusion criteria:
– to include any empirical studies of PP that involved higher
education students as the population of interest.
• Exclusion criteria:
– Papers presenting unsubstantiated claims made by the
author(s), for which no evidence was available.
– Papers about Agile/XP describing development practices
other than PP, such as test-first programming, refactoring
etc.
– Papers that only described tools (software or hardware)
that could support the PP practice.
– Papers not written in English.
– Papers involving students but outside higher education.
43
3) Assessing studies’ quality
• Reasons:
– To provide more detailed Inclusion/Exclusion criteria
– To check whether quality differences provide an
explanation for differences in study results
– As a means of weighting the importance of
individual studies when results are being
synthesized.
– To guide the interpretation of findings and
determine the strength of inferences.
– To guide recommendations for further research.
44
23. 6/19/2014
23
Assessing studies’ quality
Quality relates to the extent to which the study minimizes bias
and maximizes internal and external validity (Khan et al. 2001) .
Quality Concepts Definition (Kitchenham & Charter, 2007)
45
Terms Synonyms Definition
Bias Systematic error tendency to produce results that depart systematically
from the ‘true’ results. Unbiased results are internally
valid
Internal
Validity
Validity The extent to which the design and conduct of the
study are likely to prevent systematic error. Internal
validity is a prerequisite for external validity
External
Validity
Generalisability,
Applicability
The extent to which the effects observed in the study
are applicable outside of the study
Assessing studies’ quality
Assessing quality of studies:
◦ Methodology or design of the study
◦ Analysis of studies’ findings.
Quality checklist or instrument need to be designed to
facilitate quality assessment.
Most quality checklists include questions aimed at
assessing the extent to which articles have addressed
bias and validity.
24. 6/19/2014
24
E.g. Study Quality Assessment - Salleh et al. (2011)
47
Item Answer
1. Was the article referred? [30] Yes/No
2. Were the aim(s) of the study clearly stated? [16], [67] Yes/No/Partially
3. Were the study participants or observational units adequately described? For
example, students’programming experience, year of study etc. [44], [68]
Yes/No/Partially
4. Were the data collections carried out very well? For example, discussion of
procedures used for collection, and how the study setting may have
influenced the data collected [44], [48], [67], [68]
Yes/No/Partially
5. Were potential confounders adequately controlled for in the analysis? [67] Yes/No/Partially
6. Were the approach to and formulation of the analysis well conveyed? For
example, description of the form of the original data, rationale for choice
of method/tool/package [48], [67], [68]
Yes/No/Partially
7. Were the findings credible? For example, the study was methodologically
explained so that we can trust the findings; findings/conclusions are
resonant with other knowledge and experience [48], [44], [68]
Yes/No/Partially
E.g. Study Quality Assessment
Kitchenham et al. (2007) constructed a quality questionnaire based
on 5 issues affecting the quality of the study :
1. Is the data analysis process appropriate?
2. Did studies carry out a sensitivity or residual analysis?
3. Were accuracy statistics based on the raw data scale?
4. How good was the study comparison method?
5. The size of the within-company data set (e.g < 10 projects
considered poor quality)
25. 6/19/2014
25
4) Data Extraction
Involve reading the full text article.
Data extracted from primary studies should be recorded using data
extraction form.
The form should be designed and piloted when the protocol is defined.
Collect all the information that can be used to answer the RQ and the
study’s quality criteria.
Both quality checklist and review data can be included in the same
form.
In case of duplicates publications (reporting the same data), refer the
most complete one.
For validation, a set of papers should be reviewed by 2 or more
researchers. Compare results and resolve any conflicts.
49
Extraction Template (Cruzes et al., 2007)
Publication
• Authors
• Year
• Title
• Source
• Abstract
• Topic
• Aims
Contexts
• Subjects
• Technologies
• Industry
• Settings
• Instruments
• Study Type
Findings
• Verbatim
text/data
• Origin
• Strength of
Evidence
1 * 1 *
26. 6/19/2014
26
5) Synthesis of evidence
Involves collating and summarizing the results of the
included primary studies.
Key objectives of data synthesis (Cruzes & Dyba, 2011):
– to analyze and evaluate multiple studies
– to select appropriate methods for integrating or
providing new interpretive explanations about them
• Types of Synthesis:
– Descriptive (narrative/qualitative, thematic analysis)
– Numerical synthesis (e.g. meta-analysis)
51
Descriptive Synthesis
– “An approach to the synthesis of findings from multiple
studies that relies primarily on the use of words and text to
summarise and explain the findings of the synthesis. It
adopts a textual approach to the process of synthesis to
‘tell the story’ of the findings from the included studies.” –
Popay et al. 2006
• Methods proposed for synthesis of qualitative studies: Meta-
ethnography, thematic analysis, content analysis, narrative
synthesis etc.
52
27. 6/19/2014
27
Using Meta-Ethnography
• It is a method that involves induction and interpretation.
• The outcome is the translation of studies into one
another, which encourages the researcher to understand
and transfer ideas, concepts and metaphors across
different studies.
• Studies can relate to one another in one of three ways:
– they may be directly comparable as reciprocal translations
– they may stand in opposition to one another as refutational
translations, or
– taken together they may represent a line of argument.
• Example of SLR paper that have used meta-ethnography
Numerical Synthesis (Meta-Analysis)
• Meta-analysis can be used to aggregate results or to
pool data from different studies.
• The aim is to resolve uncertainty when the results of
studies disagree; and to increase confidence in the
results obtained from individual studies
• The outcome of a meta-analysis is an average effect
size (ES) with an indication of how variable that effect
size is between studies.
• ES can be calculated based on Hedges or Cohen’s
approach; i.e. the standardized mean difference
between the two groups.
54
28. 6/19/2014
28
Meta-Analysis
• Meta-analysis involves three main steps:
1. Decide which studies to be included in the meta-
analysis.
2. Estimate an effect size for each individual study.
3. Combine the effect sizes from the individual studies
to estimate and test the combined effect.
• Results of the meta-analysis can be presented in a
forest plot.
• Software tools: Comprehensive Meta-Analysis (CMA),
MIX, OpenMeta [Analyst], Review Manager (RevMan)
from Cochrane Collaboration, Stata, R.
Meta-analysis of PP’s effectiveness (Salleh et al.)
The pooled ES for this MA=0.67 (medium effect size).
Effect size category (Kampenes et al., 2006):
Small (effect size of 0.000 – 0.376)
Medium (effect size of 0.378 – 1)
Large (effect size of 1.002 – 3.40).
29. 6/19/2014
29
REPORTING THE RESULTS
57
1) Reporting Results In Academic
Journals/Conferences
2) Reporting Structure
3) Examples of Published SLR
Reporting SLR results in
journals/conferences
• Some conferences and journals include a specific
topic/area on SLR.
• E.g.
– Information & Software Technology (IST) has an
editor specializing in systematic reviews.
– Int’l Symposium on Empirical Software Engineering
& Measurement (ESEM)
– IEEE Transactions on Education
58
30. 6/19/2014
30
Reporting Structure
• Introduction
– General introduction about the research. State the
purpose of the review. Emphasize the reason(s) why the
RQ is important. State the significance of the review
work and how the project contributes to the BOK of the
field.
• Main Body
– Review method – briefly describe steps taken to conduct
the review
– Results – Findings from the review, i.e. the synthesis
– Discussion – implication of review for research &
practice, Threats to Validity
• Conclusions
59
Other examples of SLR
• Cross versus Within-Company Cost Estimation Studies: A
Systematic Review (Kitchenham et al., 2007)
• A Systematic Review of Theory Use in Software Engineering
Experiments (Hannay et al., 2007)
• Motivations in software engineering: a Systematic Literature
Review (Beecham et al., 2008)
• Effectiveness of Requirements Elicitation Techniques: Empirical
Results Derived from a Systematic Review (Davis et al., 2006)
• A systematic review of software development cost estimation
studies (Jorgensen & Shepperd, 2007)
• The effectiveness of pair programming: A meta-analysis (Hannay
et al., 2009).
60
31. 6/19/2014
31
References
Budgen,D., Charters, S., Turner, M.,, Brereton, P., Kitchenham, B., Linkman, S.
(2006) Investigating the applicability of the evidence-based paradigm to
software engineering, Proc. 2006 Int’l Workshop on interdisciplinary software
engineering research, pp. 7 – 13.
Cruzes, D.S., and Dyba, T. (2011) Research Synthesis in Software Engineering: A
tertiary study, Information and Software Technology, 53(5), pp. 440-455.
Cochrane Collaboration. Cochrane Reviewers’ Handbook. Version 4.2.1.
December 2003
Cohen, J. Weighted Kappa: nominal scale agreement with provision for scaled
disagreement or partial credit. Pychol Bull. (70) 1968, pp. 213-220.
Davis, A., Dieste, O., Hickey, A., Juristo, N., Moreno, A.M. (2006) Effectiveness of
Requirements Elicitation Techniques: Empirical Results Derived from a
Systematic Review, 14th IEEE Requirements Engineering Conference , pp 176 –
185. 61
References
Dyba, T., & Dingsoyr, T. (2008). Strength of evidence in systematic reviews in
software engineering. Proceedings of the 2nd Int'l Symp. Empirical Software
Engineering & Measurement (ESEM 2008), 178-187.
Dyba, T., Kitchenham, B. A., & Jorgensen, M. (2005). Evidence-based
software engineering for practitioners. IEEE Software, 22(1), 58-65.
Jalali, S. & Wohlin, C. (2012). Systematic Literature Studies: Database
Searches vs.Backward Snowballing, Proceedings of the 6th Int'l Symp.
Empirical Software Engineering & Measurement (ESEM 2012).
Kampenes, V. B., Dyba, T., Hannay, J. E., & Sjoberg, D. I. K. (2007). A
systematic review of effect size in software engineering. Information and
Software Technology, 49, 1073-1086.
62
32. 6/19/2014
32
References
Khan, K.S., ter Riet, Gerben., Glanville, Julia., Sowden, Amanda, J. and
Kleijnen, Jo. (eds) Undertaking Systematic Review of Research on
Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning
Reviews. CRD Report Number 4 (2nd Edition), NHS Centre for Reviews
and Dissemination, University of York, IBSN 1 900640 20 1, March 2001.
Kitchenham, B. & Charters, S. (2007). Procedures for performing
systematic literature reviews in software engineering. UK: Keele
University and University of Durham.
Kitchenham, B., Mendes, E., Travassos, G.H. (2007) A Systematic Review
of Cross- vs. Within-Company Cost Estimation Studies, IEEE Transactions
on Software Engineering, 33 (5), pp 316-329.
References
Martyn Shuttleworth (2009). What Is A Literature Review?. Retrieved 29
Jul. 2012 from Experiment Resources: http://www.experiment-
resources.com/what-is-a-literature-review.html
Noblit, G.W. & Hare, R.D. Meta-ethnography: Synthesizing Qualitative
Studies, Sage, 1988.
Petticrew, M., & Roberts, H. (2006). Systematic Reviews in the Social
Sciences: A Practical Guide: Blackwell Publishing.
Pickard, L. M., B. A. J. Kitchenham, et al. (1998). "Combining empirical
results in software engineering." Information and Software Technology
40(4): 811-821.
Salleh, N., Mendes, E., & Grundy, J. (2011). Empirical studies of pair
programming for CS/SE teaching in higher education: A systematic
literature review. IEEE Transactions on Software Engineering, 37(4): 509 –
525.