Download as PDF, PPTX
![Systematic Literature
Review
Systematic Review in Software Engineering Technical Report ES 679/05
articles are extracted and synthesized during the result analysis phase. Meanwhile
which one of these phases is executed, their results must be stored. Therefore,
systematic review packaging is performed through the whole process. There are two
checkpoints in the proposed systematic review process. Before executing the systematic
review, it is necessary to guarantee that the planning is suitable. The protocol must be
evaluated and if problems are found, the researcher must return to the planning stage to
review the protocol. Similarly, if problems regarding web search engines are found
during the execution phase, the systematic review may be re-executed.
Figure 2. Systematic Review Process.
The stages listed above may appear to be sequential, but it is important to recognize
that many of the stages involve iteration. In particular, many activities are initiated
during the protocol development stage, and refined when the review proper takes place.
For example [Kitchenham et al., 2002]:
Planning Execution Result Analisys
Packaging
[ protocol plan approved ]
[ protocol plan disapproved ]
[ execution approved ]
[ execution disapproved ]
Planning Conducting Reporting](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-15-320.jpg)
![Data Extraction
• Design of Data Extraction Forms
• Content of Data Collection Forms
Electronic forms are useful and can facilitate subsequent analysis.
6.4.2 Contents of Data Collection Forms
In addition to including all the questions needed to answer the review question and
quality evaluation criteria, data collection forms should provide standard information
including:
Name of Reviewer
Date of Data extraction
Title, authors, journal, publication details
Space for additional notes
Examples
Kitchenham et al. [21] used the extraction form shown in Table 7 (note the actual form also
included the quality questions).
Table 7 Data Collection form completed for Maxwell et al., 1998
Data item Value Additional notes
Data Extractor
Data Checker
Study Identifier S1
Application domain Space, military and industrial
Name of database European Space Agency (ESA)
Number of projects in
database (including within-
company projects)
108
Number of cross-company
projects
60
Number of projects in within-
company data set
29
Size metric(s):
FP (Yes/No)
Version used:
LOC (Yes/No)
Version used:
FP: No
LOC: Yes (KLOC)
Others: No](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-41-320.jpg)
![(4) A data extraction and synthesis method, representing the sys-
tematic approach to consolidate and integrate existing
knowledge.
The review protocol was developed in cooperation with the first
and second authors, and validated by the third author prior to con-
ducting the review. In the following, we describe the implementa-
tion of the review protocol.
in this stage as depicted in Fig. 1. In the first stage, relevant publi-
cations were retrieved by querying the databases mentioned above
with the search string depicted in Table 1. All database queries
were made in the first two weeks of October 2012. This yielded a
total of 1152 initial results, and 1034 publications after the
removal of duplicates, which were included in the second stage.
In the second stage, publications were excluded based on their
titles and abstract. Criteria for inclusion in the following stage were
Table 1
Composition of the search string.
AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design,
interface design
OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development
OR Software development, systems development
Fig. 1. Selection process (based on [31]).
166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
(4) A data extraction and synthesis method, representing the sys- in this stage as depicted in Fig. 1. In the first stage, relevant publi-
Table 1
Composition of the search string.
AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design,
interface design
OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development
OR Software development, systems development
Fig. 1. Selection process (based on [31]).
166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-101-320.jpg)
![on these criteria, 74 articles were considered relevant for inclusion
in the next stage. Following the strategy suggested by Webster and
Watson [36], we conducted a forward and backward search based
on these key publications. The retrieved new and potentially rele-
vant articles were fed to the second stage for further processing,
resulting in an iterative extension of the sample. This resulted in
an additional set of nine articles for inclusion in stage four. It has
to be noted that both da Silva et al.’s [20] and Sohaib and Khan’s
[23] reviews were included in the stage three sample. However,
they were excluded from the detailed analysis in stage four as they
do not present the results of primary research. In total, 83 papers
were selected for the final sample.1
In the fourth stage, the final sample passed a first categorization
and a quality assessment. During the categorization, each paper
was assigned to one of the four research foci, which will be intro-
duced in the next section. A quality assessment of each paper was
subsequently conducted to obtain additional information support-
ing the interpretation of the paper’s recommendations. This assess-
ment was based on four questions, which are given in Table 2. Each
of the four criteria was graded on a dichotomous scale as ‘‘yes’’ or
‘‘no,’’ while question 3b was answered by means of the respective
method, which was either explicitly stated in the paper or deter-
mined by the researcher based on the available information.
3.3. Paper analysis
Barksdale and McCrickard [22], we did no
people integration and social integration, a
aspects almost inseparable.
3.3.2. Identification of codes
We used the qualitative data analysis
code the papers. As a universal tool for qu
MAXQDA is usually employed to analyze tex
view transcripts, and supports a variety of qu
ods. Various authors (e.g. [31,37,38]) h
experiences concerning the use of similar s
synthesize data from a corpus of texts eme
literature review. Motivated by these exper
approach to process a large amount of text
transparently. This seems especially usefu
extant literature, which may be used to d
retrieved easily in later stages of the resear
QDA, codes can be assigned to text segme
ments. Besides descriptive statistics on the
documents, MAXQDA provides a convenien
text passages, allowing for easy data synthe
allowed us to conduct a quality analysis of
be able to retrieve papers within a certain c
manner, the categorization was done by ass
the appropriate category to the title and abs
thermore, if a criterion given in Table 2 was
the text segment providing the information
The analysis of the document corpus wa
aspects that have to be taken into account
and ASD, with the overarching aim of der
the analysis of the document corpus, observ
different aspects of integration were assign
with the aim of consolidating the perspectiv
on the same issue later on.
The dimensions introduced above forme
of high-level codes. For simplification reas
‘‘process,’’ ‘‘practices,’’ ‘‘people/social,’’ and
level codes in the following. Based on our a
Table 2
Criteria for quality assessment.
1. Is there a clear statement of the research goals (e.g. in an explicitly
verbalized research question)?
2. Is there an adequate description of the context in which the research was
carried out?
3. (Only applicable to empirical research papers):
a. Is the research method explicitly stated?
b. Which research method was chosen?
4. Are there explicit recommendations, which could be aggregated as
principles?
M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
Table 3
Initial coding system.
Level 1 Process Practices People/social Technology
Level 2 Little design up front Prototyping Close collaboration Data exchange
User testing⁄
User testing⁄
Cross-functionality IDE integration
One sprint ahead User stories Knowledge transfer
Cohesive overall design Scenarios
Parallel tracks Personas
Iterative design/development
Incremental design/development
Fig. 2. Number of publications by research type.
168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-102-320.jpg)
![3.3.3. Identification of candidate principles
In order to identify candidate principles, each code was investi-
gated concerning two characteristics. First, the frequency of occur-
rence was assessed to gain an impression of the relative
importance of each aspect. As an example, the code ‘‘prototyping’’
was assigned in 40 (48.2%) articles in the literature review, indicat-
ing that the use of prototypes was discussed in this document. In
contrast, the idea of ‘‘remote usability testing’’ was only found in
one (1.2%) article, leading to the conclusion that, while the former
patterns. These patterns represent related ideas or concepts pre-
sented in different publications along the four dimensions, leading
to the merger of independent aspects in the coding system. The
identification of such patterns was based on a qualitative analysis
and the expertise of the researchers in this domain. In this step,
also the findings of the three assessed reviews were drawn upon
to identify themes for potential principles [33].
4. Results
4.1. Number and distribution of publications
When analyzing the result set, it became apparent that, while
the integration of UCD and ASD was first discussed a decade ago,
the topic gained momentum in 2007 and, since then, a constantly
high number of relevant articles have been published every year, as
Fig. 3 illustrates. This reflects that, while the idea of integrating
UCD and ASD, has been around for some time, many integration
issues are still unresolved and research is ongoing. In particular,
at least 17 relevant articles have been published since da Silva
et al.’s [20] systematic literature was conducted in late 2010, justi-
fying an update of their findings.
Moreover, we identified the research type of each paper as pre-
Parallel tracks Personas
Iterative design/development
Incremental design/development
Fig. 2. Number of publications by research type.
Fig. 3. Number of publications by year.
3.3.3. Identification of candidate principles
In order to identify candidate principles, each code was investi-
patterns. These patterns rep
sented in different publicatio
to the merger of independen
identification of such pattern
and the expertise of the rese
also the findings of the three
to identify themes for potent
4. Results
4.1. Number and distribution o
When analyzing the resul
the integration of UCD and A
the topic gained momentum
high number of relevant artic
Fig. 2. Number of publications by research type.
Fig. 3. Number of publications by year.](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-103-320.jpg)
![process dimension, many more codes have been identified for
the other two dimensions. Moreover, the codes in the process
dimension are mainly specific to software development, while
the majority of the codes in the other dimensions are rather gen-
eric (e.g. codes like ‘‘collaboration’’ or ‘‘data exchange’’). This might
be due to the overall number of papers assigned to the ‘‘process’’
dimension being larger than in the other dimensions. Apart from
that, it might also be an indicator of the depth and maturity of dis-
course in each of the dimensions. While aspects of process integra-
tion have been intensively discussed and refined, the discussion in
no principles concerning technology integration were derived. A
different challenge emerged for the people/social dimension.
Although we found more sources, the recommendations were con-
tradictory. While ASD suggests the collective accountability of the
team [119], UCD methodologies usually suggest individual respon-
sibility, for example, allocating the responsibility of developing a
usable product to the UCD expert [122–124]. Furthermore, the
team setup led to contradictory evidence. On the one hand, two
dedicated teams, which handle design and development activities
Table 4
Overview of publications in the final sample.
Dimension Included publications Number of
publications
Process Benigni et al. [39], Budwig et al. [40], Felker et al. [41], Ferreira et al. [14], Ferreira et al. [42], Fox et al. [21], Holzinger et al. [43], Hussain
et al. [44], Hussain et al. [45], Kuusinen et al. [46], Lee & McCrickard [47], Lee et al. [48], Lee et al. [49], Losada et al. [50], Losada et al.
[51], Losada et al. [52], Memmel et al. [53], Memmel et al. [54], Miller [55], Najafi & Toyoshiba [56], Paelke & Nebe [57], Paelke & Sester
[58], Wolkerstorfer et al. [59], Zhang et al. [60]
24 (28.9%)
Practices Adikari et al. [61], Beyer et al. [62], Broschinsky & Baker [63], Carvalho [64], Chamberlain et al. [35], Cho [65], Constantine [13],
Constantine & Lockwood [66], da Silva et al. [67], Detweiler [68], Düchting et al. [69], Evnin & Pries [70], Ferré et al. [71], Fisher &
Bankston [72], Haikara [73], Hansson et al. [74], Hellmann et al. [75], Hellmann et al. [76], Hennings [77], Hodgetts [78], Humayoun
et al. [79], Hussain et al. [80], Illmensee & Muff [81], Isomursu et al. [82], Kane [83], Lárusdóttir [84], Lárusdóttir et al. [85], Lee et al.
[86], Medina-Medina et al. [87], Memmel et al. [88], Meszaros & Aston [89], Obendorf & Finck [90], Obendorf et al. [91], Patton [92],
Patton [93], Petrovic & Siegmann [94], Rafla et al. [95], Rittenbruch et al. [96]
38 (45.8%)
People &
Social
Barksdale & McCrickard [22], Barksdale & McCrickard [97], Barksdale et al. [98], Brown et al. [99], Brown et al. [100], Ferreira et al.
[101], Ferreira et al. [102], Kollmann et al. [103], Leszek & Courage [104], Lievesley & Yee [105], Singh [106], Ungar [107], Ungar & White
[108], Williams & Ferguson [109]
14 (16.9%)
Technology Feiner & Andrews [110], Gonçalves & Santos [111], Hosseini-Khayat et al. [112], Humayoun et al. [113], Lee [114], Peixoto [115], Peixoto
[116]
7 (8.4%)
Total 83 (100.0%)](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-104-320.jpg)
![There is evidence that a limited up-front design effort is
particularly necessary to provide a consistent and cohesive user
interface and navigation structure as it supports designers in find-
ing a suitable design concept from the very beginning [42,61]. Pat-
ton [93] states that user needs and goals have to be clearly defined
before conducting design and development efforts in order to
ensure the usability and usefulness of the software in ASD. Various
authors have confirmed this view, presenting user research as an
essential aspect of up-front design efforts in agile environments
[20,21,24,35,49,55]. In total, 22 (26.5%) of the papers in the review
confirm a relationship between LDUF and a cohesive overall design
of the final product, which is challenging to achieve in an agile
Suggestion. A shift from an up-front design to up-front analysis
concept is identified. In order to deliver a product that is valuable
to the customer and end-user in that it is both usable and useful,
product discovery and product creation should be separated in
UCASD. Overall, as listed in Table 5, four codes from our coding sys-
tem support our suggestion to include an additional principle in
this specific context.
Besides making room for sufficient up-front design activities as
a prerequisite for delivering a usable product with a consistent
user interaction concept, it allows for mitigating agile shortcom-
ings with regard to product discovery, fostering the delivery of a
useful product. Thus, we formulate the first principle as:
Fig. 5. Codes and articles related to the process dimension.
M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 171](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-106-320.jpg)
![not part of agile method-
ed by drawing on UCD, in
pectations, and ideas are
ude that the usefulness
sability concerns not only
ring product planning. In
e design throughout soft-
ns should be included in
05,106].
Table 5
Codes related to the separate product discovery and product creation principle.
Included codes Resulting principle
Little design up front Separate product discovery and product creation
Cycle zero
Cohesive overall design
Product vision/innovation
Sohaib and Khan [23]. Specifically, their reviews confirm the signif-
icance of iterative and incremental design and development. There
is general agreement among UCD researchers that an iterative
approach is essential for developing a product with high usability
references to iterative an
Scrum) or UCD (e.g. ISO
papers do not explicitly
approach, none of them ch
principle is articulated as
Principle 2 (Iterative a
ment). User-centered agil
design and development i
manner.
5.1.3. Parallel Interwoven C
Focus Area. Extendin
product creation principle
quent course of action afte
ment activities. As a res
design allows for specifyin
important features of a sys
to be considered in later
Table 6
Codes related to the iterative and incremental design and development principle.
Included codes Resulting principle
Iterative design/development Iterative and incremental design and
developmentIncremental design/
development
Table 7
Codes related to the parallel interwoven creation tracks principle.
Included codes Resulting principle
Parallel tracks Parallel interwoven creation tracks
Design one sprint ahead
Synchronization/integration
172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
Sohaib and Khan [23]. Specifically, their reviews confirm the signif-
icance of iterative and incremental design and development. There
is general agreement among UCD researchers that an iterative
references to iterative a
Scrum) or UCD (e.g. ISO
papers do not explicitly
approach, none of them ch
principle is articulated as
Principle 2 (Iterative a
ment). User-centered agil
design and development i
manner.
5.1.3. Parallel Interwoven C
Focus Area. Extendin
product creation principl
quent course of action afte
ment activities. As a res
design allows for specifyin
important features of a sy
Table 6
Codes related to the iterative and incremental design and development principle.
Included codes Resulting principle
Iterative design/development Iterative and incremental design and
developmentIncremental design/
development
Table 7
Codes related to the parallel interwoven creation tracks principle.
Included codes Resulting principle
Parallel tracks Parallel interwoven creation tracks
Design one sprint ahead
Synchronization/integration
172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-107-320.jpg)
![The value of individuals and interactions is documen
agile manifesto [134]. According to UCD, understanding
their tasks should be the focus of software developm
Moreover, UCD and ASD both encourage customer or u
pation in the development of new systems or software
suggest the following principle:
Fig. 7. Codes and number of articles related to the artifact-mediated communication principle.
UCASD.
Description
Separate product discovery and product creation
Iterative and incremental design and development
Parallel interwoven creation tracks
M. Brhel et al. / Information and Software Technology 61 (2015) 163–181](https://image.slidesharecdn.com/slr-ict-2016-160909011629/85/Systematic-Literature-Review-109-320.jpg)
Uploaded byTiago Silva da Silva
Systematic Literature Review
This document discusses planning and conducting a systematic literature review. It begins by explaining that systematic reviews aim to aggregate all relevant evidence on a topic in a fair and repeatable manner. The document then outlines the key steps in planning a review, including specifying the research question, developing a review protocol, and evaluating the protocol. It also covers conducting the review, such as identifying relevant research, selecting primary studies, assessing study quality, and extracting and synthesizing data. The importance of transparency and replicability in the review process is emphasized.
More Related Content
Similar to Systematic Literature Review
Systematic Literature Review
- 1. Systematic Literature Review Prof. Dr.TiagoSilva da Silva (ICT/UNIFESP) silvadasilva@unifesp.br
- 2.
- 3. Literature Review • Expertscan be wrong.
- 4. Literature Review • Expertscan be wrong. • Researcher’s choice of “related studies” can be biased.
- 5. Literature Review • Expertscan be wrong. • Researcher’s choice of “related studies” can be biased. • Informal reviews can miss important studies.
- 6. Systematic Literature Review • SystematicReview • Medical domain • Evidence-based software engineering (Kitchenham) • “We cannot have evidence-based software engineering without a sound methodology for aggregating evidence from different empirical studies. SRs provide that methodology.”
- 7. Systematic Literature Review • Themajor advantage of a SR is that it is based on a well-defined methodology. • SRs aim to find, assess, and aggregate all relevant evidence about some topic of interest in a fair, repeatable, and auditable manner. • A researcher conducting a SR selects empirical studies that are relevant to the particular research question, assesses the validity of each one, and then determines the trend shown by those studies.
- 8. Systematic Literature Review • Toinvestigate all available evidence that supports or refutes a particular “topic of interest”. • To summarize the existing evidence concerning a treatment or technology, e.g. to summarize the empirical evidence of the benefits and limitations of a specific agile method. • To identify any gaps in current research in order to suggest areas for further investigation. • To provide a framework/background in order to appropriately position new research activities. • Sometimes a single researcher, such as a PhD student, needs to do this activity alone, but that limitation raises the risk of missing relevant papers.
- 9.
- 10. Advantages • The well-definedmethodology makes it less likely that the results of the literature are biased, although it does not protect against publication bias in the primary studies.
- 11. Advantages • The well-definedmethodology makes it less likely that the results of the literature are biased, although it does not protect against publication bias in the primary studies. • They can provide information about the effects of some phenomenon across a wide range of settings and empirical methods. If studies give consistent results, systematic reviews provide evidence that the phenomenon is robust and transferable. If the studies give inconsistent results, sources of variation can be studied.
- 12.
- 13. Disadvantage • Requires considerablymore effort than traditional literature reviews.
- 14. Systematic Literature Review • Planningthe review • Conducting the review • Reporting the review
- 15. Systematic Literature Review Systematic Reviewin Software Engineering Technical Report ES 679/05 articles are extracted and synthesized during the result analysis phase. Meanwhile which one of these phases is executed, their results must be stored. Therefore, systematic review packaging is performed through the whole process. There are two checkpoints in the proposed systematic review process. Before executing the systematic review, it is necessary to guarantee that the planning is suitable. The protocol must be evaluated and if problems are found, the researcher must return to the planning stage to review the protocol. Similarly, if problems regarding web search engines are found during the execution phase, the systematic review may be re-executed. Figure 2. Systematic Review Process. The stages listed above may appear to be sequential, but it is important to recognize that many of the stages involve iteration. In particular, many activities are initiated during the protocol development stage, and refined when the review proper takes place. For example [Kitchenham et al., 2002]: Planning Execution Result Analisys Packaging [ protocol plan approved ] [ protocol plan disapproved ] [ execution approved ] [ execution disapproved ] Planning Conducting Reporting
- 16. Systematic Literature Review e beeniden- d these have dence-based nham et al., technique, ; he question; y (closeness pplicability engineering alues and n executing titute a sys- rder to pro- relevant to systematic d interpret- ar research Watson, 2002), they are less common in software engineer- ing (however, see (Glass et al., 2002) as an example of a sec- ondary study that samples literature within the software engineering domain). Indeed, at the present time, outside of information systems research, reviews in any form, as well as review journals are really not part of the computing 9. Write Review Report 10. Validate Report 2. Develop Review Protocol 3. Validate Review Protocol Phase 1: Plan Review Phase 2: Conduct Review Phase 3: Document Review 8. Synthesise Data 4. Identify Relevant Research 5. Select Primary Studies 7. Extract Required Data 6. Assess Study Quality 1. Specify Research Questions Fig. 1. Systematic literature review process.
- 17. Planning the Review 1.Identification of the need for a review 2. Commissioning a review* 3. Specifying the research question(s) 4. Developing a review protocol 5. Evaluating the review protocol*
- 18. Conducting the Review 1.Identification of research 2. Selection of primary studies 3. Study quality assessment 4. Data extraction and monitoring 5. Data synthesis
- 19. Reporting the Review 1.Specifying dissemination mechanisms 2. Formatting the main report 3. Evaluating the report*
- 20.
- 21. The need fora systematic review • What are the review’s objectives? • What sources were searched to identify primary studies? Were there any restrictions? • What were the inclusion/exclusion criteria and how were they applied? • What criteria were used to assess the quality of primary studies?
- 22. The need fora systematic review • How were quality criteria applied? • How were the data extracted from the primary studies? • How were the data synthesized? • How were differences between studies investigated? • How were the data combined? • Was it reasonable to combine the studies? • Do the conclusions flow from the evidence?
- 23. The Research Question(s) • Themost important part of any systematic review. The question(s) drive(s) the entire methodology: • The search process must identify primary studies that address the research questions. • The data extraction process must extract the data items needed to answer the questions. • The data analysis process must synthesize the data in such a way that the questions can be answered.
- 24. The Research Question(s) • Assessingthe effect of a technology. • Assessing the frequency or rate of a project development factor such as the adoption of a technology, or the frequency or rate of project success or failure. • Identifying cost and risk factors associated with a technology. • Identifying the impact of technologies on reliability, performance and cost models. • Cost benefit analysis of employing specific software development technologies or software applications.
- 25. The Research Question(s) example •Question 1: What evidence is there that cross-company estimation models are not significantly different from within- company estimation models for predicting effort for software/ Web projects? • Question 2: What characteristics of the study data sets and the data analysis methods used in the study affect the outcome of within- and cross-company effort estimation accuracy studies? • Question 3: Which experimental procedure is most appropriate for studies comparing within- and cross-company estimation models?
- 26. The Research Question(s) structure •Population: software or web-project. • Intervention: cross-company project effort estimation model. • Comparison: single-company project effort estimation model. • Outcomes: prediction or estimate accuracy.
- 27. Developing a Review Protocol •A review protocol specifies the methods that will be used to undertake a specific systematic review • Background: the rationale for the survey. • The research questions that the review is intended to answer. • The strategy that will be used to search for primary studies including search terms and resources to be searched: digital libraries, specific journals and proceedings.
- 28. Developing a Review Protocol •Study selection criteria: used to determine which studies are included in, or excluded from. • Study selection procedures: how the selection criteria will be applied e.g. how many assessors will evaluate each prospective primary study, and how disagreements among assessors will be resolved.
- 29. Developing a Review Protocol •Study quality assessment checklists and procedures: quality checklists to assess the individual studies. • Data extraction strategy: how the information required from each primary study will be obtained. • Synthesis of the extracted data: defines the synthesis strategy. • Dissemination strategy
- 30.
- 31. Identification of Research • Generatinga search strategy • Preliminary searches aimed at both identifying existing systematic reviews and assessing the volume of potentially relevant studies. • Trial searches using various combinations of search terms derived from the research question. • Checking trial research strings against lists of already known primary studies. • Consultations with experts in the field.
- 32. Identification of Research • Publicationbias • The problem that positive results are more likely to be published than negative results.The concept of positive or negative results sometimes depends on the viewpoint of the researcher. • Scanning the grey literature • Scanning conference proceedings
- 33. • Bibliography Managementand Document Retrieval • Reference Manager (http://www.refman.com/) • Endnote (http://endnote.com/) • Mendeley (http://www.mendeley.com/) • Papers (http://papersapp.com/mac/) • Documenting the Search (The process of performing a systematic literature review must be transparent and replicable (as far as possible) Identification of Research
- 34. Identification of Research literature search. Oncereference lists have been finalised the full articles of potentially useful studies will need to be obtained. A logging system is needed to make sure all relevant studies are obtained. 6.1.4 Documenting the Search The process of performing a systematic literature review must be transparent and replicable (as far as possible): The review must be documented in sufficient detail for readers to be able to assess the thoroughness of the search. The search should be documented as it occurs and changes noted and justified. The unfiltered search results should be saved and retained for possible reanalysis. Procedures for documenting the search process are given in Table 2. Table 2 Search process documentation Data Source Documentation Digital Library Name of database Search strategy for the database Date of search Years covered by search Journal Hand Searches Name of journal Years searched Any issues not searched Conference proceedings Title of proceedings Name of conference (if different) Title translation (if necessary) Journal name (if published as part of a journal) Efforts to identify unpublished studies Research groups and researchers contacted (Names and contact details) Research web sites searched (Date and URL) Other sources Date Searched/Contacted URL Any specific conditions pertaining to the search • ACM Digital Library (http:// dl.acm.org/) • IEEE Xplore (http:// ieeexplore.ieee.org/) • Science Direct (http:// www.sciencedirect.com/) • ISI Web of Science (http:// isiknowledge.com/) • Scopus (http://www.scopus.com/) • Google Scholar*
- 35. Study Selection • Studyselection criteria: to identify those primary studies that provide direct evidence about the research question. Inclusion and exclusion criteria should be based on the research question. • Inclusion: • any study that compared predictions of cross-company models with within-company models based on analysis of single company project data. • Exclusion: • studies where projects were only collected from a small number of different sources (e.g. 2 or 3 companies), • studies where models derived from a within-company data set were compared with predictions from a general cost estimation model.
- 36. Study Selection • Studyselection process: • Language • Journal • Authors • Setting • Participants or subjects • Research Design • Date of publication • Reliability of inclusion decisions: • When two or more researchers assess each paper, agreement between researchers can be measured using the Cohen Kappa statistic.
- 37. Identify relevant studies- automated and manual search Exclude studies on the basis of Title Exclude studies on the basis of Abstract Obtain primary papers and critically appraise studies n papers n papers n papers n papers
- 38. Study Quality Assessment • Toprovide still more detailed inclusion/exclusion criteria. • To investigate whether quality differences provide an explanation for differences in study results. • As a means of weighting the importance of individual studies when results are being synthesized. • To guide the interpretation of findings and determine the strength of inferences. • To guide recommendations for further research.
- 39. • Development ofQuality Instruments • Design • Conduct • Analysis • Conclusions • Example: 1.Is the data analysis process appropriate? 1.1.Was the data investigated to identify outliers and to assess distributional properties before analysis? 1.2.Was the result of the investigation used appropriately to transform the data and select appropriate data points? Study Quality Assessment
- 40. • Less than10 projects: Poor quality (score=0) • Between 10 and 20 projects: Fair quality (score=0.33) • Between 21 and 40 projects: Good quality (score=0.67) • More than 40 projects: Excellent quality (score=1) Study Quality Assessment example
- 41. Data Extraction • Designof Data Extraction Forms • Content of Data Collection Forms Electronic forms are useful and can facilitate subsequent analysis. 6.4.2 Contents of Data Collection Forms In addition to including all the questions needed to answer the review question and quality evaluation criteria, data collection forms should provide standard information including: Name of Reviewer Date of Data extraction Title, authors, journal, publication details Space for additional notes Examples Kitchenham et al. [21] used the extraction form shown in Table 7 (note the actual form also included the quality questions). Table 7 Data Collection form completed for Maxwell et al., 1998 Data item Value Additional notes Data Extractor Data Checker Study Identifier S1 Application domain Space, military and industrial Name of database European Space Agency (ESA) Number of projects in database (including within- company projects) 108 Number of cross-company projects 60 Number of projects in within- company data set 29 Size metric(s): FP (Yes/No) Version used: LOC (Yes/No) Version used: FP: No LOC: Yes (KLOC) Others: No
- 42. Data Extraction • Dataextraction procedures • Whenever feasible, data extraction should be performed independently by two or more researchers. Data from the researchers must be compared and disagreements resolved either by consensus among researchers or arbitration by an additional independent researcher. • Multiple publications of the same data • It is important not to include multiple publications of the same data in a systematic review synthesis because duplicate reports would seriously bias any results.When there are duplicate publications, the most complete should be used.
- 43. Data Synthesis • Unpublisheddata, missing data and data requiring manipulation • Collating and summarizing the results of the included primary studies.The data synthesis activities should be specified in the review protocol. • Descriptive (Narrative) Synthesis • Extracted information about the studies (i.e. intervention, population, context, sample sizes, outcomes, study quality) should be tabulated in a manner consistent with the review question.
- 44. Data Synthesis • QuantitativeSynthesis: quantitative data should also be presented in tabular form including: • Sample size for each intervention. • Estimates effect size for each intervention with standard errors for each effect. • Difference between the mean values for each intervention, and the confidence interval for the difference. • Units used for measuring the effect.
- 45. Data Synthesis • Presentationof Quantitative Results • Charts x Tables • Qualitative Synthesis • Tries to integrate studies comprising natural language results and conclusions, where different researchers may have used terms and concepts with subtly (or grossly) different meanings • Synthesis of Qualitative and Quantitative Synthesis • Synthesize the quantitative and qualitative studies separately. • Then attempt to integrate the qualitative and quantitative results by investigating whether the qualitative results can help explain the quantitative results. For example qualitative studies can suggest reasons why a treatment does or does not work in specific circumstances.
- 46. Data Synthesis • SensitivityAnalysis • High quality primary studies only. • Primary studies of particular types. • Primary studies for which data extraction presented no difficulties (i.e. excluding any studies where there was some residual disagreement about the data extracted). • The experimental method used by the primary studies.
- 47.
- 48. Reporting • Specifying theDissemination Strategy • Journals • Conferences • Formatting the Main Systematic Review Report • Technical Report • Section of a thesis • Journal or conference format • Evaluating Systematic Review Reports
- 49.
- 50. Problems associated with conductinga SLR • The protocol may not have identified all the variations in the designing and reporting of relevant primary studies. • Selection of the digital libraries to be searched and whether the search is automated or manual. • A manual search involves looking at the back issues of a set of journals (either paper or online) and deciding from the title and abstract which individual papers are candidates for inclusion in the review. • An automated search uses strings, usually based on complex Boolean formulas, to turn up papers in online catalogs.
- 51. (software OR applicationOR product OR Web OR WWW OR Internet OR World-Wide Web OR project OR development) AND (method OR process OR system OR technique OR methodology OR procedure) AND (cross company OR cross organisation OR cross organization OR cross organizational OR cross organisational OR cross- company OR cross-organisation OR cross- organization OR cross-organizational OR cross-organisational OR multi company OR multi organisation OR multi organization OR multi organizational OR multi organisational OR multi- company OR multi-organisation OR multi- organization OR multi-organizational OR multi-organisational OR multiple company OR multiple organisation OR multiple organization OR multiple organizational OR multiple organisational OR multiple-company OR multiple- organisation OR multiple-organization OR multiple-organizational OR multiple- organisational OR within company OR within organisation OR within organization OR within organizational OR within organisational OR within- company OR within-organisation OR within-organization OR within- organizational OR within-organisational OR single company OR single organisation OR single organization OR single organizational OR single organisational OR single-company OR single-organisation OR single- organization OR single-organizational OR single-organisational OR company- specific) AND (model OR modeling OR modelling) AND (effort OR cost OR resource) AND (estimation OR prediction OR assessment)
- 52. Problems associated with conductinga SLR • Digital libraries differ in subtle ways in their implementations of searches: • Digital libraries have different interfaces and different tolerances for the complex Boolean formulas typically used by SR researchers to turn up relevant papers. • Digital libraries also have different procedures for searching the body of the paper, or only the title, abstract, and keywords. In addition, of course, indexing systems only search titles, keywords, and abstracts. • Automated searches from different sources may overlap (i.e., the same paper may be found in different digital libraries), and every search includes a large number of irrelevant papers .
- 53. Problems associated with conductinga SLR • Digital libraries differ in subtle ways in their implementations of searches: • (TITLE-ABS-KEY(usability) OR TITLE-ABS-KEY("human-computer interaction") OR TITLE-ABS-KEY("computer-human interaction") OR TITLE-ABS-KEY("human factor") OR TITLE-ABS-KEY("user experience") OR TITLE-ABS-KEY("user-centered design")) AND (TITLE-ABS-KEY(agile) OR TITLE-ABS-KEY("agile method") OR TITLE-ABS-KEY("agile development") OR TITLE-ABS-KEY("agile practice") OR TITLE-ABS-KEY("agile project") OR TITLE-ABS-KEY("agile lifecycle") OR TITLE-ABS-KEY(scrum) OR TITLE-ABS- KEY("extreme programming") OR TITLE-ABS-KEY("lean development") OR TITLE-ABS- KEY("feature driven development") OR TITLE-ABS-KEY("dynamic system development method") OR TITLE-ABS-KEY("agile unified process")) AND PUBYEAR AFT 2000 AND (LIMIT-TO(SUBJAREA, "COMP") OR LIMIT-TO(SUBJAREA, "MULT")) • (usability OR "human-computer interaction" OR "computer-human interaction" OR "human factor" OR "user experience" OR "user-centered design") AND (agile OR "agile method" OR "agile development" OR "agile practice" OR "agile project" OR "agile lifecycle" OR scrum OR "extreme programming" OR "lean development" OR "feature driven development" OR "dynamic system development method" OR "agile unified process")
- 54. Problems associated with conductinga SLR • For manual searches: • Select the journals and conference proceedings you intend to search. • Justify your selection of sources. However, rather surprisingly, in one case study we found that a targeted manual search was much quicker than a broad automated search. • In practice, need for a mixed strategy. • If you do a manual search of some sources (including specialist conference proceedings), this should give you a baseline set of candidate primary studies against which you can validate the efficacy of your automated search strings. • Alternatively, a domain expert can identify a baseline set of papers.
- 55. Other types ofreviews • Systematic Mapping Study: a broad review of primary studies in a specific topic area that aims to identify what evidence is available on the topic. • Tertiary Study: a review of secondary studies related to the same research question.
- 56. Lessons Learned e beeniden- d these have dence-based nham et al., technique, ; he question; y (closeness pplicability engineering alues and n executing titute a sys- rder to pro- relevant to systematic d interpret- ar research Watson, 2002), they are less common in software engineer- ing (however, see (Glass et al., 2002) as an example of a sec- ondary study that samples literature within the software engineering domain). Indeed, at the present time, outside of information systems research, reviews in any form, as well as review journals are really not part of the computing 9. Write Review Report 10. Validate Report 2. Develop Review Protocol 3. Validate Review Protocol Phase 1: Plan Review Phase 2: Conduct Review Phase 3: Document Review 8. Synthesise Data 4. Identify Relevant Research 5. Select Primary Studies 7. Extract Required Data 6. Assess Study Quality 1. Specify Research Questions Fig. 1. Systematic literature review process.
- 57. Lessons Learned • Stage1: Specify Research Questions • L1: Expect to revise your questions during protocol development, as your understanding of the problem increases. • L2: A pre-review mapping study may help in scoping research questions.
- 58. Lessons Learned • Stage2: Develop Review Protocol • L3: All systematic review team members need to take an active part in developing the review protocol. • L4: Piloting the research protocol is essential. It will find mistakes in your data collection and aggregation procedures. It may also indicate that you need to change the methodology you intend to use to address the research questions.
- 59. Lessons Learned • Stage3:Validate Review Protocol • L5: Data extraction is assisted by having data definitions and data extraction guidelines from the protocol recorded in a separate short document. • L6: There needs to be an agreed validation process separate from the protocol piloting activity. Ideally, external reviewers should undertake this validation process.
- 60. Lessons Learned • Stage4: Identify Relevant Research • L7: There are alternative search strategies that enable you to achieve different sort of search completion criteria.You must select and justify a search strategy that is appropriate for your research question. • L8: We need to search many different electronic sources; no single source finds all of the primary studies. • L9: Current software engineering search engines are not designed to support systematic literature reviews. Unlike medical researchers, software engineering researchers need to perform resource-dependent searches.
- 61. Lessons Learned • Stage5: Select Primary Studies • L10: The standard of IT and software engineering abstracts is too poor to rely on when selecting primary studies.You should also review the conclusions.
- 62. Lessons Learned • Stage6:Assess Study Quality • L11: All the medical standards emphasise that it is necessary to assess the quality of primary studies. However, it depends on the type of systematic literature review you are undertaking. • L12: It is important to be sure how the quality assessment will be used in the subsequent data aggregation and analysis.
- 63. Lessons Learned • Stage7: Extract Required Data • L13: Having one reader act as data extractor and one act as data checker may be helpful when there are a large number of papers to review. • L14: Review team members must make sure they understand the protocol and the data extraction process.
- 64. Lessons Learned • Stage8: Synthesise Data • L15: Software engineering systematic reviews are likely to be qualitative in nature. • L16: Even when collecting quantitative information it may not be possible to perform meta-analysis of software engineering studies because the reporting protocols vary so much from study to study. • L17: Tabulating the data is a useful means of aggregation but it is necessary to explain how the aggregated data actually answers the research questions.
- 65. Lessons Learned • Stage9: Write Review Report • L18: Review teams need to keep a detailed record of decisions made throughout the review process. • L19: The software engineering community needs to establish mechanisms for publishing systematic literature reviews which may result in papers that are longer than those traditionally accepted by many software engineering outlets or that have appendices stored in electronic repositories.
- 66. Lessons Learned • Stage10: Validate Report
- 67. References • Kitchenham, B..(2004). Procedures for Performing Systematic Reviews. Joint Technical Report. • Biolchini, J.; Mian, P. G.; Natali,A. C. C.;Travassos, G. H.T.. (2005) .Systematic Review in Software Engineering.Technical Report. • Kitchneham, B.. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report. • Brereton, P.; Kitchenham, B.; Budgen, D.;Turner, M.; Khalil, M.. (2008). Lessons from applying the systematic literature review process within the software engineering domain.The Journal of Systems and Software. • Dyba ̊,T.; Dingsøyr,T.. (2008). Empirical studies of agile software development:A systematic review. Information and Software Technology. • Kitchenham, B.; Brereton, P.; Budgen, D.;Turner, M.; Bailey, J.; Linkman, S.. (2009). Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology. • Kitchenham, B.; Pretorius, R.; Budgen, D.; Brereton, P.; Turner, M.; Niazi, M.; Linkman, S.. (2010). Systematic literature reviews in software engineering – A tertiary study. Information and Software Technology. • Kitchenham, B.; Mendes, E.;Travassos, G. H.T..(2007) A Systematic Review of Cross- vs.Within- Company Cost Estimation Studies, IEEE Trans on SE.
- 68. Systematic Literature Review Prof. Dr.TiagoSilva da Silva (ICT/UNIFESP) silvadasilva@unifesp.br
- 70. User-Centered Design and AgileMethods: A Systematic Review Tiago Silva da Silva,Angela Martin, Frank Maurer, Milene Silveira To contact me after my presentation, text BDL to INTRO (46876)
- 71.
- 72. Category Keyword UCD Usability Human-Computer Interatcion Computer-HumanInteraction Human Factor User Experience User-Centered Design User Interface Agile Agile Agile Method Agile Development Agile Practice Agile Project Agile Lifecycle Scrum Extreme Programming Lean Development Feature Driven Development Dynamic System Development Agile Unified Process
- 73. Search String usability OR"human-computer interaction" OR "computer-human interaction" OR "human factor" OR "user experience" OR "user- centered design" OR "user interface" AND agile OR "agile method" OR "agile development" OR "agile practice" OR "agile project" OR "agile lifecycle" OR scrum OR "extreme programming" OR "lean development" OR "feature driven development" OR "dynamic system development" OR "agile unified process"
- 74.
- 75. Digital Library Amount of papers FirstClassification Inclusion Exclusion IEEExplore 59 24 35 Science Direct 4 0 4 CiteSeerX 59 0 59 Scopus 154 24 130 Agile 21 21 0 XP 12 12 0 Total 309 81 228 Percentage 100% 26% 74%
- 76. Digital Library Amount of papers Second Classification Total Selected Repeated Not Relevant Total81 16 7 58 Percentage 100% 20% 8% 72%
- 77.
- 78.
- 79.
- 82.
- 83.
- 85. Theoretical Experience ReportEmpirical Experimental
- 86.
- 89.
- 92.
- 95.
- 100. Exploring principles ofuser-centered agile software development: A literature review Manuel Brhel a , Hendrik Meth b , Alexander Maedche a,b,⇑ , Karl Werder a a Chair of Information Systems IV, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany b Institute for Enterprise Systems, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany a r t i c l e i n f o Article history: Received 5 September 2013 Received in revised form 7 January 2015 Accepted 7 January 2015 Available online 17 January 2015 Keywords: Agile software development User-centered design Systematic literature review a b s t r a c t Context: In the last decade, software development has been characterized by two major approaches: agile software development, which aims to achieve increased velocity and flexibility during the development process, and user-centered design, which places the goals and needs of the system’s end-users at the cen- ter of software development in order to deliver software with appropriate usability. Hybrid development models, referred to as user-centered agile software development (UCASD) in this article, propose to com- bine the merits of both approaches in order to design software that is both useful and usable. Objective: This paper aims to capture the current state of the art in UCASD approaches and to derive gen- eric principles from these approaches. More specifically, we investigate the following research question: Which principles constitute a user-centered agile software development approach? Method: We conduct a systematic review of the literature on UCASD. Identified works are analyzed using a coding scheme that differentiates four levels of UCASD: the process, practices, people/social and tech- nology dimensions. Through subsequent synthesis, we derive generic principles of UCASD. Results: We identified and analyzed 83 relevant publications. The analysis resulted in a comprehensive coding system and five principles for UCASD: (1) separate product discovery and product creation, (2) iterative and incremental design and development, (3) parallel interwoven creation tracks, (4) continuous stakeholder involvement, and (5) artifact-mediated communication. Conclusion: Our paper contributes to the software development body of knowledge by (1) providing a broad overview of existing works in the area of UCASD, (2) deriving an analysis framework (in form a cod- ing system) for works in this area, going beyond former classifications, and (3) identifying generic prin- ciples of UCASD and associating them with specific practices and processes. Ó 2015 Published by Elsevier B.V. Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 2. Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 2.1. Summary of existing literature reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 2.2. Gap analysis of existing literature reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 3. Research method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Information and Software Technology 61 (2015) 163–181 Contents lists available at ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof
- 101. (4) A dataextraction and synthesis method, representing the sys- tematic approach to consolidate and integrate existing knowledge. The review protocol was developed in cooperation with the first and second authors, and validated by the third author prior to con- ducting the review. In the following, we describe the implementa- tion of the review protocol. in this stage as depicted in Fig. 1. In the first stage, relevant publi- cations were retrieved by querying the databases mentioned above with the search string depicted in Table 1. All database queries were made in the first two weeks of October 2012. This yielded a total of 1152 initial results, and 1034 publications after the removal of duplicates, which were included in the second stage. In the second stage, publications were excluded based on their titles and abstract. Criteria for inclusion in the following stage were Table 1 Composition of the search string. AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design, interface design OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development OR Software development, systems development Fig. 1. Selection process (based on [31]). 166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 (4) A data extraction and synthesis method, representing the sys- in this stage as depicted in Fig. 1. In the first stage, relevant publi- Table 1 Composition of the search string. AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design, interface design OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development OR Software development, systems development Fig. 1. Selection process (based on [31]). 166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
- 102. on these criteria,74 articles were considered relevant for inclusion in the next stage. Following the strategy suggested by Webster and Watson [36], we conducted a forward and backward search based on these key publications. The retrieved new and potentially rele- vant articles were fed to the second stage for further processing, resulting in an iterative extension of the sample. This resulted in an additional set of nine articles for inclusion in stage four. It has to be noted that both da Silva et al.’s [20] and Sohaib and Khan’s [23] reviews were included in the stage three sample. However, they were excluded from the detailed analysis in stage four as they do not present the results of primary research. In total, 83 papers were selected for the final sample.1 In the fourth stage, the final sample passed a first categorization and a quality assessment. During the categorization, each paper was assigned to one of the four research foci, which will be intro- duced in the next section. A quality assessment of each paper was subsequently conducted to obtain additional information support- ing the interpretation of the paper’s recommendations. This assess- ment was based on four questions, which are given in Table 2. Each of the four criteria was graded on a dichotomous scale as ‘‘yes’’ or ‘‘no,’’ while question 3b was answered by means of the respective method, which was either explicitly stated in the paper or deter- mined by the researcher based on the available information. 3.3. Paper analysis Barksdale and McCrickard [22], we did no people integration and social integration, a aspects almost inseparable. 3.3.2. Identification of codes We used the qualitative data analysis code the papers. As a universal tool for qu MAXQDA is usually employed to analyze tex view transcripts, and supports a variety of qu ods. Various authors (e.g. [31,37,38]) h experiences concerning the use of similar s synthesize data from a corpus of texts eme literature review. Motivated by these exper approach to process a large amount of text transparently. This seems especially usefu extant literature, which may be used to d retrieved easily in later stages of the resear QDA, codes can be assigned to text segme ments. Besides descriptive statistics on the documents, MAXQDA provides a convenien text passages, allowing for easy data synthe allowed us to conduct a quality analysis of be able to retrieve papers within a certain c manner, the categorization was done by ass the appropriate category to the title and abs thermore, if a criterion given in Table 2 was the text segment providing the information The analysis of the document corpus wa aspects that have to be taken into account and ASD, with the overarching aim of der the analysis of the document corpus, observ different aspects of integration were assign with the aim of consolidating the perspectiv on the same issue later on. The dimensions introduced above forme of high-level codes. For simplification reas ‘‘process,’’ ‘‘practices,’’ ‘‘people/social,’’ and level codes in the following. Based on our a Table 2 Criteria for quality assessment. 1. Is there a clear statement of the research goals (e.g. in an explicitly verbalized research question)? 2. Is there an adequate description of the context in which the research was carried out? 3. (Only applicable to empirical research papers): a. Is the research method explicitly stated? b. Which research method was chosen? 4. Are there explicit recommendations, which could be aggregated as principles? M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 Table 3 Initial coding system. Level 1 Process Practices People/social Technology Level 2 Little design up front Prototyping Close collaboration Data exchange User testing⁄ User testing⁄ Cross-functionality IDE integration One sprint ahead User stories Knowledge transfer Cohesive overall design Scenarios Parallel tracks Personas Iterative design/development Incremental design/development Fig. 2. Number of publications by research type. 168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
- 103. 3.3.3. Identification ofcandidate principles In order to identify candidate principles, each code was investi- gated concerning two characteristics. First, the frequency of occur- rence was assessed to gain an impression of the relative importance of each aspect. As an example, the code ‘‘prototyping’’ was assigned in 40 (48.2%) articles in the literature review, indicat- ing that the use of prototypes was discussed in this document. In contrast, the idea of ‘‘remote usability testing’’ was only found in one (1.2%) article, leading to the conclusion that, while the former patterns. These patterns represent related ideas or concepts pre- sented in different publications along the four dimensions, leading to the merger of independent aspects in the coding system. The identification of such patterns was based on a qualitative analysis and the expertise of the researchers in this domain. In this step, also the findings of the three assessed reviews were drawn upon to identify themes for potential principles [33]. 4. Results 4.1. Number and distribution of publications When analyzing the result set, it became apparent that, while the integration of UCD and ASD was first discussed a decade ago, the topic gained momentum in 2007 and, since then, a constantly high number of relevant articles have been published every year, as Fig. 3 illustrates. This reflects that, while the idea of integrating UCD and ASD, has been around for some time, many integration issues are still unresolved and research is ongoing. In particular, at least 17 relevant articles have been published since da Silva et al.’s [20] systematic literature was conducted in late 2010, justi- fying an update of their findings. Moreover, we identified the research type of each paper as pre- Parallel tracks Personas Iterative design/development Incremental design/development Fig. 2. Number of publications by research type. Fig. 3. Number of publications by year. 3.3.3. Identification of candidate principles In order to identify candidate principles, each code was investi- patterns. These patterns rep sented in different publicatio to the merger of independen identification of such pattern and the expertise of the rese also the findings of the three to identify themes for potent 4. Results 4.1. Number and distribution o When analyzing the resul the integration of UCD and A the topic gained momentum high number of relevant artic Fig. 2. Number of publications by research type. Fig. 3. Number of publications by year.
- 104. process dimension, manymore codes have been identified for the other two dimensions. Moreover, the codes in the process dimension are mainly specific to software development, while the majority of the codes in the other dimensions are rather gen- eric (e.g. codes like ‘‘collaboration’’ or ‘‘data exchange’’). This might be due to the overall number of papers assigned to the ‘‘process’’ dimension being larger than in the other dimensions. Apart from that, it might also be an indicator of the depth and maturity of dis- course in each of the dimensions. While aspects of process integra- tion have been intensively discussed and refined, the discussion in no principles concerning technology integration were derived. A different challenge emerged for the people/social dimension. Although we found more sources, the recommendations were con- tradictory. While ASD suggests the collective accountability of the team [119], UCD methodologies usually suggest individual respon- sibility, for example, allocating the responsibility of developing a usable product to the UCD expert [122–124]. Furthermore, the team setup led to contradictory evidence. On the one hand, two dedicated teams, which handle design and development activities Table 4 Overview of publications in the final sample. Dimension Included publications Number of publications Process Benigni et al. [39], Budwig et al. [40], Felker et al. [41], Ferreira et al. [14], Ferreira et al. [42], Fox et al. [21], Holzinger et al. [43], Hussain et al. [44], Hussain et al. [45], Kuusinen et al. [46], Lee & McCrickard [47], Lee et al. [48], Lee et al. [49], Losada et al. [50], Losada et al. [51], Losada et al. [52], Memmel et al. [53], Memmel et al. [54], Miller [55], Najafi & Toyoshiba [56], Paelke & Nebe [57], Paelke & Sester [58], Wolkerstorfer et al. [59], Zhang et al. [60] 24 (28.9%) Practices Adikari et al. [61], Beyer et al. [62], Broschinsky & Baker [63], Carvalho [64], Chamberlain et al. [35], Cho [65], Constantine [13], Constantine & Lockwood [66], da Silva et al. [67], Detweiler [68], Düchting et al. [69], Evnin & Pries [70], Ferré et al. [71], Fisher & Bankston [72], Haikara [73], Hansson et al. [74], Hellmann et al. [75], Hellmann et al. [76], Hennings [77], Hodgetts [78], Humayoun et al. [79], Hussain et al. [80], Illmensee & Muff [81], Isomursu et al. [82], Kane [83], Lárusdóttir [84], Lárusdóttir et al. [85], Lee et al. [86], Medina-Medina et al. [87], Memmel et al. [88], Meszaros & Aston [89], Obendorf & Finck [90], Obendorf et al. [91], Patton [92], Patton [93], Petrovic & Siegmann [94], Rafla et al. [95], Rittenbruch et al. [96] 38 (45.8%) People & Social Barksdale & McCrickard [22], Barksdale & McCrickard [97], Barksdale et al. [98], Brown et al. [99], Brown et al. [100], Ferreira et al. [101], Ferreira et al. [102], Kollmann et al. [103], Leszek & Courage [104], Lievesley & Yee [105], Singh [106], Ungar [107], Ungar & White [108], Williams & Ferguson [109] 14 (16.9%) Technology Feiner & Andrews [110], Gonçalves & Santos [111], Hosseini-Khayat et al. [112], Humayoun et al. [113], Lee [114], Peixoto [115], Peixoto [116] 7 (8.4%) Total 83 (100.0%)
- 105. Fig. 4. Finalcoding system for the process, people/social, technology, and practices dimensions. 170 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
- 106. There is evidencethat a limited up-front design effort is particularly necessary to provide a consistent and cohesive user interface and navigation structure as it supports designers in find- ing a suitable design concept from the very beginning [42,61]. Pat- ton [93] states that user needs and goals have to be clearly defined before conducting design and development efforts in order to ensure the usability and usefulness of the software in ASD. Various authors have confirmed this view, presenting user research as an essential aspect of up-front design efforts in agile environments [20,21,24,35,49,55]. In total, 22 (26.5%) of the papers in the review confirm a relationship between LDUF and a cohesive overall design of the final product, which is challenging to achieve in an agile Suggestion. A shift from an up-front design to up-front analysis concept is identified. In order to deliver a product that is valuable to the customer and end-user in that it is both usable and useful, product discovery and product creation should be separated in UCASD. Overall, as listed in Table 5, four codes from our coding sys- tem support our suggestion to include an additional principle in this specific context. Besides making room for sufficient up-front design activities as a prerequisite for delivering a usable product with a consistent user interaction concept, it allows for mitigating agile shortcom- ings with regard to product discovery, fostering the delivery of a useful product. Thus, we formulate the first principle as: Fig. 5. Codes and articles related to the process dimension. M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 171
- 107. not part ofagile method- ed by drawing on UCD, in pectations, and ideas are ude that the usefulness sability concerns not only ring product planning. In e design throughout soft- ns should be included in 05,106]. Table 5 Codes related to the separate product discovery and product creation principle. Included codes Resulting principle Little design up front Separate product discovery and product creation Cycle zero Cohesive overall design Product vision/innovation Sohaib and Khan [23]. Specifically, their reviews confirm the signif- icance of iterative and incremental design and development. There is general agreement among UCD researchers that an iterative approach is essential for developing a product with high usability references to iterative an Scrum) or UCD (e.g. ISO papers do not explicitly approach, none of them ch principle is articulated as Principle 2 (Iterative a ment). User-centered agil design and development i manner. 5.1.3. Parallel Interwoven C Focus Area. Extendin product creation principle quent course of action afte ment activities. As a res design allows for specifyin important features of a sys to be considered in later Table 6 Codes related to the iterative and incremental design and development principle. Included codes Resulting principle Iterative design/development Iterative and incremental design and developmentIncremental design/ development Table 7 Codes related to the parallel interwoven creation tracks principle. Included codes Resulting principle Parallel tracks Parallel interwoven creation tracks Design one sprint ahead Synchronization/integration 172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 Sohaib and Khan [23]. Specifically, their reviews confirm the signif- icance of iterative and incremental design and development. There is general agreement among UCD researchers that an iterative references to iterative a Scrum) or UCD (e.g. ISO papers do not explicitly approach, none of them ch principle is articulated as Principle 2 (Iterative a ment). User-centered agil design and development i manner. 5.1.3. Parallel Interwoven C Focus Area. Extendin product creation principl quent course of action afte ment activities. As a res design allows for specifyin important features of a sy Table 6 Codes related to the iterative and incremental design and development principle. Included codes Resulting principle Iterative design/development Iterative and incremental design and developmentIncremental design/ development Table 7 Codes related to the parallel interwoven creation tracks principle. Included codes Resulting principle Parallel tracks Parallel interwoven creation tracks Design one sprint ahead Synchronization/integration 172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
- 108. need to beadopted to assure the synchronization n of such tracks. ods and user-centered development, Chamberlain et firm the importance of continuous stakeholder in Fig. 6. Codes and number of articles related to the continuous stakeholder involvement principle.
- 109. The value ofindividuals and interactions is documen agile manifesto [134]. According to UCD, understanding their tasks should be the focus of software developm Moreover, UCD and ASD both encourage customer or u pation in the development of new systems or software suggest the following principle: Fig. 7. Codes and number of articles related to the artifact-mediated communication principle. UCASD. Description Separate product discovery and product creation Iterative and incremental design and development Parallel interwoven creation tracks M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
- 110. Fig. 8. Codesand number of articles related to dimension ‘‘Practices’’.