Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Systematic Literature Review

1,345 views

Published on

Introdução sobre Revisão Sistemática da Literatura

Published in: Education
  • Be the first to comment

Systematic Literature Review

  1. 1. Systematic Literature Review Prof. Dr.Tiago Silva da Silva (ICT/UNIFESP) silvadasilva@unifesp.br
  2. 2. Literature Review
  3. 3. Literature Review • Experts can be wrong.
  4. 4. Literature Review • Experts can be wrong. • Researcher’s choice of “related studies” can be biased.
  5. 5. Literature Review • Experts can be wrong. • Researcher’s choice of “related studies” can be biased. • Informal reviews can miss important studies.
  6. 6. Systematic Literature Review • Systematic Review • Medical domain • Evidence-based software engineering (Kitchenham) • “We cannot have evidence-based software engineering without a sound methodology for aggregating evidence from different empirical studies. SRs provide that methodology.”
  7. 7. Systematic Literature Review • The major advantage of a SR is that it is based on a well-defined methodology. • SRs aim to find, assess, and aggregate all relevant evidence about some topic of interest in a fair, repeatable, and auditable manner. • A researcher conducting a SR selects empirical studies that are relevant to the particular research question, assesses the validity of each one, and then determines the trend shown by those studies.
  8. 8. Systematic Literature Review • To investigate all available evidence that supports or refutes a particular “topic of interest”. • To summarize the existing evidence concerning a treatment or technology, e.g. to summarize the empirical evidence of the benefits and limitations of a specific agile method. • To identify any gaps in current research in order to suggest areas for further investigation. • To provide a framework/background in order to appropriately position new research activities. • Sometimes a single researcher, such as a PhD student, needs to do this activity alone, but that limitation raises the risk of missing relevant papers.
  9. 9. Advantages
  10. 10. Advantages • The well-defined methodology makes it less likely that the results of the literature are biased, although it does not protect against publication bias in the primary studies.
  11. 11. Advantages • The well-defined methodology makes it less likely that the results of the literature are biased, although it does not protect against publication bias in the primary studies. • They can provide information about the effects of some phenomenon across a wide range of settings and empirical methods. If studies give consistent results, systematic reviews provide evidence that the phenomenon is robust and transferable. If the studies give inconsistent results, sources of variation can be studied.
  12. 12. Disadvantage
  13. 13. Disadvantage • Requires considerably more effort than traditional literature reviews.
  14. 14. Systematic Literature Review • Planning the review • Conducting the review • Reporting the review
  15. 15. Systematic Literature Review Systematic Review in Software Engineering Technical Report ES 679/05 articles are extracted and synthesized during the result analysis phase. Meanwhile which one of these phases is executed, their results must be stored. Therefore, systematic review packaging is performed through the whole process. There are two checkpoints in the proposed systematic review process. Before executing the systematic review, it is necessary to guarantee that the planning is suitable. The protocol must be evaluated and if problems are found, the researcher must return to the planning stage to review the protocol. Similarly, if problems regarding web search engines are found during the execution phase, the systematic review may be re-executed. Figure 2. Systematic Review Process. The stages listed above may appear to be sequential, but it is important to recognize that many of the stages involve iteration. In particular, many activities are initiated during the protocol development stage, and refined when the review proper takes place. For example [Kitchenham et al., 2002]: Planning Execution Result Analisys Packaging [ protocol plan approved ] [ protocol plan disapproved ] [ execution approved ] [ execution disapproved ] Planning Conducting Reporting
  16. 16. Systematic Literature Review e been iden- d these have dence-based nham et al., technique, ; he question; y (closeness pplicability engineering alues and n executing titute a sys- rder to pro- relevant to systematic d interpret- ar research Watson, 2002), they are less common in software engineer- ing (however, see (Glass et al., 2002) as an example of a sec- ondary study that samples literature within the software engineering domain). Indeed, at the present time, outside of information systems research, reviews in any form, as well as review journals are really not part of the computing 9. Write Review Report 10. Validate Report 2. Develop Review Protocol 3. Validate Review Protocol Phase 1: Plan Review Phase 2: Conduct Review Phase 3: Document Review 8. Synthesise Data 4. Identify Relevant Research 5. Select Primary Studies 7. Extract Required Data 6. Assess Study Quality 1. Specify Research Questions Fig. 1. Systematic literature review process.
  17. 17. Planning the Review 1. Identification of the need for a review 2. Commissioning a review* 3. Specifying the research question(s) 4. Developing a review protocol 5. Evaluating the review protocol*
  18. 18. Conducting the Review 1. Identification of research 2. Selection of primary studies 3. Study quality assessment 4. Data extraction and monitoring 5. Data synthesis
  19. 19. Reporting the Review 1. Specifying dissemination mechanisms 2. Formatting the main report 3. Evaluating the report*
  20. 20. Planning
  21. 21. The need for a systematic review • What are the review’s objectives? • What sources were searched to identify primary studies? Were there any restrictions? • What were the inclusion/exclusion criteria and how were they applied? • What criteria were used to assess the quality of primary studies?
  22. 22. The need for a systematic review • How were quality criteria applied? • How were the data extracted from the primary studies? • How were the data synthesized? • How were differences between studies investigated? • How were the data combined? • Was it reasonable to combine the studies? • Do the conclusions flow from the evidence?
  23. 23. The Research Question(s) • The most important part of any systematic review. The question(s) drive(s) the entire methodology: • The search process must identify primary studies that address the research questions. • The data extraction process must extract the data items needed to answer the questions. • The data analysis process must synthesize the data in such a way that the questions can be answered.
  24. 24. The Research Question(s) • Assessing the effect of a technology. • Assessing the frequency or rate of a project development factor such as the adoption of a technology, or the frequency or rate of project success or failure. • Identifying cost and risk factors associated with a technology. • Identifying the impact of technologies on reliability, performance and cost models. • Cost benefit analysis of employing specific software development technologies or software applications.
  25. 25. The Research Question(s) example • Question 1: What evidence is there that cross-company estimation models are not significantly different from within- company estimation models for predicting effort for software/ Web projects? • Question 2: What characteristics of the study data sets and the data analysis methods used in the study affect the outcome of within- and cross-company effort estimation accuracy studies? • Question 3: Which experimental procedure is most appropriate for studies comparing within- and cross-company estimation models?
  26. 26. The Research Question(s) structure • Population: software or web-project. • Intervention: cross-company project effort estimation model. • Comparison: single-company project effort estimation model. • Outcomes: prediction or estimate accuracy.
  27. 27. Developing a Review Protocol • A review protocol specifies the methods that will be used to undertake a specific systematic review • Background: the rationale for the survey. • The research questions that the review is intended to answer. • The strategy that will be used to search for primary studies including search terms and resources to be searched: digital libraries, specific journals and proceedings.
  28. 28. Developing a Review Protocol • Study selection criteria: used to determine which studies are included in, or excluded from. • Study selection procedures: how the selection criteria will be applied e.g. how many assessors will evaluate each prospective primary study, and how disagreements among assessors will be resolved.
  29. 29. Developing a Review Protocol • Study quality assessment checklists and procedures: quality checklists to assess the individual studies. • Data extraction strategy: how the information required from each primary study will be obtained. • Synthesis of the extracted data: defines the synthesis strategy. • Dissemination strategy
  30. 30. Conducting
  31. 31. Identification of Research • Generating a search strategy • Preliminary searches aimed at both identifying existing systematic reviews and assessing the volume of potentially relevant studies. • Trial searches using various combinations of search terms derived from the research question. • Checking trial research strings against lists of already known primary studies. • Consultations with experts in the field.
  32. 32. Identification of Research • Publication bias • The problem that positive results are more likely to be published than negative results.The concept of positive or negative results sometimes depends on the viewpoint of the researcher. • Scanning the grey literature • Scanning conference proceedings
  33. 33. • Bibliography Management and Document Retrieval • Reference Manager (http://www.refman.com/) • Endnote (http://endnote.com/) • Mendeley (http://www.mendeley.com/) • Papers (http://papersapp.com/mac/) • Documenting the Search (The process of performing a systematic literature review must be transparent and replicable (as far as possible) Identification of Research
  34. 34. Identification of Research literature search. Once reference lists have been finalised the full articles of potentially useful studies will need to be obtained. A logging system is needed to make sure all relevant studies are obtained. 6.1.4 Documenting the Search The process of performing a systematic literature review must be transparent and replicable (as far as possible): The review must be documented in sufficient detail for readers to be able to assess the thoroughness of the search. The search should be documented as it occurs and changes noted and justified. The unfiltered search results should be saved and retained for possible reanalysis. Procedures for documenting the search process are given in Table 2. Table 2 Search process documentation Data Source Documentation Digital Library Name of database Search strategy for the database Date of search Years covered by search Journal Hand Searches Name of journal Years searched Any issues not searched Conference proceedings Title of proceedings Name of conference (if different) Title translation (if necessary) Journal name (if published as part of a journal) Efforts to identify unpublished studies Research groups and researchers contacted (Names and contact details) Research web sites searched (Date and URL) Other sources Date Searched/Contacted URL Any specific conditions pertaining to the search • ACM Digital Library (http:// dl.acm.org/) • IEEE Xplore (http:// ieeexplore.ieee.org/) • Science Direct (http:// www.sciencedirect.com/) • ISI Web of Science (http:// isiknowledge.com/) • Scopus (http://www.scopus.com/) • Google Scholar*
  35. 35. Study Selection • Study selection criteria: to identify those primary studies that provide direct evidence about the research question. Inclusion and exclusion criteria should be based on the research question. • Inclusion: • any study that compared predictions of cross-company models with within-company models based on analysis of single company project data. • Exclusion: • studies where projects were only collected from a small number of different sources (e.g. 2 or 3 companies), • studies where models derived from a within-company data set were compared with predictions from a general cost estimation model.
  36. 36. Study Selection • Study selection process: • Language • Journal • Authors • Setting • Participants or subjects • Research Design • Date of publication • Reliability of inclusion decisions: • When two or more researchers assess each paper, agreement between researchers can be measured using the Cohen Kappa statistic.
  37. 37. Identify relevant studies - automated and manual search Exclude studies on the basis of Title Exclude studies on the basis of Abstract Obtain primary papers and critically appraise studies n papers n papers n papers n papers
  38. 38. Study Quality Assessment • To provide still more detailed inclusion/exclusion criteria. • To investigate whether quality differences provide an explanation for differences in study results. • As a means of weighting the importance of individual studies when results are being synthesized. • To guide the interpretation of findings and determine the strength of inferences. • To guide recommendations for further research.
  39. 39. • Development of Quality Instruments • Design • Conduct • Analysis • Conclusions • Example: 1.Is the data analysis process appropriate? 1.1.Was the data investigated to identify outliers and to assess distributional properties before analysis? 1.2.Was the result of the investigation used appropriately to transform the data and select appropriate data points? Study Quality Assessment
  40. 40. • Less than 10 projects: Poor quality (score=0) • Between 10 and 20 projects: Fair quality (score=0.33) • Between 21 and 40 projects: Good quality (score=0.67) • More than 40 projects: Excellent quality (score=1) Study Quality Assessment example
  41. 41. Data Extraction • Design of Data Extraction Forms • Content of Data Collection Forms Electronic forms are useful and can facilitate subsequent analysis. 6.4.2 Contents of Data Collection Forms In addition to including all the questions needed to answer the review question and quality evaluation criteria, data collection forms should provide standard information including: Name of Reviewer Date of Data extraction Title, authors, journal, publication details Space for additional notes Examples Kitchenham et al. [21] used the extraction form shown in Table 7 (note the actual form also included the quality questions). Table 7 Data Collection form completed for Maxwell et al., 1998 Data item Value Additional notes Data Extractor Data Checker Study Identifier S1 Application domain Space, military and industrial Name of database European Space Agency (ESA) Number of projects in database (including within- company projects) 108 Number of cross-company projects 60 Number of projects in within- company data set 29 Size metric(s): FP (Yes/No) Version used: LOC (Yes/No) Version used: FP: No LOC: Yes (KLOC) Others: No
  42. 42. Data Extraction • Data extraction procedures • Whenever feasible, data extraction should be performed independently by two or more researchers. Data from the researchers must be compared and disagreements resolved either by consensus among researchers or arbitration by an additional independent researcher. • Multiple publications of the same data • It is important not to include multiple publications of the same data in a systematic review synthesis because duplicate reports would seriously bias any results.When there are duplicate publications, the most complete should be used.
  43. 43. Data Synthesis • Unpublished data, missing data and data requiring manipulation • Collating and summarizing the results of the included primary studies.The data synthesis activities should be specified in the review protocol. • Descriptive (Narrative) Synthesis • Extracted information about the studies (i.e. intervention, population, context, sample sizes, outcomes, study quality) should be tabulated in a manner consistent with the review question.
  44. 44. Data Synthesis • Quantitative Synthesis: quantitative data should also be presented in tabular form including: • Sample size for each intervention. • Estimates effect size for each intervention with standard errors for each effect. • Difference between the mean values for each intervention, and the confidence 
 interval for the difference. • Units used for measuring the effect.
  45. 45. Data Synthesis • Presentation of Quantitative Results • Charts x Tables • Qualitative Synthesis • Tries to integrate studies comprising natural language results and conclusions, where different researchers may have used terms and concepts with subtly (or grossly) different meanings • Synthesis of Qualitative and Quantitative Synthesis • Synthesize the quantitative and qualitative studies separately. • Then attempt to integrate the qualitative and quantitative results by investigating whether the qualitative results can help explain the quantitative results. For example qualitative studies can suggest reasons why a treatment does or does not work in specific circumstances.
  46. 46. Data Synthesis • Sensitivity Analysis • High quality primary studies only. • Primary studies of particular types. • Primary studies for which data extraction presented no difficulties (i.e. excluding any studies where there was some residual disagreement about the data extracted). • The experimental method used by the primary studies.
  47. 47. Reporting
  48. 48. Reporting • Specifying the Dissemination Strategy • Journals • Conferences • Formatting the Main Systematic Review Report • Technical Report • Section of a thesis • Journal or conference format • Evaluating Systematic Review Reports
  49. 49. Problems associated with conducting a SLR
  50. 50. Problems associated with conducting a SLR • The protocol may not have identified all the variations in the designing and reporting of relevant primary studies. • Selection of the digital libraries to be searched and whether the search is automated or manual. • A manual search involves looking at the back issues of a set of journals (either paper or online) and deciding from the title and abstract which individual papers are candidates for inclusion in the review. • An automated search uses strings, usually based on complex Boolean formulas, to turn up papers in online catalogs.
  51. 51. (software OR application OR product OR Web OR WWW OR Internet OR World-Wide Web OR project OR development) AND (method OR process OR system OR technique OR methodology OR procedure) AND (cross company OR cross organisation OR cross organization OR cross organizational OR cross organisational OR cross- company OR cross-organisation OR cross- organization OR cross-organizational OR cross-organisational OR multi company OR multi organisation OR multi organization OR multi organizational OR multi organisational OR multi- company OR multi-organisation OR multi- organization OR multi-organizational OR multi-organisational OR multiple company OR multiple organisation OR multiple organization OR multiple organizational OR multiple organisational OR multiple-company OR multiple- organisation OR multiple-organization OR multiple-organizational OR multiple- organisational OR within company OR within organisation OR within organization OR within organizational OR within organisational OR within- company OR within-organisation OR within-organization OR within- organizational OR within-organisational OR single company OR single organisation OR single organization OR single organizational OR single organisational OR single-company OR single-organisation OR single- organization OR single-organizational OR single-organisational OR company- specific) AND (model OR modeling OR modelling) AND (effort OR cost OR resource) AND (estimation OR prediction OR assessment)
  52. 52. Problems associated with conducting a SLR • Digital libraries differ in subtle ways in their implementations of searches: • Digital libraries have different interfaces and different tolerances for the complex Boolean formulas typically used by SR researchers to turn up relevant papers. • Digital libraries also have different procedures for searching the body of the paper, or only the title, abstract, and keywords. In addition, of course, indexing systems only search titles, keywords, and abstracts. • Automated searches from different sources may overlap (i.e., the same paper may be found in different digital libraries), and every search includes a large number of irrelevant papers .
  53. 53. Problems associated with conducting a SLR • Digital libraries differ in subtle ways in their implementations of searches: • (TITLE-ABS-KEY(usability) OR TITLE-ABS-KEY("human-computer interaction") OR TITLE-ABS-KEY("computer-human interaction") OR TITLE-ABS-KEY("human factor") OR TITLE-ABS-KEY("user experience") OR TITLE-ABS-KEY("user-centered design")) AND (TITLE-ABS-KEY(agile) OR TITLE-ABS-KEY("agile method") OR TITLE-ABS-KEY("agile development") OR TITLE-ABS-KEY("agile practice") OR TITLE-ABS-KEY("agile project") OR TITLE-ABS-KEY("agile lifecycle") OR TITLE-ABS-KEY(scrum) OR TITLE-ABS- KEY("extreme programming") OR TITLE-ABS-KEY("lean development") OR TITLE-ABS- KEY("feature driven development") OR TITLE-ABS-KEY("dynamic system development method") OR TITLE-ABS-KEY("agile unified process")) AND PUBYEAR AFT 2000 AND (LIMIT-TO(SUBJAREA, "COMP") OR LIMIT-TO(SUBJAREA, "MULT")) • (usability OR "human-computer interaction" OR "computer-human interaction" OR "human factor" OR "user experience" OR "user-centered design") AND (agile OR "agile method" OR "agile development" OR "agile practice" OR "agile project" OR "agile lifecycle" OR scrum OR "extreme programming" OR "lean development" OR "feature driven development" OR "dynamic system development method" OR "agile unified process")
  54. 54. Problems associated with conducting a SLR • For manual searches: • Select the journals and conference proceedings you intend to search. • Justify your selection of sources. However, rather surprisingly, in one case study we found that a targeted manual search was much quicker than a broad automated search. • In practice, need for a mixed strategy. • If you do a manual search of some sources (including specialist conference proceedings), this should give you a baseline set of candidate primary studies against which you can validate the efficacy of your automated search strings. • Alternatively, a domain expert can identify a baseline set of papers.
  55. 55. Other types of reviews • Systematic Mapping Study: a broad review of primary studies in a specific topic area that aims to identify what evidence is available on the topic. • Tertiary Study: a review of secondary studies related to the same research question.
  56. 56. Lessons Learned e been iden- d these have dence-based nham et al., technique, ; he question; y (closeness pplicability engineering alues and n executing titute a sys- rder to pro- relevant to systematic d interpret- ar research Watson, 2002), they are less common in software engineer- ing (however, see (Glass et al., 2002) as an example of a sec- ondary study that samples literature within the software engineering domain). Indeed, at the present time, outside of information systems research, reviews in any form, as well as review journals are really not part of the computing 9. Write Review Report 10. Validate Report 2. Develop Review Protocol 3. Validate Review Protocol Phase 1: Plan Review Phase 2: Conduct Review Phase 3: Document Review 8. Synthesise Data 4. Identify Relevant Research 5. Select Primary Studies 7. Extract Required Data 6. Assess Study Quality 1. Specify Research Questions Fig. 1. Systematic literature review process.
  57. 57. Lessons Learned • Stage 1: Specify Research Questions • L1: Expect to revise your questions during protocol development, as your understanding of the problem increases. • L2: A pre-review mapping study may help in scoping research questions.
  58. 58. Lessons Learned • Stage 2: Develop Review Protocol • L3: All systematic review team members need to take an active part in developing the review protocol. • L4: Piloting the research protocol is essential. It will find mistakes in your data collection and aggregation procedures. It may also indicate that you need to change the methodology you intend to use to address the research questions.
  59. 59. Lessons Learned • Stage 3:Validate Review Protocol • L5: Data extraction is assisted by having data definitions and data extraction guidelines from the protocol recorded in a separate short document. • L6: There needs to be an agreed validation process separate from the protocol piloting activity. Ideally, external reviewers should undertake this validation process.
  60. 60. Lessons Learned • Stage 4: Identify Relevant Research • L7: There are alternative search strategies that enable you to achieve different sort of search completion criteria.You must select and justify a search strategy that is appropriate for your research question. • L8: We need to search many different electronic sources; no single source finds all of the primary studies. • L9: Current software engineering search engines are not designed to support systematic literature reviews. Unlike medical researchers, software engineering researchers need to perform resource-dependent searches.
  61. 61. Lessons Learned • Stage 5: Select Primary Studies • L10: The standard of IT and software engineering abstracts is too poor to rely on when selecting primary studies.You should also review the conclusions.
  62. 62. Lessons Learned • Stage 6:Assess Study Quality • L11: All the medical standards emphasise that it is necessary to assess the quality of primary studies. However, it depends on the type of systematic literature review you are undertaking. • L12: It is important to be sure how the quality assessment will be used in the subsequent data aggregation and analysis.
  63. 63. Lessons Learned • Stage 7: Extract Required Data • L13:  Having one reader act as data extractor and one act as data checker may be helpful when there are a large number of papers to review. • L14:  Review team members must make sure they understand the protocol and the data extraction process.
  64. 64. Lessons Learned • Stage 8: Synthesise Data • L15:  Software engineering systematic reviews are likely to be qualitative in nature. • L16:  Even when collecting quantitative information it may not be possible to perform meta-analysis of software engineering studies because the reporting protocols vary so much from study to study. • L17:  Tabulating the data is a useful means of aggregation but it is necessary to explain how the aggregated data actually answers the research questions.
  65. 65. Lessons Learned • Stage 9: Write Review Report • L18:  Review teams need to keep a detailed record of decisions made throughout the review process.
 • L19:  The software engineering community needs to establish mechanisms for publishing systematic literature reviews which may result in papers that are longer than those traditionally accepted by many software engineering outlets or that have appendices stored in electronic repositories.
  66. 66. Lessons Learned • Stage 10: Validate Report
  67. 67. References • Kitchenham, B.. (2004). Procedures for Performing Systematic Reviews. Joint Technical Report. • Biolchini, J.; Mian, P. G.; Natali,A. C. C.;Travassos, G. H.T.. (2005) .Systematic Review in Software Engineering.Technical Report. • Kitchneham, B.. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report. • Brereton, P.; Kitchenham, B.; Budgen, D.;Turner, M.; Khalil, M.. (2008). Lessons from applying the systematic literature review process within the software engineering domain.The Journal of Systems and Software. • Dyba ̊,T.; Dingsøyr,T.. (2008). Empirical studies of agile software development:A systematic review. Information and Software Technology. • Kitchenham, B.; Brereton, P.; Budgen, D.;Turner, M.; Bailey, J.; Linkman, S.. (2009). Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology. • Kitchenham, B.; Pretorius, R.; Budgen, D.; Brereton, P.; Turner, M.; Niazi, M.; Linkman, S.. (2010). Systematic literature reviews in software engineering – A tertiary study. Information and Software Technology. • Kitchenham, B.; Mendes, E.;Travassos, G. H.T..(2007) A Systematic Review of Cross- vs.Within- Company Cost Estimation Studies, IEEE Trans on SE.
  68. 68. Systematic Literature Review Prof. Dr.Tiago Silva da Silva (ICT/UNIFESP) silvadasilva@unifesp.br
  69. 69. User-Centered Design and Agile Methods: A Systematic Review Tiago Silva da Silva,Angela Martin, Frank Maurer, Milene Silveira To contact me after my presentation, text BDL to INTRO (46876)
  70. 70. Defining the terms...
  71. 71. Category Keyword UCD Usability Human-Computer Interatcion Computer-Human Interaction Human Factor User Experience User-Centered Design User Interface Agile Agile Agile Method Agile Development Agile Practice Agile Project Agile Lifecycle Scrum Extreme Programming Lean Development Feature Driven Development Dynamic System Development Agile Unified Process
  72. 72. Search String usability OR "human-computer interaction" OR "computer-human interaction" OR "human factor" OR "user experience" OR "user- centered design" OR "user interface" AND agile OR "agile method" OR "agile development" OR "agile practice" OR "agile project" OR "agile lifecycle" OR scrum OR "extreme programming" OR "lean development" OR "feature driven development" OR "dynamic system development" OR "agile unified process"
  73. 73. Inclusion Exclusion
  74. 74. Digital Library Amount of papers First Classification Inclusion Exclusion IEEExplore 59 24 35 Science Direct 4 0 4 CiteSeerX 59 0 59 Scopus 154 24 130 Agile 21 21 0 XP 12 12 0 Total 309 81 228 Percentage 100% 26% 74%
  75. 75. Digital Library Amount of papers Second Classification Total Selected Repeated Not Relevant Total 81 16 7 58 Percentage 100% 20% 8% 72%
  76. 76. 309
  77. 77. 309 81
  78. 78. 309 81 58
  79. 79. Descriptive Information Content-related Information
  80. 80. Descriptive Information
  81. 81. Theoretical Experience Report Empirical Experimental
  82. 82. Focus
  83. 83. Approach
  84. 84. Results
  85. 85. Artefacts, Practices, Needs...
  86. 86. Exploring principles of user-centered agile software development: A literature review Manuel Brhel a , Hendrik Meth b , Alexander Maedche a,b,⇑ , Karl Werder a a Chair of Information Systems IV, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany b Institute for Enterprise Systems, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany a r t i c l e i n f o Article history: Received 5 September 2013 Received in revised form 7 January 2015 Accepted 7 January 2015 Available online 17 January 2015 Keywords: Agile software development User-centered design Systematic literature review a b s t r a c t Context: In the last decade, software development has been characterized by two major approaches: agile software development, which aims to achieve increased velocity and flexibility during the development process, and user-centered design, which places the goals and needs of the system’s end-users at the cen- ter of software development in order to deliver software with appropriate usability. Hybrid development models, referred to as user-centered agile software development (UCASD) in this article, propose to com- bine the merits of both approaches in order to design software that is both useful and usable. Objective: This paper aims to capture the current state of the art in UCASD approaches and to derive gen- eric principles from these approaches. More specifically, we investigate the following research question: Which principles constitute a user-centered agile software development approach? Method: We conduct a systematic review of the literature on UCASD. Identified works are analyzed using a coding scheme that differentiates four levels of UCASD: the process, practices, people/social and tech- nology dimensions. Through subsequent synthesis, we derive generic principles of UCASD. Results: We identified and analyzed 83 relevant publications. The analysis resulted in a comprehensive coding system and five principles for UCASD: (1) separate product discovery and product creation, (2) iterative and incremental design and development, (3) parallel interwoven creation tracks, (4) continuous stakeholder involvement, and (5) artifact-mediated communication. Conclusion: Our paper contributes to the software development body of knowledge by (1) providing a broad overview of existing works in the area of UCASD, (2) deriving an analysis framework (in form a cod- ing system) for works in this area, going beyond former classifications, and (3) identifying generic prin- ciples of UCASD and associating them with specific practices and processes. Ó 2015 Published by Elsevier B.V. Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 2. Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 2.1. Summary of existing literature reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 2.2. Gap analysis of existing literature reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 3. Research method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Information and Software Technology 61 (2015) 163–181 Contents lists available at ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof
  87. 87. (4) A data extraction and synthesis method, representing the sys- tematic approach to consolidate and integrate existing knowledge. The review protocol was developed in cooperation with the first and second authors, and validated by the third author prior to con- ducting the review. In the following, we describe the implementa- tion of the review protocol. in this stage as depicted in Fig. 1. In the first stage, relevant publi- cations were retrieved by querying the databases mentioned above with the search string depicted in Table 1. All database queries were made in the first two weeks of October 2012. This yielded a total of 1152 initial results, and 1034 publications after the removal of duplicates, which were included in the second stage. In the second stage, publications were excluded based on their titles and abstract. Criteria for inclusion in the following stage were Table 1 Composition of the search string. AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design, interface design OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development OR Software development, systems development Fig. 1. Selection process (based on [31]). 166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 (4) A data extraction and synthesis method, representing the sys- in this stage as depicted in Fig. 1. In the first stage, relevant publi- Table 1 Composition of the search string. AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design, interface design OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software development OR Software development, systems development Fig. 1. Selection process (based on [31]). 166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
  88. 88. on these criteria, 74 articles were considered relevant for inclusion in the next stage. Following the strategy suggested by Webster and Watson [36], we conducted a forward and backward search based on these key publications. The retrieved new and potentially rele- vant articles were fed to the second stage for further processing, resulting in an iterative extension of the sample. This resulted in an additional set of nine articles for inclusion in stage four. It has to be noted that both da Silva et al.’s [20] and Sohaib and Khan’s [23] reviews were included in the stage three sample. However, they were excluded from the detailed analysis in stage four as they do not present the results of primary research. In total, 83 papers were selected for the final sample.1 In the fourth stage, the final sample passed a first categorization and a quality assessment. During the categorization, each paper was assigned to one of the four research foci, which will be intro- duced in the next section. A quality assessment of each paper was subsequently conducted to obtain additional information support- ing the interpretation of the paper’s recommendations. This assess- ment was based on four questions, which are given in Table 2. Each of the four criteria was graded on a dichotomous scale as ‘‘yes’’ or ‘‘no,’’ while question 3b was answered by means of the respective method, which was either explicitly stated in the paper or deter- mined by the researcher based on the available information. 3.3. Paper analysis Barksdale and McCrickard [22], we did no people integration and social integration, a aspects almost inseparable. 3.3.2. Identification of codes We used the qualitative data analysis code the papers. As a universal tool for qu MAXQDA is usually employed to analyze tex view transcripts, and supports a variety of qu ods. Various authors (e.g. [31,37,38]) h experiences concerning the use of similar s synthesize data from a corpus of texts eme literature review. Motivated by these exper approach to process a large amount of text transparently. This seems especially usefu extant literature, which may be used to d retrieved easily in later stages of the resear QDA, codes can be assigned to text segme ments. Besides descriptive statistics on the documents, MAXQDA provides a convenien text passages, allowing for easy data synthe allowed us to conduct a quality analysis of be able to retrieve papers within a certain c manner, the categorization was done by ass the appropriate category to the title and abs thermore, if a criterion given in Table 2 was the text segment providing the information The analysis of the document corpus wa aspects that have to be taken into account and ASD, with the overarching aim of der the analysis of the document corpus, observ different aspects of integration were assign with the aim of consolidating the perspectiv on the same issue later on. The dimensions introduced above forme of high-level codes. For simplification reas ‘‘process,’’ ‘‘practices,’’ ‘‘people/social,’’ and level codes in the following. Based on our a Table 2 Criteria for quality assessment. 1. Is there a clear statement of the research goals (e.g. in an explicitly verbalized research question)? 2. Is there an adequate description of the context in which the research was carried out? 3. (Only applicable to empirical research papers): a. Is the research method explicitly stated? b. Which research method was chosen? 4. Are there explicit recommendations, which could be aggregated as principles? M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 Table 3 Initial coding system. Level 1 Process Practices People/social Technology Level 2 Little design up front Prototyping Close collaboration Data exchange User testing⁄ User testing⁄ Cross-functionality IDE integration One sprint ahead User stories Knowledge transfer Cohesive overall design Scenarios Parallel tracks Personas Iterative design/development Incremental design/development Fig. 2. Number of publications by research type. 168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
  89. 89. 3.3.3. Identification of candidate principles In order to identify candidate principles, each code was investi- gated concerning two characteristics. First, the frequency of occur- rence was assessed to gain an impression of the relative importance of each aspect. As an example, the code ‘‘prototyping’’ was assigned in 40 (48.2%) articles in the literature review, indicat- ing that the use of prototypes was discussed in this document. In contrast, the idea of ‘‘remote usability testing’’ was only found in one (1.2%) article, leading to the conclusion that, while the former patterns. These patterns represent related ideas or concepts pre- sented in different publications along the four dimensions, leading to the merger of independent aspects in the coding system. The identification of such patterns was based on a qualitative analysis and the expertise of the researchers in this domain. In this step, also the findings of the three assessed reviews were drawn upon to identify themes for potential principles [33]. 4. Results 4.1. Number and distribution of publications When analyzing the result set, it became apparent that, while the integration of UCD and ASD was first discussed a decade ago, the topic gained momentum in 2007 and, since then, a constantly high number of relevant articles have been published every year, as Fig. 3 illustrates. This reflects that, while the idea of integrating UCD and ASD, has been around for some time, many integration issues are still unresolved and research is ongoing. In particular, at least 17 relevant articles have been published since da Silva et al.’s [20] systematic literature was conducted in late 2010, justi- fying an update of their findings. Moreover, we identified the research type of each paper as pre- Parallel tracks Personas Iterative design/development Incremental design/development Fig. 2. Number of publications by research type. Fig. 3. Number of publications by year. 3.3.3. Identification of candidate principles In order to identify candidate principles, each code was investi- patterns. These patterns rep sented in different publicatio to the merger of independen identification of such pattern and the expertise of the rese also the findings of the three to identify themes for potent 4. Results 4.1. Number and distribution o When analyzing the resul the integration of UCD and A the topic gained momentum high number of relevant artic Fig. 2. Number of publications by research type. Fig. 3. Number of publications by year.
  90. 90. process dimension, many more codes have been identified for the other two dimensions. Moreover, the codes in the process dimension are mainly specific to software development, while the majority of the codes in the other dimensions are rather gen- eric (e.g. codes like ‘‘collaboration’’ or ‘‘data exchange’’). This might be due to the overall number of papers assigned to the ‘‘process’’ dimension being larger than in the other dimensions. Apart from that, it might also be an indicator of the depth and maturity of dis- course in each of the dimensions. While aspects of process integra- tion have been intensively discussed and refined, the discussion in no principles concerning technology integration were derived. A different challenge emerged for the people/social dimension. Although we found more sources, the recommendations were con- tradictory. While ASD suggests the collective accountability of the team [119], UCD methodologies usually suggest individual respon- sibility, for example, allocating the responsibility of developing a usable product to the UCD expert [122–124]. Furthermore, the team setup led to contradictory evidence. On the one hand, two dedicated teams, which handle design and development activities Table 4 Overview of publications in the final sample. Dimension Included publications Number of publications Process Benigni et al. [39], Budwig et al. [40], Felker et al. [41], Ferreira et al. [14], Ferreira et al. [42], Fox et al. [21], Holzinger et al. [43], Hussain et al. [44], Hussain et al. [45], Kuusinen et al. [46], Lee & McCrickard [47], Lee et al. [48], Lee et al. [49], Losada et al. [50], Losada et al. [51], Losada et al. [52], Memmel et al. [53], Memmel et al. [54], Miller [55], Najafi & Toyoshiba [56], Paelke & Nebe [57], Paelke & Sester [58], Wolkerstorfer et al. [59], Zhang et al. [60] 24 (28.9%) Practices Adikari et al. [61], Beyer et al. [62], Broschinsky & Baker [63], Carvalho [64], Chamberlain et al. [35], Cho [65], Constantine [13], Constantine & Lockwood [66], da Silva et al. [67], Detweiler [68], Düchting et al. [69], Evnin & Pries [70], Ferré et al. [71], Fisher & Bankston [72], Haikara [73], Hansson et al. [74], Hellmann et al. [75], Hellmann et al. [76], Hennings [77], Hodgetts [78], Humayoun et al. [79], Hussain et al. [80], Illmensee & Muff [81], Isomursu et al. [82], Kane [83], Lárusdóttir [84], Lárusdóttir et al. [85], Lee et al. [86], Medina-Medina et al. [87], Memmel et al. [88], Meszaros & Aston [89], Obendorf & Finck [90], Obendorf et al. [91], Patton [92], Patton [93], Petrovic & Siegmann [94], Rafla et al. [95], Rittenbruch et al. [96] 38 (45.8%) People & Social Barksdale & McCrickard [22], Barksdale & McCrickard [97], Barksdale et al. [98], Brown et al. [99], Brown et al. [100], Ferreira et al. [101], Ferreira et al. [102], Kollmann et al. [103], Leszek & Courage [104], Lievesley & Yee [105], Singh [106], Ungar [107], Ungar & White [108], Williams & Ferguson [109] 14 (16.9%) Technology Feiner & Andrews [110], Gonçalves & Santos [111], Hosseini-Khayat et al. [112], Humayoun et al. [113], Lee [114], Peixoto [115], Peixoto [116] 7 (8.4%) Total 83 (100.0%)
  91. 91. Fig. 4. Final coding system for the process, people/social, technology, and practices dimensions. 170 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
  92. 92. There is evidence that a limited up-front design effort is particularly necessary to provide a consistent and cohesive user interface and navigation structure as it supports designers in find- ing a suitable design concept from the very beginning [42,61]. Pat- ton [93] states that user needs and goals have to be clearly defined before conducting design and development efforts in order to ensure the usability and usefulness of the software in ASD. Various authors have confirmed this view, presenting user research as an essential aspect of up-front design efforts in agile environments [20,21,24,35,49,55]. In total, 22 (26.5%) of the papers in the review confirm a relationship between LDUF and a cohesive overall design of the final product, which is challenging to achieve in an agile Suggestion. A shift from an up-front design to up-front analysis concept is identified. In order to deliver a product that is valuable to the customer and end-user in that it is both usable and useful, product discovery and product creation should be separated in UCASD. Overall, as listed in Table 5, four codes from our coding sys- tem support our suggestion to include an additional principle in this specific context. Besides making room for sufficient up-front design activities as a prerequisite for delivering a usable product with a consistent user interaction concept, it allows for mitigating agile shortcom- ings with regard to product discovery, fostering the delivery of a useful product. Thus, we formulate the first principle as: Fig. 5. Codes and articles related to the process dimension. M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 171
  93. 93. not part of agile method- ed by drawing on UCD, in pectations, and ideas are ude that the usefulness sability concerns not only ring product planning. In e design throughout soft- ns should be included in 05,106]. Table 5 Codes related to the separate product discovery and product creation principle. Included codes Resulting principle Little design up front Separate product discovery and product creation Cycle zero Cohesive overall design Product vision/innovation Sohaib and Khan [23]. Specifically, their reviews confirm the signif- icance of iterative and incremental design and development. There is general agreement among UCD researchers that an iterative approach is essential for developing a product with high usability references to iterative an Scrum) or UCD (e.g. ISO papers do not explicitly approach, none of them ch principle is articulated as Principle 2 (Iterative a ment). User-centered agil design and development i manner. 5.1.3. Parallel Interwoven C Focus Area. Extendin product creation principle quent course of action afte ment activities. As a res design allows for specifyin important features of a sys to be considered in later Table 6 Codes related to the iterative and incremental design and development principle. Included codes Resulting principle Iterative design/development Iterative and incremental design and developmentIncremental design/ development Table 7 Codes related to the parallel interwoven creation tracks principle. Included codes Resulting principle Parallel tracks Parallel interwoven creation tracks Design one sprint ahead Synchronization/integration 172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 Sohaib and Khan [23]. Specifically, their reviews confirm the signif- icance of iterative and incremental design and development. There is general agreement among UCD researchers that an iterative references to iterative a Scrum) or UCD (e.g. ISO papers do not explicitly approach, none of them ch principle is articulated as Principle 2 (Iterative a ment). User-centered agil design and development i manner. 5.1.3. Parallel Interwoven C Focus Area. Extendin product creation principl quent course of action afte ment activities. As a res design allows for specifyin important features of a sy Table 6 Codes related to the iterative and incremental design and development principle. Included codes Resulting principle Iterative design/development Iterative and incremental design and developmentIncremental design/ development Table 7 Codes related to the parallel interwoven creation tracks principle. Included codes Resulting principle Parallel tracks Parallel interwoven creation tracks Design one sprint ahead Synchronization/integration 172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
  94. 94. need to be adopted to assure the synchronization n of such tracks. ods and user-centered development, Chamberlain et firm the importance of continuous stakeholder in Fig. 6. Codes and number of articles related to the continuous stakeholder involvement principle.
  95. 95. The value of individuals and interactions is documen agile manifesto [134]. According to UCD, understanding their tasks should be the focus of software developm Moreover, UCD and ASD both encourage customer or u pation in the development of new systems or software suggest the following principle: Fig. 7. Codes and number of articles related to the artifact-mediated communication principle. UCASD. Description Separate product discovery and product creation Iterative and incremental design and development Parallel interwoven creation tracks M. Brhel et al. / Information and Software Technology 61 (2015) 163–181
  96. 96. Fig. 8. Codes and number of articles related to dimension ‘‘Practices’’.

×