Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective

320 views

Published on

Continuous Integration (CI) and Continuous Delivery
(CD) are widespread in both industrial and open-source
software (OSS) projects. Recent research characterized build
failures in CI and identified factors potentially correlated to them. However, most observations and findings of previous work are exclusively based on OSS projects or data from a single industrial organization. This paper provides a first attempt to compare the CI processes and occurrences of build failures in 349 Java OSS projects and 418 projects from a financial organization, ING Nederland. Through the analysis of 34,182 failing builds (26% of the total number of observed builds), we derived a taxonomy of failures that affect the observed CI processes. Using cluster analysis, we observed that in some cases OSS and ING projects share similar build failure patterns (e.g., few compilation failures as compared to frequent testing failures), while in other cases completely different patterns emerge. In short, we explain how OSS and ING CI processes exhibit commonalities, yet are substantially different in their design and in the failures they report.

  • Be the first to comment

  • Be the first to like this

A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective

  1. 1. A tale of ci build failures: an Open Source and a Financial Organization Perspective Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, Sebastiano Panichella 1 @ccvassallovassallo@ifi.uzh.ch
  2. 2. Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. (Martin Fowler) 2
  3. 3. Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. (Martin Fowler) 3
  4. 4. Software building: Continuous Integration 4 4 Build(CI)Source
  5. 5. Software building: Continuous Integration 5 5 Build(CI)Source
  6. 6. • On average, a build failure takes 57 min to fix. • The overall cost of build failures ranging from 904.64 to 2034.92 man-hours (over a period of 6 months). • They monitored roughly 7200 man-hours. . . Kerzazi, F. Khomh, and B. Adams, Why do automated builds break? an empirical study, in 30th IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 41–50, IEEE, 2014. 
 Relevance of Build Breakage 6 6
  7. 7. Build breaks types 7 Failing tests are the dominant reason for unsuccessful builds (Rausch et al., MSR 2017) (Beller et al., MSR 2017) IndustryOSS 40% of the failures occur during static analysis (Miller et al., AGILE 2008) Dependencies between components are the most relevant cause of compilation related failures (Seo et al., ICSE 2014) 7
  8. 8. 8 What are differences and commonalities in the distribution of build failure types occurring in OSS and industry? 8
  9. 9. 9 9
  10. 10. 10 RQ1 What types of failures affect builds of OSS and industrial projects? RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? 10
  11. 11. Data Selection 11 • 418 Maven (mostly Java) projects • 12,871 builds of which 3,390 (≈ 26%) failed. • 349 Maven (Java) projects • 116,741 builds, of which 30,792 (≈ 26%) failed 11 IndustryOSS
  12. 12. 12 Data Selection • 418 Maven (mostly Java) projects • 12,871 builds of which 3,390 (≈ 26%) failed. • 349 Maven (Java) projects • 116,741 builds, of which 30,792 (≈ 26%) failed IndustryOSS Build failure logs were the only resources we could access 12
  13. 13. 13 Data Preprocessing 13
  14. 14. 14 Data Preprocessing 14 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project es-common: Compilation failure [ERROR] /home/travis/build/zhangkaitao/es/common/src/main/java/com/sishuok/es/common/utils/html/HTMLUtils.java:[14,8] class HtmlUtils is public, should be declared in a file named HtmlUtils.java [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
  15. 15. Keyword grouping Build Log classification Result Validation ! " # $ Build Failure Catalog 15 15 CODE ANALYSIS (STATIC) org.codehaus.mojo:sonar-maven-plugin:sonar org.codehaus.mojo:findbugs-maven-plugin:findbugs Inter-rater agreement • Industry: k=0.8 (strong agreement) • OSS: k=0.62 (strong agreement)
  16. 16. 16 CLEAN VALIDATION PRE-PROCESSING (RESOURCES) COMPILATION TESTING PACKAGING DOCUMENTATION SUPPORT EXTERNAL TASKS CODE ANALYSIS RELEASE PREPARATION DEPLOYMENT DEPENDENCIES PRODUCTION TEST SUPPORT UNIT TESTING INTEGRATION TESTING NON FUNCTIONAL TESTING CROSSCUTTING STATIC DYNAMIC LOCAL REMOTE RQ1 What type of failures affect builds of OSS and industrial projects? 16
  17. 17. 17 Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? Industry 17
  18. 18. 18 Compilation (production) Compilation (test) Compilation (support) % of build failures 0% 5% 10% 15% 20% 25% 0.2% 1.8% 7.1% 0.0% 2.3% 4.2% ING OSS Compilation Errors are fairly limited Industry RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? 18
  19. 19. 19 Testing (unit) Testing (integration) Testing (non functional) % of build failures 0% 5% 10% 15% 20% 25% 0.0% 5.0% 28.0% 2.7% 13.3% 5.2% ING OSS OSS projects exhibit more unit than integration testing related failures. In industry it’s the opposite. Early discovery of non-functional failures in industry. RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? Industry 19
  20. 20. 20 Static Analysis % of build failures 0% 5% 10% 15% 20% 25% 4.2% 16.4% ING OSS Static Analysis tools: on CI server in OSS, remotely in industry. RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? Industry 20
  21. 21. 21 Deployment (Local) Deployment (Remote) Release Preparation % of build failures 0% 5% 10% 15% 20% 25% 0.0% 0.5% 0.0% 21.1% 10.0% 0.4% ING OSS Release preparation and deployment failures are very common in industry, less so in OSS RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? Industry 21
  22. 22. 22 • Projects clustering (using K-means algorithm) • Optimal value of silhouette statistic: 6 • Each cluster dominated either by industrial or oss projects • except Dependencies #projects 0 45 90 135 180 Cluster CodeAnalysis ReleasePreparation Dependencies CrosscuttingTesting UnitTesting Compilation ING OSS 25% 75% 100% 59% 41% 85% 31% 69% 94% 6% 15% 22 RQ2 How frequent are the different types of build failures in the observed OSS and industrial project? Industry
  23. 23. 23 • Dependencies related failures occur with the same frequency. Key findings • In OSS projects a lot of failures are due to unit testing: try to catch those issues earlier! • Need for a better release/deployment strategy in OSS. • Static analysis on separate server: well collected data and less overloading of CI server. • Towards early discovery of non functional testing failures. 23
  24. 24. @ccvassallovassallo@ifi.uzh.ch 24 Build breaks types X Failing tests are the dominant reason for unsuccessful builds (Rausch et al., MSR 2017) (Beller et al., MSR 2017) IndustryOSS 40% of the failures occur during static analysis (Miller et al., AGILE 2008) Dependencies between components are the most relevant cause of compilation related failures (Seo et al., ICSE 2014) X X RQ1 What types of failures affect builds of OSS and industrial projects? RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? X X CLEAN VALIDATION PRE-PROCESSING (RESOURCES) COMPILATION TESTING PACKAGING DOCUMENTATION SUPPORT EXTERNAL TASKS CODE ANALYSIS RELEASE PREPARATION DEPLOYMENT DEPENDENCIES PRODUCTION TEST SUPPORT UNIT TESTING INTEGRATION TESTING NON FUNCTIONAL TESTING CROSSCUTTING STATIC DYNAMIC LOCAL REMOTE RQ1 What type of failures affect builds of OSS and industrial projects? X X Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS RQ2 How frequent are the different types of build failures in the observed OSS and industrial projects? Industry X

×