SlideShare a Scribd company logo
1 of 22
Download to read offline
An Empirical Analysis of Build Failures in the
Continuous Integration Worfklows
of Java-Based Open-Source Software
Thomas Rausch, Waldemar Hummer, Philipp Leitner*, Stefan Schulte
Distributed Systems Group
Vienna University of Technology, Austria
http://dsg.tuwien.ac.at
* Software Evolution and Architecture Lab
University of Zurich, Switzerland
http://www.ifi.uzh.ch/en/seal.html
2
Continuous Integration
VCS
CI Server Build
Feedback
Logs
Vasilescu et al. (2015).
Quality and Productivity Outcomes Relating to
Continuous Integration in GitHub
“Our main finding is that continuous
integration improves the productivity of project
teams”
Kerzazi et al. (2014).
Why do Automated Builds Break? An Empirical Study
“We [...] quantified the cost of such build
breakage as more than 336.18 man-hours”
3
4
Related Work
5
Understanding Build Failures
What types of errors cause CI build failures?
Which development practices can be
associated with CI build failures?
6
Research Setting
Project Name Description
Apache Storm Distributed Computation
Butterknife Android Dependency Injection
Crate.IO Scalable SQL database
JabRef BibTeX management GUI
jcabi-github Wrapper of GitHub API
Hystrix Latency and fault tolerance library
Presto Distributed SQL query engine
Openmicroscopy Microscopy data environment
RxAndroid RxJava bindings for Android
Sponge API Minecraft plugin API
Spring Boot Java Application Framework
Square OkHttp HTTP+HTTP/2 client for Android
Square Retofit HTTP client for Android
Wordpress-Android WordPress for Android
7
Data Acquisition
a
b
c
d
Topology Mapping
CI build history
Change history
8
Understanding Build Failures
What types of errors cause CI build failures?
Which development practices can be
associated with CI build failures?
9
Error Categorization and Quantification
 Goal
○ Categorization of errors
○ Frequency of occurrence of error types

Approach
○ Systematic exploration of ~54 000 logfiles
○ Categorization scheme based on log message patterns
[INFO] Compiling 67 source files to /home/travis/.../target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/.../redis/RedisAutoConfiguration.java:[143,10] cannot find symbol
[INFO] 1 error
[INFO] Compiling 67 source files to /home/travis/.../target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/.../redis/RedisAutoConfiguration.java:[143,10] cannot find symbol
[INFO] 1 error
[INFO] Compiling 67 source files to /home/travis/.../target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/.../redis/RedisAutoConfiguration.java:[143,10] cannot find symbol
[INFO] 1 error
10
Error Categories
unknown Errors without a clearly identifiable cause 9
itestfailure An automated integration test failed 4
doc Documentation (e.g., JavaDoc) problem 3
license License criteria not met (missing header) 3
compatibility API incompatibility 2
androidsdk Android SDK-related error 1
buildout Error specific to Crate.IO python module 1
Label Description Occurrences
testfailure An automated test failed 12
compile Compilation error 12
git VCS interaction error 12
buildconfig Faulty build config 11
crash Build environment crash or timeout 11
dependency Dependency error 11
quality Coding-rule violation (e.g., Checkstyle) 10
11
Distribution of Common Error Types
Faulty VCS
interaction
Faulty build
configuration Dependency
error
Compilation
error
Coding-rule
violation
Failing test
Crash
40%
30%
20%
10%
0%
12
Distribution of Common Error Types
Apache Storm
Butterknife
Crate.IO
Hystrix
Error
testfailure
compile
git dependency crash
buildconfig quality others
Percentage
JabRef
jcabi-github
Presto
RxAndroid
SpongeAPI
Spring Boot
Square OkHttp
Square Retrofit
0% 25% 50% 75% 100%
13
Understanding Build Failures
What types of errors cause CI build failures?
Which development practices can be
associated with CI build failures?
14
Change Metrics
.java .txt
Changes
 Complexity
○ Churn, number of files, ...

File types
○ README.txt vs.
IntegrationTest.java
 Date and time

Author
○ Experience, commit
frequency, ...
15
Process Metrics
b1
b2
b4
t
b3
a
b
e
c
f
d
g
VCS
commit graph
CI build
information
 Build History
○ Build climate
 Build Type
○ Pull request, merge, ...

Pull Request Scenarios
○ Rebase, squash, ...
16
Statistical Correlation Analysis
 For each project individually
 Non-parametric correlation tests
○ Pearson’s chi-square test
○ Mann—Whitney U test

Calculate effect sizes
○ Cramér’s V
○ Rank-biserial correlation
17
PassedBuild outcome Failed
Failed Passed
Previous build result
Percentageofbuilds
Findings
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build history
b
b’
18
PassedBuild outcome Failed
Failed Passed
Previous build result
Percentageofbuilds
Findings
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build history
b
b’
19
Findings
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build history
No evidence that either history manipula-
tion operations or parallel development
to a PR affect the PR’s build outcome.
No evidence that either history manipula-
tion operations or parallel development
to a PR affect the PR’s build outcome.
Pull request scenarios
20
Findings
Even objectively harmless changes can
break builds. This indicates unwanted
flakiness of tests or the build environment.
Even objectively harmless changes can
break builds. This indicates unwanted
flakiness of tests or the build environment.
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
Build failures mostly occur consecutively.
Phases of build instability perpetuate
failures.
File types
Build history
 577 builds from Spring Boot
 Changelog file change only
 14% original failures
○ 52% test failures
○ 45% environment crash
○ 3% dependency error
No evidence that either history manipula-
tion operations or parallel development
to a PR affect the PR’s build outcome.
No evidence that either history manipula-
tion operations or parallel development
to a PR affect the PR’s build outcome.
Pull request scenarios
21
Summary
 Categorization of error types (beyond failed/errored)
 Quantification of error type occurrence
 Statistical analysis of impact factors
 Uncovered challenges that arise when mining CI data
22
Dipl.-Ing.
Thomas Rausch
Research Assistant
TU Wien
Distributed Systems Group
Argentinierstraße 8/184-1, 1040, Vienna, Austria
T: +43 1 58801 184 838
E: rausch@dsg.tuwien.ac.at
dsg.tuwien.ac.at/staff/trausch

More Related Content

Similar to An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software

Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...
Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...
Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...Keheliya Gallaba
 
Part 2 improving your software development v1.0
Part 2   improving your software development v1.0Part 2   improving your software development v1.0
Part 2 improving your software development v1.0Jasmine Conseil
 
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...Gene Kim
 
Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...
Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...
Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...Atlassian
 
Continuous Integration (Jenkins/Hudson)
Continuous Integration (Jenkins/Hudson)Continuous Integration (Jenkins/Hudson)
Continuous Integration (Jenkins/Hudson)Dennys Hsieh
 
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...DevOps Enterprise Summmit
 
Continuous Everything
Continuous EverythingContinuous Everything
Continuous EverythingAndrea Tino
 
Delivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous IntegrationDelivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous IntegrationAspire Systems
 
DevOps for Your Mobile App
DevOps for Your Mobile AppDevOps for Your Mobile App
DevOps for Your Mobile AppSeth Valdetero
 
PVS-Studio for Linux (CoreHard presentation)
PVS-Studio for Linux (CoreHard presentation)PVS-Studio for Linux (CoreHard presentation)
PVS-Studio for Linux (CoreHard presentation)Andrey Karpov
 
Continuous Delivery Applied
Continuous Delivery AppliedContinuous Delivery Applied
Continuous Delivery AppliedExcella
 
Continuous Delivery Applied (Agile Richmond)
Continuous Delivery Applied (Agile Richmond)Continuous Delivery Applied (Agile Richmond)
Continuous Delivery Applied (Agile Richmond)Mike McGarr
 
Continuous delivery for databases
Continuous delivery for databasesContinuous delivery for databases
Continuous delivery for databasesDevOpsGroup
 
Enabling Continuous Integration with Azure Pipelines
Enabling Continuous Integration with Azure PipelinesEnabling Continuous Integration with Azure Pipelines
Enabling Continuous Integration with Azure PipelinesEng Teong Cheah
 
havcs-410-101 a-2-10-srt-pg_2
havcs-410-101 a-2-10-srt-pg_2havcs-410-101 a-2-10-srt-pg_2
havcs-410-101 a-2-10-srt-pg_2raryal
 
Continous Integration: A Case Study
Continous Integration: A Case StudyContinous Integration: A Case Study
Continous Integration: A Case StudyTalentica Software
 
A Continuous Delivery Safety Net for Databases
A Continuous Delivery Safety Net for DatabasesA Continuous Delivery Safety Net for Databases
A Continuous Delivery Safety Net for DatabasesIBM UrbanCode Products
 
Flight East 2018 Presentation–Continuous Integration––An Overview
Flight East 2018 Presentation–Continuous Integration––An OverviewFlight East 2018 Presentation–Continuous Integration––An Overview
Flight East 2018 Presentation–Continuous Integration––An OverviewSynopsys Software Integrity Group
 

Similar to An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software (20)

Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...
Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...
Noise and Heterogeneity in Historical Build Data: An Empirical Study of Travi...
 
Azure from scratch part 4
Azure from scratch part 4Azure from scratch part 4
Azure from scratch part 4
 
Part 2 improving your software development v1.0
Part 2   improving your software development v1.0Part 2   improving your software development v1.0
Part 2 improving your software development v1.0
 
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
 
Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...
Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...
Build it, Test it, Ship it: Continuous Delivery at Turner Broadcasting System...
 
Continuous Integration (Jenkins/Hudson)
Continuous Integration (Jenkins/Hudson)Continuous Integration (Jenkins/Hudson)
Continuous Integration (Jenkins/Hudson)
 
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
 
Continuous Everything
Continuous EverythingContinuous Everything
Continuous Everything
 
Delivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous IntegrationDelivering Quality Software with Continuous Integration
Delivering Quality Software with Continuous Integration
 
DevOps for Your Mobile App
DevOps for Your Mobile AppDevOps for Your Mobile App
DevOps for Your Mobile App
 
PVS-Studio for Linux (CoreHard presentation)
PVS-Studio for Linux (CoreHard presentation)PVS-Studio for Linux (CoreHard presentation)
PVS-Studio for Linux (CoreHard presentation)
 
Continuous Delivery Applied
Continuous Delivery AppliedContinuous Delivery Applied
Continuous Delivery Applied
 
Continuous Delivery Applied (Agile Richmond)
Continuous Delivery Applied (Agile Richmond)Continuous Delivery Applied (Agile Richmond)
Continuous Delivery Applied (Agile Richmond)
 
Continuous delivery for databases
Continuous delivery for databasesContinuous delivery for databases
Continuous delivery for databases
 
Continuous Delivery Applied
Continuous Delivery AppliedContinuous Delivery Applied
Continuous Delivery Applied
 
Enabling Continuous Integration with Azure Pipelines
Enabling Continuous Integration with Azure PipelinesEnabling Continuous Integration with Azure Pipelines
Enabling Continuous Integration with Azure Pipelines
 
havcs-410-101 a-2-10-srt-pg_2
havcs-410-101 a-2-10-srt-pg_2havcs-410-101 a-2-10-srt-pg_2
havcs-410-101 a-2-10-srt-pg_2
 
Continous Integration: A Case Study
Continous Integration: A Case StudyContinous Integration: A Case Study
Continous Integration: A Case Study
 
A Continuous Delivery Safety Net for Databases
A Continuous Delivery Safety Net for DatabasesA Continuous Delivery Safety Net for Databases
A Continuous Delivery Safety Net for Databases
 
Flight East 2018 Presentation–Continuous Integration––An Overview
Flight East 2018 Presentation–Continuous Integration––An OverviewFlight East 2018 Presentation–Continuous Integration––An Overview
Flight East 2018 Presentation–Continuous Integration––An Overview
 

More from Thomas Rausch

Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...Thomas Rausch
 
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...Thomas Rausch
 
Towards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AITowards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AIThomas Rausch
 
Edge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AIEdge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AIThomas Rausch
 
Portable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge ComputersPortable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge ComputersThomas Rausch
 
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing ApplicationsEMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing ApplicationsThomas Rausch
 
Message-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing ApplicationsMessage-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing ApplicationsThomas Rausch
 
Build Failure Prediction in Continuous Integration Workflows
Build Failure Prediction in Continuous Integration WorkflowsBuild Failure Prediction in Continuous Integration Workflows
Build Failure Prediction in Continuous Integration WorkflowsThomas Rausch
 
Git Introduction Tutorial
Git Introduction TutorialGit Introduction Tutorial
Git Introduction TutorialThomas Rausch
 

More from Thomas Rausch (9)

Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...
 
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
 
Towards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AITowards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AI
 
Edge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AIEdge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AI
 
Portable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge ComputersPortable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge Computers
 
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing ApplicationsEMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
 
Message-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing ApplicationsMessage-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing Applications
 
Build Failure Prediction in Continuous Integration Workflows
Build Failure Prediction in Continuous Integration WorkflowsBuild Failure Prediction in Continuous Integration Workflows
Build Failure Prediction in Continuous Integration Workflows
 
Git Introduction Tutorial
Git Introduction TutorialGit Introduction Tutorial
Git Introduction Tutorial
 

Recently uploaded

Loudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxLoudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxpriyankatabhane
 
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPRPirithiRaju
 
DNA isolation molecular biology practical.pptx
DNA isolation molecular biology practical.pptxDNA isolation molecular biology practical.pptx
DNA isolation molecular biology practical.pptxGiDMOh
 
Immunoblott technique for protein detection.ppt
Immunoblott technique for protein detection.pptImmunoblott technique for protein detection.ppt
Immunoblott technique for protein detection.pptAmirRaziq1
 
Observational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsObservational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsSérgio Sacani
 
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...Christina Parmionova
 
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11GelineAvendao
 
complex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdfcomplex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdfSubhamKumar3239
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxRitchAndruAgustin
 
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep LearningCombining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learningvschiavoni
 
linear Regression, multiple Regression and Annova
linear Regression, multiple Regression and Annovalinear Regression, multiple Regression and Annova
linear Regression, multiple Regression and AnnovaMansi Rastogi
 
DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...
DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...
DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...HafsaHussainp
 
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...Sérgio Sacani
 
The Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionThe Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionJadeNovelo1
 
Introduction of Human Body & Structure of cell.pptx
Introduction of Human Body & Structure of cell.pptxIntroduction of Human Body & Structure of cell.pptx
Introduction of Human Body & Structure of cell.pptxMedical College
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPirithiRaju
 
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdfKDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdfGABYFIORELAMALPARTID1
 

Recently uploaded (20)

Loudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxLoudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptx
 
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
 
DNA isolation molecular biology practical.pptx
DNA isolation molecular biology practical.pptxDNA isolation molecular biology practical.pptx
DNA isolation molecular biology practical.pptx
 
Immunoblott technique for protein detection.ppt
Immunoblott technique for protein detection.pptImmunoblott technique for protein detection.ppt
Immunoblott technique for protein detection.ppt
 
Observational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsObservational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive stars
 
PLASMODIUM. PPTX
PLASMODIUM. PPTXPLASMODIUM. PPTX
PLASMODIUM. PPTX
 
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
Charateristics of the Angara-A5 spacecraft launched from the Vostochny Cosmod...
 
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
WEEK 4 PHYSICAL SCIENCE QUARTER 3 FOR G11
 
complex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdfcomplex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdf
 
AZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTXAZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTX
 
Interferons.pptx.
Interferons.pptx.Interferons.pptx.
Interferons.pptx.
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
 
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep LearningCombining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
 
linear Regression, multiple Regression and Annova
linear Regression, multiple Regression and Annovalinear Regression, multiple Regression and Annova
linear Regression, multiple Regression and Annova
 
DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...
DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...
DOG BITE management in pediatrics # for Pediatric pgs# topic presentation # f...
 
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
 
The Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionThe Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and Function
 
Introduction of Human Body & Structure of cell.pptx
Introduction of Human Body & Structure of cell.pptxIntroduction of Human Body & Structure of cell.pptx
Introduction of Human Body & Structure of cell.pptx
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPR
 
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdfKDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
 

An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software

  • 1. An Empirical Analysis of Build Failures in the Continuous Integration Worfklows of Java-Based Open-Source Software Thomas Rausch, Waldemar Hummer, Philipp Leitner*, Stefan Schulte Distributed Systems Group Vienna University of Technology, Austria http://dsg.tuwien.ac.at * Software Evolution and Architecture Lab University of Zurich, Switzerland http://www.ifi.uzh.ch/en/seal.html
  • 2. 2 Continuous Integration VCS CI Server Build Feedback Logs Vasilescu et al. (2015). Quality and Productivity Outcomes Relating to Continuous Integration in GitHub “Our main finding is that continuous integration improves the productivity of project teams” Kerzazi et al. (2014). Why do Automated Builds Break? An Empirical Study “We [...] quantified the cost of such build breakage as more than 336.18 man-hours”
  • 3. 3
  • 5. 5 Understanding Build Failures What types of errors cause CI build failures? Which development practices can be associated with CI build failures?
  • 6. 6 Research Setting Project Name Description Apache Storm Distributed Computation Butterknife Android Dependency Injection Crate.IO Scalable SQL database JabRef BibTeX management GUI jcabi-github Wrapper of GitHub API Hystrix Latency and fault tolerance library Presto Distributed SQL query engine Openmicroscopy Microscopy data environment RxAndroid RxJava bindings for Android Sponge API Minecraft plugin API Spring Boot Java Application Framework Square OkHttp HTTP+HTTP/2 client for Android Square Retofit HTTP client for Android Wordpress-Android WordPress for Android
  • 7. 7 Data Acquisition a b c d Topology Mapping CI build history Change history
  • 8. 8 Understanding Build Failures What types of errors cause CI build failures? Which development practices can be associated with CI build failures?
  • 9. 9 Error Categorization and Quantification  Goal ○ Categorization of errors ○ Frequency of occurrence of error types  Approach ○ Systematic exploration of ~54 000 logfiles ○ Categorization scheme based on log message patterns [INFO] Compiling 67 source files to /home/travis/.../target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/travis/.../redis/RedisAutoConfiguration.java:[143,10] cannot find symbol [INFO] 1 error [INFO] Compiling 67 source files to /home/travis/.../target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/travis/.../redis/RedisAutoConfiguration.java:[143,10] cannot find symbol [INFO] 1 error [INFO] Compiling 67 source files to /home/travis/.../target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/travis/.../redis/RedisAutoConfiguration.java:[143,10] cannot find symbol [INFO] 1 error
  • 10. 10 Error Categories unknown Errors without a clearly identifiable cause 9 itestfailure An automated integration test failed 4 doc Documentation (e.g., JavaDoc) problem 3 license License criteria not met (missing header) 3 compatibility API incompatibility 2 androidsdk Android SDK-related error 1 buildout Error specific to Crate.IO python module 1 Label Description Occurrences testfailure An automated test failed 12 compile Compilation error 12 git VCS interaction error 12 buildconfig Faulty build config 11 crash Build environment crash or timeout 11 dependency Dependency error 11 quality Coding-rule violation (e.g., Checkstyle) 10
  • 11. 11 Distribution of Common Error Types Faulty VCS interaction Faulty build configuration Dependency error Compilation error Coding-rule violation Failing test Crash 40% 30% 20% 10% 0%
  • 12. 12 Distribution of Common Error Types Apache Storm Butterknife Crate.IO Hystrix Error testfailure compile git dependency crash buildconfig quality others Percentage JabRef jcabi-github Presto RxAndroid SpongeAPI Spring Boot Square OkHttp Square Retrofit 0% 25% 50% 75% 100%
  • 13. 13 Understanding Build Failures What types of errors cause CI build failures? Which development practices can be associated with CI build failures?
  • 14. 14 Change Metrics .java .txt Changes  Complexity ○ Churn, number of files, ...  File types ○ README.txt vs. IntegrationTest.java  Date and time  Author ○ Experience, commit frequency, ...
  • 15. 15 Process Metrics b1 b2 b4 t b3 a b e c f d g VCS commit graph CI build information  Build History ○ Build climate  Build Type ○ Pull request, merge, ...  Pull Request Scenarios ○ Rebase, squash, ...
  • 16. 16 Statistical Correlation Analysis  For each project individually  Non-parametric correlation tests ○ Pearson’s chi-square test ○ Mann—Whitney U test  Calculate effect sizes ○ Cramér’s V ○ Rank-biserial correlation
  • 17. 17 PassedBuild outcome Failed Failed Passed Previous build result Percentageofbuilds Findings Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build history b b’
  • 18. 18 PassedBuild outcome Failed Failed Passed Previous build result Percentageofbuilds Findings Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build history b b’
  • 19. 19 Findings Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build history No evidence that either history manipula- tion operations or parallel development to a PR affect the PR’s build outcome. No evidence that either history manipula- tion operations or parallel development to a PR affect the PR’s build outcome. Pull request scenarios
  • 20. 20 Findings Even objectively harmless changes can break builds. This indicates unwanted flakiness of tests or the build environment. Even objectively harmless changes can break builds. This indicates unwanted flakiness of tests or the build environment. Build failures mostly occur consecutively. Phases of build instability perpetuate failures. Build failures mostly occur consecutively. Phases of build instability perpetuate failures. File types Build history  577 builds from Spring Boot  Changelog file change only  14% original failures ○ 52% test failures ○ 45% environment crash ○ 3% dependency error No evidence that either history manipula- tion operations or parallel development to a PR affect the PR’s build outcome. No evidence that either history manipula- tion operations or parallel development to a PR affect the PR’s build outcome. Pull request scenarios
  • 21. 21 Summary  Categorization of error types (beyond failed/errored)  Quantification of error type occurrence  Statistical analysis of impact factors  Uncovered challenges that arise when mining CI data
  • 22. 22 Dipl.-Ing. Thomas Rausch Research Assistant TU Wien Distributed Systems Group Argentinierstraße 8/184-1, 1040, Vienna, Austria T: +43 1 58801 184 838 E: rausch@dsg.tuwien.ac.at dsg.tuwien.ac.at/staff/trausch