SlideShare a Scribd company logo
1 of 48
Download to read offline
Build Failure Prediction in
Continuous Integration Workflows
Master’s Thesis Presentation
Software Engineering & Internet Computing
Thomas Rausch
Advisor: Stefan Schulte
Co-Advisor: Waldemar Hummer
I
Introduction
Thomas Rausch 3
Build
t
Build
Code Product
Thomas Rausch 4
Continuous Integration (CI)
●
Build at every change
●
Identify problems early
●
Allow frequent releases
VCS
CI Server
Feedback
Build
P. M. Duvall, S. Matyas, and A. Glover, Continuous integration:
improving software quality and reducing risk. Pearson Education, 2007.
Thomas Rausch 5
Build Failure
Build
Wasted resources
t
Examine Determine Fix
ErrorBuild log
Thomas Rausch 6
Software Defect Prediction
Complex
code
Bad
practices
Unstable
environment
Software
defect
M. D’Ambros, M. Lanza, and R. Robbes, Evaluating defect prediction approaches:
a benchmark and an extensive comparison, vol. 17, no. 4–5. 2012.
Thomas Rausch 7
Build Failure Prediction?
Complex
code
Bad
practices
Unstable
environment
Build
failure
?
Unstable
environment
?
Thomas Rausch 8
Research Questions
Which errors cause CI build failures?
What factors influence build outcomes?
Can we predict build failures?
II
Solution Approach
Thomas Rausch 10
Empirical Study
Research Setting
●
14 open source projects that employ CI
Topology mapping
a
b
c
d
VCS
Builds Logs
Change history
CI build history
Thomas Rausch 11
Outline
I Introduction
II Solution Approach
Study
III Systematic study of build errors
IV Factors influencing build failures
V Build failure prediction
VI Summary & Conclusion
III
Systematic Study of
Build Errors
Thomas Rausch 13
Build Steps
validatefetch compile inspect test
Thomas Rausch 14
Build Failure
validatefetch compile inspect test
Thomas Rausch 15
Build Error
Build log output
validatefetch compile inspect test
“[ERROR] Compilation Error”
Thomas Rausch 16
Build Error Frequency
Faulty VCS
interaction
Faulty build
configuration Dependency
error
Compilation
error
Coding-rule
violation
Failing test
Crash
40%
30%
20%
10%
0%
62%
Thomas Rausch 17
Build Error
t
validatefetch compile inspect test
Thomas Rausch 18
Frequency
Seconds
Error Probability Distribution
t
IV
Factors Influencing
Build Results
Thomas Rausch 20
Causes for Build Failures
Complex
code
Bad
practices
Unstable
environment
Build
failure
?
Thomas Rausch 21
Measurable Properties
Complex
code
Bad
practices
Unstable
environment
) = ?fn(
Thomas Rausch 22
Measurable Properties
Complex
code
Bad
practices
Unstable
environment
= fn(?)
Thomas Rausch 24
Measurable Properties
Change Metrics
●
What was done to the software?
Process Metrics
●
How were the changes applied?
Thomas Rausch 25
Measurable Properties
Change Metrics
●
Complexity
●
File types
●
Date and time
●
Author
Changes
.java .txt
Thomas Rausch 26
Measurable Properties
Process Metrics
●
Build history
●
Build type
●
Integration scenario
b1
b2
b4
t
b3
a
b
e
c
f
d
g
VCS
commit graph
CI build
information
Topology mapping
Thomas Rausch 27
Statistical Correlation Analyses
Categorical Variables
●
{c1, …,cn} ~ { , } Pearson’s chi-square test
Numerical Variables
●
R ~ { , } Mann–Whitney U test
D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures.
crc Press, 2003.
Thomas Rausch 28
Results
Metric Meta
Build history +++++
Build type ++
Author ++
Change complexity +
Date and time +
File types ~
Integration Scenario ~
Meta statistics
●
Using Fisher’s method
●
Relative relation strength
Van Zwet, W. R., & Oosterhoff, J. (1967). On the combination of independent test
statistics. The Annals of Mathematical Statistics, 38(3), 659–680. article.
Thomas Rausch 29
PassedBuild outcome Failed
Failed Passed
Previous build result
Percentageofbuilds
Results
Metric Meta
Build history +++++
Build type ++
Author ++
Change complexity +
Date and time +
File types ~
Integration Scenario ~
b
b’
Thomas Rausch 30
PassedBuild outcome Failed
Failed Passed
Previous build result
Percentageofbuilds
Results
Metric Meta
Build history +++++
Build type ++
Author ++
Change complexity +
Date and time +
File types ~
Integration Scenario ~
b
b’
Thomas Rausch 31
Results
Metric Meta
Build history +++++
Build type ++
Author ++
Change complexity +
Date and time +
File types ~
Integration Scenario ~
Example
●
Documentation file (changelog)
●
577 builds – doc. change only
●
14% original failures
●
52% test failures
●
45% environment crash
●
3% dependency error
Similar behavior in all projects
Noise skews statistics
V
Build Failure Prediction
Thomas Rausch 33
Machine Learning
?
Classifier
Observations
?
?
Thomas Rausch 34
Experiment Design
Well-known algorithms
●
Naive Bayes
●
C4.5 Decision Trees
●
Random Forest
Feature sets
●
Process metrics
●
Change metrics
●
Combined
Prediction for
●
Binary (failed/passed)
●
Multi-class (error type)
Baseline
●
0-R classifier
frequency table
predicts the average
Thomas Rausch 35
Binary classification result
Average F1
-scores
Binary classification
●
F1-score, 1.0 = perfect
●
Ranging from
●
0.71 – 0.91
Algorithm 0-R NB C4.5 RF
All CM PM
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Thomas Rausch 36
Multi-class classification results
Multi-class classification
●
●
RMSE
●
Error between
actual and predicted
Average RMSE
p=[ p1 ... pn]
Algorithm 0-R NB C4.5 RF
All CM PM
0.0
0.3
0.1
0.2
Thomas Rausch 37
Update Predictions
False positive elimination
●
Likelihood for error declines
●
Plausibility of prediction declines
tmax t50 t75
VI
Summary & Conclusion
Thomas Rausch 39
Research Questions - Answered
Which errors cause CI build failures?
●
Failing tests (41%)
●
Coding-rule violations (11%)
●
Compilation errors (10%)
●
A number of errors involving the CI environment
Thomas Rausch 40
Research Questions - Answered
What factors influence build outcomes?
●
Process Metrics
●
Build history failures persist
●
Build type merges vs. forward engineering
●
Change Metrics
●
Actually, not so much
Thomas Rausch 41
Research Questions - Answered
Can we predict build failures?
●
Yes!
●
We can update predictions during the execution
State Rate min max
Passed 87% 45% 99%
Failed 66% 27% 96%
Thomas Rausch 42
Contributions
In-depth analysis of CI workflow
●
Multiplicity of build errors
●
Factors influencing build failures
Topology mapping
Baseline for CI build failure prediction
Thomas Rausch 43
Future Work
Examine more influence factors
●
Developer behavior
●
Project management workflows
Improve learning methods
Incorporate prediction into CI tools
●
Improve feedback mechanism
●
Improve development productivity
The end?
Questions & Answers
Thomas Rausch
t.rausch@infosys.tuwien.ac.at
Thomas Rausch 45
Research Subjects
Project Name Description
Apache Storm Distributed Computation Framework
Butterknife Android Dependency Injection Library
Crate.IO Scalable SQL database
JabRef Graphical Java application for managing BibTeX databases
jcabi-github Object Oriented Wrapper of GitHub API
Hystrix Latency and fault tolerance library for distributed systems
Presto Distributed SQL query engine for big data
Openmicroscopy Microscopy data environment
RxAndroid RxJava bindings for Android
Sponge API Minecraft plugin API
Spring Boot Java Application Framework
Square OkHttp HTTP+HTTP/2 client for Android and Java
Square Retofit HTTP client for Android and Java
Wordpress-Android WordPress for Android
Thomas Rausch 46
Integration Scenarios
a
b
c
f
e d
g
a
b
c
e
d
f
a
b
c
e
d
a
b
c
d
Initialization Update
Thomas Rausch 47
Related Work
Build Failure Prediction
●
Hassan and Zhang 2006
●
Outdated assumptions
●
No CI workflow considerations
●
Kerzazi et al. 2014
●
Only statistical analysis, no prediction
●
Only binary build outcome
●
Wolf 2009, Schröter 2010
●
Socio-technical factors
●
Closed source enterprise software
Thomas Rausch 48
Data Source
{
"id": 22555277,
"commit_id": 6534711,
"number": "784",
"pull_request_number": "1912",
"pull_request_title": "Example PR",
"started_at": "2014-04-08T19:37:44Z",
"finished_at": "2014-04-08T19:52:56Z",
"duration": 2648,
"state": "failed"
"commit": { ... }
}
Travis-CI
●
Hosted CI service
●
RESTful API
●
Integrated with GitHub
Thomas Rausch 49
Runtime Evolution

More Related Content

Similar to Build Failure Prediction in Continuous Integration Workflows

An Empirical Study on the Adequacy of Testing in Open Source Projects
An Empirical Study on the Adequacy of Testing in Open Source ProjectsAn Empirical Study on the Adequacy of Testing in Open Source Projects
An Empirical Study on the Adequacy of Testing in Open Source ProjectsPavneet Singh Kochhar
 
The Road to Data-Informed Agile Development Processes
The Road to Data-Informed Agile Development ProcessesThe Road to Data-Informed Agile Development Processes
The Road to Data-Informed Agile Development ProcessesChristoph Matthies
 
Can we induce change with what we measure?
Can we induce change with what we measure?Can we induce change with what we measure?
Can we induce change with what we measure?Michaela Greiler
 
Design Assignment Part B
Design Assignment Part BDesign Assignment Part B
Design Assignment Part BSimon Reynolds
 
Agile Software Development Practices: Perceptions & Project Data
Agile Software Development Practices: Perceptions & Project DataAgile Software Development Practices: Perceptions & Project Data
Agile Software Development Practices: Perceptions & Project DataChristoph Matthies
 
Promise 2011: "Empirical validation of human factors on predicting issue reso...
Promise 2011: "Empirical validation of human factors on predicting issue reso...Promise 2011: "Empirical validation of human factors on predicting issue reso...
Promise 2011: "Empirical validation of human factors on predicting issue reso...CS, NcState
 
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...Rodi Jolak
 
Declarative Performance Testing Automation - Automating Performance Testing f...
Declarative Performance Testing Automation - Automating Performance Testing f...Declarative Performance Testing Automation - Automating Performance Testing f...
Declarative Performance Testing Automation - Automating Performance Testing f...Vincenzo Ferme
 
Feedback in Scrum: Data-Informed Retrospectives
Feedback in Scrum: Data-Informed Retrospectives Feedback in Scrum: Data-Informed Retrospectives
Feedback in Scrum: Data-Informed Retrospectives Christoph Matthies
 
PhD Proposal talk
PhD Proposal talkPhD Proposal talk
PhD Proposal talkRay Buse
 
13 analysis of_software_architectures
13 analysis of_software_architectures13 analysis of_software_architectures
13 analysis of_software_architecturesMajong DevJfu
 
Introduction to Software Engineering
Introduction to Software EngineeringIntroduction to Software Engineering
Introduction to Software EngineeringRizwan411667
 
Celebrating 30 years of ISSRE
Celebrating 30 years of ISSRECelebrating 30 years of ISSRE
Celebrating 30 years of ISSREISSREConf
 
Celebrating 30 years of ISSRE
Celebrating 30 years of ISSRECelebrating 30 years of ISSRE
Celebrating 30 years of ISSREISSREConf
 
Measuring the Effects of Design Patterns by David Kemp
Measuring the Effects of Design Patterns by David KempMeasuring the Effects of Design Patterns by David Kemp
Measuring the Effects of Design Patterns by David Kempmelbournepatterns
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven DevelopmentJosh Candish
 
Construction cost engineering Assignment Help and Construction cost engineeri...
Construction cost engineering Assignment Help and Construction cost engineeri...Construction cost engineering Assignment Help and Construction cost engineeri...
Construction cost engineering Assignment Help and Construction cost engineeri...john mayer
 
Model Checking as a Service: Towards Pragmatic Hidden Formal Methods
Model Checking as a Service: Towards Pragmatic Hidden Formal MethodsModel Checking as a Service: Towards Pragmatic Hidden Formal Methods
Model Checking as a Service: Towards Pragmatic Hidden Formal MethodsIncQuery Labs
 
Software engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research resultsSoftware engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research resultsNikolai Avteniev
 
Software Analytics = Sharing Information
Software Analytics = Sharing InformationSoftware Analytics = Sharing Information
Software Analytics = Sharing InformationThomas Zimmermann
 

Similar to Build Failure Prediction in Continuous Integration Workflows (20)

An Empirical Study on the Adequacy of Testing in Open Source Projects
An Empirical Study on the Adequacy of Testing in Open Source ProjectsAn Empirical Study on the Adequacy of Testing in Open Source Projects
An Empirical Study on the Adequacy of Testing in Open Source Projects
 
The Road to Data-Informed Agile Development Processes
The Road to Data-Informed Agile Development ProcessesThe Road to Data-Informed Agile Development Processes
The Road to Data-Informed Agile Development Processes
 
Can we induce change with what we measure?
Can we induce change with what we measure?Can we induce change with what we measure?
Can we induce change with what we measure?
 
Design Assignment Part B
Design Assignment Part BDesign Assignment Part B
Design Assignment Part B
 
Agile Software Development Practices: Perceptions & Project Data
Agile Software Development Practices: Perceptions & Project DataAgile Software Development Practices: Perceptions & Project Data
Agile Software Development Practices: Perceptions & Project Data
 
Promise 2011: "Empirical validation of human factors on predicting issue reso...
Promise 2011: "Empirical validation of human factors on predicting issue reso...Promise 2011: "Empirical validation of human factors on predicting issue reso...
Promise 2011: "Empirical validation of human factors on predicting issue reso...
 
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
 
Declarative Performance Testing Automation - Automating Performance Testing f...
Declarative Performance Testing Automation - Automating Performance Testing f...Declarative Performance Testing Automation - Automating Performance Testing f...
Declarative Performance Testing Automation - Automating Performance Testing f...
 
Feedback in Scrum: Data-Informed Retrospectives
Feedback in Scrum: Data-Informed Retrospectives Feedback in Scrum: Data-Informed Retrospectives
Feedback in Scrum: Data-Informed Retrospectives
 
PhD Proposal talk
PhD Proposal talkPhD Proposal talk
PhD Proposal talk
 
13 analysis of_software_architectures
13 analysis of_software_architectures13 analysis of_software_architectures
13 analysis of_software_architectures
 
Introduction to Software Engineering
Introduction to Software EngineeringIntroduction to Software Engineering
Introduction to Software Engineering
 
Celebrating 30 years of ISSRE
Celebrating 30 years of ISSRECelebrating 30 years of ISSRE
Celebrating 30 years of ISSRE
 
Celebrating 30 years of ISSRE
Celebrating 30 years of ISSRECelebrating 30 years of ISSRE
Celebrating 30 years of ISSRE
 
Measuring the Effects of Design Patterns by David Kemp
Measuring the Effects of Design Patterns by David KempMeasuring the Effects of Design Patterns by David Kemp
Measuring the Effects of Design Patterns by David Kemp
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
Construction cost engineering Assignment Help and Construction cost engineeri...
Construction cost engineering Assignment Help and Construction cost engineeri...Construction cost engineering Assignment Help and Construction cost engineeri...
Construction cost engineering Assignment Help and Construction cost engineeri...
 
Model Checking as a Service: Towards Pragmatic Hidden Formal Methods
Model Checking as a Service: Towards Pragmatic Hidden Formal MethodsModel Checking as a Service: Towards Pragmatic Hidden Formal Methods
Model Checking as a Service: Towards Pragmatic Hidden Formal Methods
 
Software engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research resultsSoftware engineering practices and software quality empirical research results
Software engineering practices and software quality empirical research results
 
Software Analytics = Sharing Information
Software Analytics = Sharing InformationSoftware Analytics = Sharing Information
Software Analytics = Sharing Information
 

More from Thomas Rausch

Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...Thomas Rausch
 
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...Thomas Rausch
 
Towards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AITowards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AIThomas Rausch
 
Edge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AIEdge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AIThomas Rausch
 
Portable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge ComputersPortable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge ComputersThomas Rausch
 
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing ApplicationsEMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing ApplicationsThomas Rausch
 
Message-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing ApplicationsMessage-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing ApplicationsThomas Rausch
 
An Empirical Analysis of Build Failures in the Continuous Integration Workflo...
An Empirical Analysis of Build Failures in the Continuous Integration Workflo...An Empirical Analysis of Build Failures in the Continuous Integration Workflo...
An Empirical Analysis of Build Failures in the Continuous Integration Workflo...Thomas Rausch
 
Git Introduction Tutorial
Git Introduction TutorialGit Introduction Tutorial
Git Introduction TutorialThomas Rausch
 

More from Thomas Rausch (9)

Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...Test cloud application deployments locally and in CI without staging environm...
Test cloud application deployments locally and in CI without staging environm...
 
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...
 
Towards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AITowards a Serverless Platform for Edge AI
Towards a Serverless Platform for Edge AI
 
Edge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AIEdge Intelligence: The Convergence of Humans, Things and AI
Edge Intelligence: The Convergence of Humans, Things and AI
 
Portable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge ComputersPortable Energy-Aware Cluster-Based Edge Computers
Portable Energy-Aware Cluster-Based Edge Computers
 
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing ApplicationsEMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
EMMA: Distributed QoS-Aware MQTT Middleware for Edge Computing Applications
 
Message-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing ApplicationsMessage-Oriented Middleware for Edge Computing Applications
Message-Oriented Middleware for Edge Computing Applications
 
An Empirical Analysis of Build Failures in the Continuous Integration Workflo...
An Empirical Analysis of Build Failures in the Continuous Integration Workflo...An Empirical Analysis of Build Failures in the Continuous Integration Workflo...
An Empirical Analysis of Build Failures in the Continuous Integration Workflo...
 
Git Introduction Tutorial
Git Introduction TutorialGit Introduction Tutorial
Git Introduction Tutorial
 

Recently uploaded

Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Seán Kennedy
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改yuu sss
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档208367051
 
在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证
在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证
在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证nhjeo1gg
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一F La
 
Heart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectHeart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectBoston Institute of Analytics
 
在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证
在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证
在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证nhjeo1gg
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Thomas Poetter
 
MK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docxMK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docxUnduhUnggah1
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一
办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一
办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一z xss
 
Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 2Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 217djon017
 
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degreeyuu sss
 
Multiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfMultiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfchwongval
 
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...GQ Research
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
 
毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...
毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...
毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...ttt fff
 
Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...Seán Kennedy
 
Top 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In QueensTop 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In Queensdataanalyticsqueen03
 
SWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptxSWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptxviniciusperissetr
 

Recently uploaded (20)

Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
 
在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证
在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证
在线办理UM毕业证迈阿密大学毕业证成绩单留信学历认证
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
 
Heart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectHeart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis Project
 
在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证
在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证
在线办理WLU毕业证罗瑞尔大学毕业证成绩单留信学历认证
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
 
MK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docxMK KOMUNIKASI DATA (TI)komdat komdat.docx
MK KOMUNIKASI DATA (TI)komdat komdat.docx
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一
办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一
办理(UC毕业证书)堪培拉大学毕业证成绩单原版一比一
 
Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 2Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 2
 
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
 
Multiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfMultiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdf
 
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
 
毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...
毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...
毕业文凭制作#回国入职#diploma#degree美国加州州立大学北岭分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#de...
 
Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...
 
Top 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In QueensTop 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In Queens
 
SWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptxSWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptx
 

Build Failure Prediction in Continuous Integration Workflows

  • 1. Build Failure Prediction in Continuous Integration Workflows Master’s Thesis Presentation Software Engineering & Internet Computing Thomas Rausch Advisor: Stefan Schulte Co-Advisor: Waldemar Hummer
  • 4. Thomas Rausch 4 Continuous Integration (CI) ● Build at every change ● Identify problems early ● Allow frequent releases VCS CI Server Feedback Build P. M. Duvall, S. Matyas, and A. Glover, Continuous integration: improving software quality and reducing risk. Pearson Education, 2007.
  • 5. Thomas Rausch 5 Build Failure Build Wasted resources t Examine Determine Fix ErrorBuild log
  • 6. Thomas Rausch 6 Software Defect Prediction Complex code Bad practices Unstable environment Software defect M. D’Ambros, M. Lanza, and R. Robbes, Evaluating defect prediction approaches: a benchmark and an extensive comparison, vol. 17, no. 4–5. 2012.
  • 7. Thomas Rausch 7 Build Failure Prediction? Complex code Bad practices Unstable environment Build failure ? Unstable environment ?
  • 8. Thomas Rausch 8 Research Questions Which errors cause CI build failures? What factors influence build outcomes? Can we predict build failures?
  • 10. Thomas Rausch 10 Empirical Study Research Setting ● 14 open source projects that employ CI Topology mapping a b c d VCS Builds Logs Change history CI build history
  • 11. Thomas Rausch 11 Outline I Introduction II Solution Approach Study III Systematic study of build errors IV Factors influencing build failures V Build failure prediction VI Summary & Conclusion
  • 13. Thomas Rausch 13 Build Steps validatefetch compile inspect test
  • 14. Thomas Rausch 14 Build Failure validatefetch compile inspect test
  • 15. Thomas Rausch 15 Build Error Build log output validatefetch compile inspect test “[ERROR] Compilation Error”
  • 16. Thomas Rausch 16 Build Error Frequency Faulty VCS interaction Faulty build configuration Dependency error Compilation error Coding-rule violation Failing test Crash 40% 30% 20% 10% 0% 62%
  • 17. Thomas Rausch 17 Build Error t validatefetch compile inspect test
  • 18. Thomas Rausch 18 Frequency Seconds Error Probability Distribution t
  • 20. Thomas Rausch 20 Causes for Build Failures Complex code Bad practices Unstable environment Build failure ?
  • 21. Thomas Rausch 21 Measurable Properties Complex code Bad practices Unstable environment ) = ?fn(
  • 22. Thomas Rausch 22 Measurable Properties Complex code Bad practices Unstable environment = fn(?)
  • 23. Thomas Rausch 24 Measurable Properties Change Metrics ● What was done to the software? Process Metrics ● How were the changes applied?
  • 24. Thomas Rausch 25 Measurable Properties Change Metrics ● Complexity ● File types ● Date and time ● Author Changes .java .txt
  • 25. Thomas Rausch 26 Measurable Properties Process Metrics ● Build history ● Build type ● Integration scenario b1 b2 b4 t b3 a b e c f d g VCS commit graph CI build information Topology mapping
  • 26. Thomas Rausch 27 Statistical Correlation Analyses Categorical Variables ● {c1, …,cn} ~ { , } Pearson’s chi-square test Numerical Variables ● R ~ { , } Mann–Whitney U test D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. crc Press, 2003.
  • 27. Thomas Rausch 28 Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ Meta statistics ● Using Fisher’s method ● Relative relation strength Van Zwet, W. R., & Oosterhoff, J. (1967). On the combination of independent test statistics. The Annals of Mathematical Statistics, 38(3), 659–680. article.
  • 28. Thomas Rausch 29 PassedBuild outcome Failed Failed Passed Previous build result Percentageofbuilds Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ b b’
  • 29. Thomas Rausch 30 PassedBuild outcome Failed Failed Passed Previous build result Percentageofbuilds Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ b b’
  • 30. Thomas Rausch 31 Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ Example ● Documentation file (changelog) ● 577 builds – doc. change only ● 14% original failures ● 52% test failures ● 45% environment crash ● 3% dependency error Similar behavior in all projects Noise skews statistics
  • 32. Thomas Rausch 33 Machine Learning ? Classifier Observations ? ?
  • 33. Thomas Rausch 34 Experiment Design Well-known algorithms ● Naive Bayes ● C4.5 Decision Trees ● Random Forest Feature sets ● Process metrics ● Change metrics ● Combined Prediction for ● Binary (failed/passed) ● Multi-class (error type) Baseline ● 0-R classifier frequency table predicts the average
  • 34. Thomas Rausch 35 Binary classification result Average F1 -scores Binary classification ● F1-score, 1.0 = perfect ● Ranging from ● 0.71 – 0.91 Algorithm 0-R NB C4.5 RF All CM PM 0.3 0.4 0.5 0.6 0.7 0.8 0.9
  • 35. Thomas Rausch 36 Multi-class classification results Multi-class classification ● ● RMSE ● Error between actual and predicted Average RMSE p=[ p1 ... pn] Algorithm 0-R NB C4.5 RF All CM PM 0.0 0.3 0.1 0.2
  • 36. Thomas Rausch 37 Update Predictions False positive elimination ● Likelihood for error declines ● Plausibility of prediction declines tmax t50 t75
  • 38. Thomas Rausch 39 Research Questions - Answered Which errors cause CI build failures? ● Failing tests (41%) ● Coding-rule violations (11%) ● Compilation errors (10%) ● A number of errors involving the CI environment
  • 39. Thomas Rausch 40 Research Questions - Answered What factors influence build outcomes? ● Process Metrics ● Build history failures persist ● Build type merges vs. forward engineering ● Change Metrics ● Actually, not so much
  • 40. Thomas Rausch 41 Research Questions - Answered Can we predict build failures? ● Yes! ● We can update predictions during the execution State Rate min max Passed 87% 45% 99% Failed 66% 27% 96%
  • 41. Thomas Rausch 42 Contributions In-depth analysis of CI workflow ● Multiplicity of build errors ● Factors influencing build failures Topology mapping Baseline for CI build failure prediction
  • 42. Thomas Rausch 43 Future Work Examine more influence factors ● Developer behavior ● Project management workflows Improve learning methods Incorporate prediction into CI tools ● Improve feedback mechanism ● Improve development productivity
  • 43. The end? Questions & Answers Thomas Rausch t.rausch@infosys.tuwien.ac.at
  • 44. Thomas Rausch 45 Research Subjects Project Name Description Apache Storm Distributed Computation Framework Butterknife Android Dependency Injection Library Crate.IO Scalable SQL database JabRef Graphical Java application for managing BibTeX databases jcabi-github Object Oriented Wrapper of GitHub API Hystrix Latency and fault tolerance library for distributed systems Presto Distributed SQL query engine for big data Openmicroscopy Microscopy data environment RxAndroid RxJava bindings for Android Sponge API Minecraft plugin API Spring Boot Java Application Framework Square OkHttp HTTP+HTTP/2 client for Android and Java Square Retofit HTTP client for Android and Java Wordpress-Android WordPress for Android
  • 45. Thomas Rausch 46 Integration Scenarios a b c f e d g a b c e d f a b c e d a b c d Initialization Update
  • 46. Thomas Rausch 47 Related Work Build Failure Prediction ● Hassan and Zhang 2006 ● Outdated assumptions ● No CI workflow considerations ● Kerzazi et al. 2014 ● Only statistical analysis, no prediction ● Only binary build outcome ● Wolf 2009, Schröter 2010 ● Socio-technical factors ● Closed source enterprise software
  • 47. Thomas Rausch 48 Data Source { "id": 22555277, "commit_id": 6534711, "number": "784", "pull_request_number": "1912", "pull_request_title": "Example PR", "started_at": "2014-04-08T19:37:44Z", "finished_at": "2014-04-08T19:52:56Z", "duration": 2648, "state": "failed" "commit": { ... } } Travis-CI ● Hosted CI service ● RESTful API ● Integrated with GitHub