Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Build Failure Prediction in
Continuous Integration Workflows
Master’s Thesis Presentation
Software Engineering & Internet ...
I
Introduction
Thomas Rausch 3
Build
t
Build
Code Product
Thomas Rausch 4
Continuous Integration (CI)
●
Build at every change
●
Identify problems early
●
Allow frequent releases
VC...
Thomas Rausch 5
Build Failure
Build
Wasted resources
t
Examine Determine Fix
ErrorBuild log
Thomas Rausch 6
Software Defect Prediction
Complex
code
Bad
practices
Unstable
environment
Software
defect
M. D’Ambros, M....
Thomas Rausch 7
Build Failure Prediction?
Complex
code
Bad
practices
Unstable
environment
Build
failure
?
Unstable
environ...
Thomas Rausch 8
Research Questions
Which errors cause CI build failures?
What factors influence build outcomes?
Can we pre...
II
Solution Approach
Thomas Rausch 10
Empirical Study
Research Setting
●
14 open source projects that employ CI
Topology mapping
a
b
c
d
VCS
Bu...
Thomas Rausch 11
Outline
I Introduction
II Solution Approach
Study
III Systematic study of build errors
IV Factors influen...
III
Systematic Study of
Build Errors
Thomas Rausch 13
Build Steps
validatefetch compile inspect test
Thomas Rausch 14
Build Failure
validatefetch compile inspect test
Thomas Rausch 15
Build Error
Build log output
validatefetch compile inspect test
“[ERROR] Compilation Error”
Thomas Rausch 16
Build Error Frequency
Faulty VCS
interaction
Faulty build
configuration Dependency
error
Compilation
erro...
Thomas Rausch 17
Build Error
t
validatefetch compile inspect test
Thomas Rausch 18
Frequency
Seconds
Error Probability Distribution
t
IV
Factors Influencing
Build Results
Thomas Rausch 20
Causes for Build Failures
Complex
code
Bad
practices
Unstable
environment
Build
failure
?
Thomas Rausch 21
Measurable Properties
Complex
code
Bad
practices
Unstable
environment
) = ?fn(
Thomas Rausch 22
Measurable Properties
Complex
code
Bad
practices
Unstable
environment
= fn(?)
Thomas Rausch 24
Measurable Properties
Change Metrics
●
What was done to the software?
Process Metrics
●
How were the chan...
Thomas Rausch 25
Measurable Properties
Change Metrics
●
Complexity
●
File types
●
Date and time
●
Author
Changes
.java .txt
Thomas Rausch 26
Measurable Properties
Process Metrics
●
Build history
●
Build type
●
Integration scenario
b1
b2
b4
t
b3
a...
Thomas Rausch 27
Statistical Correlation Analyses
Categorical Variables
●
{c1, …,cn} ~ { , } Pearson’s chi-square test
Num...
Thomas Rausch 28
Results
Metric Meta
Build history +++++
Build type ++
Author ++
Change complexity +
Date and time +
File ...
Thomas Rausch 29
PassedBuild outcome Failed
Failed Passed
Previous build result
Percentageofbuilds
Results
Metric Meta
Bui...
Thomas Rausch 30
PassedBuild outcome Failed
Failed Passed
Previous build result
Percentageofbuilds
Results
Metric Meta
Bui...
Thomas Rausch 31
Results
Metric Meta
Build history +++++
Build type ++
Author ++
Change complexity +
Date and time +
File ...
V
Build Failure Prediction
Thomas Rausch 33
Machine Learning
?
Classifier
Observations
?
?
Thomas Rausch 34
Experiment Design
Well-known algorithms
●
Naive Bayes
●
C4.5 Decision Trees
●
Random Forest
Feature sets
...
Thomas Rausch 35
Binary classification result
Average F1
-scores
Binary classification
●
F1-score, 1.0 = perfect
●
Ranging...
Thomas Rausch 36
Multi-class classification results
Multi-class classification
●
●
RMSE
●
Error between
actual and predict...
Thomas Rausch 37
Update Predictions
False positive elimination
●
Likelihood for error declines
●
Plausibility of predictio...
VI
Summary & Conclusion
Thomas Rausch 39
Research Questions - Answered
Which errors cause CI build failures?
●
Failing tests (41%)
●
Coding-rule v...
Thomas Rausch 40
Research Questions - Answered
What factors influence build outcomes?
●
Process Metrics
●
Build history fa...
Thomas Rausch 41
Research Questions - Answered
Can we predict build failures?
●
Yes!
●
We can update predictions during th...
Thomas Rausch 42
Contributions
In-depth analysis of CI workflow
●
Multiplicity of build errors
●
Factors influencing build...
Thomas Rausch 43
Future Work
Examine more influence factors
●
Developer behavior
●
Project management workflows
Improve le...
The end?
Questions & Answers
Thomas Rausch
t.rausch@infosys.tuwien.ac.at
Thomas Rausch 45
Research Subjects
Project Name Description
Apache Storm Distributed Computation Framework
Butterknife And...
Thomas Rausch 46
Integration Scenarios
a
b
c
f
e d
g
a
b
c
e
d
f
a
b
c
e
d
a
b
c
d
Initialization Update
Thomas Rausch 47
Related Work
Build Failure Prediction
●
Hassan and Zhang 2006
●
Outdated assumptions
●
No CI workflow con...
Thomas Rausch 48
Data Source
{
"id": 22555277,
"commit_id": 6534711,
"number": "784",
"pull_request_number": "1912",
"pull...
Thomas Rausch 49
Runtime Evolution
Upcoming SlideShare
Loading in …5
×

Build Failure Prediction in Continuous Integration Workflows

673 views

Published on

Final presentation of my master's thesis on Build Failure Prediction in Continuous Integration Workflows

Published in: Data & Analytics
  • Be the first to comment

Build Failure Prediction in Continuous Integration Workflows

  1. 1. Build Failure Prediction in Continuous Integration Workflows Master’s Thesis Presentation Software Engineering & Internet Computing Thomas Rausch Advisor: Stefan Schulte Co-Advisor: Waldemar Hummer
  2. 2. I Introduction
  3. 3. Thomas Rausch 3 Build t Build Code Product
  4. 4. Thomas Rausch 4 Continuous Integration (CI) ● Build at every change ● Identify problems early ● Allow frequent releases VCS CI Server Feedback Build P. M. Duvall, S. Matyas, and A. Glover, Continuous integration: improving software quality and reducing risk. Pearson Education, 2007.
  5. 5. Thomas Rausch 5 Build Failure Build Wasted resources t Examine Determine Fix ErrorBuild log
  6. 6. Thomas Rausch 6 Software Defect Prediction Complex code Bad practices Unstable environment Software defect M. D’Ambros, M. Lanza, and R. Robbes, Evaluating defect prediction approaches: a benchmark and an extensive comparison, vol. 17, no. 4–5. 2012.
  7. 7. Thomas Rausch 7 Build Failure Prediction? Complex code Bad practices Unstable environment Build failure ? Unstable environment ?
  8. 8. Thomas Rausch 8 Research Questions Which errors cause CI build failures? What factors influence build outcomes? Can we predict build failures?
  9. 9. II Solution Approach
  10. 10. Thomas Rausch 10 Empirical Study Research Setting ● 14 open source projects that employ CI Topology mapping a b c d VCS Builds Logs Change history CI build history
  11. 11. Thomas Rausch 11 Outline I Introduction II Solution Approach Study III Systematic study of build errors IV Factors influencing build failures V Build failure prediction VI Summary & Conclusion
  12. 12. III Systematic Study of Build Errors
  13. 13. Thomas Rausch 13 Build Steps validatefetch compile inspect test
  14. 14. Thomas Rausch 14 Build Failure validatefetch compile inspect test
  15. 15. Thomas Rausch 15 Build Error Build log output validatefetch compile inspect test “[ERROR] Compilation Error”
  16. 16. Thomas Rausch 16 Build Error Frequency Faulty VCS interaction Faulty build configuration Dependency error Compilation error Coding-rule violation Failing test Crash 40% 30% 20% 10% 0% 62%
  17. 17. Thomas Rausch 17 Build Error t validatefetch compile inspect test
  18. 18. Thomas Rausch 18 Frequency Seconds Error Probability Distribution t
  19. 19. IV Factors Influencing Build Results
  20. 20. Thomas Rausch 20 Causes for Build Failures Complex code Bad practices Unstable environment Build failure ?
  21. 21. Thomas Rausch 21 Measurable Properties Complex code Bad practices Unstable environment ) = ?fn(
  22. 22. Thomas Rausch 22 Measurable Properties Complex code Bad practices Unstable environment = fn(?)
  23. 23. Thomas Rausch 24 Measurable Properties Change Metrics ● What was done to the software? Process Metrics ● How were the changes applied?
  24. 24. Thomas Rausch 25 Measurable Properties Change Metrics ● Complexity ● File types ● Date and time ● Author Changes .java .txt
  25. 25. Thomas Rausch 26 Measurable Properties Process Metrics ● Build history ● Build type ● Integration scenario b1 b2 b4 t b3 a b e c f d g VCS commit graph CI build information Topology mapping
  26. 26. Thomas Rausch 27 Statistical Correlation Analyses Categorical Variables ● {c1, …,cn} ~ { , } Pearson’s chi-square test Numerical Variables ● R ~ { , } Mann–Whitney U test D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. crc Press, 2003.
  27. 27. Thomas Rausch 28 Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ Meta statistics ● Using Fisher’s method ● Relative relation strength Van Zwet, W. R., & Oosterhoff, J. (1967). On the combination of independent test statistics. The Annals of Mathematical Statistics, 38(3), 659–680. article.
  28. 28. Thomas Rausch 29 PassedBuild outcome Failed Failed Passed Previous build result Percentageofbuilds Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ b b’
  29. 29. Thomas Rausch 30 PassedBuild outcome Failed Failed Passed Previous build result Percentageofbuilds Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ b b’
  30. 30. Thomas Rausch 31 Results Metric Meta Build history +++++ Build type ++ Author ++ Change complexity + Date and time + File types ~ Integration Scenario ~ Example ● Documentation file (changelog) ● 577 builds – doc. change only ● 14% original failures ● 52% test failures ● 45% environment crash ● 3% dependency error Similar behavior in all projects Noise skews statistics
  31. 31. V Build Failure Prediction
  32. 32. Thomas Rausch 33 Machine Learning ? Classifier Observations ? ?
  33. 33. Thomas Rausch 34 Experiment Design Well-known algorithms ● Naive Bayes ● C4.5 Decision Trees ● Random Forest Feature sets ● Process metrics ● Change metrics ● Combined Prediction for ● Binary (failed/passed) ● Multi-class (error type) Baseline ● 0-R classifier frequency table predicts the average
  34. 34. Thomas Rausch 35 Binary classification result Average F1 -scores Binary classification ● F1-score, 1.0 = perfect ● Ranging from ● 0.71 – 0.91 Algorithm 0-R NB C4.5 RF All CM PM 0.3 0.4 0.5 0.6 0.7 0.8 0.9
  35. 35. Thomas Rausch 36 Multi-class classification results Multi-class classification ● ● RMSE ● Error between actual and predicted Average RMSE p=[ p1 ... pn] Algorithm 0-R NB C4.5 RF All CM PM 0.0 0.3 0.1 0.2
  36. 36. Thomas Rausch 37 Update Predictions False positive elimination ● Likelihood for error declines ● Plausibility of prediction declines tmax t50 t75
  37. 37. VI Summary & Conclusion
  38. 38. Thomas Rausch 39 Research Questions - Answered Which errors cause CI build failures? ● Failing tests (41%) ● Coding-rule violations (11%) ● Compilation errors (10%) ● A number of errors involving the CI environment
  39. 39. Thomas Rausch 40 Research Questions - Answered What factors influence build outcomes? ● Process Metrics ● Build history failures persist ● Build type merges vs. forward engineering ● Change Metrics ● Actually, not so much
  40. 40. Thomas Rausch 41 Research Questions - Answered Can we predict build failures? ● Yes! ● We can update predictions during the execution State Rate min max Passed 87% 45% 99% Failed 66% 27% 96%
  41. 41. Thomas Rausch 42 Contributions In-depth analysis of CI workflow ● Multiplicity of build errors ● Factors influencing build failures Topology mapping Baseline for CI build failure prediction
  42. 42. Thomas Rausch 43 Future Work Examine more influence factors ● Developer behavior ● Project management workflows Improve learning methods Incorporate prediction into CI tools ● Improve feedback mechanism ● Improve development productivity
  43. 43. The end? Questions & Answers Thomas Rausch t.rausch@infosys.tuwien.ac.at
  44. 44. Thomas Rausch 45 Research Subjects Project Name Description Apache Storm Distributed Computation Framework Butterknife Android Dependency Injection Library Crate.IO Scalable SQL database JabRef Graphical Java application for managing BibTeX databases jcabi-github Object Oriented Wrapper of GitHub API Hystrix Latency and fault tolerance library for distributed systems Presto Distributed SQL query engine for big data Openmicroscopy Microscopy data environment RxAndroid RxJava bindings for Android Sponge API Minecraft plugin API Spring Boot Java Application Framework Square OkHttp HTTP+HTTP/2 client for Android and Java Square Retofit HTTP client for Android and Java Wordpress-Android WordPress for Android
  45. 45. Thomas Rausch 46 Integration Scenarios a b c f e d g a b c e d f a b c e d a b c d Initialization Update
  46. 46. Thomas Rausch 47 Related Work Build Failure Prediction ● Hassan and Zhang 2006 ● Outdated assumptions ● No CI workflow considerations ● Kerzazi et al. 2014 ● Only statistical analysis, no prediction ● Only binary build outcome ● Wolf 2009, Schröter 2010 ● Socio-technical factors ● Closed source enterprise software
  47. 47. Thomas Rausch 48 Data Source { "id": 22555277, "commit_id": 6534711, "number": "784", "pull_request_number": "1912", "pull_request_title": "Example PR", "started_at": "2014-04-08T19:37:44Z", "finished_at": "2014-04-08T19:52:56Z", "duration": 2648, "state": "failed" "commit": { ... } } Travis-CI ● Hosted CI service ● RESTful API ● Integrated with GitHub
  48. 48. Thomas Rausch 49 Runtime Evolution

×