One of the main activities in software development is monitoring, which plays a vital role in verifying the proper implementation of processes, identifying errors, and discovering opportunities for improvement. Continuous Integration (CI) covers a set of widely adopted practices that enhance software development. However, there are indications that developers may not adequately monitor all CI practices. In this thesis, we delve deep into the ocean of monitoring CI practices. Our goal is to discover how this monitoring is conducted, demonstrate the advantages of monitoring CI practices, and highlight the challenges that need to be overcome. We conducted a series of studies to analyze the impact of specific CI practices on the volume of Pull Requests and bug-related Issues, as well as the importance attributed to these practices by developers and the support for monitoring provided by the most popular CI tools. Finally, we evaluate monitoring in real-world scenarios, conducting a case study on three projects from three different organizations, where we investigate more deeply the developers' interest in monitoring CI practices, their benefits, challenges, and the evolution of CI practices over a two-month period. In our first study, we analyze the impact of specific CI practices on the volume of Pull Requests and bug-related Issues. Our findings revealed a positive correlation between CI practices and the increase in the number of merged pull requests. We also identified a significant correlation between the number of bug-related issues and the number of bugs. Additionally, our findings suggest that higher values of CI practices can indicate better quality in the development process. In our second study, we investigate the importance attributed to these practices by developers and the support for monitoring provided by the most popular CI services. We find that developers usually monitor only code coverage and basic build metadata (e.g. build duration and build status). Developers expressed interest in monitoring CI practices if provided the opportunity. Furthermore, we have identified that several leading CI services still have initial support in monitoring CI practices. Finally, we evaluate monitoring in real-world scenarios, conducting a case study on three projects from three different organizations, where we investigate more deeply the developers' interest in monitoring CI practices, their benefits, challenges, and the evolution of CI practices over a two-month period. The case study revealed that monitoring CI practices provides several benefits to the project and is cost-effective. Participants demonstrated a strong desire to integrate CI monitoring dashboards with popular CI services.

1. A Deep Dive into Continuous
Integration Monitoring Practices
UNIVERSIDADE FEDERAL DO RIO GRANDE DO NORTE
CENTRO DE CIÊNCIAS EXATAS E DA TERRA
DEPARTAMENTO DE INFORMÁTICA E MATEMÁTICAAPLICADA
PROGRAMA DE PÓS-GRADUAÇÃO EM SISTEMAS E COMPUTAÇÃO
Jadson José dos Santos
PhD Candidate
Doctoral Defense
Advisor: Prof. Dr. Uirá Kulesza
Co-Advisor: Prof. Dr. Daniel Alencar da Costa
Natal-RN
4:30PM Thursday, Apr. 25, 2024
Background Wallpaper Credits: Jon Hutchinson. https://www.teachwire.net/news/6-ideas-for-taking-a-curriculum-deep-dive-with-resources/

2. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 2
INTRODUCTION
“Continuous Integration (CI) is a
software development practice
where members of a team integrate
their work frequently, usually each
person integrates at least daily.”
Martin Fowler

3. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 3
• There is extensive research into the benefits and
costs of Continuous Integration
INTRODUCTION
Hilton et. al. 2016.
Usage, costs, and benefits of continuous integration in open-source projects.
Costs
• CI systems have too high
maintenance costs
• CI takes too long to set up
• Our project doesn’t have automated
tests
• The developers on my project are
not familiar enough with CI
• …
Benefits
• CI makes us less worried about
breaking our builds
• CI helps us catch bugs earlier
• CI helps us deploy more often
• CI makes integration easier
• CI lets us spend less time
debugging
• …

4. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 4
LITERATURE
REVIEW
Zhao et al. 2017
After the transition to
using Travis CI: (i) the
increasing number of
merge commits, (ii)
test suite sizes seem
to increase, etc...
Vasilescu et al. 2015
After CI: (i) Teams are significantly
more effective at merging pull
requests. (ii) Core developers are able
to discover significantly more bugs
than in teams not using CI
Bernardo et al. 2018
After CI: a large increase in the
number of submitted, merged and
delivered PRs per release.

5. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 5
• However
• Most of studies focused in miner repositories and
have examined the overall impact of CI
• However … there is a lack of studies specifically
addressing the monitoring CI practices
LITERATURE
REVIEW

6. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 6
LITERATURE
REVIEW
42.5%
Of the primary studies did not provide explicit criteria
for identifying CI usage in projects.
56.25%
Among studies that applied only one criterion, CI
service configuration was the most common
Soares et al. 2021

7. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 7
LITERATURE
REVIEW
Felidré et al 2019
(i) The average code coverage of Java projects it
was 63%;
(ii) 60% of the projects have infrequent commits;
(iii) 85% of the projects have at least one broken
build that take a long time to be fixed
(iv) most of projects have a build that executes with
more than 10 minutes.
A fact known as CI THEATER.

8. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 8
AGENDA
OBJECTIVES
METHODOLOGY
STUDY 1
STUDY 2
STUDY 3
CONCLUSION

9. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 9
OBJECTIVES
This thesis aims to answer the following general research question:
• How important is monitoring CI practices?
• We hypothesize that development teams typically do not adequately monitor
project CI practices
• Consistently monitoring these practices over time can lead to more efficient
and optimized CI processes

10. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 10
METHODOLOGY

11. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 11
Study 1: Investigating the Impact of Continuous
Integration Practices on the Volume of Merged
Pull Requests and Closed Bug-Related Issues
of Open-Source Projects
Background Wallpaper Credits: Lombardy.https://www.en.regione.lombardia.it/wps/portal/site/en-regione-lombardia/discover-lombardy/statistic-and-observatories

12. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 12
OBJECTIVES
• Verify the potential influence of 5 CI practice* in the Volume of
merged Pull Requests (related to project productivity) and in the
Volume of bug-related Issue (related to project quality)
Study 1
2 years
First commit
travis.yml
Last Build
* 1) build duration, 2) build activity, 3) build health, 4) time to fix a broken build, and 5) commit activity

13. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 13
METHODOLOGY*
*
Similarly
to
Zhao
et
al.’s
work
Study 1
Most starred GitHub projects Relevant projects**
27,883 3,143 2,029 776
90
Have used Travis-CI 2 years using Travis-CI
Filter active projects **** Collect Info ***
*** PRs, Issues, commits, and builds
** more than 100 stars, 100 PRs, 100 Issues,10MB **** at least 1 build in all 24 month of study
Build LR Models and D.A.

14. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 14
METHODOLOGY
• RQ1: Which CI practices contribute to higher volume of merged
pull requests?
• We fit linear regression models (Ordinary Least Squares, OLS) to find
associations between the number of merged PRs and the 5 CI practices.
Study 1

15. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 15
RESULTS
• RQ1: Which CI practices contribute to higher volume of merged
pull requests?
Regression model
R2 0.353
Study 1

16. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 16
METHODOLOGY
• RQ2: Which CI practice contribute to higher volume of closed bug-
related issues?
• The number of bug-related IR present an excess number of zeros. We applied
zero-inflated negative binomial regression models (ZINB)
Study 1
24 months
0 0 0 0 2 0 0 0 0 1 0 0 0 1 0 0 0
Number of closed
bug-related issues

17. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 17
RESULTS
• RQ2: Which CI practice contribute to higher volume of closed bug-related issues?
*
Study 1

18. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 18
METHODOLOGY
• RQ3: Can consistently applied CI practices be an indicator of process quality?
Study 1
Projects where
practices are
above the
overall median
Projects where
practices are
below the
overall median
* a confidence level of 95% and a confidence interval of 10%
Randomly selected
91 and 88 PR*
PR Comments

19. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 19
RESULTS
• RQ3: Can consistently applied CI practices be an indicator of process quality?
Projects with worst CI practices Projects with best CI practices
Study 1

20. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 20
Background Wallpaper Credits: Lisa Melonçon. https://tek-ritr.com/survey-research/

21. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 21
OBJECTIVES
To better understand the importance and prevalence of
monitoring 7 CI practices, we:
• Document Analysis 1: Investigate whether developers express
the need for monitoring CI practices in their project repositories
• Survey: Where we ask questions regarding the monitoring of CI
practices
• Document Analysis 2: Perform a document analysis of the
official documentation of current state-of-the art CI tools related
to monitoring capabilities
Study 2

22. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 22
METHODOLOGY*
Study 2
Most starred GitHub projects * Using GitHub Actions
11,014 7,658 2,488
121
2 years using CI **
Filter active projects **** Relevant projects***
** with 6 months recent builds
* with more than 100 stars, 100 PRs, 100 Issues,10MB
*** more then 100 builds, issues, PRs, commits and coverage inputs inside study period
Identify Survey Participants
776
**** build information for all 24 months
Identify CI workflows
500
(28 answers)

23. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 23
METHODOLOGY
RQ1: Do developers express the need to monitor CI practices in
PRs?
Study 2
PR Comments
(4,892 comments)
Randomly selected
382 PRs*
73,004 PRs
from 121 projects
* a confidence level of 95% and a confidence interval of 5%

24. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 24
RESULTS
• RQ1:Do developers express the need to monitor CI practices in
PRs?
Study 2

25. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 25
• RQ2: What is the perceived importance of monitoring CI practices?
• Most projects (12/19): automatic execution of the CI build and the automatic execution
of tests define the project CI level
• Test related practices (31/85) are the most monitored CI practices
Study 2

26. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 26
RESULTS
• RQ2: What is the perceived importance of monitoring CI practices?
• Evaluation of CI practices
• Most most developers (20/28),
71%, have given positive
comment about the 7 CI
practices
• Almost half of developers
(11/24) would like to improve
CI practices values
Study 2

27. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 27
RESULTS
• RQ2: What is the perceived importance of monitoring CI practices?
Developer 10
Yes, these metrics will provide valuable information in the
status of our Project over time.
Developer 15
depends on practice, for example “commit frequency” is not
important at all, but fixing a broken build is”
Developer 18
Absolutely! It saves time in the long run and increases
confidence when refactoring is applied”
Study 2

28. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 28
RESULTS
• RQ2: What is the perceived importance of monitoring CI practices?
• Evaluation of CI
practices
• 68% of the
developers (19/28)
would like to
receive regular
updates of our
survey because
they think the
monitor the 7 CI
practices is useful
Study 2

29. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 29
METHODOLOGY RQ3: Which monitoring features are currently supported by existing
CI tools?
• We conduct a Grey Literature Review (GLR) to compile the existing monitoring
capabilities of existing CI services and third-party tools
• We conduct a search on the official documentation of most popular CI services*
• We examined the sections within the documentation that could be directly
correlated with the concept of monitoring
• We collect all references to third-party tools cited in the official CI services
documentation
• We conduct a search on the official documentation of third-party tools
* TRAVISCI, GHACTIONS, CIRCLECI, APPVEYOR, AZURE, GITLAB and JENKINS.
(Golzadeh, M., Decan, A., Mens, T.: On the rise and fall of ci services in github.
In: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER))
Document Analysis
+
Thematic Analysis
Study 2

30. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 30
RESULTS • RQ3: Which monitoring features are currently supported by
existing CI tools?
• CI services currently offer limited built-in support for monitoring CI
• Build metadata tracking is the most commonly supported monitoring practice (“Build
Duration” and “Build Status”)
• There are various complementary tools that can enhance this monitoring with a
predominant focus on fundamental information about the performance of the
CI pipeline
• However, they often come with additional costs and complexity
Study 2

31. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 31
RESULTS
Study 2

32. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 32
Study 3: Do you monitor CI
practices? I don’t know. You tell me:
A case study
Background Wallpaper Credits: Gratis vector handgetekende case study-sjabloon voor bestemmingspagina’s. Freepik. Free License

33. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 33
OBJECTIVES • We conducted a multiple case study with mixed methods.
• Collection of metrics in three public Brazilian organizations (5 participants).
• 5 Research Questions
• Over a period of 8 weeks, we conducted a series of interviews with team
members to gain insights into the benefits and challenges of monitoring CI
practices
RQ1, RQ2,
RQ3, RQ4,
RQ5
RQ1 RQ5 RQ4
Study 3

34. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 34
MONITORING
TOOL • We developed an extensive* suite of metrics tool that automatically
collect the evolution of 7 CI practices over time
* Supports: 17 metrics (7 CI + 4 DORA + 6 Basics) and integration 4 tools with GHActions, GITLAB, Sonarqube and CodeCov
Study 3

35. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 35
RESULTS
• Thematic Analysis
Study 3

36. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 36
RESULTS
• RQ1: Were the developers satisfied with the project’s CI maturity
levels?
• Most of the projects (
!
"
) were satisfied with the CI maturity level prior to the case
study.
• They did not utilize any metric-driven assessment to evaluate CI maturity
Participant P1
The tool, like the study, also emphasizes
the importance of providing an accurate
idea of how the CI is performing,
enabling individuals to understand the
metrics and reference values.
Participant P2
It’s something we’re still maturing,
we started from scratch.... Then, one
thing ended up helping the other,
seeing these metrics and what
needed improvement in our projects
Study 3

37. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 37
RESULTS
• RQ2: What benefits does CI monitoring bring?
• Give a overview of CI stage of a project
• Highlight problems related to CI
• Awaken a desire for improvements
• Improve the quality of CI adoption
• Increase communication within the team.
• CI monitoring also proved to be not costly to apply
Participant P1
It brings precisely this visibility of those aspects that can be
considered a problem. As I was talking about this coverage, we already
have this information on the Sonarqube. But the other ones are new
aspects that bring a more qualitative view of the CI practices we did
not have in other tools ...
Study 3

38. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 38
RESULTS
• RQ2: What benefits does CI monitoring bring?
Participant P2
It helped to highlight... The coverage is
evident; it was interesting because it showed
that there was coverage data, and suddenly, it
disappeared. I think it’s helping.
Participant P3
I think it ... pointing out that we are having problems with the code coverage
metric. Regarding the build, we already felt that the build took a long time and that
the VMs were slow, but not that much, more than 8 hours of build time. We have
tests, tests that involve the instantiation of other modules in VMs, ...., but almost
10h is a very high value.
Participant P5
It brings more confidence related
to some CI needs, for sure. It provided a
better view of this process.
Study 3

39. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 39
RESULTS
• RQ3: What are the challenges/disadvantages for using it on a daily
basis?
• The time dedicated to other development tasks as the main difficulty in
monitoring CI practice
• The lack of integration with other CI tools
• Legacy projects and the internal culture
• A complex development environment, reflected in the instability of the CI
environment
Participant P1
I get a little worried about not having time to
present what has been completed if I’m going to
do all the tests, then we end up having to fit the
tests into the free time we have in the
sprint
Study 3

40. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 40
RESULTS
• RQ4: Are developers interested in using the tool?
Participant P3
I received the email and
the problem of code
coverage appeared, as I
already knew about code
coverage, I did not pay
much attention to the tool
because I thought that
perhaps it would be more
of a priority to look at the
issue of the pipeline
Study 3

41. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 41
RESULTS
• RQ5: What was the evolution of CI practices during the use of the tool?
Organization A
Participant P1
… We already knew
about the coverage
problem from the
beginning, but it
became very evident.
With this information,
we began to act to
resolve
this problem
Study 3

42. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 42
RESULTS
• RQ5: What was the evolution of CI practices during the use of the tool?
Organization B
Participant P2
… Last week we had
some problems with
infrastructure, we
ended up having to
do some configurations
and such. Because of it,
the build health metric
started to have a
lower value
Study 3

43. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 43
RESULTS
• RQ5: What was the evolution of CI practices during the use of the tool?
Organization C
Participant P5
… There was also a
period at the end of last
year of the build being
generated in a broken
way … We had already
observed this in Gitlab,
but the monitoring tool
also contributed, in a
way, complemented this
need to fix it
Study 3

44. 4/27/24 A Deep Dive into Continuous Integration Monitoring Practices 44
Background Wallpaper Credits: https://stock.adobe.com/br/search?k=overview

45. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 45
Thesis
Main
Contributions Better Development Process
Quality: Our findings suggest that
higher values of CI practices may
indicate better quality in the
development process
Overlooked of CI monitoring: Most of
developers consider just run automatic builds
and coverage (tests execution) to evaluate
maturity of CI process
Monitoring CI Bring Benefits: We discovered strong evidences
indicating that developers place significant importance on monitoring
Continuous Integration (CI) practices and, this monitoring brings
several benefits for the project

46. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 46
Thesis
Main
Contributions
Basic Tools Support: Leading CI services and
complementary tools still offer basic support for monitoring
CI practices. Focus on monitoring fundamental
information related to the performance of the CI pipeline

47. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 47
Thesis
Main
Contributions
Monitoring Suite Tool: We developed an extensive suite
of metric tool that automatically calculate the values of seven
CI practices.
https://github.com/jadsonjs/holter-devops

48. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 48
Future
Work
?

49. 27/04/2024 A Deep Dive into Continuous Integration Monitoring Practices 49
What’s
Next? Extensions for new Practices:
Extending this work to assess additional
CI/CD practices in areas like DevOps
Extensions for new
Contexts: Our evaluations
can be replicated in more
extensive projects, testing the
robustness of our results
Tooling: 1) Improvements to our monitoring suite tool can be made. 2) We
strongly recommend that CI services (such as Gitlab, Github, CircleCi,
Jenkins, among others) strive to support CI practices monitoring natively
Measure CI maturity:
Deeper exploration of the
optimal practices values for
different contexts,
establishing ideal values to
determine the CI maturity
level
Extensions for CI practices benefits:
Exploring other ways of evaluating the
benefits of CI practices in software
development projects

50. 27/04/2024 50
A Deep Dive into Continuous Integration Monitoring Practices
Thank you http://caseufrn.github.io/
Background Wallpaper Credits: https://www.localdivethailand.com/phuket-diving-frequently-asked-questions/