This document summarizes the PVS-Studio development team's experience moving from CruiseControl.NET (CCNet) to Jenkins as their continuous integration server. Some key issues with CCNet included that it was no longer being developed and had unstable source code management. Jenkins provided more flexibility through plugins and allowed separating build steps and logs through the Multijob plugin. This helped replicate CCNet's task visualization. Overall, Jenkins met their needs and provided ongoing support through active development.

1. Moving from CruiseControl.NET to Jenkins
in the PVS-Studio development team
Author: Svyatoslav Razmyslov
Date: 07.02.2017
Now it's hard to imagine software development without automated project builds and testing. There are
various ready-made solutions to minimize the time expenses for the integration of the modifications
into the project. In this article I am going to speak about the way PVS-Studio team changed the
continuous integration server from CruiseControl.NET to Jenkins I will also be talking about the motives
behind this decision, the goals we tried to pursue and the issues we had to deal with during that
process.
Introduction
Continuous integration - is an automated process of building, deploying and testing software. This
practice is popular both among large development teams and individual developers. There is quite a
number of solutions for this practice. In this article we are going to talk about free open source projects
CruiseControl.NET and Jenkins.
CruiseControl.NET(CCNet) - a tool for continuous integration of software implemented on the .NET
Framework. There are also variants of the tool in Java (CruiseControl) and a version for the Ruby-
environments (CruiseControl.rb). You can view and manage the information about the builds through
web-interface or a desktop utility. It integrates with different version control systems. Is an open source
project but, unfortunately, is not developing since 2013.
Jenkins- a tool for continuous integration with open source, written in Java. It was forked from the
Hudson project after an argument with Oracle. Providing functions of continuous integration, it allows
automating a part of the development process, which does not require the involvement of human
developers. The abilities of Jenkins can be extended through plugins. At the moment the project is
actively developed and supported by the developers and the community.
Although this article may look like a review in the "CCNet vs. Jenkins" style, the choice is already made
for a server with Jenkins. The main reason why we changed the tool for continuous integration is that

2. the CruiseControl.NET project is no longer developing. There were also other issues when working with
CCNet that we are going to cover later.
Our project has quite a long history - recently we had a 10-year anniversary, you may read the story in
the article "PVS-Studio project - 10 years of failures and successes". We were using CCNet for more than
5 years. We got so used to its interface, settings and functions, that Jenkins seemed to be really
uncomfortable. First, we started using it with the release of PVS-Studio for Linux. The switch to the
Jenkins platform was preceded by a long study of the tool. Some time was spent on looking for the
similar functions from CCNet. Further on, I will describe the most interesting moments from our work.
Our claims for CCNet
1. CCNet is no longer developing. It can be used, but in case you want to expand the functionality
or fix existing/potential errors, you will have to do that by yourself.
2. The Source Code Management mode is working unstably, namely, for the automatic launch
when there are changes in the version control system. In case of problems with the network, in
this mode a project gets the status "Failed", even if it wasn't run. In practice, the problem occurs
so often (sadly, our office doesn't have the most stable Internet access), that this mode
becomes impossible to use.
3. When querying SCM on the edits, in case of an error in the version control system (for example,
if some directory was removed from a repository, specified in the settings, causing a tree
conflict), the execution of the project will be immediately interrupted; still, the status will be
kept "Success" - a project will cease working, but its status will remain "green " in the web
interface and desktop utility. In such a mode the launch of tests can be not done for weeks and
there is a risk that no one will pay attention to it, thinking that tests are working correctly.
4. The general server performance log is too verbose and unstructured: it's hard to understand
which step of the build failed and find a log specifically for this step. When running multiple
projects in parallel, the build log gets "mixed". The XML log of the build of a separate project is
available in the web interface, but quite often is not really detailed and doesn't contain all the
commands that were run.
5. Inefficient paralleling of subtasks inside a project. The subtasks are run in parallel by groups by
the number of the processor cores. If long and quick commands get to the group, then the new
tasks won't be run, until all the tasks from the previous run aren't completed.
Comparison of usage scenarios
Server settings
The settings of the CCNet projects (configuration of the server) were stored in one xml file, and various
passwords in a different one. Although the settings file reached the size of ~4500 lines, it was still quite
convenient to use it. By pressing Alt+2 in Notepad++, the whole list of projects can be minimized and
you can edit the one you need (figure 1).
Figure 1 - Editing the CCNet settings in Notepad++

3. Although the file had some duplicate code, there were no difficulties with the server support.
Here is how the SCM block was configured:
<svn>
<username>&SVN_USERNAME;</username>
<password>&SVN_PASSWORD;</password>
<trunkUrl>&SVN_ROOT;...</trunkUrl>
<workingDirectory>&PROJECT_ROOT;...</workingDirectory>
<executable>&SVN_FOLDER;</executable>
<deleteObstructions>true</deleteObstructions>
<cleanUp>true</cleanUp>
<revert>true</revert>
<timeout units="minutes">30</timeout>
</svn>
Here is how the MSBuild block was configured:
<msbuild>
<description>PVS-Studio 2015</description>
<workingDirectory>&PROJECT_ROOT;...</workingDirectory>
<projectFile>...PVS-Studio-vs2015.sln</projectFile>
<buildArgs>/p:Configuration=Release</buildArgs>
<targets>Build</targets>
<timeout>600</timeout>
<executable>&MSBUILD14_PATH;</executable>
</msbuild>
Here is how the block for common tasks was configured
<exec>
<description>PVS-Studio 2015 sign</description>
<executable>&PROJECT_ROOT;...SignToolWrapper.exe</executable>
<baseDirectory>&PROJECT_ROOT;...</baseDirectory>
<buildArgs>"&SIGNTOOL;" ... PVS-Studio-vs2015.dll"</buildArgs>
<buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>
On the basis of such a project file, CCNet displays very conveniently all the executable steps (albeit only
in the desktop tray utility, for some reason the web-interface didn't support this). In Jenkins we had
some troubles with the "high-level" display of stages of completing an integration project, but we will
speak about that later.
In Jenkins you have to store quite a number of settings files: a server configuration, settings files for
some plugins, each project has its own configuration file. Although all of these files exist in the xml
format, they aren't very convenient to view (at least in comparison with CCNet), because all the
commands in the tags are written as plain text. The truth is, this is largely connected with the ideology
of the use of the instrument. In CCNet the config is written manually, so it can be "nicely" formatted.
Jenkins suggests editing project settings using its web interface and automatically generates configs.
Here is an example of how the commands look like in the Jenkins configs:
<hudson.tasks.BatchFile>
<command>CD &quot;%BUILD_FOLDERS%Builder&quot;&#xd;
PVS-Studio_setup.exe /VERYSILENT /SUPPRESSMSGBOXES ...&#xd;

4. Publisher_setup.exe /VERYSILENT /SUPPRESSMSGBOXES</command>
</hudson.tasks.BatchFile>
And this is quite a small example.
View the task statuses
As I have written before, in CCNet the projects are configured with Task blocks. Here is how a
successfully completed task looks with all the steps (figure 2).
Figure 2 - viewing the status of tasks in the CCTray (a desktop client for CCNet)
An error in any of the blocks can be clearly seen in the hierarchy of subtasks. It is very convenient and
illustrative visualization of the integration process. We almost never had to search for logs, it was clear
looking at the description what requires checking on the local computer. We could not find such a
feature in Jenkins, that's why we had to study this question really thoroughly before moving to Jenkins.
We can draw such analogy between CCNet and Jenkins: there is a project in CCNet (Project), the 'step' in
this project is called a Task (as you can see on the figure above). In Jenkins there is also a project (Job),
but its 'steps' are called the same - Steps (figure 3).
Figure 3 - Corresponding names in CCNet and Jenkins

5. Unfortunately, the web interface of Jenkins cannot visualize individual steps - the Job has only a full
console log of all steps together. A big disadvantage is that you cannot see in Jenkins which of the steps
failed - you need to watch the full log of the Job build. And as people get used to good things very
quickly, we didn't want to abandon the old usage scenario. At that time we discovered Multijob Plugin
This plugin allowed us making the following improvements:
1. Using Jobs as Steps in other Jobs. Thus we got universal Jobs that allowed separating the log of certain
subtasks and the main thing to view statuses of separate subtasks. The Jenkins web interface can
visualize quite well the completion of individual Jobs within a Multijob - it's exactly what we were
looking for. Figure 4 shows an example of a completed Multijob.
Figure 4 - viewing a completed Multijob.

6. 2. We managed to get rid of duplicate code using universal Jobs. For example, there is a compilation of
some utility: for the distribution, for tests, for the code analysis. In CCNet these were same Task blocks
in 3 different projects. To compile this utility in Jenkins there is a Job that is used by several Multijobs.
3. During the creation of projects in Jenkins, we have the following workflow. We divide all the Jobs into
Multijob "projects" and universal "steps". The names of the universal Jobs have a prefix "job_" and do
not suggest using as a separate (stand-alone) project. They also don't have loading the source code from
the repository. The names of Multijobs have the "proj_" prefix and include loading of the source code
and running only other Jobs. (We try to avoid Steps, because they are not visualized).
The universal Jobs are run with the following parameter:
WORKSPACE=$WORKSPACE
It means that the Job will be run in the working directory of the Multijob.
Thus we can get a log of the source files updates and logs of all the steps of the build separately. It
would be hard and meaningless to follow this ideology for all the projects. It is done only for several
biggest and most important projects that should be studied in detail in case of problems.
4. You can configure conditional and parallel launch of the Jobs in Multijob. Multijobs can run Multijobs
using the same ways. Thus you can combine launches of different projects: to start the build of all
installers or all the tests.
Viewing the build logs
It was extremely inconvenient to view the build logs of the projects, because they were mixed with the
server output and had a special markup. There is no such a problem in Jenkins and there is a possibility
to separate the logs for subtasks in some projects.
Getting the revision of the source code
In Jenkins a separate revision of the version is defined for every added link to the SVN repository. I.e. if
we add several directories, the numbers can vary greatly, but we need only one maximum number.
According to the documentation, we should work with this as follows:
If you have multiple modules checked out, use the svnversion command. If you have multiple modules
checked out, you can use the svnversion command to get the revision information, or you can use the
SVN_REVISION_<n> environment variables, where <n> is a 1-based index matching the locations
configured.
It was done in this very way: of all the obtained values SVN_REVISION_<n> , the maximum value was
chosen and added to the built programs.
Useful plugins for Jenkins
The enhancement of the tool with the help of the plugins allows flexible configuration of the server. The
plugins to run the tests in Visual Studio are perhaps the only ones that we refused to use. They had
additional mandatory launch parameters, that we didn't use that's why it was easier to create a
universal Job that would just start the tests from a command line. Unfortunately, Jenkins, being "out of
the box" couldn't do much of what we got so used to in CCNet. However, we manage "to get back" all
the necessary functionality.
Here is a list of those plugins that we found useful, with a small description:
1. Multijob plugin- allows using other Jobs as stages of the build with the possibility of sequential
and parallel execution.

7. 2. Environment Injector Plugin - a plugin that is used to specify global passwords. They are used as
environment variables, and the plugin hides the variable value in the log.
3. Pre SCM BuildStep Plugin - adding extra steps before executing the commands of the version
control system.
4. MSBuild Plugin - a handy plugin to build projects using MSBuild. In the settings you specify the
paths to different versions of MSBuild only once, later in the project you can easily add build
steps.
5. Parameterized Trigger plugin - add parameters of a project launch. For example, you can make a
choice of a trunk/stable branch to build a distribution.
6. Post Build Script Plug-in - performing additional steps after the build.
7. Throttle Concurrent Builds Plug-in - this plugin allows adjusting the number of concurrently
running project builds either globally or within a specific category. It lets you get the
functionality in Jenkins similar to the queues in CCNet - an ability to execute in parallel several
projects from different categories (queues), while providing consistent execution of the projects
within a single queue. For example, we have Installers queues (distributions) and Tests. We
want to have an ability to build some distribution in parallel with running the tests, but at the
same time, tests shouldn't work in parallel - there won't be enough "cores" on the server.
8. Build Name Setter Plugin - allows specifying the name in a necessary format:
Major.Minor.Revision.Build.
9. Dashboard View - allows adding a custom Jobs display in the browser. As we have universal
Jobs, that are of no use to run manually, we created a list without them with the help of this
plugin.
10. ZenTimestamp Plugin - a handy plugin that adds timestamps in the build logs.
Overview of desktop clients
To receive notifications from the CCNet we used a client for Windows - CCTray.
Here are variants that are now available to work with Jenkins:
1. CCTray - this program can be used for a Jenkins server as well. The projects will look approximately
the same as they were before (figure 5).
Figure 5 - CCTray screenshot
The description of CCTray as a Jenkins client:
 It is not developing, the same as CCNet;
 Can't show subtasks (works only for CCNet);
 Cannot run the projects;
 You can go to the project page by clicking on the title;
 Configurable display of projects (Icons, List, Details);
 Open source.

8. 2. CatLight (figure 6)
Figure 6 - a screenshot of CatLight
Client description:
 At this moment it is a beta version, the final version will be paid;
 There are crashes during the setup and work, glitches in the interface;
 When the PC "wakes up" after the hibernation, the status on the dashboard doesn't get
automatically updated.
 Can't show subtasks for Multijobs;
 Cannot run the projects;
 The project display is not customizable (the only possible display - figure 6);
 You can go to the project page by clicking on the title;
 Ability to view the status of last 5 launches and jump-click to them;
 Ability to view the progress of the running project;
 If several servers are added, they are conveniently separated by a line;
 Versions for Windows, Linux and Mac.
3. Kato (figure 7)
Figure 7 - screenshot of Kato
Client description:
 Can't show subtasks for Multijobs;
 Can run the projects; Unfortunately, does not support projects with a parameterized launch -
the utility crashes when trying to run such a project;
 Projects from different servers are displayed in one list and are not separated in any way (not
always convenient);
 Configurable project display (List, Grid);
 You can go to the project page by clicking on the title;

9.  You can view the latest log directly in the client, but due to lack of monospace text it is not very
convenient;
 Open source.
 Only for Windows.
4. CCMenu - a client only for Mac, open source. It is not relevant for us, but perhaps, some may need it.
Conclusion
Using CI is useful in any project. There is a great free tool Jenkins that was reviewed in the article, and
also there are a lot of other free and paid Cl. It is rather pleasant to use a developing the project: a large
number of upgrades are released for Jenkins and plugins. New solutions get created, as for example,
Blue Ocean project that is still in Beta. You may find it on the main page of the Jenkins site.
Although, the clients for monitoring Jenkins projects weren't much impressive to me. A lot of obvious
features are missing. Perhaps desktop clients are particularly in demand and it is more correct to use
only some web interface.
When moving to a new server, we couldn't use Jenkins as a Windows service, because in this mode the
UI tests do not run. We found a way out by setting up a server as a console application with a hidden
window.
If you have any comments to this material of some interesting solutions of the problems we've
mentioned here, we'll be glad to get them in the comment section of via a feedback form.