Informational article which will discuss the issues with code signing solutions as they relate to ci/cd workflows (including DIY and HSM solutions). Targeted Persona: mostly technical decision makers and operational champions (devops/devsecops).

1. The Challenge of Integrating Security Solutions with CI/CD Workflows
Created for UnBound Security
In Mobile World Conference (MWC), 2019, Satya Nadella, the Executive Chairman and CEO of
Microsoft famously reiterated “Every company is now a software company”. This message
was also stated by Watts S. Humphrey, the father of quality in software and CMMI, about two
decades ago, when he said “Every business is a software business”.
In today’s native digital world, we know that a Bank is a software company and a car is a
computer – computing is a core part of every industry.
Those of us who have been around in the software industry well understand that DevOps or
CI/CD workflows form the bedrock of this software. It is the CI/CD pipelines or workflows
that churn out software faster, better and more secure.
As the role of software becomes more pervasive, the role of security across the software
development lifecycle becomes paramount. Enterprises have been taking security very
seriously, since a breach can lead to loss of repute as well as heavy financial losses.
The Covid-19 pandemic has further accelerated security deployments and investments
across Enterprises.
The 2021 State of Security Operations survey highlights this changing trend:
2021 State of Security Operations Research Report (microfocus.com)
The report states:
• 85% of respondents say their companies increased their security budgets
• The same percentage increased their adoption of cloud-security services and
technologies
• 82% say they have increased the adoption of threat intelligence

2. With the heavy focus and investments on security, threat intelligence and detection is one
of the key components. Verifying that the deployed code is legitimate so that threats such
as supply chain attacks can be mitigated is most critical. The need to ensure that the
software was provided from the stated vendor and was not tampered by a malicious
adversary and contains malware or any unwanted code is most critical.
The method for protecting code is using digital signatures and PKI – specifically using code
signing certificates.
There are a number of approaches for securing code signing certificates:
• DIY: A Do It Yourself, custom built code signing solution integrating with your CI/CD
pipeline. While its tempting to build something that fits perfectly into your
ecosystem, it’s a classic build-vs-buy decision.
• Hardware Security Module (HSM): These are FIPS-40 certified, dedicated hardware
devices, requiring special expertise to deploy and maintain. Hyper scalers typically
provide their own HSM’s and that makes an Enterprise solution (on premise + cloud)
much more difficult to manage
• 3rd
Party vendors: Use HSMs as a root of trust for a 3rd party code signing solution.
While this centralises the Management of code signing certificates from 3rd
party, it’s
still a hardware based solution. It also restricts CI/CD integration, due to limited
availability of “signer utilities” by HSM
• SECaaS: This provides centralised management of code signing certificates without
the HSMs or any other backend software. In certain cases, SECaaC services requires
the code signing certificates to be generated by a certain, specific Certificate
Authority (CA), and does not allow to use code signing certificates from any CA.
• Niche solution: There are specialised solutions that Centrally manage code signing
certificates without dedicated hardware, support signing of any code, integrate with
CI/CD platforms and include additional security layers, such as scan the file for
malwares before it is signed, in order to mitigate supply chain attacks.
Let’s now explore how all these aspects are brought together in the CI/CD pipeline.
That’s the place where all dimensions of security is embedded in.
Illustration of how security is integrated through the CI/CD pipeline

3. 1. Agile backlog: The Security NRF Requirements are captured in the Agile tool (Jira,
Rally, Azure DevOps etc.), so that they can be implemented at the appropriate layer
(infra/app/db etc.) during the SDLC
2. Development IDE: Depending on what security tools the Enterprise uses, their
corresponding IDE (Eclipse, IntelliJ IDEA, Visual Studio etc.) plugins are available, so
that Developers can “left-shift” - detect and fix issues before the code leaves the IDE
itself!
3. Build Tools: Build tools like maven, gradle etc. have security checks embedded in like
OWASP vulnerability etc.
4. CI System: CI system like Jenkins, TeamCity, Azure DevOps etc. have plugins for
SAST/DAST security tools like SonarQube, Veracode etc. This is where the first stage
gate is typically implemented. If a critical vulnerability is found, the build is
“terminated” and the entire pipeline stalled
5. Environment Provisioning: While spinning up environments using infra-as-code tools
like terraform, chef, puppet etc, care has to be taken to adhere to Enterprise security
guidelines
6. Database Deploys: Tools like Liquibase or Datical treat database-as-code and enable
code reviews and automated deploys, with stage gate implementation
7. App Deploys: Whether you use tools like Jenkins, TeamCity, Azure DevOps, or cloud
native ArgoCD or more sophisticated IBM UrbanCode Deploy, they have integrations
with various security facets like verification of code signatures etc.
8. Testing: This is where you might have specific focus on security testing using tools
such as Acunetix, Netsparker, ZED Attack Proxy (ZAP)), and Pen testing with
Netsparker, Wireshark, Burp Suite etc.
9. Production System: Monitoring of Production Systems and their security posture
through Argus, Splunk, SolarWinds, Nagios, OSSEC etc.
Given the plethora of tools for each tenet in the CI/CD space, you would imagine that
embedding security into the pipeline would be a pretty standard and mature practice.
However, this is far from the reality of most Enterprises.
Here are the top 5 reasons why it is challenging to integrate security solutions into CI/CD
pipelines:
1. No gold standards for security: With the huge variations in technology and toolset
landscape across Enterprises, there are no gold standards for security. Of course,
there are regulatory and compliance requirements. Some industry guidelines are
available, however, their implementation specifics are often left open, to suite

4. inclusion
2. It’s beyond the SAST tools of the world: As a DevOps Consultant, I often hear this –
“Of course we embed security into the SDLC – we use SonarQube”. That’s a great
place to start; however, there’s an entire world out there beyond this one!
3. Loss of control: Traditionally, there was an elusive Security Team, which got invoked
during the Pre-Release phase, and who had the all-encompassing power to stop a
Release! Now, with “Continuous Release” making Release itself a non-event and
everything-as-code embedding security, the Security Teams tend to feel threatened
4. Developer resistance: With everything-as-code, the boundaries of a developer’s role
are getting blurred. They are increasingly becoming responsible for not just the
application, but also the infrastructure, configuration, deployments etc. And not all
developers like this evolution
5. Complexity of the game: As Enterprises become composable and cloud native, their
complexities grow disproportionately. It’s hard before it becomes simple.
As an example, the cryptographic key management is very complex and demands a
niche solution. It needs a unified key management and protection platform which
controls and manages all keys anywhere – on-premise, in the cloud, any cloud. Easily
integrates to existing solutions, and includes native support for all standard libraries,
including KMIP. That’s a pretty big ask!
Marc Andreessen, the co-author of Mosaic, the first widely used web browser; co-founder
of Netscape famously said a decade ago – “Software is eating the world”. As this statement
continues to be true in this decade and more, the importance of integrating security with
CI/CD pipelines is now more than ever.