2. People, process, and technology
Step 1: make sure your CI does not harm your security
Step 2: only then it can be used to improve your security
People
ProcessTechnology
4. Basic CI cycle
Developer Development machine Version Control repo CI Server
Write code +
tests
Pull latest code + tests
Local
build +
test run
Commit
Poll
Generate feedback
Feedback
Dashboard or gadget
“Continuous Integration assumes a high degree of tests
which are automated into the software”, Martin Fowler
5. Isolate your environment
• Phishing link in email => keylogger installed => source code gone (or backdoor
deployed)
• Experimenting with development network => accounting department affected before
EOY
• Extra challenges: remote work or BYOD
6. Basic CI cycle
Developer Development machine Version Control repo CI Server
Write code +
tests
Pull latest code + tests
Local
build +
test run
Commit
Poll
Generate feedback
Feedback
Dashboard or gadget
7. Version control server
• It has one job only – remove or disable everything else
• No shared or generic accounts
• Matching business process to close accounts
8. Basic CI cycle
Developer Development machine Version Control repo CI Server
Write code +
tests
Pull latest code + tests
Local
build +
test run
Commit
Poll
Generate feedback
Feedback
Dashboard or gadget
9. Integration build server
• Who is responsible for keeping it up to date?
• Where do external components come from?
• Check vendor advice on compiler and linker options
On September 18, 2014, RiskIQ
detected credential-stealing malware
being loaded onto users’ computers
through a drive-by download at
jQuery.com.
Not too long ago, news
broke about a malicious
version of Xcode dubbed
XcodeGhost that had been
surreptitiously injecting
malicious code into iOS
developers’ apps.
SK-CSIRT identified malicious software libraries in the
official Python package
repository, PyPI, posing as well known libraries. A
prominent example is a fake
package urllib-1.21.1.tar.gz, based upon a well known
package
urllib3-1.21.1.tar.gz.
10. Basic CI cycle
Developer Development machine Version Control repo CI Server
Write code +
tests
Pull latest code + tests
Local
build +
test run
Commit
Poll
Generate feedback
Feedback
Dashboard or gadget
11. Feedback mechanism
• IoT electronic toys are notoriously insecure
• Custom integration scripts - are you cutting corners?
12. Do no harm
• Do not acquire CI components « by accident »
• Not everything is secure out of the box
• Dormant account today is an attacker-controlled account tomorrow
- Compromising a staging continuous integration server (Jenkins) from a developer's
machine, which then gave us access to the company's production Chef repository,
allowing us to push a reverse shell to a production server.
- Using access to a dev/staging Jenkins server to modify code in repositories after they've
been through code review, which were then replicated to production.
https://www.blackhat.com/us-17/briefings.html#developing-trust-and-gitting-betrayed
13. Treat people right
• Don’t leave new starters to figure out CI rules on their own
• Controls are safety net, not surveillance
• Anyone on security team?
• Help devs to automate “boring” things
• Learn to say “yes”
• Security goals must support business goals
• Anyone in management?
• Don’t abuse metrics
16. Code reviews
• No change too small
• Leave trivial checks to tools
• Not a separate task, but in DoD for each task
• Reject & rework is part of “normal”
17. Root-cause analysis
• What happens to externally reported issues?
• The first security feedback to introduce
• What was missing in our CI process? => Improve
18. Chain of custody
• Can you trust your release notes?
• Has every “unit of work” in the release gone through all the checks?
• Was it modified since “time of check”?
19. Reference implementations
• Give dev teams libraries / tools that are secure by default
• Leverage frameworks (e.g. output encoding)
• Build your own
• Potential areas to consider:
• Managing secrets
• Anything related to crypto
• Authentication / authorization
• SQL, file system access, shell exec()
Core takeaway:
• “<X> is hard to do securely, have to be aware of threats 1, 2, and …”
• Build a secure by default implementation
• Automate the tests
21. If you didn’t laugh at the previous one…
• Static analysis – reason about code based on looking at it
• Dynamic analysis – run code and observe how it behaves
• Fundamental trade-offs:
• Static analysis – high coverage, imprecise (false positives)
• Dynamic analysis – code coverage is a challenge, but precise (the issues it finds are true
positives)
22. Pick your poison
Grep
• Localized
• Not syntax-aware
• Regex
Lint/AST
• Syntax aware
• Some semantics
• Can express rules that use
code constructs
SAST
• Builds detailed model of
code
• Complex rules that rely
on control and/or data
flow
23. Static analysis
• Run on dev machine needs to be fast, low to no false positives
• Ideally – integrated with IDE
• Run in CI needs to provide fast feedback
• Can be implemented with git hooks or as a part of the test suite
• Integration with version control is useful
• Integration with bug tracker is useful (but…)
• Ability to enable analysis by defect type is very useful
26. Points of contact
Irene Michlin
Principal Security Consultant
M: +44 (0) 7972 333 148
E: irene.michlin@nccgroup.trust
T: @IreneMichlin
Thanks and credit to Clint Gibler - @clintgibler and Antti Vähä-Sipilä - @@anttivs
https://www.nccgroup.trust/uk/our-research/securing-the-continuous-integration-
process/
https://www.nccgroup.trust/uk/our-research/best-practices-for-the-use-of-static-code-
analysis-within-a-real-world-secure-development-lifecycle/
Editor's Notes
On people, I only have one piece of advice - Hire the right people and treat them right. The rest of the talk is on process and technology.
The choice is enormous, not going to say “This is the recommended toolchain”. Not on the agenda, not on the hidden agenda.
It’s rather what to look for in my tools, and how to put them together so that first of all I’m not harming my organisation’s security. And the secondary goal – I have this neat CI pipeline going, can I use it to improve my software’s security, amongst other things.
To ensure we are on the same page for the rest of the talk.
What happens in dev, stays in dev
Physical or virtual, dev machines must be segregated from the main corporate environment.
Let’s talk about version control server
The choice of the version control pattern depends on multiple factors, such as frequency of releases, number of teams, structure of the code base etc. Generally speaking, patterns that require frequent commits to a shared branch are more suited for implementing CI.
SSO is better than proliferation of accounts per each tool.
RBAC with as much granularity as does not interfere with your work.
Needs administrative component to terminate the accounts as required.
Integration server
It only needs R/O access to source control
Set local repo for external libraries. Use scanners.
Standard builds remove chasing the “it worked on my machine” problems
Feedback mechanism
Who is listening? Unless it is a low-tech mechanism “All the developers are in the same room and they look at a physical dashboard when they feel like it”, you have to consider the security of the mechanism.
The tools are held together with what is often called “glue skills” – the developers are putting together ad hoc scripts, punching holes in firewalls, hardcode passwords.
Even if the script itself isn’t dangerous, if it’s world-writable but runs at high privilege, it has a high potential to become dangerous.
Glue skills are great to have in your team, but they can also be dangerous. Treat these scripts as first class citizens – under source control and under the same security standard as everything else.
My example is not an attack on a particular tool, literally I’ve searched for “broken build gadgets” and picked a first example.
CI environments are rarely planned upfront. The tools are introduced one by one and sometimes people only learn about everything they have during an audit, which is a bit too late.
Secure system lifecycle. The product choice and evaluation can be quick and informal, but it has to be a conscious choice.
Installation and configuration – don’t enable features you don’t need
Ensure account management lifecycle covers all the CI tools.
Luckily the industry is moving on from the “people are the weakest link” concept.
The figure above is based on several popular maturity matrices for continuous integration, as well as NCC Group experience with variety of clients.
Won’t have time to go through them all, only couple.
There is now understanding in the software development industry that security features are not enough for building a secure product. To achieve an acceptable level of overall security, the product needs all the features to be secure: the whole attack surface of the product must be considered from a security point of view, not only features that are directly security related, such as login functionality.
The same is true for securing continuous integration. In the matrix above, the dedicated Security row lists explicit security related activities that must happen at some point towards CI maturity. However, every cell on the other rows needs to be evaluated for the security of its tools and processes as well.
Frequently asked “are manual code reviews waste of time”? Not if done well.
If your decision has only one possible outcome, then why bother with the decision?
It’s waste of time.
Another source of waste of time in review is “saving up” to end of week/end of iteration. It’s all about quick feedback.
People like to rush into bug bounty programs
Don't trouble trouble until trouble troubles you
Whether you release each change individually in CD, or in batches, the same questions apply
I’ve seen a company with a checklist for each api function – authentication, authorisation, input, output. Makes code reivew expensive, boring, and error prone. They’ve moved all the “boilerplate” to be given to developers for free.
Test case that automatically checks for missing access controls and goes through the checklists
Either you love dogs very much, or you haven’t come across static analysis
There is a range of complexity in “light-weight” vs “heavy-weight” static analysis approaches
From simple text matching to understanding code syntax, semantics, and can reason about data and control flow
Each of these have strengths and contexts in which they are useful
The figure above is based on several popular maturity matrices for continuous integration, as well as NCC Group experience with variety of clients.
Won’t have time to go through them all, only couple.
There is now understanding in the software development industry that security features are not enough for building a secure product. To achieve an acceptable level of overall security, the product needs all the features to be secure: the whole attack surface of the product must be considered from a security point of view, not only features that are directly security related, such as login functionality.
The same is true for securing continuous integration. In the matrix above, the dedicated Security row lists explicit security related activities that must happen at some point towards CI maturity. However, every cell on the other rows needs to be evaluated for the security of its tools and processes as well.
Don’t throw all the tools at the dev team at the same time.
Fuzzers, static analysis, dynamic analysis all take time to learn and configure and initially will bring the productivity down, not up
Start with “new” code, ensure your debt isn’t growing.
No two journeys towards maturity are the same. It may make sense for your organisation to take a practice from Intermediate or Advanced before doing everything in Beginner. It meant to inform, not to restrict.
Hard to come with good questions on the spot, so I won’t force you. I’m here today, obviously, and if you have questions later, find me on twitter or by email.