Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Securing
Application
Deployments in
CI/CD Environments
Hello!
I am Binu Ramakrishnan
Principal Security Engineer
at Yahoo
You can find me at:
@securitysauce
Overview
◎CI/CD platform overview.
◎High level objectives.
◎Threat modeling.
◎Security patterns and best practices.
CI/CD
CI/CD Deployments
◎Single-tenant
◎Multi-tenant
Single-tenant
Dedicated build environments.
Multi-tenant
Shared build environment.
Driven by economies of scale.
CI/CD in Nutshell
Enterprise CI/CD
High Level Security
Objectives
Verifiable
Chain of
Trust
No Elevated
Privileges for
Build Jobs
Threat Modeling
CI/CD Admin
Exclusive access to CI/CD
platform
Actors
Developer
User/customer of the
platform.
CI/CD Platform
Developer
Li...
Trust
Boundaries.
https://www.flickr.com/photos/nasamarshall/14596371842
Trust
Boundaries
Web interface
◎ UI
◎ REST APIs
Application
Trust
Boundaries
Isolation based on Docker containers or
VMs.
Host
Trust
Boundaries
Segmentation
Network
Attack Surface
Entry Points
Exit Points
Entry Points
◎ Build web interface.
◎ Commit and build notification handlers.
◎ Source and package dependencies.
Internal Entry Points
◎ Build slave endpoint (to launch builds; SSH).
◎ Other control plane endpoints/REST APIs.
Exit Points
◎ Deployable artifacts.
◎ Build notifications - eg emails, IRC and Hipchat
messages.
◎ Build console logs.
◎ G...
Threats
Common Threats
◎ User account compromise & insider
threats.
○ User and platform admins.
◎ Network Intrusion.
○ CI/CD inter...
Build Slave Compromise
Large attack surface, spread across multiple networks (iphone, Android,
Server app etc.)
How?
◎ A n...
Shared/Non-ephemeral Keys
◎ Locally stored long-lived keys to access protected
services.
Two forms:
◉ Shared key
◉ Per-job...
SSH over Unrestricted Shell
Allows arbitrary commands to get
executed on a remote host.
Building External Code
An attacker can take this path to get into internal
networks, either by adding backdoors or exploit...
Baseline security controls that are
expected to be there in 2016
Existing Security Controls
◎ Authenticated endpoints.
◎ Web application security practices - CSRF,
HTTPS, HTTPOnly cookies...
Security Patterns for
Risk Reduction
Ephemeral Keys
https://www.flickr.com/photos/articnomad/241620406
Stateless Auth
Architecture
https://www.flickr.com/photos/18946008@N06/14551311971/
◎ Events:
○ Commit trigger.
○ Manual trigger from build UI.
○ Automated/cron job.
○ Trigger a downstream job.
◎ Upstream s...
Audit Trails
https://www.flickr.com/photos/adriensifre/8403355648
Container,
ToolChain
Hardening
http://www.publicdomainfiles.com/show_file.php?id=13493588417214
Network Segmentation
https://www.flickr.com/photos/bretagne-
balades/15355029654
Do not pull PII or other sensitive info to build
machine.
Minimal Builds
Few more...
SSH: Use
Restricted
Shells
Headless SSH access for
automated deployment
should use a restricted
shell.
Roll Ke...
Acknowledgements
◎ Christopher Harrell
◎ St John Johnson
◎ Mike Shema
◎ Jeremiah Wuenschel
Let’s recap major concepts
Ephemeral
Keys
Ephemeral keys are the
future. Service providers
should start supporting
ephemer...
Thanks!
Any questions?
You can find me at:
@securitysauce
Appendix-A
Use
◎ Source code
◎ Pre-built packages
Risks
◎ Targeted back doors
◎ Vulnerabilities
Hard Problem: Securing Supply Chain
T...
Appendix-B
Credits
Special thanks to all the people who made and
released these awesome resources for free:
◎ Presentation template b...
Securing Application Deployments in CI/CD Environments (Updated slides: http://bit.ly/2fOipxt)
Securing Application Deployments in CI/CD Environments (Updated slides: http://bit.ly/2fOipxt)
Upcoming SlideShare
Loading in …5
×

Securing Application Deployments in CI/CD Environments (Updated slides: http://bit.ly/2fOipxt)

1,465 views

Published on

In a multi-tenant CI/CD environment, developers trust and delegate CI/CD systems to deploy their applications to production. But, what is the basis of this trust? How the trust is enforced from commit-to-deploy? What is the trustworthiness of the application deployed by CI/CD through automation? This talk highlights security risks with CI/CD deployments and offer solutions to mitigate those risks

Published in: Technology

Securing Application Deployments in CI/CD Environments (Updated slides: http://bit.ly/2fOipxt)

  1. 1. Securing Application Deployments in CI/CD Environments
  2. 2. Hello! I am Binu Ramakrishnan Principal Security Engineer at Yahoo You can find me at: @securitysauce
  3. 3. Overview ◎CI/CD platform overview. ◎High level objectives. ◎Threat modeling. ◎Security patterns and best practices.
  4. 4. CI/CD
  5. 5. CI/CD Deployments ◎Single-tenant ◎Multi-tenant
  6. 6. Single-tenant Dedicated build environments.
  7. 7. Multi-tenant Shared build environment. Driven by economies of scale.
  8. 8. CI/CD in Nutshell
  9. 9. Enterprise CI/CD
  10. 10. High Level Security Objectives
  11. 11. Verifiable Chain of Trust
  12. 12. No Elevated Privileges for Build Jobs
  13. 13. Threat Modeling
  14. 14. CI/CD Admin Exclusive access to CI/CD platform Actors Developer User/customer of the platform. CI/CD Platform Developer Limited admin access to platform.
  15. 15. Trust Boundaries. https://www.flickr.com/photos/nasamarshall/14596371842
  16. 16. Trust Boundaries Web interface ◎ UI ◎ REST APIs Application
  17. 17. Trust Boundaries Isolation based on Docker containers or VMs. Host
  18. 18. Trust Boundaries Segmentation Network
  19. 19. Attack Surface Entry Points Exit Points
  20. 20. Entry Points ◎ Build web interface. ◎ Commit and build notification handlers. ◎ Source and package dependencies.
  21. 21. Internal Entry Points ◎ Build slave endpoint (to launch builds; SSH). ◎ Other control plane endpoints/REST APIs.
  22. 22. Exit Points ◎ Deployable artifacts. ◎ Build notifications - eg emails, IRC and Hipchat messages. ◎ Build console logs. ◎ Git repo update with build status and badges.
  23. 23. Threats
  24. 24. Common Threats ◎ User account compromise & insider threats. ○ User and platform admins. ◎ Network Intrusion. ○ CI/CD internal and external endpoints.
  25. 25. Build Slave Compromise Large attack surface, spread across multiple networks (iphone, Android, Server app etc.) How? ◎ A network level compromise, exploiting a vulnerability in build slave. ◎ Jobs break out of build container. Impact ◎ Access to production servers. ◎ Listen to the network, spoof identity and access unauthorized data.
  26. 26. Shared/Non-ephemeral Keys ◎ Locally stored long-lived keys to access protected services. Two forms: ◉ Shared key ◉ Per-job keys (eg. OAuth tokens)
  27. 27. SSH over Unrestricted Shell Allows arbitrary commands to get executed on a remote host.
  28. 28. Building External Code An attacker can take this path to get into internal networks, either by adding backdoors or exploiting known vulnerabilities with open source software.
  29. 29. Baseline security controls that are expected to be there in 2016
  30. 30. Existing Security Controls ◎ Authenticated endpoints. ◎ Web application security practices - CSRF, HTTPS, HTTPOnly cookies, XSS protection etc. ◎ Source control: granular authorization for repos. ◎ Use KMS to manage secrets; no hardcoded secrets.
  31. 31. Security Patterns for Risk Reduction
  32. 32. Ephemeral Keys https://www.flickr.com/photos/articnomad/241620406
  33. 33. Stateless Auth Architecture https://www.flickr.com/photos/18946008@N06/14551311971/
  34. 34. ◎ Events: ○ Commit trigger. ○ Manual trigger from build UI. ○ Automated/cron job. ○ Trigger a downstream job. ◎ Upstream service stores downstream service credentials (OAuth, Shared Keys etc.). ◎ Equal trust on all components in the pipeline
  35. 35. Audit Trails https://www.flickr.com/photos/adriensifre/8403355648
  36. 36. Container, ToolChain Hardening http://www.publicdomainfiles.com/show_file.php?id=13493588417214
  37. 37. Network Segmentation https://www.flickr.com/photos/bretagne- balades/15355029654
  38. 38. Do not pull PII or other sensitive info to build machine. Minimal Builds
  39. 39. Few more... SSH: Use Restricted Shells Headless SSH access for automated deployment should use a restricted shell. Roll Keys Periodically Establish a process to periodically roll trust anchor keys (and do it periodically). Restrict Job Console Logs Restrict build job console logs only to authorized users Enable 2FA Admins must follow good security hygiene and use 2FA to access platform application and hosts Prune Admin Access List Keep admin list small for build systems and Git repo access Vulnerability Patch Mgmt Maintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure
  40. 40. Acknowledgements ◎ Christopher Harrell ◎ St John Johnson ◎ Mike Shema ◎ Jeremiah Wuenschel
  41. 41. Let’s recap major concepts Ephemeral Keys Ephemeral keys are the future. Service providers should start supporting ephemeral keys for authorization Stateless Auth Architecture Augment the trust dependency of 1:1 relationship between the pipeline components with workflow job tokens. Audit Logs A verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD. Minimal Builds Avoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum Network Segmentation Network level Isolation of CI/CD machines from other machines ToolChain Hardening Build tools and Docker containers must be adequately hardened.
  42. 42. Thanks! Any questions? You can find me at: @securitysauce
  43. 43. Appendix-A
  44. 44. Use ◎ Source code ◎ Pre-built packages Risks ◎ Targeted back doors ◎ Vulnerabilities Hard Problem: Securing Supply Chain This threat is more to do with applications than the CI/CD platform itself. Open source components constitute a large part of modern Internet based applications.
  45. 45. Appendix-B
  46. 46. Credits Special thanks to all the people who made and released these awesome resources for free: ◎ Presentation template by SlidesCarnival ◎ Photographs by Unsplash & Death to the Stock Photo (license)

×