Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Securing application deployments in multi-tenant CI/CD environments

1,144 views

Published on

The goal of the talk is to introduce you to, the security risks and challenges associated with operating or using a multi-tenant CI/CD platform, and offers security patterns and best practices to harden it.

Video: http://oreil.ly/2hVCilH

Published in: Technology

Securing application deployments in multi-tenant CI/CD environments

  1. 1. Securing Application Deployments in Multi-tenant CI/CD Environments O’Reilly Security Conference NY 2016
  2. 2. HELLO! I am Binu Ramakrishnan Principal Security Engineer at Yahoo You can find me at: @securitysauce github.com/prbinu
  3. 3. Overview ▹ A brief introduction to CI/CD platform ▹ Challenges with multi-tenant CI/CD environments ▹ Threat modeling ▹ Security patterns and best practices
  4. 4. CI/CD A CI/CD pipeline is a set of jobs chained to achieve continuous delivery, and deployment of applications to production.
  5. 5. CI/CD Deployments Single-tenant | Multi-tenant
  6. 6. Single-tenant ▹ Dedicated build environments ▹ Distributed ▹ Tenant and provider are the same ▹ Limited security impact
  7. 7. Multi-tenant ▹ Shared build environments ▹ Driven by economies of scale ▹ Focused on operational efficiency ▹ Allow enterprise to centrally enforce dev stds & compliance controls ▹ Gateway to enterprise networks
  8. 8. Multi-tenant CI/CD - Security Risks ▹ This platform can be an attractive target for attackers because: ▸ Consolidation of large number of long-lived keys in one place ▸ It acts as a corridor that directly connects to large number of machines, bypassing firewalls ▹ A successful attack can have a significant impact to an enterprise ▸ Leads to exposure of user’s personal info, application credentials, TLS private keys etc.
  9. 9. Denial of Service (DoS) Attacks * http://bit.ly/2exRpWm “Enterprises relying on SaaS apps had no choice but to sit and wait and hope their providers got back online as soon as possible.” *
  10. 10. CI/CD in Nutshell
  11. 11. Enterprise CI/CD
  12. 12. Threat Modeling
  13. 13. CI/CD Admin Exclusive access to the CI/CD platform Actors User/Developer Person who uses the platform and run their build pipelines. CI/CD Platform Developer Limited admin access to the platform.
  14. 14. Trust Boundaries ▹ Application - web interface, REST APIs ▹ Host - isolation between host OS and the build job ▹ Network - segment build machines from other machines and from Internet
  15. 15. Entry Points ▹ Build web interface ▹ Commit handlers ▹ Internal endpoints ▸ Build slave endpoint (to launch builds; SSH) ▹ Deployable artifacts ▹ Build notifications - eg. emails, IRC and Hipchat messages ▹ Build console logs . Attack Surface Exit Points
  16. 16. Threats 16
  17. 17. Common Threats ▹ User account compromise & insider threats ▸ User and platform admins ▹ Network Intrusion ▸ CI/CD internal and external endpoints
  18. 18. Local Storage of Non-ephemeral Keys ▹ Long-lived keys are stored locally to allow build jobs to access protected services. ▹ OAuth tokens, Shared secrets, SSH private keys etc.
  19. 19. Weak Security Controls Build jobs require access to services as part of their execution (eg. artifacts store). Protection mechanism used by those services are less effective in following cases: ▸ IP whitelist based protection ▸ Shared secrets: shared (and accessible) by all build jobs The above controls are susceptible to spoofing in multi-tenant environments.
  20. 20. Build Slave Compromise Large attack surface, spread across multiple networks (iphone, Android, Server app etc.) How? ▹ A network level compromise, exploiting a vulnerability on build slaves ▹ Jobs break out of build container Impact ▹ Listen to the network, spoof identity and access locally stored credentials ▹ Gain access to unauthorized machines and steal sensitive data
  21. 21. Building External Code An attacker can take this path to get into internal network, either by adding backdoors or exploiting known vulnerabilities with open source software.
  22. 22. Security Patterns for Risk Reduction
  23. 23. Ephemeral Keys https://www.flickr.com/photos/articnomad/241620406 JWT, OAuth, SSH-CA, AWS access key etc. ▹ Automated deployments require delegation ▹ Token Delegation: a controlled replay of tokens ▹ Ephemeral keys go well with token delegation model
  24. 24. Capability Delegation & Stateless Auth https://www.flickr.com/photos/18946008@N06/14551311971/ By passing capabilities (in the form of tokens) with build events, we eliminate the need to store long-lived keys in CI/CD platform
  25. 25. ▹ Separate out auth components from the core build platform ▹ The Auth service intercepts and authenticates build events, and issues job tokens to downstream build platform ▹ Stateless: Build platform no longer stores job specific long-lived keys ▹ Distributed: Per tenant Auth service for cloud based environments, hosted inside tenant’s network ▹ No consolidation of keys in one place
  26. 26. Audit Trails ▹ Immutable and append-only ▹ Traceability: logs should capture build pipeline events end to end from commit to final deploy. ▹ Trail traits: who, how, when, sequence of events, start and end state. ▹ Audit trails produced by the Auth service can provide a verifiable chain of trust from commit to deploy.
  27. 27. Container & ToolChain Hardening
  28. 28. Network Segmentation Colocating CI/CD platform with other corporate machines allow malicious build jobs to raid your internal network. The recommendation is to zone CI/CD platform from other corporate systems.
  29. 29. Minimal Builds ▹ Do not expose PII or other sensitive info to build machines. ▹ Do complex tests outside of build, in a separate environment you control.
  30. 30. Few more... SSH: Use Restricted Shells Headless SSH access for automated deployment should use a restricted shell. Roll Keys Periodically Establish a process to periodically roll trust anchor keys (and do it periodically). Restrict Job Console Logs Restrict build job console logs only to authorized users Enable 2FA Admins must follow good security hygiene and use 2FA to access platform application and hosts. Prune Admin Access List Keep admin list small for build systems and Git repo access. Vulnerability Patch Mgmt Maintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure.
  31. 31. Acknowledgements ▹ Christopher Harrell ▹ St John Johnson ▹ Mike Shema ▹ Jeremiah Wuenschel
  32. 32. Let’s recap major concepts Ephemeral Keys Replace long-lived keys with time-bound scoped tokens. Service providers should start supporting tokens for auth & authz. Capability Delegation Delegate job capabilities with build events, and eliminate the need to store long-lived keys in CI/CD platform. Audit Trails A verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD. Minimal Builds Avoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum. Network Segmentation Network level Isolation of CI/CD machines. Build ToolChain Hardening Build tools and Docker containers must be adequately hardened.
  33. 33. THANKS! Any questions? You can find me at: @securitysauce github.com/prbinu
  34. 34. CREDITS Special thanks to all the people who made and released these awesome resources for free: ▹ Presentation template by SlidesCarnival ▹ Photographs by Death to the Stock Photo (license)

×