Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Securing Application Deployments in CI/CD Environments (Updated slides:

In a multi-tenant CI/CD environment, developers trust and delegate CI/CD systems to deploy their applications to production. But, what is the basis of this trust? How the trust is enforced from commit-to-deploy? What is the trustworthiness of the application deployed by CI/CD through automation? This talk highlights security risks with CI/CD deployments and offer solutions to mitigate those risks

  • Be the first to comment

Securing Application Deployments in CI/CD Environments (Updated slides:

  1. 1. Securing Application Deployments in CI/CD Environments
  2. 2. Hello! I am Binu Ramakrishnan Principal Security Engineer at Yahoo You can find me at: @securitysauce
  3. 3. Overview ◎CI/CD platform overview. ◎High level objectives. ◎Threat modeling. ◎Security patterns and best practices.
  4. 4. CI/CD
  5. 5. CI/CD Deployments ◎Single-tenant ◎Multi-tenant
  6. 6. Single-tenant Dedicated build environments.
  7. 7. Multi-tenant Shared build environment. Driven by economies of scale.
  8. 8. CI/CD in Nutshell
  9. 9. Enterprise CI/CD
  10. 10. High Level Security Objectives
  11. 11. Verifiable Chain of Trust
  12. 12. No Elevated Privileges for Build Jobs
  13. 13. Threat Modeling
  14. 14. CI/CD Admin Exclusive access to CI/CD platform Actors Developer User/customer of the platform. CI/CD Platform Developer Limited admin access to platform.
  15. 15. Trust Boundaries.
  16. 16. Trust Boundaries Web interface ◎ UI ◎ REST APIs Application
  17. 17. Trust Boundaries Isolation based on Docker containers or VMs. Host
  18. 18. Trust Boundaries Segmentation Network
  19. 19. Attack Surface Entry Points Exit Points
  20. 20. Entry Points ◎ Build web interface. ◎ Commit and build notification handlers. ◎ Source and package dependencies.
  21. 21. Internal Entry Points ◎ Build slave endpoint (to launch builds; SSH). ◎ Other control plane endpoints/REST APIs.
  22. 22. Exit Points ◎ Deployable artifacts. ◎ Build notifications - eg emails, IRC and Hipchat messages. ◎ Build console logs. ◎ Git repo update with build status and badges.
  23. 23. Threats
  24. 24. Common Threats ◎ User account compromise & insider threats. ○ User and platform admins. ◎ Network Intrusion. ○ CI/CD internal and external endpoints.
  25. 25. Build Slave Compromise Large attack surface, spread across multiple networks (iphone, Android, Server app etc.) How? ◎ A network level compromise, exploiting a vulnerability in build slave. ◎ Jobs break out of build container. Impact ◎ Access to production servers. ◎ Listen to the network, spoof identity and access unauthorized data.
  26. 26. Shared/Non-ephemeral Keys ◎ Locally stored long-lived keys to access protected services. Two forms: ◉ Shared key ◉ Per-job keys (eg. OAuth tokens)
  27. 27. SSH over Unrestricted Shell Allows arbitrary commands to get executed on a remote host.
  28. 28. Building External Code An attacker can take this path to get into internal networks, either by adding backdoors or exploiting known vulnerabilities with open source software.
  29. 29. Baseline security controls that are expected to be there in 2016
  30. 30. Existing Security Controls ◎ Authenticated endpoints. ◎ Web application security practices - CSRF, HTTPS, HTTPOnly cookies, XSS protection etc. ◎ Source control: granular authorization for repos. ◎ Use KMS to manage secrets; no hardcoded secrets.
  31. 31. Security Patterns for Risk Reduction
  32. 32. Ephemeral Keys
  33. 33. Stateless Auth Architecture
  34. 34. ◎ Events: ○ Commit trigger. ○ Manual trigger from build UI. ○ Automated/cron job. ○ Trigger a downstream job. ◎ Upstream service stores downstream service credentials (OAuth, Shared Keys etc.). ◎ Equal trust on all components in the pipeline
  35. 35. Audit Trails
  36. 36. Container, ToolChain Hardening
  37. 37. Network Segmentation balades/15355029654
  38. 38. Do not pull PII or other sensitive info to build machine. Minimal Builds
  39. 39. Few more... SSH: Use Restricted Shells Headless SSH access for automated deployment should use a restricted shell. Roll Keys Periodically Establish a process to periodically roll trust anchor keys (and do it periodically). Restrict Job Console Logs Restrict build job console logs only to authorized users Enable 2FA Admins must follow good security hygiene and use 2FA to access platform application and hosts Prune Admin Access List Keep admin list small for build systems and Git repo access Vulnerability Patch Mgmt Maintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure
  40. 40. Acknowledgements ◎ Christopher Harrell ◎ St John Johnson ◎ Mike Shema ◎ Jeremiah Wuenschel
  41. 41. Let’s recap major concepts Ephemeral Keys Ephemeral keys are the future. Service providers should start supporting ephemeral keys for authorization Stateless Auth Architecture Augment the trust dependency of 1:1 relationship between the pipeline components with workflow job tokens. Audit Logs A verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD. Minimal Builds Avoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum Network Segmentation Network level Isolation of CI/CD machines from other machines ToolChain Hardening Build tools and Docker containers must be adequately hardened.
  42. 42. Thanks! Any questions? You can find me at: @securitysauce
  43. 43. Appendix-A
  44. 44. Use ◎ Source code ◎ Pre-built packages Risks ◎ Targeted back doors ◎ Vulnerabilities Hard Problem: Securing Supply Chain This threat is more to do with applications than the CI/CD platform itself. Open source components constitute a large part of modern Internet based applications.
  45. 45. Appendix-B
  46. 46. Credits Special thanks to all the people who made and released these awesome resources for free: ◎ Presentation template by SlidesCarnival ◎ Photographs by Unsplash & Death to the Stock Photo (license)