Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)

732 views

Published on

Slides from Jeff Mitchell's talk "Hiding in Plain Sight: Managing Secrets in a Container Environment" at ContainerDays Boston 2016: http://dynamicinfradays.org/events/2016-boston/programme.html#secrets

Published in: Technology

ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)

  1. 1. Managing Secrets in Container Environments Jeff Mitchell HashiCorp @jefferai
  2. 2. About Me ● Project Lead on Vault ○ A HashiCorp tool for managing secrets ○ https://vaultproject.io ● Previously, Architect at Akamai ○ Heavy focus on containerization and managing container security
  3. 3. What this talk is... ● A discussion of paradigms and considerations necessary for managing secrets in container environments ● Applied security, not crypto theory ○ Goal: “Know what you need to know” when figuring out your organization’s security policy
  4. 4. ...and what this talk isn’t ● One True Answer™ ○ Support requirements impact available technology/workflows ○ Each organization must set their own security policy and implement practices that satisfy it ● A discussion of container/hypervisor/OS security ○ Let’s talk Linux containerization
  5. 5. ...and what this talk isn’t ● Not a deep dive into security rabbit holes (3-4 turtles max)
  6. 6. ...and what this talk isn’t ● Not a deep dive into security rabbit holes (3-4 turtles max)
  7. 7. Scheduler Secret Management Tool Scheduler Agent Container Problem Space
  8. 8. Bear with me for four slides while I wax philosophical…
  9. 9. What is “Security”? ● Security is the practice of risk management ○ Accepting some risks, guarding against violations of norms ● Anything that can elevate risk is a threat ○ Modeling threats informs security policy ● Risk increases with system complexity ○ More points of failure, confusion, and ingress = harder to secure
  10. 10. What is a “Secret”? ● A secret is something that will elevate your risk if exposed to unauthorized entities ○ Undesired consequences are harm ○ Harm may come in many forms ■ Unauthorized data access ■ Identity spoofing ■ Private data egress ■ Regulatory fines ● An exposed secret is a threat
  11. 11. Secrets vs. Identifiers ● Not all things that can be disclosed are secrets, some are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS Key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk
  12. 12. What is “Trust”? ● A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is companion to modeling threats ● Two concepts: ○ Circle of trust ○ Chain of trust
  13. 13. Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?
  14. 14. Circle of Trust Only allowed long-term storage is in circle of trust RAM root Secret Management Tool Persistent Storage Random Users General apps/services NSA Random Wi- Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud?
  15. 15. Chain of Trust ● The set of links (e.g. network hops) that any particular secret travels through from entity A to entity B ○ Source/destination must be in circle of trust ● Any link is an access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Employee Post-It™ ○ Conspiracy of One/Compromised employee
  16. 16. Problem Space ● We can now establish the problem space: ○ Managing secrets in an environment means establishing trust chains in the environment ○ Links in the chains have associated risk ■ Minimize hops and minimize risk-per-link ■ Risk cannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place
  17. 17. Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM
  18. 18. Scheduler Secret Management Tool Scheduler Agent Container ?
  19. 19. Scheduler Secret Management Tool Scheduler Agent Container ?
  20. 20. Scheduler Secret Management Tool Scheduler Agent Container ?
  21. 21. Secret Protection ● Establishing a train of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we only need to do this for one secret!
  22. 22. Secure Introduction ● First secret authenticates us to allow direct access to more ● If you can protect this secret, you can protect any secret ○ (Generally) ● This concept is secure introduction (SI)
  23. 23. Secret Protection ● How do you protect secrets (perform SI)? ○ First establish success criteria based on acceptable risk ● For this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access
  24. 24. Rotation ● As lifetime increases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets ○ Debugging ● Secrets should be rotated “frequently” ○ User passwords vs. machine secrets ■ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ■ Less frequently/more used = more likely overseen ■ Machines: it’s just data - plan/build for rotation and rotate often
  25. 25. Distribution ● The literal movement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.
  26. 26. Limit Exposure ● Principle of least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set
  27. 27. Access Detection ● Things have a way of being leaky ● Env vars: common way to pass in secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans
  28. 28. Access Detection ● Equally as important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better
  29. 29. Break-Glass ● Compromised? ○ Stop all further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!
  30. 30. ● So...this is complicated… ● ...but there’s good news!
  31. 31. ● Recent explosion of FOSS Secret Management (SM) tools ○ Vault ○ KeyWhiz ○ Knox ○ Conjur (Docker only) ○ Many more (apologies if I missed your favorite)... Secret Management Tools
  32. 32. ● If you only take away two things from this talk, make sure they are the following:
  33. 33. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto
  34. 34. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto 2. Use it in a Secret Management tool
  35. 35. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto 2. Use it in a Secret Management tool NO! NO!
  36. 36. ● If you only take away two things from this talk, make sure they are the following: 1. Use a Secret Management tool 2. Don’t roll your own
  37. 37. Secret Management Tools ● Why? ○ Central, secure storage (avoid sprawl) ○ Central management ○ Codified, secure access mechanisms ○ Centralized auditing ■ Ensure knowledge of what secrets have been seen by what users/services ○ Secret rotation/revocation/expiration
  38. 38. Big Caveat ● This isn’t a vendor talk, but I have a bias ● I don’t have exhaustive knowledge of non-Vault SMs ● Anything I say a SM can do can be done with Vault, and may or may not be possible with other solutions ○ They may absolutely do things Vault can’t
  39. 39. SM + SI ● Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● Containers can support existing SI paradigms ○ If you can securely get credentials to it, but imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways
  40. 40. SM + SI ● Platform agnostic (with platform-based enhancements) ○ Change authentication mechanism “front-end” ■ Containers ■ EC2 ■ Bare metal ■ Users
  41. 41. SMs <3 Schedulers ● Anyone working with containers at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes (All-In-One)
  42. 42. SMs <3 Schedulers ● Schedulers are sources of truth and provide hooks ● SM tools and schedulers can be a magical combination ○ May not be direct integration, glue code needed ○ OTOH avoiding tight coupling means better reuse across environments
  43. 43. Scheduler Secret Management Tool Scheduler Agent Container
  44. 44. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional)
  45. 45. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
  46. 46. Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
  47. 47. Preconditions ● Suppose a SM authentication token has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ● Claim: These primitives allow excellent SI for containers
  48. 48. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
  49. 49. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
  50. 50. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
  51. 51. Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
  52. 52. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
  53. 53. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap
  54. 54. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol
  55. 55. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds
  56. 56. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container
  57. 57. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds
  58. 58. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container
  59. 59. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert
  60. 60. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container
  61. 61. How’d we do? ● Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access
  62. 62. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap
  63. 63. How’d we do? ● Can detect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e. g. fail job) ○ Denied access from invalid token: raise high-priority security alert and check audit logs: ■ Access of outer token storage in audit logs? Unauthorized use ■ No access? Probably timing issue, but investigate carefully
  64. 64. How’d we do? ● Mechanism not container-specific ○ CM tool drops wrapping/outer token on EC2 instance ○ File injected into chroot ○ Etc.
  65. 65. Other Paradigms ● Remember, security is risk management ● Depending on your tech stack: ○ Different primitives might be available ○ May be constrained ● Never forget your success criteria
  66. 66. Wrap-Up ● Plan your risk tolerance/security policy/success criteria in advance ○ Do it now! ● Use a secret management tool ○ You’re smart, but it’s hard ● Schedulers provide perfect integration points
  67. 67. Questions
  68. 68. Thanks!

×