![](http://img.youtube.com/vi/["cUBEjCqPvk4"]/1.jpg)
Just for you: FREE 60-day trial to the world’s largest digital library.
The SlideShare family just got bigger. Enjoy access to millions of ebooks, audiobooks, magazines, and more from Scribd.
Cancel anytime.
- 1. Managing Secrets in Container Environments Jeff Mitchell HashiCorp @jefferai
- 2. About Me ● Project Lead on Vault ○ A HashiCorp tool for managing secrets ○ https://vaultproject.io ● Previously, Architect at Akamai ○ Heavy focus on containerization and managing container security
- 3. What this talk is... ● A discussion of paradigms and considerations necessary for managing secrets in container environments ● Applied security, not crypto theory ○ Goal: “Know what you need to know” when figuring out your organization’s security policy
- 4. ...and what this talk isn’t ● One True Answer™ ○ Support requirements impact available technology/workflows ○ Each organization must set their own security policy and implement practices that satisfy it ● A discussion of container/hypervisor/OS security ○ Let’s talk Linux containerization
- 5. ...and what this talk isn’t ● Not a deep dive into security rabbit holes (3-4 turtles max)
- 6. ...and what this talk isn’t ● Not a deep dive into security rabbit holes (3-4 turtles max)
- 7. Scheduler Secret Management Tool Scheduler Agent Container Problem Space
- 8. Bear with me for four slides while I wax philosophical…
- 9. What is “Security”? ● Security is the practice of risk management ○ Accepting some risks, guarding against violations of norms ● Anything that can elevate risk is a threat ○ Modeling threats informs security policy ● Risk increases with system complexity ○ More points of failure, confusion, and ingress = harder to secure
- 10. What is a “Secret”? ● A secret is something that will elevate your risk if exposed to unauthorized entities ○ Undesired consequences are harm ○ Harm may come in many forms ■ Unauthorized data access ■ Identity spoofing ■ Private data egress ■ Regulatory fines ● An exposed secret is a threat
- 11. Secrets vs. Identifiers ● Not all things that can be disclosed are secrets, some are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS Key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk
- 12. What is “Trust”? ● A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is companion to modeling threats ● Two concepts: ○ Circle of trust ○ Chain of trust
- 13. Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?
- 14. Circle of Trust Only allowed long-term storage is in circle of trust RAM root Secret Management Tool Persistent Storage Random Users General apps/services NSA Random Wi- Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud?
- 15. Chain of Trust ● The set of links (e.g. network hops) that any particular secret travels through from entity A to entity B ○ Source/destination must be in circle of trust ● Any link is an access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Employee Post-It™ ○ Conspiracy of One/Compromised employee
- 16. Problem Space ● We can now establish the problem space: ○ Managing secrets in an environment means establishing trust chains in the environment ○ Links in the chains have associated risk ■ Minimize hops and minimize risk-per-link ■ Risk cannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place
- 17. Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM
- 18. Scheduler Secret Management Tool Scheduler Agent Container ?
- 19. Scheduler Secret Management Tool Scheduler Agent Container ?
- 20. Scheduler Secret Management Tool Scheduler Agent Container ?
- 21. Secret Protection ● Establishing a train of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we only need to do this for one secret!
- 22. Secure Introduction ● First secret authenticates us to allow direct access to more ● If you can protect this secret, you can protect any secret ○ (Generally) ● This concept is secure introduction (SI)
- 23. Secret Protection ● How do you protect secrets (perform SI)? ○ First establish success criteria based on acceptable risk ● For this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access
- 24. Rotation ● As lifetime increases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets ○ Debugging ● Secrets should be rotated “frequently” ○ User passwords vs. machine secrets ■ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ■ Less frequently/more used = more likely overseen ■ Machines: it’s just data - plan/build for rotation and rotate often
- 25. Distribution ● The literal movement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.
- 26. Limit Exposure ● Principle of least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set
- 27. Access Detection ● Things have a way of being leaky ● Env vars: common way to pass in secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans
- 28. Access Detection ● Equally as important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better
- 29. Break-Glass ● Compromised? ○ Stop all further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!
- 30. ● So...this is complicated… ● ...but there’s good news!
- 31. ● Recent explosion of FOSS Secret Management (SM) tools ○ Vault ○ KeyWhiz ○ Knox ○ Conjur (Docker only) ○ Many more (apologies if I missed your favorite)... Secret Management Tools
- 32. ● If you only take away two things from this talk, make sure they are the following:
- 33. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto
- 34. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto 2. Use it in a Secret Management tool
- 35. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto 2. Use it in a Secret Management tool NO! NO!
- 36. ● If you only take away two things from this talk, make sure they are the following: 1. Use a Secret Management tool 2. Don’t roll your own
- 37. Secret Management Tools ● Why? ○ Central, secure storage (avoid sprawl) ○ Central management ○ Codified, secure access mechanisms ○ Centralized auditing ■ Ensure knowledge of what secrets have been seen by what users/services ○ Secret rotation/revocation/expiration
- 38. Big Caveat ● This isn’t a vendor talk, but I have a bias ● I don’t have exhaustive knowledge of non-Vault SMs ● Anything I say a SM can do can be done with Vault, and may or may not be possible with other solutions ○ They may absolutely do things Vault can’t
- 39. SM + SI ● Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● Containers can support existing SI paradigms ○ If you can securely get credentials to it, but imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways
- 40. SM + SI ● Platform agnostic (with platform-based enhancements) ○ Change authentication mechanism “front-end” ■ Containers ■ EC2 ■ Bare metal ■ Users
- 41. SMs <3 Schedulers ● Anyone working with containers at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes (All-In-One)
- 42. SMs <3 Schedulers ● Schedulers are sources of truth and provide hooks ● SM tools and schedulers can be a magical combination ○ May not be direct integration, glue code needed ○ OTOH avoiding tight coupling means better reuse across environments
- 43. Scheduler Secret Management Tool Scheduler Agent Container
- 44. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional)
- 45. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
- 46. Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
- 47. Preconditions ● Suppose a SM authentication token has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ● Claim: These primitives allow excellent SI for containers
- 48. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
- 49. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
- 50. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
- 51. Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container
- 52. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
- 53. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap
- 54. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol
- 55. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds
- 56. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container
- 57. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds
- 58. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container
- 59. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert
- 60. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container
- 61. How’d we do? ● Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access
- 62. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap
- 63. How’d we do? ● Can detect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e. g. fail job) ○ Denied access from invalid token: raise high-priority security alert and check audit logs: ■ Access of outer token storage in audit logs? Unauthorized use ■ No access? Probably timing issue, but investigate carefully
- 64. How’d we do? ● Mechanism not container-specific ○ CM tool drops wrapping/outer token on EC2 instance ○ File injected into chroot ○ Etc.
- 65. Other Paradigms ● Remember, security is risk management ● Depending on your tech stack: ○ Different primitives might be available ○ May be constrained ● Never forget your success criteria
- 66. Wrap-Up ● Plan your risk tolerance/security policy/success criteria in advance ○ Do it now! ● Use a secret management tool ○ You’re smart, but it’s hard ● Schedulers provide perfect integration points
- 67. Questions
- 68. Thanks!
Recommended
More Related Content
Similar to ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)
Related Books
Free with a 30 day trial from Scribd