ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)

1. Managing Secrets in Container Environments Jeff Mitchell HashiCorp @jefferai

2. About Me ● Project Lead on Vault ○ A HashiCorp tool for managing secrets ○ https://vaultproject.io ● Previously, Architect at Akamai ○ Heavy focus on containerization and managing container security

3. What this talk is... ● A discussion of paradigms and considerations necessary for managing secrets in container environments ● Applied security, not crypto theory ○ Goal: “Know what you need to know” when figuring out your organization’s security policy

4. ...and what this talk isn’t ● One True Answer™ ○ Support requirements impact available technology/workflows ○ Each organization must set their own security policy and implement practices that satisfy it ● A discussion of container/hypervisor/OS security ○ Let’s talk Linux containerization

5. ...and what this talk isn’t ● Not a deep dive into security rabbit holes (3-4 turtles max)

6. ...and what this talk isn’t ● Not a deep dive into security rabbit holes (3-4 turtles max)

7. Scheduler Secret Management Tool Scheduler Agent Container Problem Space

8. Bear with me for four slides while I wax philosophical…

9. What is “Security”? ● Security is the practice of risk management ○ Accepting some risks, guarding against violations of norms ● Anything that can elevate risk is a threat ○ Modeling threats informs security policy ● Risk increases with system complexity ○ More points of failure, confusion, and ingress = harder to secure

10. What is a “Secret”? ● A secret is something that will elevate your risk if exposed to unauthorized entities ○ Undesired consequences are harm ○ Harm may come in many forms ■ Unauthorized data access ■ Identity spoofing ■ Private data egress ■ Regulatory fines ● An exposed secret is a threat

11. Secrets vs. Identifiers ● Not all things that can be disclosed are secrets, some are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS Key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk

12. What is “Trust”? ● A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is companion to modeling threats ● Two concepts: ○ Circle of trust ○ Chain of trust

13. Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?

14. Circle of Trust Only allowed long-term storage is in circle of trust RAM root Secret Management Tool Persistent Storage Random Users General apps/services NSA Random Wi- Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud?

15. Chain of Trust ● The set of links (e.g. network hops) that any particular secret travels through from entity A to entity B ○ Source/destination must be in circle of trust ● Any link is an access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Employee Post-It™ ○ Conspiracy of One/Compromised employee

16. Problem Space ● We can now establish the problem space: ○ Managing secrets in an environment means establishing trust chains in the environment ○ Links in the chains have associated risk ■ Minimize hops and minimize risk-per-link ■ Risk cannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place

17. Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM

18. Scheduler Secret Management Tool Scheduler Agent Container ?

21. Secret Protection ● Establishing a train of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we only need to do this for one secret!

22. Secure Introduction ● First secret authenticates us to allow direct access to more ● If you can protect this secret, you can protect any secret ○ (Generally) ● This concept is secure introduction (SI)

23. Secret Protection ● How do you protect secrets (perform SI)? ○ First establish success criteria based on acceptable risk ● For this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access

24. Rotation ● As lifetime increases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets ○ Debugging ● Secrets should be rotated “frequently” ○ User passwords vs. machine secrets ■ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ■ Less frequently/more used = more likely overseen ■ Machines: it’s just data - plan/build for rotation and rotate often

25. Distribution ● The literal movement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.

26. Limit Exposure ● Principle of least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set

27. Access Detection ● Things have a way of being leaky ● Env vars: common way to pass in secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans

28. Access Detection ● Equally as important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better

29. Break-Glass ● Compromised? ○ Stop all further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!

30. ● So...this is complicated… ● ...but there’s good news!

31. ● Recent explosion of FOSS Secret Management (SM) tools ○ Vault ○ KeyWhiz ○ Knox ○ Conjur (Docker only) ○ Many more (apologies if I missed your favorite)... Secret Management Tools

32. ● If you only take away two things from this talk, make sure they are the following:

33. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto

34. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto 2. Use it in a Secret Management tool

35. ● If you only take away two things from this talk, make sure they are the following: 1. Write your own crypto 2. Use it in a Secret Management tool NO! NO!

36. ● If you only take away two things from this talk, make sure they are the following: 1. Use a Secret Management tool 2. Don’t roll your own

37. Secret Management Tools ● Why? ○ Central, secure storage (avoid sprawl) ○ Central management ○ Codified, secure access mechanisms ○ Centralized auditing ■ Ensure knowledge of what secrets have been seen by what users/services ○ Secret rotation/revocation/expiration

38. Big Caveat ● This isn’t a vendor talk, but I have a bias ● I don’t have exhaustive knowledge of non-Vault SMs ● Anything I say a SM can do can be done with Vault, and may or may not be possible with other solutions ○ They may absolutely do things Vault can’t

39. SM + SI ● Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● Containers can support existing SI paradigms ○ If you can securely get credentials to it, but imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways

40. SM + SI ● Platform agnostic (with platform-based enhancements) ○ Change authentication mechanism “front-end” ■ Containers ■ EC2 ■ Bare metal ■ Users

41. SMs <3 Schedulers ● Anyone working with containers at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes (All-In-One)

42. SMs <3 Schedulers ● Schedulers are sources of truth and provide hooks ● SM tools and schedulers can be a magical combination ○ May not be direct integration, glue code needed ○ OTOH avoiding tight coupling means better reuse across environments

43. Scheduler Secret Management Tool Scheduler Agent Container

44. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional)

45. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens

46. Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens

47. Preconditions ● Suppose a SM authentication token has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ● Claim: These primitives allow excellent SI for containers

48. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container

49. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container

50. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container

51. Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container

52. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens

53. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap

54. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol

55. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds

56. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container

57. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds

58. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container

59. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert

60. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container

61. How’d we do? ● Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access

62. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: none Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap

63. How’d we do? ● Can detect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e. g. fail job) ○ Denied access from invalid token: raise high-priority security alert and check audit logs: ■ Access of outer token storage in audit logs? Unauthorized use ■ No access? Probably timing issue, but investigate carefully

64. How’d we do? ● Mechanism not container-specific ○ CM tool drops wrapping/outer token on EC2 instance ○ File injected into chroot ○ Etc.

65. Other Paradigms ● Remember, security is risk management ● Depending on your tech stack: ○ Different primitives might be available ○ May be constrained ● Never forget your success criteria

66. Wrap-Up ● Plan your risk tolerance/security policy/success criteria in advance ○ Do it now! ● Use a secret management tool ○ You’re smart, but it’s hard ● Schedulers provide perfect integration points

67. Questions

68. Thanks!