ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

1. Think Like a Vault Developer: Secure Introduction & Containers Jeff Mitchell HashiCorp @jefferai

2. ● Understand basic tenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals

3. ● Understand basic tenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals BORING

4. ● Applied security in action: secure introduction, response wrapping and orchestrators ○ Not a Nomad/Vault talk but will use these as examples Goals

5. Non-Goals ● Discussion of crypto theory ○ Crypto properties (algorithm selection, etc.) are important for applied security ○ Can stay high-level ● Comparison of container/hypervisor/OS security ○ Let’s talk Linux/Docker

6. Non-Goals ● Deep dive into security rabbit holes ○ Let’s keep paranoia in check

7. Tenets / Defining Security

8. “Security” ● Security is the practice of risk management ○ Deciding which risks can be accepted and which cannot ○ Guarding against violations of norms ● Risk increases with system complexity ○ More points of failure, confusion, ingress and egress

9. “Threat” ● Anything that can elevate risk is a threat ● Modeling threats informs security policy

10. “Secret” ● A secret is something that will elevate your risk if exposed to unauthorized entities ● Undesired consequences are harm ○ Unauthorized data access ○ Identity spoofing ○ Private data egress ○ Regulatory fines

11. Exposure ● An exposed secret ○ Elevates risk ○ May cause harm ● An exposed secret is a threat

12. Secrets vs. Identifiers ● Some things that can be disclosed are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk

13. What is “Trust”? ● A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is a companion to modeling threats ● Two concepts

14. Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?

15. Circle of Trust Things not in circle of trust: Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?

16. Circle of Trust Only allowed long-term storage is in circle of trust Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?

17. Chain of Trust ● The set of links (e.g. network hops) that any particular secret travels through from source entity A to destination entity B ● Source/destination must be in circle of trust

18. Links ● Any link in a chain of trust is a potential access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Conspiracy of One/Compromised employee ○ Employee Post-It™

19. Problem Space ● We can now establish a problem space: ○ Managing secrets in a given environment means establishing trust chains in the environment ● Links in the chains have associated risk ○ Minimize hops and minimize risk-per-link

20. Problem Space ● In other words: figuring out how to securely get a secret from producer to consumer

21. Problem Space ● In other words: figuring out how to securely get a secret from producer to consumer DUH

22. Problem Space ● Thinking securely means you have to take the roundabout approach ○ Understanding trust, trust levels, and trust chains along the way is incredibly important ● Only after understanding trust can you figure out where and how to minimize risk

23. Problem Space ● Risk cannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place

24. Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM

25. Scheduler Secret Management Tool Scheduler Agent Container ?

28. Security Properties / Success Criteria

29. Secret Protection ● Establishing a chain of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we (often) only need to do this for one secret! ○ First secret authenticates us to allow direct access to more

30. Secure Introduction ● If you can protect this secret, you can protect any secret ○ ...generally ● This concept is secure introduction (SI)

31. Secure Introduction ● How do you protect secrets (perform SI)? ● First: establish success criteria based on your organization’s acceptable risk ○ Any given organization needs a security policy and associated success criteria that meets their individual needs

32. Secure Introduction ● Reasons your organization doesn’t need to establish a security policy:

33. Secure Introduction ● Reasons your organization doesn’t need to establish a security policy: ○ None

34. Success Criteria ● The security properties that must be implemented/met to implement the accepted security policy ● For secure introduction for this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access

35. Rotation/Expiration ● As lifetime increases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets (WPA-TKIP, RC4) ○ Debugging

36. Rotation/Expiration ● Secrets should be rotated “frequently” and have lifetimes as short as possible ● User passwords vs. machine secrets ○ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ○ Less frequently/more used = more likely overseen ○ Machines: it’s just data - plan/build for rotation and rotate often

37. Distribution ● The literal movement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.

38. Limit Exposure ● Principle of least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set

39. Access Detection ● Things have a way of being leaky ● Env vars are a common way to pass in container secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans

40. Access Detection ● Equally as important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better

41. Break-Glass ● Compromised? ○ Stop all further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!

42. ● You do not have to face this alone

43. ● Explosion of FOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more Secret Management Tools

44. ● Explosion of FOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more ○ Vault! Secret Management Tools

45. ● If you only take away two things from this talk, make sure they are the following:

46. ● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto

47. ● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool

48. ● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool NO! NO!

49. ● If you only take away two things from this talk, make sure they are the following: ● Use a Secret Management tool ● Don’t roll your own

50. Secret Management Tools ● Discussion of SM tools true for Vault and its capabilities ● Vault has a focus on providing security primitives ○ Enables creation of complex workflows ○ We’ll see how this provides necessary flexibility

51. Secret Management Tools ● Central: ○ Secure storage (avoid sprawl) ○ Management ○ Auditing (who has seen/accessed what, when) ● Codified, secure access mechanisms ● Secret rotation/revocation/expiration

52. SM + SI ● Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● 100k containers can support existing SI paradigms ○ But imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways

53. SMs <3 Schedulers ● Anyone working with tasks (containers) at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes

54. SMs <3 Schedulers ● Schedulers are sources of truth...and provide hooks ● SM tools and schedulers can be a magical combination

55. Scheduler Secret Management Tool Scheduler Agent Container Vault

56. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Vault

57. Scheduler Vault Scheduler Agent Container SI #1 (Traditional): AWS-EC2 with Vault Secure Intro Client Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens

58. Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault

59. Preconditions ● A Vault token has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ■ ‘cubbyhole’ ● Claim: These primitives allow excellent secret coverage

60. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault

61. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault

62. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault

63. Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault

64. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault

65. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap Vault

66. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Vault

67. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds Vault

68. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container Vault

69. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds Vault

70. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container Vault

71. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert Vault

72. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container Vault

73. How’d we do? ● Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access

74. Access Detection ● Can detect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e.g. fail job) ○ Invalid token? Check current time vs. issue time + TTL: ■ Expired? YELLOW FLAG: investigate, but probably just bad timing ■ Not expired? RED FLAG: Unauthorized use

75. Response Wrapping ● This paradigm is so useful that it’s available for nearly any Vault call and known as response wrapping ● Mechanism not container- or authentication-specific ○ CM tool drops wrapped secret on EC2 instance ○ File with wrapped value injected into chroot ○ Etc.

76. Job-Finish Revocation ● Wrapped responses containing tokens also return the token’s accessor (reference) ● Scheduling agent/system can correlate accessor with job ○ Revoke token / child leases immediately on job finish

77. Integration ● Nomad integration uses a variation of this scheme but basic concept is the same ○ User-supplied token at job submission time checked for access to avoid privilege escalation via job submission ○ Agent can unwrap secrets and mount into ramdisk ○ Etc.

78. Integration ● HashiStack tools follow the Unix model; we set out to tackle this problem in a generic way ● Any scheduler can implement Nomad’s approach (or a variation) ○ Primitives to do so are there in Vault ○ Kubernetes (https://github.com/kelseyhightower/vault-controller) ○ Mesos (https://github.com/ChannelMeter/vault-gatekeeper-mesos)

79. Wrap-Up ● Don’t ignore risk, minimize it ○ Understand your circles and chains of trust ● Use Vault, or some other SM tool ● Orchestration tools are great facilitators of secure introduction

80. Thanks!

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

DynamicInfraDays

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)