Think Like a Vault Developer:
Secure Introduction & Containers
Jeff Mitchell
HashiCorp
@jefferai
● Understand basic tenets of security, as the Vault team
thinks about them
● Explore properties necessary for secret manag...
● Understand basic tenets of security, as the Vault team
thinks about them
● Explore properties necessary for secret manag...
● Applied security in action: secure introduction, response
wrapping and orchestrators
○ Not a Nomad/Vault talk but will u...
Non-Goals
● Discussion of crypto theory
○ Crypto properties (algorithm selection, etc.) are important for
applied security...
Non-Goals
● Deep dive into security rabbit holes
○ Let’s keep paranoia in check
Tenets / Defining Security
“Security”
● Security is the practice of risk management
○ Deciding which risks can be accepted and which cannot
○ Guardin...
“Threat”
● Anything that can elevate risk is a threat
● Modeling threats informs security policy
“Secret”
● A secret is something that will elevate your risk if exposed
to unauthorized entities
● Undesired consequences ...
Exposure
● An exposed secret
○ Elevates risk
○ May cause harm
● An exposed secret is a threat
Secrets vs. Identifiers
● Some things that can be disclosed are identifiers
○ Username (identifier) vs. Password (secret)
...
What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities ...
Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?
Circle of Trust
Things not in circle of trust:
Persistent
Storage
Random
Users
General
apps/services
NSA
Random
Wi-Fi Hots...
Circle of Trust
Only allowed long-term storage is in circle of trust
Persistent
Storage
Random
Users
General
apps/services...
Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from source entity A to d...
Links
● Any link in a chain of trust is a potential access
point/interception point
○ Accidental logging
○ Exploitation by...
Problem Space
● We can now establish a problem space:
○ Managing secrets in a given environment means establishing trust
c...
Problem Space
● In other words: figuring out how to securely get a secret
from producer to consumer
Problem Space
● In other words: figuring out how to securely get a secret
from producer to consumer
DUH
Problem Space
● Thinking securely means you have to take the roundabout
approach
○ Understanding trust, trust levels, and ...
Problem Space
● Risk cannot be fully mitigated; must assume any given
secret will eventually be divulged
● Ultimate goal: ...
Scheduler
Secret Management Tool
Scheduler Agent
Container
Goal: Securely
move secret from
originator to new
container’s R...
Scheduler
Secret Management Tool
Scheduler Agent
Container
?
Scheduler
Secret Management Tool
Scheduler Agent
Container
?
Scheduler
Secret Management Tool
Scheduler Agent
Container
?
Security Properties /
Success Criteria
Secret Protection
● Establishing a chain of trust requires defining the
requirements it must fulfill to keep secrets prote...
Secure Introduction
● If you can protect this secret, you can protect any secret
○ ...generally
● This concept is secure i...
Secure Introduction
● How do you protect secrets (perform SI)?
● First: establish success criteria based on your
organizat...
Secure Introduction
● Reasons your organization doesn’t need to establish a
security policy:
Secure Introduction
● Reasons your organization doesn’t need to establish a
security policy:
○ None
Success Criteria
● The security properties that must be implemented/met to
implement the accepted security policy
● For se...
Rotation/Expiration
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets (WPA...
Rotation/Expiration
● Secrets should be rotated “frequently” and have lifetimes
as short as possible
● User passwords vs. ...
Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, ...
Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not r...
Access Detection
● Things have a way of being leaky
● Env vars are a common way to pass in container secrets.
Also:
○ Ofte...
Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audi...
Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after...
● You do not have to face this alone
● Explosion of FOSS Secret Management (SM) tools
○ KeyWhiz
○ Knox
○ Conjur
○ Many more
Secret Management Tools
● Explosion of FOSS Secret Management (SM) tools
○ KeyWhiz
○ Knox
○ Conjur
○ Many more
○ Vault!
Secret Management Tools
● If you only take away two things from this talk, make sure
they are the following:
● If you only take away two things from this talk, make sure
they are the following:
● Write your own crypto
● If you only take away two things from this talk, make sure
they are the following:
● Write your own crypto
● Use it in a...
● If you only take away two things from this talk, make sure
they are the following:
● Write your own crypto
● Use it in a...
● If you only take away two things from this talk, make sure
they are the following:
● Use a Secret Management tool
● Don’...
Secret Management Tools
● Discussion of SM tools true for Vault and its capabilities
● Vault has a focus on providing secu...
Secret Management Tools
● Central:
○ Secure storage (avoid sprawl)
○ Management
○ Auditing (who has seen/accessed what, wh...
SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilitie...
SMs <3 Schedulers
● Anyone working with tasks (containers) at scale uses a
scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kub...
SMs <3 Schedulers
● Schedulers are sources of truth...and provide hooks
● SM tools and schedulers can be a magical combina...
Scheduler
Secret Management Tool
Scheduler Agent
Container
Vault
Scheduler
Secret Management Tool
Scheduler Agent
Container
SI #1 (Traditional)
Vault
Scheduler
Vault
Scheduler Agent
Container
SI #1 (Traditional):
AWS-EC2 with
Vault Secure Intro Client
Uses: ∞
TTL: 8h (ren...
Scheduler
Secret Management Tool
Scheduler Agent
Container
SI #2 (New)
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_toke...
Preconditions
● A Vault token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Uses: ∞
TTL: 8h (re...
Scheduler
App: db_writer
Security Policy: app_db_rw_pol
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)...
Scheduler
App: db_writer
Security Policy: app_db_rw_pol
Secret Management Tool
Create Token:
Policy: app_db_rw_pol
Schedul...
Scheduler
Secret Management Tool
Create Token:
Policy: app_db_rw_pol
Uses: 1
TTL: 30s (non-renewable)
Policy: read_cubby
P...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
ContainerImage: db_...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Container
Uses: 1
T...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Uses: ∞
TTL: 8h (re...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Container
Uses: ∞
T...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Uses: ∞
TTL: 8h (re...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Container
Uses: ∞
T...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Uses: ∞
TTL: 8h (re...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Container
Uses: ∞
T...
Scheduler
Secret Management Tool
Scheduler Agent
Uses: ∞
TTL: 8h (renewable)
Policy: create_app_tokens
Uses: ∞
TTL: 8h (re...
How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with...
Access Detection
● Can detect unauthorized access of “real” authentication
token due to time and use limit
● Application:
...
Response Wrapping
● This paradigm is so useful that it’s available for nearly any
Vault call and known as response wrappin...
Job-Finish Revocation
● Wrapped responses containing tokens also return the
token’s accessor (reference)
● Scheduling agen...
Integration
● Nomad integration uses a variation of this scheme but
basic concept is the same
○ User-supplied token at job...
Integration
● HashiStack tools follow the Unix model; we set out to
tackle this problem in a generic way
● Any scheduler c...
Wrap-Up
● Don’t ignore risk, minimize it
○ Understand your circles and chains of trust
● Use Vault, or some other SM tool
...
Thanks!
ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)
Upcoming SlideShare
Loading in …5
×

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

341 views

Published on

Slides from Jeff Mitchell's talk "The Secure Introduction Problem: Getting Secrets Into Containers" at ContainerDays NYC 2016: http://dynamicinfradays.org/events/2016-nyc/programme.html#secrets

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
341
On SlideShare
0
From Embeds
0
Number of Embeds
134
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

  1. 1. Think Like a Vault Developer: Secure Introduction & Containers Jeff Mitchell HashiCorp @jefferai
  2. 2. ● Understand basic tenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals
  3. 3. ● Understand basic tenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals BORING
  4. 4. ● Applied security in action: secure introduction, response wrapping and orchestrators ○ Not a Nomad/Vault talk but will use these as examples Goals
  5. 5. Non-Goals ● Discussion of crypto theory ○ Crypto properties (algorithm selection, etc.) are important for applied security ○ Can stay high-level ● Comparison of container/hypervisor/OS security ○ Let’s talk Linux/Docker
  6. 6. Non-Goals ● Deep dive into security rabbit holes ○ Let’s keep paranoia in check
  7. 7. Tenets / Defining Security
  8. 8. “Security” ● Security is the practice of risk management ○ Deciding which risks can be accepted and which cannot ○ Guarding against violations of norms ● Risk increases with system complexity ○ More points of failure, confusion, ingress and egress
  9. 9. “Threat” ● Anything that can elevate risk is a threat ● Modeling threats informs security policy
  10. 10. “Secret” ● A secret is something that will elevate your risk if exposed to unauthorized entities ● Undesired consequences are harm ○ Unauthorized data access ○ Identity spoofing ○ Private data egress ○ Regulatory fines
  11. 11. Exposure ● An exposed secret ○ Elevates risk ○ May cause harm ● An exposed secret is a threat
  12. 12. Secrets vs. Identifiers ● Some things that can be disclosed are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk
  13. 13. What is “Trust”? ● A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is a companion to modeling threats ● Two concepts
  14. 14. Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?
  15. 15. Circle of Trust Things not in circle of trust: Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
  16. 16. Circle of Trust Only allowed long-term storage is in circle of trust Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
  17. 17. Chain of Trust ● The set of links (e.g. network hops) that any particular secret travels through from source entity A to destination entity B ● Source/destination must be in circle of trust
  18. 18. Links ● Any link in a chain of trust is a potential access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Conspiracy of One/Compromised employee ○ Employee Post-It™
  19. 19. Problem Space ● We can now establish a problem space: ○ Managing secrets in a given environment means establishing trust chains in the environment ● Links in the chains have associated risk ○ Minimize hops and minimize risk-per-link
  20. 20. Problem Space ● In other words: figuring out how to securely get a secret from producer to consumer
  21. 21. Problem Space ● In other words: figuring out how to securely get a secret from producer to consumer DUH
  22. 22. Problem Space ● Thinking securely means you have to take the roundabout approach ○ Understanding trust, trust levels, and trust chains along the way is incredibly important ● Only after understanding trust can you figure out where and how to minimize risk
  23. 23. Problem Space ● Risk cannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place
  24. 24. Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM
  25. 25. Scheduler Secret Management Tool Scheduler Agent Container ?
  26. 26. Scheduler Secret Management Tool Scheduler Agent Container ?
  27. 27. Scheduler Secret Management Tool Scheduler Agent Container ?
  28. 28. Security Properties / Success Criteria
  29. 29. Secret Protection ● Establishing a chain of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we (often) only need to do this for one secret! ○ First secret authenticates us to allow direct access to more
  30. 30. Secure Introduction ● If you can protect this secret, you can protect any secret ○ ...generally ● This concept is secure introduction (SI)
  31. 31. Secure Introduction ● How do you protect secrets (perform SI)? ● First: establish success criteria based on your organization’s acceptable risk ○ Any given organization needs a security policy and associated success criteria that meets their individual needs
  32. 32. Secure Introduction ● Reasons your organization doesn’t need to establish a security policy:
  33. 33. Secure Introduction ● Reasons your organization doesn’t need to establish a security policy: ○ None
  34. 34. Success Criteria ● The security properties that must be implemented/met to implement the accepted security policy ● For secure introduction for this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access
  35. 35. Rotation/Expiration ● As lifetime increases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets (WPA-TKIP, RC4) ○ Debugging
  36. 36. Rotation/Expiration ● Secrets should be rotated “frequently” and have lifetimes as short as possible ● User passwords vs. machine secrets ○ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ○ Less frequently/more used = more likely overseen ○ Machines: it’s just data - plan/build for rotation and rotate often
  37. 37. Distribution ● The literal movement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.
  38. 38. Limit Exposure ● Principle of least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set
  39. 39. Access Detection ● Things have a way of being leaky ● Env vars are a common way to pass in container secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans
  40. 40. Access Detection ● Equally as important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better
  41. 41. Break-Glass ● Compromised? ○ Stop all further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!
  42. 42. ● You do not have to face this alone
  43. 43. ● Explosion of FOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more Secret Management Tools
  44. 44. ● Explosion of FOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more ○ Vault! Secret Management Tools
  45. 45. ● If you only take away two things from this talk, make sure they are the following:
  46. 46. ● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto
  47. 47. ● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool
  48. 48. ● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool NO! NO!
  49. 49. ● If you only take away two things from this talk, make sure they are the following: ● Use a Secret Management tool ● Don’t roll your own
  50. 50. Secret Management Tools ● Discussion of SM tools true for Vault and its capabilities ● Vault has a focus on providing security primitives ○ Enables creation of complex workflows ○ We’ll see how this provides necessary flexibility
  51. 51. Secret Management Tools ● Central: ○ Secure storage (avoid sprawl) ○ Management ○ Auditing (who has seen/accessed what, when) ● Codified, secure access mechanisms ● Secret rotation/revocation/expiration
  52. 52. SM + SI ● Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● 100k containers can support existing SI paradigms ○ But imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways
  53. 53. SMs <3 Schedulers ● Anyone working with tasks (containers) at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes
  54. 54. SMs <3 Schedulers ● Schedulers are sources of truth...and provide hooks ● SM tools and schedulers can be a magical combination
  55. 55. Scheduler Secret Management Tool Scheduler Agent Container Vault
  56. 56. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Vault
  57. 57. Scheduler Vault Scheduler Agent Container SI #1 (Traditional): AWS-EC2 with Vault Secure Intro Client Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
  58. 58. Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
  59. 59. Preconditions ● A Vault token has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ■ ‘cubbyhole’ ● Claim: These primitives allow excellent secret coverage
  60. 60. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  61. 61. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  62. 62. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  63. 63. Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  64. 64. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
  65. 65. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap Vault
  66. 66. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Vault
  67. 67. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds Vault
  68. 68. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container Vault
  69. 69. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds Vault
  70. 70. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container Vault
  71. 71. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert Vault
  72. 72. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container Vault
  73. 73. How’d we do? ● Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access
  74. 74. Access Detection ● Can detect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e.g. fail job) ○ Invalid token? Check current time vs. issue time + TTL: ■ Expired? YELLOW FLAG: investigate, but probably just bad timing ■ Not expired? RED FLAG: Unauthorized use
  75. 75. Response Wrapping ● This paradigm is so useful that it’s available for nearly any Vault call and known as response wrapping ● Mechanism not container- or authentication-specific ○ CM tool drops wrapped secret on EC2 instance ○ File with wrapped value injected into chroot ○ Etc.
  76. 76. Job-Finish Revocation ● Wrapped responses containing tokens also return the token’s accessor (reference) ● Scheduling agent/system can correlate accessor with job ○ Revoke token / child leases immediately on job finish
  77. 77. Integration ● Nomad integration uses a variation of this scheme but basic concept is the same ○ User-supplied token at job submission time checked for access to avoid privilege escalation via job submission ○ Agent can unwrap secrets and mount into ramdisk ○ Etc.
  78. 78. Integration ● HashiStack tools follow the Unix model; we set out to tackle this problem in a generic way ● Any scheduler can implement Nomad’s approach (or a variation) ○ Primitives to do so are there in Vault ○ Kubernetes (https://github.com/kelseyhightower/vault-controller) ○ Mesos (https://github.com/ChannelMeter/vault-gatekeeper-mesos)
  79. 79. Wrap-Up ● Don’t ignore risk, minimize it ○ Understand your circles and chains of trust ● Use Vault, or some other SM tool ● Orchestration tools are great facilitators of secure introduction
  80. 80. Thanks!

×