SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 30 day free trial to unlock unlimited reading.
ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)
Slides from Jeff Mitchell's talk "Hiding in Plain Sight: Managing Secrets in a Container Environment" at ContainerDays Boston 2016: http://dynamicinfradays.org/events/2016-boston/programme.html#secrets
1.
Managing Secrets in
Container Environments
Jeff Mitchell
HashiCorp
@jefferai
2.
About Me
● Project Lead on Vault
○ A HashiCorp tool for managing secrets
○ https://vaultproject.io
● Previously, Architect at Akamai
○ Heavy focus on containerization and managing container security
3.
What this talk is...
● A discussion of paradigms and considerations necessary
for managing secrets in container environments
● Applied security, not crypto theory
○ Goal: “Know what you need to know” when figuring out your
organization’s security policy
4.
...and what this talk isn’t
● One True Answer™
○ Support requirements impact available technology/workflows
○ Each organization must set their own security policy and
implement practices that satisfy it
● A discussion of container/hypervisor/OS security
○ Let’s talk Linux containerization
5.
...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)
6.
...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)
7.
Scheduler
Secret Management Tool
Scheduler Agent
Container
Problem Space
8.
Bear with me for four slides while I wax
philosophical…
9.
What is “Security”?
● Security is the practice of risk management
○ Accepting some risks, guarding against violations of norms
● Anything that can elevate risk is a threat
○ Modeling threats informs security policy
● Risk increases with system complexity
○ More points of failure, confusion, and ingress = harder to secure
10.
What is a “Secret”?
● A secret is something that will elevate your risk if exposed
to unauthorized entities
○ Undesired consequences are harm
○ Harm may come in many forms
■ Unauthorized data access
■ Identity spoofing
■ Private data egress
■ Regulatory fines
● An exposed secret is a threat
11.
Secrets vs. Identifiers
● Not all things that can be disclosed are secrets, some are
identifiers
○ Username (identifier) vs. Password (secret)
○ TLS certificate (identifier) vs. TLS Key (secret)
○ GitHub user name (identifier) vs. API key (secret)
● Identifiers aren’t completely risk-free
○ Have chosen to ignore that risk
12.
What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities is companion to modeling threats
● Two concepts:
○ Circle of trust
○ Chain of trust
13.
Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?
14.
Circle of Trust
Only allowed long-term storage is in circle of trust
RAM
root
Secret
Management
Tool
Persistent
Storage
Random
Users
General
apps/services
NSA
Random Wi-
Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
15.
Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from entity A to entity B
○ Source/destination must be in circle of trust
● Any link is an access point/interception point
○ Accidental logging
○ Exploitation by attacker
○ Lookup by operator
○ Employee Post-It™
○ Conspiracy of One/Compromised employee
16.
Problem Space
● We can now establish the problem space:
○ Managing secrets in an environment means establishing trust
chains in the environment
○ Links in the chains have associated risk
■ Minimize hops and minimize risk-per-link
■ Risk cannot be fully mitigated; must assume any given secret will
eventually be divulged
● Ultimate goal: zero trust
○ Don’t give the opportunity for risks to occur in the first place
17.
Scheduler
Secret Management Tool
Scheduler Agent
Container
Goal: Securely
move secret from
originator to new
container’s RAM
21.
Secret Protection
● Establishing a train of trust requires defining the
requirements it must fulfill to keep secrets protected
● Good news: we only need to do this for one secret!
22.
Secure Introduction
● First secret authenticates us to allow direct access to
more
● If you can protect this secret, you can protect any secret
○ (Generally)
● This concept is secure introduction (SI)
23.
Secret Protection
● How do you protect secrets (perform SI)?
○ First establish success criteria based on acceptable risk
● For this talk:
○ Don’t let them live forever (rotate/expire)
○ Distribute them securely
○ Limit exposure if disclosed
○ Have a break-glass procedure
○ Detect unauthorized access
24.
Rotation
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets
○ Debugging
● Secrets should be rotated “frequently”
○ User passwords vs. machine secrets
■ xkcd’s ______ ______ ______ ______ → bad policies + frequent
rotation = written down user passwords
■ Less frequently/more used = more likely overseen
■ Machines: it’s just data - plan/build for rotation and rotate often
25.
Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, always covered
○ Encryption
○ Wrapping
○ Etc.
26.
Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not root
○ API credentials: minimal function set
27.
Access Detection
● Things have a way of being leaky
● Env vars: common way to pass in secrets. Also:
○ Often logged, sometimes multiple places
○ Easily discoverable by operator
■ Both Docker commands and non-Docker commands can spill the
beans
28.
Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audit logs are great…
○ ...but do you look at them?
● Active detection (when possible) is even better
29.
Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after re-establishing identities of legitimate
secret-holders
○ …
● Figure this out during the planning process -- not after!
30.
● So...this is complicated…
● ...but there’s good news!
31.
● Recent explosion of FOSS Secret Management (SM) tools
○ Vault
○ KeyWhiz
○ Knox
○ Conjur (Docker only)
○ Many more (apologies if I missed your favorite)...
Secret Management Tools
32.
● If you only take away two things from this talk, make sure
they are the following:
33.
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
34.
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
2. Use it in a Secret Management tool
35.
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
2. Use it in a Secret Management tool
NO!
NO!
36.
● If you only take away two things from this talk, make sure
they are the following:
1. Use a Secret Management tool
2. Don’t roll your own
37.
Secret Management Tools
● Why?
○ Central, secure storage (avoid sprawl)
○ Central management
○ Codified, secure access mechanisms
○ Centralized auditing
■ Ensure knowledge of what secrets have been seen by what
users/services
○ Secret rotation/revocation/expiration
38.
Big Caveat
● This isn’t a vendor talk, but I have a bias
● I don’t have exhaustive knowledge of non-Vault SMs
● Anything I say a SM can do can be done with Vault, and
may or may not be possible with other solutions
○ They may absolutely do things Vault can’t
39.
SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilities/primitives
● Containers can support existing SI paradigms
○ If you can securely get credentials to it, but imagine…
■ Managing 100k LDAP/AD users churning at high rate
■ Generating/dropping 100k Kerberos keytabs
○ ...and you’ve solved SI anyways
40.
SM + SI
● Platform agnostic (with platform-based enhancements)
○ Change authentication mechanism “front-end”
■ Containers
■ EC2
■ Bare metal
■ Users
41.
SMs <3 Schedulers
● Anyone working with containers at scale uses a scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kubernetes (All-In-One)
42.
SMs <3 Schedulers
● Schedulers are sources of truth and provide hooks
● SM tools and schedulers can be a magical combination
○ May not be direct integration, glue code needed
○ OTOH avoiding tight coupling means better reuse across
environments
47.
Preconditions
● Suppose a SM authentication token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set of authorization policies
■ e.g. use first secret (auth token) to get application secrets
○ Consistent ID in audit logs
○ Token-scoped secure storage tied to token lifetime
● Claim: These primitives allow excellent SI for containers
61.
How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with only copy of inner token value
○ Distribute them securely
✓ Inner token covered the entire way
○ Limit exposure if disclosed
✓ Only specific policies granted, can only access specific secrets
○ Have a break-glass procedure
✓ Can lock down access at SM tool
✓ Audit logs ensure we know area of exposure
○ Detect unauthorized access
63.
How’d we do?
● Can detect unauthorized access of “real” authentication token due to
time and use limit
● Application:
○ Reads inner token: success
○ Reads storage but no inner token found: log error and recover (e.
g. fail job)
○ Denied access from invalid token: raise high-priority security alert
and check audit logs:
■ Access of outer token storage in audit logs? Unauthorized use
■ No access? Probably timing issue, but investigate carefully
64.
How’d we do?
● Mechanism not container-specific
○ CM tool drops wrapping/outer token on EC2 instance
○ File injected into chroot
○ Etc.
65.
Other Paradigms
● Remember, security is risk management
● Depending on your tech stack:
○ Different primitives might be available
○ May be constrained
● Never forget your success criteria
66.
Wrap-Up
● Plan your risk tolerance/security policy/success criteria in
advance
○ Do it now!
● Use a secret management tool
○ You’re smart, but it’s hard
● Schedulers provide perfect integration points
Slides from Jeff Mitchell's talk "Hiding in Plain Sight: Managing Secrets in a Container Environment" at ContainerDays Boston 2016: http://dynamicinfradays.org/events/2016-boston/programme.html#secrets
1.
Managing Secrets in
Container Environments
Jeff Mitchell
HashiCorp
@jefferai
2.
About Me
● Project Lead on Vault
○ A HashiCorp tool for managing secrets
○ https://vaultproject.io
● Previously, Architect at Akamai
○ Heavy focus on containerization and managing container security
3.
What this talk is...
● A discussion of paradigms and considerations necessary
for managing secrets in container environments
● Applied security, not crypto theory
○ Goal: “Know what you need to know” when figuring out your
organization’s security policy
4.
...and what this talk isn’t
● One True Answer™
○ Support requirements impact available technology/workflows
○ Each organization must set their own security policy and
implement practices that satisfy it
● A discussion of container/hypervisor/OS security
○ Let’s talk Linux containerization
5.
...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)
6.
...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)
7.
Scheduler
Secret Management Tool
Scheduler Agent
Container
Problem Space
8.
Bear with me for four slides while I wax
philosophical…
9.
What is “Security”?
● Security is the practice of risk management
○ Accepting some risks, guarding against violations of norms
● Anything that can elevate risk is a threat
○ Modeling threats informs security policy
● Risk increases with system complexity
○ More points of failure, confusion, and ingress = harder to secure
10.
What is a “Secret”?
● A secret is something that will elevate your risk if exposed
to unauthorized entities
○ Undesired consequences are harm
○ Harm may come in many forms
■ Unauthorized data access
■ Identity spoofing
■ Private data egress
■ Regulatory fines
● An exposed secret is a threat
11.
Secrets vs. Identifiers
● Not all things that can be disclosed are secrets, some are
identifiers
○ Username (identifier) vs. Password (secret)
○ TLS certificate (identifier) vs. TLS Key (secret)
○ GitHub user name (identifier) vs. API key (secret)
● Identifiers aren’t completely risk-free
○ Have chosen to ignore that risk
12.
What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities is companion to modeling threats
● Two concepts:
○ Circle of trust
○ Chain of trust
13.
Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?
14.
Circle of Trust
Only allowed long-term storage is in circle of trust
RAM
root
Secret
Management
Tool
Persistent
Storage
Random
Users
General
apps/services
NSA
Random Wi-
Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
15.
Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from entity A to entity B
○ Source/destination must be in circle of trust
● Any link is an access point/interception point
○ Accidental logging
○ Exploitation by attacker
○ Lookup by operator
○ Employee Post-It™
○ Conspiracy of One/Compromised employee
16.
Problem Space
● We can now establish the problem space:
○ Managing secrets in an environment means establishing trust
chains in the environment
○ Links in the chains have associated risk
■ Minimize hops and minimize risk-per-link
■ Risk cannot be fully mitigated; must assume any given secret will
eventually be divulged
● Ultimate goal: zero trust
○ Don’t give the opportunity for risks to occur in the first place
17.
Scheduler
Secret Management Tool
Scheduler Agent
Container
Goal: Securely
move secret from
originator to new
container’s RAM
21.
Secret Protection
● Establishing a train of trust requires defining the
requirements it must fulfill to keep secrets protected
● Good news: we only need to do this for one secret!
22.
Secure Introduction
● First secret authenticates us to allow direct access to
more
● If you can protect this secret, you can protect any secret
○ (Generally)
● This concept is secure introduction (SI)
23.
Secret Protection
● How do you protect secrets (perform SI)?
○ First establish success criteria based on acceptable risk
● For this talk:
○ Don’t let them live forever (rotate/expire)
○ Distribute them securely
○ Limit exposure if disclosed
○ Have a break-glass procedure
○ Detect unauthorized access
24.
Rotation
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets
○ Debugging
● Secrets should be rotated “frequently”
○ User passwords vs. machine secrets
■ xkcd’s ______ ______ ______ ______ → bad policies + frequent
rotation = written down user passwords
■ Less frequently/more used = more likely overseen
■ Machines: it’s just data - plan/build for rotation and rotate often
25.
Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, always covered
○ Encryption
○ Wrapping
○ Etc.
26.
Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not root
○ API credentials: minimal function set
27.
Access Detection
● Things have a way of being leaky
● Env vars: common way to pass in secrets. Also:
○ Often logged, sometimes multiple places
○ Easily discoverable by operator
■ Both Docker commands and non-Docker commands can spill the
beans
28.
Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audit logs are great…
○ ...but do you look at them?
● Active detection (when possible) is even better
29.
Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after re-establishing identities of legitimate
secret-holders
○ …
● Figure this out during the planning process -- not after!
30.
● So...this is complicated…
● ...but there’s good news!
31.
● Recent explosion of FOSS Secret Management (SM) tools
○ Vault
○ KeyWhiz
○ Knox
○ Conjur (Docker only)
○ Many more (apologies if I missed your favorite)...
Secret Management Tools
32.
● If you only take away two things from this talk, make sure
they are the following:
33.
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
34.
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
2. Use it in a Secret Management tool
35.
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
2. Use it in a Secret Management tool
NO!
NO!
36.
● If you only take away two things from this talk, make sure
they are the following:
1. Use a Secret Management tool
2. Don’t roll your own
37.
Secret Management Tools
● Why?
○ Central, secure storage (avoid sprawl)
○ Central management
○ Codified, secure access mechanisms
○ Centralized auditing
■ Ensure knowledge of what secrets have been seen by what
users/services
○ Secret rotation/revocation/expiration
38.
Big Caveat
● This isn’t a vendor talk, but I have a bias
● I don’t have exhaustive knowledge of non-Vault SMs
● Anything I say a SM can do can be done with Vault, and
may or may not be possible with other solutions
○ They may absolutely do things Vault can’t
39.
SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilities/primitives
● Containers can support existing SI paradigms
○ If you can securely get credentials to it, but imagine…
■ Managing 100k LDAP/AD users churning at high rate
■ Generating/dropping 100k Kerberos keytabs
○ ...and you’ve solved SI anyways
40.
SM + SI
● Platform agnostic (with platform-based enhancements)
○ Change authentication mechanism “front-end”
■ Containers
■ EC2
■ Bare metal
■ Users
41.
SMs <3 Schedulers
● Anyone working with containers at scale uses a scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kubernetes (All-In-One)
42.
SMs <3 Schedulers
● Schedulers are sources of truth and provide hooks
● SM tools and schedulers can be a magical combination
○ May not be direct integration, glue code needed
○ OTOH avoiding tight coupling means better reuse across
environments
47.
Preconditions
● Suppose a SM authentication token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set of authorization policies
■ e.g. use first secret (auth token) to get application secrets
○ Consistent ID in audit logs
○ Token-scoped secure storage tied to token lifetime
● Claim: These primitives allow excellent SI for containers
61.
How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with only copy of inner token value
○ Distribute them securely
✓ Inner token covered the entire way
○ Limit exposure if disclosed
✓ Only specific policies granted, can only access specific secrets
○ Have a break-glass procedure
✓ Can lock down access at SM tool
✓ Audit logs ensure we know area of exposure
○ Detect unauthorized access
63.
How’d we do?
● Can detect unauthorized access of “real” authentication token due to
time and use limit
● Application:
○ Reads inner token: success
○ Reads storage but no inner token found: log error and recover (e.
g. fail job)
○ Denied access from invalid token: raise high-priority security alert
and check audit logs:
■ Access of outer token storage in audit logs? Unauthorized use
■ No access? Probably timing issue, but investigate carefully
64.
How’d we do?
● Mechanism not container-specific
○ CM tool drops wrapping/outer token on EC2 instance
○ File injected into chroot
○ Etc.
65.
Other Paradigms
● Remember, security is risk management
● Depending on your tech stack:
○ Different primitives might be available
○ May be constrained
● Never forget your success criteria
66.
Wrap-Up
● Plan your risk tolerance/security policy/success criteria in
advance
○ Do it now!
● Use a secret management tool
○ You’re smart, but it’s hard
● Schedulers provide perfect integration points