ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Container Environment" (Jeff Mitchell)
May. 28, 2016•0 likes
5 likes
Be the first to like this
Show More
•993 views
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Report
Technology
Slides from Jeff Mitchell's talk "Hiding in Plain Sight: Managing Secrets in a Container Environment" at ContainerDays Boston 2016: http://dynamicinfradays.org/events/2016-boston/programme.html#secrets
About Me
● Project Lead on Vault
○ A HashiCorp tool for managing secrets
○ https://vaultproject.io
● Previously, Architect at Akamai
○ Heavy focus on containerization and managing container security
What this talk is...
● A discussion of paradigms and considerations necessary
for managing secrets in container environments
● Applied security, not crypto theory
○ Goal: “Know what you need to know” when figuring out your
organization’s security policy
...and what this talk isn’t
● One True Answer™
○ Support requirements impact available technology/workflows
○ Each organization must set their own security policy and
implement practices that satisfy it
● A discussion of container/hypervisor/OS security
○ Let’s talk Linux containerization
...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)
...and what this talk isn’t
● Not a deep dive into security rabbit holes
(3-4 turtles max)
Bear with me for four slides while I wax
philosophical…
What is “Security”?
● Security is the practice of risk management
○ Accepting some risks, guarding against violations of norms
● Anything that can elevate risk is a threat
○ Modeling threats informs security policy
● Risk increases with system complexity
○ More points of failure, confusion, and ingress = harder to secure
What is a “Secret”?
● A secret is something that will elevate your risk if exposed
to unauthorized entities
○ Undesired consequences are harm
○ Harm may come in many forms
■ Unauthorized data access
■ Identity spoofing
■ Private data egress
■ Regulatory fines
● An exposed secret is a threat
Secrets vs. Identifiers
● Not all things that can be disclosed are secrets, some are
identifiers
○ Username (identifier) vs. Password (secret)
○ TLS certificate (identifier) vs. TLS Key (secret)
○ GitHub user name (identifier) vs. API key (secret)
● Identifiers aren’t completely risk-free
○ Have chosen to ignore that risk
What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities is companion to modeling threats
● Two concepts:
○ Circle of trust
○ Chain of trust
Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?
Circle of Trust
Only allowed long-term storage is in circle of trust
RAM
root
Secret
Management
Tool
Persistent
Storage
Random
Users
General
apps/services
NSA
Random Wi-
Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from entity A to entity B
○ Source/destination must be in circle of trust
● Any link is an access point/interception point
○ Accidental logging
○ Exploitation by attacker
○ Lookup by operator
○ Employee Post-It™
○ Conspiracy of One/Compromised employee
Problem Space
● We can now establish the problem space:
○ Managing secrets in an environment means establishing trust
chains in the environment
○ Links in the chains have associated risk
■ Minimize hops and minimize risk-per-link
■ Risk cannot be fully mitigated; must assume any given secret will
eventually be divulged
● Ultimate goal: zero trust
○ Don’t give the opportunity for risks to occur in the first place
Secret Protection
● Establishing a train of trust requires defining the
requirements it must fulfill to keep secrets protected
● Good news: we only need to do this for one secret!
Secure Introduction
● First secret authenticates us to allow direct access to
more
● If you can protect this secret, you can protect any secret
○ (Generally)
● This concept is secure introduction (SI)
Secret Protection
● How do you protect secrets (perform SI)?
○ First establish success criteria based on acceptable risk
● For this talk:
○ Don’t let them live forever (rotate/expire)
○ Distribute them securely
○ Limit exposure if disclosed
○ Have a break-glass procedure
○ Detect unauthorized access
Rotation
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets
○ Debugging
● Secrets should be rotated “frequently”
○ User passwords vs. machine secrets
■ xkcd’s ______ ______ ______ ______ → bad policies + frequent
rotation = written down user passwords
■ Less frequently/more used = more likely overseen
■ Machines: it’s just data - plan/build for rotation and rotate often
Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, always covered
○ Encryption
○ Wrapping
○ Etc.
Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not root
○ API credentials: minimal function set
Access Detection
● Things have a way of being leaky
● Env vars: common way to pass in secrets. Also:
○ Often logged, sometimes multiple places
○ Easily discoverable by operator
■ Both Docker commands and non-Docker commands can spill the
beans
Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audit logs are great…
○ ...but do you look at them?
● Active detection (when possible) is even better
Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after re-establishing identities of legitimate
secret-holders
○ …
● Figure this out during the planning process -- not after!
● Recent explosion of FOSS Secret Management (SM) tools
○ Vault
○ KeyWhiz
○ Knox
○ Conjur (Docker only)
○ Many more (apologies if I missed your favorite)...
Secret Management Tools
● If you only take away two things from this talk, make sure
they are the following:
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
2. Use it in a Secret Management tool
● If you only take away two things from this talk, make sure
they are the following:
1. Write your own crypto
2. Use it in a Secret Management tool
NO!
NO!
● If you only take away two things from this talk, make sure
they are the following:
1. Use a Secret Management tool
2. Don’t roll your own
Secret Management Tools
● Why?
○ Central, secure storage (avoid sprawl)
○ Central management
○ Codified, secure access mechanisms
○ Centralized auditing
■ Ensure knowledge of what secrets have been seen by what
users/services
○ Secret rotation/revocation/expiration
Big Caveat
● This isn’t a vendor talk, but I have a bias
● I don’t have exhaustive knowledge of non-Vault SMs
● Anything I say a SM can do can be done with Vault, and
may or may not be possible with other solutions
○ They may absolutely do things Vault can’t
SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilities/primitives
● Containers can support existing SI paradigms
○ If you can securely get credentials to it, but imagine…
■ Managing 100k LDAP/AD users churning at high rate
■ Generating/dropping 100k Kerberos keytabs
○ ...and you’ve solved SI anyways
SM + SI
● Platform agnostic (with platform-based enhancements)
○ Change authentication mechanism “front-end”
■ Containers
■ EC2
■ Bare metal
■ Users
SMs <3 Schedulers
● Anyone working with containers at scale uses a scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kubernetes (All-In-One)
SMs <3 Schedulers
● Schedulers are sources of truth and provide hooks
● SM tools and schedulers can be a magical combination
○ May not be direct integration, glue code needed
○ OTOH avoiding tight coupling means better reuse across
environments
Preconditions
● Suppose a SM authentication token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set of authorization policies
■ e.g. use first secret (auth token) to get application secrets
○ Consistent ID in audit logs
○ Token-scoped secure storage tied to token lifetime
● Claim: These primitives allow excellent SI for containers
How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with only copy of inner token value
○ Distribute them securely
✓ Inner token covered the entire way
○ Limit exposure if disclosed
✓ Only specific policies granted, can only access specific secrets
○ Have a break-glass procedure
✓ Can lock down access at SM tool
✓ Audit logs ensure we know area of exposure
○ Detect unauthorized access
How’d we do?
● Can detect unauthorized access of “real” authentication token due to
time and use limit
● Application:
○ Reads inner token: success
○ Reads storage but no inner token found: log error and recover (e.
g. fail job)
○ Denied access from invalid token: raise high-priority security alert
and check audit logs:
■ Access of outer token storage in audit logs? Unauthorized use
■ No access? Probably timing issue, but investigate carefully
How’d we do?
● Mechanism not container-specific
○ CM tool drops wrapping/outer token on EC2 instance
○ File injected into chroot
○ Etc.
Other Paradigms
● Remember, security is risk management
● Depending on your tech stack:
○ Different primitives might be available
○ May be constrained
● Never forget your success criteria
Wrap-Up
● Plan your risk tolerance/security policy/success criteria in
advance
○ Do it now!
● Use a secret management tool
○ You’re smart, but it’s hard
● Schedulers provide perfect integration points