SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)
Slides from Jeff Mitchell's talk "The Secure Introduction Problem: Getting Secrets Into Containers" at ContainerDays NYC 2016: http://dynamicinfradays.org/events/2016-nyc/programme.html#secrets
Slides from Jeff Mitchell's talk "The Secure Introduction Problem: Getting Secrets Into Containers" at ContainerDays NYC 2016: http://dynamicinfradays.org/events/2016-nyc/programme.html#secrets
ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)
1.
Think Like a Vault Developer:
Secure Introduction & Containers
Jeff Mitchell
HashiCorp
@jefferai
2.
● Understand basic tenets of security, as the Vault team
thinks about them
● Explore properties necessary for secret management and
(initial) secret distribution
Goals
3.
● Understand basic tenets of security, as the Vault team
thinks about them
● Explore properties necessary for secret management and
(initial) secret distribution
Goals
BORING
4.
● Applied security in action: secure introduction, response
wrapping and orchestrators
○ Not a Nomad/Vault talk but will use these as examples
Goals
5.
Non-Goals
● Discussion of crypto theory
○ Crypto properties (algorithm selection, etc.) are important for
applied security
○ Can stay high-level
● Comparison of container/hypervisor/OS security
○ Let’s talk Linux/Docker
6.
Non-Goals
● Deep dive into security rabbit holes
○ Let’s keep paranoia in check
8.
“Security”
● Security is the practice of risk management
○ Deciding which risks can be accepted and which cannot
○ Guarding against violations of norms
● Risk increases with system complexity
○ More points of failure, confusion, ingress and egress
9.
“Threat”
● Anything that can elevate risk is a threat
● Modeling threats informs security policy
10.
“Secret”
● A secret is something that will elevate your risk if exposed
to unauthorized entities
● Undesired consequences are harm
○ Unauthorized data access
○ Identity spoofing
○ Private data egress
○ Regulatory fines
11.
Exposure
● An exposed secret
○ Elevates risk
○ May cause harm
● An exposed secret is a threat
12.
Secrets vs. Identifiers
● Some things that can be disclosed are identifiers
○ Username (identifier) vs. Password (secret)
○ TLS certificate (identifier) vs. TLS key (secret)
○ GitHub user name (identifier) vs. API key (secret)
● Identifiers aren’t completely risk-free
○ Have chosen to ignore that risk
13.
What is “Trust”?
● A trusted entity is one that will not divulge the secrets it
has access to
● Modeling trusted entities is a companion to modeling
threats
● Two concepts
14.
Circle of Trust
Entities we trust with any secret. For this talk:
RAM
root
Secret
Management
ToolEmployees?
CPU
Cloud?
15.
Circle of Trust
Things not in circle of trust:
Persistent
Storage
Random
Users
General
apps/services
NSA
Random
Wi-Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
RAM
root
Secret
Management
Tool
CPU
Cloud?
Employees?
16.
Circle of Trust
Only allowed long-term storage is in circle of trust
Persistent
Storage
Random
Users
General
apps/services
NSA
Random
Wi-Fi Hotspot
Your Mother’s
Notepad.txt Employees?
CPU
Cloud?
RAM
root
Secret
Management
Tool
CPU
Cloud?
Employees?
17.
Chain of Trust
● The set of links (e.g. network hops) that any particular
secret travels through from source entity A to destination
entity B
● Source/destination must be in circle of trust
18.
Links
● Any link in a chain of trust is a potential access
point/interception point
○ Accidental logging
○ Exploitation by attacker
○ Lookup by operator
○ Conspiracy of One/Compromised employee
○ Employee Post-It™
19.
Problem Space
● We can now establish a problem space:
○ Managing secrets in a given environment means establishing trust
chains in the environment
● Links in the chains have associated risk
○ Minimize hops and minimize risk-per-link
20.
Problem Space
● In other words: figuring out how to securely get a secret
from producer to consumer
21.
Problem Space
● In other words: figuring out how to securely get a secret
from producer to consumer
DUH
22.
Problem Space
● Thinking securely means you have to take the roundabout
approach
○ Understanding trust, trust levels, and trust chains along the way is
incredibly important
● Only after understanding trust can you figure out where
and how to minimize risk
23.
Problem Space
● Risk cannot be fully mitigated; must assume any given
secret will eventually be divulged
● Ultimate goal: zero trust
○ Don’t give the opportunity for risks to occur in the first place
24.
Scheduler
Secret Management Tool
Scheduler Agent
Container
Goal: Securely
move secret from
originator to new
container’s RAM
29.
Secret Protection
● Establishing a chain of trust requires defining the
requirements it must fulfill to keep secrets protected
● Good news: we (often) only need to do this for one secret!
○ First secret authenticates us to allow direct access to more
30.
Secure Introduction
● If you can protect this secret, you can protect any secret
○ ...generally
● This concept is secure introduction (SI)
31.
Secure Introduction
● How do you protect secrets (perform SI)?
● First: establish success criteria based on your
organization’s acceptable risk
○ Any given organization needs a security policy and associated
success criteria that meets their individual needs
32.
Secure Introduction
● Reasons your organization doesn’t need to establish a
security policy:
33.
Secure Introduction
● Reasons your organization doesn’t need to establish a
security policy:
○ None
34.
Success Criteria
● The security properties that must be implemented/met to
implement the accepted security policy
● For secure introduction for this talk:
○ Don’t let them live forever (rotate/expire)
○ Distribute them securely
○ Limit exposure if disclosed
○ Have a break-glass procedure
○ Detect unauthorized access
35.
Rotation/Expiration
● As lifetime increases, chance for exposure → ∞
○ Caches/logs
○ Cracked over time/enough packets (WPA-TKIP, RC4)
○ Debugging
36.
Rotation/Expiration
● Secrets should be rotated “frequently” and have lifetimes
as short as possible
● User passwords vs. machine secrets
○ xkcd’s ______ ______ ______ ______ → bad policies + frequent
rotation = written down user passwords
○ Less frequently/more used = more likely overseen
○ Machines: it’s just data - plan/build for rotation and rotate often
37.
Distribution
● The literal movement along the chain of trust
○ To/From
■ People
■ Machines
● Base level: never plaintext, always covered
○ Encryption
○ Wrapping
○ Etc.
38.
Limit Exposure
● Principle of least privilege
○ DB credentials: only specific tables/operations
○ Login credentials: not root
○ API credentials: minimal function set
39.
Access Detection
● Things have a way of being leaky
● Env vars are a common way to pass in container secrets.
Also:
○ Often logged, sometimes multiple places
○ Easily discoverable by operator
■ Both Docker commands and non-Docker commands can spill the
beans
40.
Access Detection
● Equally as important as protecting a secret is knowing if
an unintended party has intercepted it
● Audit logs are great…
○ ...but do you look at them?
● Active detection (when possible) is even better
41.
Break-Glass
● Compromised?
○ Stop all further access to protected resources
○ Perform forensics
○ Rotate all secrets after re-establishing identities of legitimate
secret-holders
○ …
● Figure this out during the planning process -- not after!
43.
● Explosion of FOSS Secret Management (SM) tools
○ KeyWhiz
○ Knox
○ Conjur
○ Many more
Secret Management Tools
44.
● Explosion of FOSS Secret Management (SM) tools
○ KeyWhiz
○ Knox
○ Conjur
○ Many more
○ Vault!
Secret Management Tools
45.
● If you only take away two things from this talk, make sure
they are the following:
46.
● If you only take away two things from this talk, make sure
they are the following:
● Write your own crypto
47.
● If you only take away two things from this talk, make sure
they are the following:
● Write your own crypto
● Use it in a Secret Management tool
48.
● If you only take away two things from this talk, make sure
they are the following:
● Write your own crypto
● Use it in a Secret Management tool
NO!
NO!
49.
● If you only take away two things from this talk, make sure
they are the following:
● Use a Secret Management tool
● Don’t roll your own
50.
Secret Management Tools
● Discussion of SM tools true for Vault and its capabilities
● Vault has a focus on providing security primitives
○ Enables creation of complex workflows
○ We’ll see how this provides necessary flexibility
52.
SM + SI
● Explicit focus on the secure introduction problem
○ Core competency from ground up
○ Necessary tools/capabilities/primitives
● 100k containers can support existing SI paradigms
○ But imagine…
■ Managing 100k LDAP/AD users churning at high rate
■ Generating/dropping 100k Kerberos keytabs
○ ...and you’ve solved SI anyways
53.
SMs <3 Schedulers
● Anyone working with tasks (containers) at scale uses a
scheduler
○ Nomad
○ Mesos
○ Fleet
○ Swarm
○ Kubernetes
54.
SMs <3 Schedulers
● Schedulers are sources of truth...and provide hooks
● SM tools and schedulers can be a magical combination
59.
Preconditions
● A Vault token has:
○ Unlimited or limited use-count
○ Limited time-to-live (TTL), possibly renewable
○ Set of authorization policies
■ e.g. use first secret (auth token) to get application secrets
○ Consistent ID in audit logs
○ Token-scoped secure storage tied to token lifetime
■ ‘cubbyhole’
● Claim: These primitives allow excellent secret coverage
73.
How’d we do?
● Look at previous list:
○ Don’t let them live forever (rotate/expire)
✓ Outer token expires (use limit) with only copy of inner token value
○ Distribute them securely
✓ Inner token covered the entire way
○ Limit exposure if disclosed
✓ Only specific policies granted, can only access specific secrets
○ Have a break-glass procedure
✓ Can lock down access at SM tool
✓ Audit logs ensure we know area of exposure
○ Detect unauthorized access
74.
Access Detection
● Can detect unauthorized access of “real” authentication
token due to time and use limit
● Application:
○ Reads inner token: success
○ Reads storage but no inner token found: log error and recover
(e.g. fail job)
○ Invalid token? Check current time vs. issue time + TTL:
■ Expired? YELLOW FLAG: investigate, but probably just bad timing
■ Not expired? RED FLAG: Unauthorized use
75.
Response Wrapping
● This paradigm is so useful that it’s available for nearly any
Vault call and known as response wrapping
● Mechanism not container- or authentication-specific
○ CM tool drops wrapped secret on EC2 instance
○ File with wrapped value injected into chroot
○ Etc.
76.
Job-Finish Revocation
● Wrapped responses containing tokens also return the
token’s accessor (reference)
● Scheduling agent/system can correlate accessor with job
○ Revoke token / child leases immediately on job finish
77.
Integration
● Nomad integration uses a variation of this scheme but
basic concept is the same
○ User-supplied token at job submission time checked for access to
avoid privilege escalation via job submission
○ Agent can unwrap secrets and mount into ramdisk
○ Etc.
78.
Integration
● HashiStack tools follow the Unix model; we set out to
tackle this problem in a generic way
● Any scheduler can implement Nomad’s approach (or a
variation)
○ Primitives to do so are there in Vault
○ Kubernetes (https://github.com/kelseyhightower/vault-controller)
○ Mesos
(https://github.com/ChannelMeter/vault-gatekeeper-mesos)
79.
Wrap-Up
● Don’t ignore risk, minimize it
○ Understand your circles and chains of trust
● Use Vault, or some other SM tool
● Orchestration tools are great facilitators of secure
introduction