DynamicInfraDays, profile picture
Uploaded byDynamicInfraDays
2,443 views

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

The document discusses key security principles related to secret management, emphasizing the establishment of trust chains and secure introduction within environments utilizing tools like HashiCorp Vault. It outlines essential practices for protecting secrets, including risk minimization, secure distribution, and monitoring unauthorized access. The ultimate goal is to achieve a zero-trust model in secret management, ensuring secrets are securely moved and handled throughout their lifecycle.

Technology◦
Related topics:
Information Security•

Embed presentation

Think Like a Vault Developer: Secure Introduction & Containers Jeff Mitchell HashiCorp @jefferai
● Understand basic tenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals
● Understand basic tenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals BORING
● Applied security in action: secure introduction, response wrapping and orchestrators ○ Not a Nomad/Vault talk but will use these as examples Goals
Non-Goals ● Discussion of crypto theory ○ Crypto properties (algorithm selection, etc.) are important for applied security ○ Can stay high-level ● Comparison of container/hypervisor/OS security ○ Let’s talk Linux/Docker
Non-Goals ● Deep dive into security rabbit holes ○ Let’s keep paranoia in check
Tenets / Defining Security
“Security” ● Security is the practice of risk management ○ Deciding which risks can be accepted and which cannot ○ Guarding against violations of norms ● Risk increases with system complexity ○ More points of failure, confusion, ingress and egress
“Threat” ● Anything that can elevate risk is a threat ● Modeling threats informs security policy
“Secret” ● A secret is something that will elevate your risk if exposed to unauthorized entities ● Undesired consequences are harm ○ Unauthorized data access ○ Identity spoofing ○ Private data egress ○ Regulatory fines
Exposure ● An exposed secret ○ Elevates risk ○ May cause harm ● An exposed secret is a threat
Secrets vs. Identifiers ● Some things that can be disclosed are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk
What is “Trust”? ● A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is a companion to modeling threats ● Two concepts
Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?
Circle of Trust Things not in circle of trust: Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
Circle of Trust Only allowed long-term storage is in circle of trust Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
Chain of Trust ● The set of links (e.g. network hops) that any particular secret travels through from source entity A to destination entity B ● Source/destination must be in circle of trust
Links ● Any link in a chain of trust is a potential access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Conspiracy of One/Compromised employee ○ Employee Post-It™
Problem Space ● We can now establish a problem space: ○ Managing secrets in a given environment means establishing trust chains in the environment ● Links in the chains have associated risk ○ Minimize hops and minimize risk-per-link
Problem Space ● In other words: figuring out how to securely get a secret from producer to consumer
Problem Space ● In other words: figuring out how to securely get a secret from producer to consumer DUH
Problem Space ● Thinking securely means you have to take the roundabout approach ○ Understanding trust, trust levels, and trust chains along the way is incredibly important ● Only after understanding trust can you figure out where and how to minimize risk
Problem Space ● Risk cannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place
Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM
Scheduler Secret Management Tool Scheduler Agent Container ?
Scheduler Secret Management Tool Scheduler Agent Container ?
Scheduler Secret Management Tool Scheduler Agent Container ?
Security Properties / Success Criteria
Secret Protection ● Establishing a chain of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we (often) only need to do this for one secret! ○ First secret authenticates us to allow direct access to more
Secure Introduction ● If you can protect this secret, you can protect any secret ○ ...generally ● This concept is secure introduction (SI)
Secure Introduction ● How do you protect secrets (perform SI)? ● First: establish success criteria based on your organization’s acceptable risk ○ Any given organization needs a security policy and associated success criteria that meets their individual needs
Secure Introduction ● Reasons your organization doesn’t need to establish a security policy:
Secure Introduction ● Reasons your organization doesn’t need to establish a security policy: ○ None
Success Criteria ● The security properties that must be implemented/met to implement the accepted security policy ● For secure introduction for this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access
Rotation/Expiration ● As lifetime increases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets (WPA-TKIP, RC4) ○ Debugging
Rotation/Expiration ● Secrets should be rotated “frequently” and have lifetimes as short as possible ● User passwords vs. machine secrets ○ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ○ Less frequently/more used = more likely overseen ○ Machines: it’s just data - plan/build for rotation and rotate often
Distribution ● The literal movement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.
Limit Exposure ● Principle of least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set
Access Detection ● Things have a way of being leaky ● Env vars are a common way to pass in container secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans
Access Detection ● Equally as important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better
Break-Glass ● Compromised? ○ Stop all further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!
● You do not have to face this alone
● Explosion of FOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more Secret Management Tools
● Explosion of FOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more ○ Vault! Secret Management Tools
● If you only take away two things from this talk, make sure they are the following:
● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto
● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool
● If you only take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool NO! NO!
● If you only take away two things from this talk, make sure they are the following: ● Use a Secret Management tool ● Don’t roll your own
Secret Management Tools ● Discussion of SM tools true for Vault and its capabilities ● Vault has a focus on providing security primitives ○ Enables creation of complex workflows ○ We’ll see how this provides necessary flexibility
Secret Management Tools ● Central: ○ Secure storage (avoid sprawl) ○ Management ○ Auditing (who has seen/accessed what, when) ● Codified, secure access mechanisms ● Secret rotation/revocation/expiration
SM + SI ● Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● 100k containers can support existing SI paradigms ○ But imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways
SMs <3 Schedulers ● Anyone working with tasks (containers) at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes
SMs <3 Schedulers ● Schedulers are sources of truth...and provide hooks ● SM tools and schedulers can be a magical combination
Scheduler Secret Management Tool Scheduler Agent Container Vault
Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Vault
Scheduler Vault Scheduler Agent Container SI #1 (Traditional): AWS-EC2 with Vault Secure Intro Client Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
Preconditions ● A Vault token has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ■ ‘cubbyhole’ ● Claim: These primitives allow excellent secret coverage
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert Vault
Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container Vault
How’d we do? ● Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access
Access Detection ● Can detect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e.g. fail job) ○ Invalid token? Check current time vs. issue time + TTL: ■ Expired? YELLOW FLAG: investigate, but probably just bad timing ■ Not expired? RED FLAG: Unauthorized use
Response Wrapping ● This paradigm is so useful that it’s available for nearly any Vault call and known as response wrapping ● Mechanism not container- or authentication-specific ○ CM tool drops wrapped secret on EC2 instance ○ File with wrapped value injected into chroot ○ Etc.
Job-Finish Revocation ● Wrapped responses containing tokens also return the token’s accessor (reference) ● Scheduling agent/system can correlate accessor with job ○ Revoke token / child leases immediately on job finish
Integration ● Nomad integration uses a variation of this scheme but basic concept is the same ○ User-supplied token at job submission time checked for access to avoid privilege escalation via job submission ○ Agent can unwrap secrets and mount into ramdisk ○ Etc.
Integration ● HashiStack tools follow the Unix model; we set out to tackle this problem in a generic way ● Any scheduler can implement Nomad’s approach (or a variation) ○ Primitives to do so are there in Vault ○ Kubernetes (https://github.com/kelseyhightower/vault-controller) ○ Mesos (https://github.com/ChannelMeter/vault-gatekeeper-mesos)
Wrap-Up ● Don’t ignore risk, minimize it ○ Understand your circles and chains of trust ● Use Vault, or some other SM tool ● Orchestration tools are great facilitators of secure introduction
Thanks!

More Related Content

PDF
ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Cont...
byDynamicInfraDays
 
PPTX
Vault
byJean-Philippe Bélanger
 
PDF
Managing secrets at scale
byAlex Schoof
 
PPTX
Nodejsvault austin2019
byTaswar Bhatti
 
PPTX
Using ansible vault to protect your secrets
byExcella
 
PDF
6 Months Sailing with Docker in Production
byHung Lin
 
PDF
Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault
byJeff Horwitz
 
PDF
Using Vault to decouple MySQL Secrets
byDerek Downey
 
ContainerDays Boston 2016: "Hiding in Plain Sight: Managing Secrets in a Cont...
byDynamicInfraDays
 
Vault
byJean-Philippe Bélanger
 
Managing secrets at scale
byAlex Schoof
 
Nodejsvault austin2019
byTaswar Bhatti
 
Using ansible vault to protect your secrets
byExcella
 
6 Months Sailing with Docker in Production
byHung Lin
 
Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault
byJeff Horwitz
 
Using Vault to decouple MySQL Secrets
byDerek Downey
 

What's hot

PPTX
Cloud patterns forwardjs April Ottawa 2019
byTaswar Bhatti
 
PPTX
Using Vault for your Nodejs Secrets
byTaswar Bhatti
 
PDF
Neil Saunders (Beamly) - Securing your AWS Infrastructure with Hashicorp Vault
byOutlyer
 
PDF
DevOpsDays - DevOps: Security 干我何事？
bysmalltown
 
PPTX
Keeping a Secret with HashiCorp Vault
byMitchell Pronschinske
 
PPTX
Hashicorp Vault ppt
byShrey Agarwal
 
PDF
HashiCorp Vault Workshop：幫 Credentials 找個窩
bysmalltown
 
PDF
HashiCorp's Vault - The Examples
byMichał Czeraszkiewicz
 
ODP
Security and why you need to review yours.
byDavid Busby, CISSP
 
PDF
Secret Management with Hashicorp’s Vault
byAWS Germany
 
PPTX
Laverna vs etherpad
byantitree
 
PDF
HashiCorp Vault Plugin Infrastructure
byNicolas Corrarello
 
PDF
Issuing temporary credentials for my sql using hashicorp vault
byOlinData
 
PPTX
Owasp crypto tools and projects
byOwaspCzech
 
PPTX
Vault - Secret and Key Management
byAnthony Ikeda
 
PDF
Sullivan white boxcrypto-baythreat-2013
byCloudflare
 
PPTX
Toni de la Fuente - Automate or die! How to survive to an attack in the Cloud...
byRootedCON
 
PPTX
The Travelling Pentester: Diaries of the Shortest Path to Compromise
byWill Schroeder
 
PPTX
Building an Empire with PowerShell
byWill Schroeder
 
PDF
A tale of application development
byNicolas Corrarello
 
Cloud patterns forwardjs April Ottawa 2019
byTaswar Bhatti
 
Using Vault for your Nodejs Secrets
byTaswar Bhatti
 
Neil Saunders (Beamly) - Securing your AWS Infrastructure with Hashicorp Vault
byOutlyer
 
DevOpsDays - DevOps: Security 干我何事？
bysmalltown
 
Keeping a Secret with HashiCorp Vault
byMitchell Pronschinske
 
Hashicorp Vault ppt
byShrey Agarwal
 
HashiCorp Vault Workshop：幫 Credentials 找個窩
bysmalltown
 
HashiCorp's Vault - The Examples
byMichał Czeraszkiewicz
 
Security and why you need to review yours.
byDavid Busby, CISSP
 
Secret Management with Hashicorp’s Vault
byAWS Germany
 
Laverna vs etherpad
byantitree
 
HashiCorp Vault Plugin Infrastructure
byNicolas Corrarello
 
Issuing temporary credentials for my sql using hashicorp vault
byOlinData
 
Owasp crypto tools and projects
byOwaspCzech
 
Vault - Secret and Key Management
byAnthony Ikeda
 
Sullivan white boxcrypto-baythreat-2013
byCloudflare
 
Toni de la Fuente - Automate or die! How to survive to an attack in the Cloud...
byRootedCON
 
The Travelling Pentester: Diaries of the Shortest Path to Compromise
byWill Schroeder
 
Building an Empire with PowerShell
byWill Schroeder
 
A tale of application development
byNicolas Corrarello
 

Viewers also liked

PDF
Introduction to Infrastructure as Code & Automation / Introduction to Chef
byNathen Harvey
 
PDF
How GitLab and HackerOne help organizations innovate faster without compromis...
byHackerOne
 
PDF
Why You Need to Rethink Container Security
byFlawCheck
 
PDF
Docker London: Container Security
byPhil Estes
 
PDF
Veer's Container Security
byJim Barlow
 
PDF
Monetising Your Skill
by'Detola Amure
 
PPTX
Understanding container security
byJohn Kinsella
 
PDF
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
byZach Hill
 
PDF
Docker Security: Are Your Containers Tightly Secured to the Ship?
byMichael Boelen
 
PDF
Practical Approaches to Container Security
byShea Stewart
 
PDF
Atomic CLI scan
byLalatendu Mohanty
 
PPTX
Docker Security workshop slides
byDocker, Inc.
 
PPTX
AWS Security Architecture - Overview
bySai Kesavamatham
 
PPTX
Security best practices for kubernetes deployment
byMichael Cherny
 
PPTX
Monitoring, Logging and Tracing on Kubernetes
byMartin Etmajer
 
PDF
Docker Security - Secure Container Deployment on Linux
byMichael Boelen
 
PDF
London HUG 19/5 - Kubernetes and vault
byLondon HashiCorp User Group
 
PDF
Container Orchestration Wars
byKarl Isenberg
 
PPTX
Docker Security Overview
bySreenivas Makam
 
Introduction to Infrastructure as Code & Automation / Introduction to Chef
byNathen Harvey
 
How GitLab and HackerOne help organizations innovate faster without compromis...
byHackerOne
 
Why You Need to Rethink Container Security
byFlawCheck
 
Docker London: Container Security
byPhil Estes
 
Veer's Container Security
byJim Barlow
 
Monetising Your Skill
by'Detola Amure
 
Understanding container security
byJohn Kinsella
 
Open Source Tools for Container Security and Compliance @Docker LA Meetup 2/13
byZach Hill
 
Docker Security: Are Your Containers Tightly Secured to the Ship?
byMichael Boelen
 
Practical Approaches to Container Security
byShea Stewart
 
Atomic CLI scan
byLalatendu Mohanty
 
Docker Security workshop slides
byDocker, Inc.
 
AWS Security Architecture - Overview
bySai Kesavamatham
 
Security best practices for kubernetes deployment
byMichael Cherny
 
Monitoring, Logging and Tracing on Kubernetes
byMartin Etmajer
 
Docker Security - Secure Container Deployment on Linux
byMichael Boelen
 
London HUG 19/5 - Kubernetes and vault
byLondon HashiCorp User Group
 
Container Orchestration Wars
byKarl Isenberg
 
Docker Security Overview
bySreenivas Makam
 

Similar to ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

PDF
Secure Secret Management on a Budget: Reasoning about Scalable SM with Vault ...
byMary Racter
 
PDF
Automation Patterns for Scalable Secret Management
byMary Racter
 
PDF
XP Days 2019: First secret delivery for modern cloud-native applications
byVlad Fedosov
 
PDF
Rugged Building Materials and Creating Agility with Security
byDavid Etue
 
PDF
Security concepts
byDeepak Raj
 
PPT
Securing Your Data In The Cloud
byOmer Trajman
 
PPTX
"Cryptography, Data Protection, and Security For Start-Ups In The Post Snowde...
byHackIT Ukraine
 
PDF
Making Security Approachable for Developers and Operators
byArmonDadgar
 
PDF
Designing Security Architecture Solutions 1st Jay Ramachandran
bypianondorman
 
PPTX
A Throwaway Deck for Cloud Security Essentials 2.0 delivered at RSA 2016
byShannon Lietz
 
PDF
Practical Cloud Security A Guide For Secure Design And Deployment 1st Edition...
byjaromdembo
 
PDF
Designing Security Architecture Solutions 1st Edition Jay Ramachandran
bywohkwrtliu0660
 
PDF
IRJET- Data and Technical Security Issues in Cloud Computing Databases
byIRJET Journal
 
PDF
Practical Cloud Security A Guide for Secure Design and Deployment 1st Edition...
bybvpxmqwie0546
 
PPTX
Practical Security for the Cloud
byChirag Joshi, CISA, CISM, CRISC
 
PPTX
Cloud security From Infrastructure to People-ware
byTzar Umang
 
PPTX
Securing Open Source Databases
byGazzang
 
PPT
Aws training in bangalore
byapponix123
 
PDF
Information Security Risk Management
byipspat
 
PPT
Cloud Computing & Security
byAwais Mansoor Chohan
 
Secure Secret Management on a Budget: Reasoning about Scalable SM with Vault ...
byMary Racter
 
Automation Patterns for Scalable Secret Management
byMary Racter
 
XP Days 2019: First secret delivery for modern cloud-native applications
byVlad Fedosov
 
Rugged Building Materials and Creating Agility with Security
byDavid Etue
 
Security concepts
byDeepak Raj
 
Securing Your Data In The Cloud
byOmer Trajman
 
"Cryptography, Data Protection, and Security For Start-Ups In The Post Snowde...
byHackIT Ukraine
 
Making Security Approachable for Developers and Operators
byArmonDadgar
 
Designing Security Architecture Solutions 1st Jay Ramachandran
bypianondorman
 
A Throwaway Deck for Cloud Security Essentials 2.0 delivered at RSA 2016
byShannon Lietz
 
Practical Cloud Security A Guide For Secure Design And Deployment 1st Edition...
byjaromdembo
 
Designing Security Architecture Solutions 1st Edition Jay Ramachandran
bywohkwrtliu0660
 
IRJET- Data and Technical Security Issues in Cloud Computing Databases
byIRJET Journal
 
Practical Cloud Security A Guide for Secure Design and Deployment 1st Edition...
bybvpxmqwie0546
 
Practical Security for the Cloud
byChirag Joshi, CISA, CISM, CRISC
 
Cloud security From Infrastructure to People-ware
byTzar Umang
 
Securing Open Source Databases
byGazzang
 
Aws training in bangalore
byapponix123
 
Information Security Risk Management
byipspat
 
Cloud Computing & Security
byAwais Mansoor Chohan
 

More from DynamicInfraDays

PDF
ContainerDays NYC 2016: "From Hello World to Real World: Building a Productio...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2016: "Securing Your Docker Image Registry for Production" ...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2016: "OpenWhisk: A Serverless Computing Platform" (Rodric ...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2016: "State of the Persistence Art: Present Best Practices...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2016: "Observability and Manageability in a Container Envir...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2016: "Containers in Azure: Understanding the Microsoft Con...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2016: "Introduction to Application Automation with Habitat"...
byDynamicInfraDays
 
PDF
ContainerDays Boston 2016: "Docker For the Developer" (Borja Burgos)
byDynamicInfraDays
 
PDF
ContainerDays Boston 2016: "Autopilot: Running Real-world Applications in Con...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2015: "Container Orchestration Compared: Kubernetes and Doc...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2015: "What It Really Takes to Build a Container Platform" ...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2015: "How Yodle Cleaned Up the Mess Using Containers and M...
byDynamicInfraDays
 
PDF
ContainerDays NYC 2015: "Easing Your Way Into Docker: Lessons From a Journey ...
byDynamicInfraDays
 
PDF
ContainerDays Boston 2015: "CoreOS: Building the Layers of the Scalable Clust...
byDynamicInfraDays
 
PDF
ContainerDays Boston 2015: "Continuous Delivery with Containers" (Nick Gauthier)
byDynamicInfraDays
 
PDF
ContainerDays Boston 2015: "A Brief History of Containers" (Jeff Victor & Kir...
byDynamicInfraDays
 
ContainerDays NYC 2016: "From Hello World to Real World: Building a Productio...
byDynamicInfraDays
 
ContainerDays NYC 2016: "Securing Your Docker Image Registry for Production" ...
byDynamicInfraDays
 
ContainerDays NYC 2016: "OpenWhisk: A Serverless Computing Platform" (Rodric ...
byDynamicInfraDays
 
ContainerDays NYC 2016: "State of the Persistence Art: Present Best Practices...
byDynamicInfraDays
 
ContainerDays NYC 2016: "Observability and Manageability in a Container Envir...
byDynamicInfraDays
 
ContainerDays NYC 2016: "Containers in Azure: Understanding the Microsoft Con...
byDynamicInfraDays
 
ContainerDays NYC 2016: "Introduction to Application Automation with Habitat"...
byDynamicInfraDays
 
ContainerDays Boston 2016: "Docker For the Developer" (Borja Burgos)
byDynamicInfraDays
 
ContainerDays Boston 2016: "Autopilot: Running Real-world Applications in Con...
byDynamicInfraDays
 
ContainerDays NYC 2015: "Container Orchestration Compared: Kubernetes and Doc...
byDynamicInfraDays
 
ContainerDays NYC 2015: "What It Really Takes to Build a Container Platform" ...
byDynamicInfraDays
 
ContainerDays NYC 2015: "How Yodle Cleaned Up the Mess Using Containers and M...
byDynamicInfraDays
 
ContainerDays NYC 2015: "Easing Your Way Into Docker: Lessons From a Journey ...
byDynamicInfraDays
 
ContainerDays Boston 2015: "CoreOS: Building the Layers of the Scalable Clust...
byDynamicInfraDays
 
ContainerDays Boston 2015: "Continuous Delivery with Containers" (Nick Gauthier)
byDynamicInfraDays
 
ContainerDays Boston 2015: "A Brief History of Containers" (Jeff Victor & Kir...
byDynamicInfraDays
 

Recently uploaded

PDF
[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...
bywintari3
 
PPTX
Guardrails in Action - Ensuring Safe AI with Azure AI Content Safety.pptx
byUdaiappa Ramachandran
 
PDF
The Necessity of Digital Forensics, the Digital Forensics Process & Laborator...
bychrstnpetwinchester
 
PDF
[DevFest Strasbourg 2025] - NodeJs Can do that !!
bymoassiongbon
 
PDF
Integrating AI with Meaningful Human Collaboration
byCadabra Studio
 
PDF
Transforming Supply Chains with Amazon Bedrock AgentCore (AWS Swiss User Grou...
byChris Bingham
 
PDF
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
PDF
5 Common Supply Chain Attacks and How They Work | CyberPro Magazine
byCyberPro Magazine
 
PPTX
The power of Slack and MuleSoft | Bangalore MuleSoft Meetup #60
byshyamraj55
 
PPTX
Connecting the unconnectable: Exploring LoRaWAN for IoT
byasasiraarthur
 
PDF
[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...
byWintari Yasmin
 
PDF
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
PDF
Accessibility & Inclusion: What Comes Next. Presentation of the Digital Acces...
byAssociazione Digital Days
 
PPTX
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
PDF
Transcript: The partnership effect: Libraries and publishers on collaborating...
byBookNet Canada
 
PDF
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
PDF
Mastering UiPath Maestro – Session 2 – Building a Live Use Case - Session 2
byDianaGray10
 
PDF
[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...
byWintari Yasmin
 
PDF
So You Want to Work at Google | DevFest Seattle 2025
byMargaret Maynard-Reid
 
PPTX
kernel PPT (Explanation of Windows Kernal).pptx
byINFOBYTES1
 
[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...
bywintari3
 
Guardrails in Action - Ensuring Safe AI with Azure AI Content Safety.pptx
byUdaiappa Ramachandran
 
The Necessity of Digital Forensics, the Digital Forensics Process & Laborator...
bychrstnpetwinchester
 
[DevFest Strasbourg 2025] - NodeJs Can do that !!
bymoassiongbon
 
Integrating AI with Meaningful Human Collaboration
byCadabra Studio
 
Transforming Supply Chains with Amazon Bedrock AgentCore (AWS Swiss User Grou...
byChris Bingham
 
"DISC as GPS for team leaders: how to lead a team from storming to performing...
byFwdays
 
5 Common Supply Chain Attacks and How They Work | CyberPro Magazine
byCyberPro Magazine
 
The power of Slack and MuleSoft | Bangalore MuleSoft Meetup #60
byshyamraj55
 
Connecting the unconnectable: Exploring LoRaWAN for IoT
byasasiraarthur
 
[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...
byWintari Yasmin
 
ODSC AI West: Agent Optimization: Beyond Context engineering
byShashikant Jagtap
 
Accessibility & Inclusion: What Comes Next. Presentation of the Digital Acces...
byAssociazione Digital Days
 
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
Transcript: The partnership effect: Libraries and publishers on collaborating...
byBookNet Canada
 
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
Mastering UiPath Maestro – Session 2 – Building a Live Use Case - Session 2
byDianaGray10
 
[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...
byWintari Yasmin
 
So You Want to Work at Google | DevFest Seattle 2025
byMargaret Maynard-Reid
 
kernel PPT (Explanation of Windows Kernal).pptx
byINFOBYTES1
 

ContainerDays NYC 2016: "The Secure Introduction Problem: Getting Secrets Into Containers" (Jeff Mitchell)

  • 1.
    Think Like aVault Developer: Secure Introduction & Containers Jeff Mitchell HashiCorp @jefferai
  • 2.
    ● Understand basictenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals
  • 3.
    ● Understand basictenets of security, as the Vault team thinks about them ● Explore properties necessary for secret management and (initial) secret distribution Goals BORING
  • 4.
    ● Applied securityin action: secure introduction, response wrapping and orchestrators ○ Not a Nomad/Vault talk but will use these as examples Goals
  • 5.
    Non-Goals ● Discussion ofcrypto theory ○ Crypto properties (algorithm selection, etc.) are important for applied security ○ Can stay high-level ● Comparison of container/hypervisor/OS security ○ Let’s talk Linux/Docker
  • 6.
    Non-Goals ● Deep diveinto security rabbit holes ○ Let’s keep paranoia in check
  • 7.
  • 8.
    “Security” ● Security isthe practice of risk management ○ Deciding which risks can be accepted and which cannot ○ Guarding against violations of norms ● Risk increases with system complexity ○ More points of failure, confusion, ingress and egress
  • 9.
    “Threat” ● Anything thatcan elevate risk is a threat ● Modeling threats informs security policy
  • 10.
    “Secret” ● A secretis something that will elevate your risk if exposed to unauthorized entities ● Undesired consequences are harm ○ Unauthorized data access ○ Identity spoofing ○ Private data egress ○ Regulatory fines
  • 11.
    Exposure ● An exposedsecret ○ Elevates risk ○ May cause harm ● An exposed secret is a threat
  • 12.
    Secrets vs. Identifiers ●Some things that can be disclosed are identifiers ○ Username (identifier) vs. Password (secret) ○ TLS certificate (identifier) vs. TLS key (secret) ○ GitHub user name (identifier) vs. API key (secret) ● Identifiers aren’t completely risk-free ○ Have chosen to ignore that risk
  • 13.
    What is “Trust”? ●A trusted entity is one that will not divulge the secrets it has access to ● Modeling trusted entities is a companion to modeling threats ● Two concepts
  • 14.
    Circle of Trust Entitieswe trust with any secret. For this talk: RAM root Secret Management ToolEmployees? CPU Cloud?
  • 15.
    Circle of Trust Thingsnot in circle of trust: Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
  • 16.
    Circle of Trust Onlyallowed long-term storage is in circle of trust Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
  • 17.
    Chain of Trust ●The set of links (e.g. network hops) that any particular secret travels through from source entity A to destination entity B ● Source/destination must be in circle of trust
  • 18.
    Links ● Any linkin a chain of trust is a potential access point/interception point ○ Accidental logging ○ Exploitation by attacker ○ Lookup by operator ○ Conspiracy of One/Compromised employee ○ Employee Post-It™
  • 19.
    Problem Space ● Wecan now establish a problem space: ○ Managing secrets in a given environment means establishing trust chains in the environment ● Links in the chains have associated risk ○ Minimize hops and minimize risk-per-link
  • 20.
    Problem Space ● Inother words: figuring out how to securely get a secret from producer to consumer
  • 21.
    Problem Space ● Inother words: figuring out how to securely get a secret from producer to consumer DUH
  • 22.
    Problem Space ● Thinkingsecurely means you have to take the roundabout approach ○ Understanding trust, trust levels, and trust chains along the way is incredibly important ● Only after understanding trust can you figure out where and how to minimize risk
  • 23.
    Problem Space ● Riskcannot be fully mitigated; must assume any given secret will eventually be divulged ● Ultimate goal: zero trust ○ Don’t give the opportunity for risks to occur in the first place
  • 24.
    Scheduler Secret Management Tool SchedulerAgent Container Goal: Securely move secret from originator to new container’s RAM
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
    Secret Protection ● Establishinga chain of trust requires defining the requirements it must fulfill to keep secrets protected ● Good news: we (often) only need to do this for one secret! ○ First secret authenticates us to allow direct access to more
  • 30.
    Secure Introduction ● Ifyou can protect this secret, you can protect any secret ○ ...generally ● This concept is secure introduction (SI)
  • 31.
    Secure Introduction ● Howdo you protect secrets (perform SI)? ● First: establish success criteria based on your organization’s acceptable risk ○ Any given organization needs a security policy and associated success criteria that meets their individual needs
  • 32.
    Secure Introduction ● Reasonsyour organization doesn’t need to establish a security policy:
  • 33.
    Secure Introduction ● Reasonsyour organization doesn’t need to establish a security policy: ○ None
  • 34.
    Success Criteria ● Thesecurity properties that must be implemented/met to implement the accepted security policy ● For secure introduction for this talk: ○ Don’t let them live forever (rotate/expire) ○ Distribute them securely ○ Limit exposure if disclosed ○ Have a break-glass procedure ○ Detect unauthorized access
  • 35.
    Rotation/Expiration ● As lifetimeincreases, chance for exposure → ∞ ○ Caches/logs ○ Cracked over time/enough packets (WPA-TKIP, RC4) ○ Debugging
  • 36.
    Rotation/Expiration ● Secrets shouldbe rotated “frequently” and have lifetimes as short as possible ● User passwords vs. machine secrets ○ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ○ Less frequently/more used = more likely overseen ○ Machines: it’s just data - plan/build for rotation and rotate often
  • 37.
    Distribution ● The literalmovement along the chain of trust ○ To/From ■ People ■ Machines ● Base level: never plaintext, always covered ○ Encryption ○ Wrapping ○ Etc.
  • 38.
    Limit Exposure ● Principleof least privilege ○ DB credentials: only specific tables/operations ○ Login credentials: not root ○ API credentials: minimal function set
  • 39.
    Access Detection ● Thingshave a way of being leaky ● Env vars are a common way to pass in container secrets. Also: ○ Often logged, sometimes multiple places ○ Easily discoverable by operator ■ Both Docker commands and non-Docker commands can spill the beans
  • 40.
    Access Detection ● Equallyas important as protecting a secret is knowing if an unintended party has intercepted it ● Audit logs are great… ○ ...but do you look at them? ● Active detection (when possible) is even better
  • 41.
    Break-Glass ● Compromised? ○ Stopall further access to protected resources ○ Perform forensics ○ Rotate all secrets after re-establishing identities of legitimate secret-holders ○ … ● Figure this out during the planning process -- not after!
  • 43.
    ● You donot have to face this alone
  • 44.
    ● Explosion ofFOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more Secret Management Tools
  • 45.
    ● Explosion ofFOSS Secret Management (SM) tools ○ KeyWhiz ○ Knox ○ Conjur ○ Many more ○ Vault! Secret Management Tools
  • 46.
    ● If youonly take away two things from this talk, make sure they are the following:
  • 47.
    ● If youonly take away two things from this talk, make sure they are the following: ● Write your own crypto
  • 48.
    ● If youonly take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool
  • 49.
    ● If youonly take away two things from this talk, make sure they are the following: ● Write your own crypto ● Use it in a Secret Management tool NO! NO!
  • 50.
    ● If youonly take away two things from this talk, make sure they are the following: ● Use a Secret Management tool ● Don’t roll your own
  • 51.
    Secret Management Tools ●Discussion of SM tools true for Vault and its capabilities ● Vault has a focus on providing security primitives ○ Enables creation of complex workflows ○ We’ll see how this provides necessary flexibility
  • 52.
    Secret Management Tools ●Central: ○ Secure storage (avoid sprawl) ○ Management ○ Auditing (who has seen/accessed what, when) ● Codified, secure access mechanisms ● Secret rotation/revocation/expiration
  • 53.
    SM + SI ●Explicit focus on the secure introduction problem ○ Core competency from ground up ○ Necessary tools/capabilities/primitives ● 100k containers can support existing SI paradigms ○ But imagine… ■ Managing 100k LDAP/AD users churning at high rate ■ Generating/dropping 100k Kerberos keytabs ○ ...and you’ve solved SI anyways
  • 54.
    SMs <3 Schedulers ●Anyone working with tasks (containers) at scale uses a scheduler ○ Nomad ○ Mesos ○ Fleet ○ Swarm ○ Kubernetes
  • 55.
    SMs <3 Schedulers ●Schedulers are sources of truth...and provide hooks ● SM tools and schedulers can be a magical combination
  • 56.
  • 57.
    Scheduler Secret Management Tool SchedulerAgent Container SI #1 (Traditional) Vault
  • 58.
    Scheduler Vault Scheduler Agent Container SI #1(Traditional): AWS-EC2 with Vault Secure Intro Client Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
  • 59.
    Scheduler Secret Management Tool SchedulerAgent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
  • 60.
    Preconditions ● A Vaulttoken has: ○ Unlimited or limited use-count ○ Limited time-to-live (TTL), possibly renewable ○ Set of authorization policies ■ e.g. use first secret (auth token) to get application secrets ○ Consistent ID in audit logs ○ Token-scoped secure storage tied to token lifetime ■ ‘cubbyhole’ ● Claim: These primitives allow excellent secret coverage
  • 61.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  • 62.
    Scheduler App: db_writer Security Policy:app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  • 63.
    Scheduler App: db_writer Security Policy:app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  • 64.
    Scheduler Secret Management Tool CreateToken: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
  • 65.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ContainerImage: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
  • 66.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap Vault
  • 67.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Vault
  • 68.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds Vault
  • 69.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container Vault
  • 70.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds Vault
  • 71.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container Vault
  • 72.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert Vault
  • 73.
    Scheduler Secret Management Tool SchedulerAgent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container Vault
  • 74.
    How’d we do? ●Look at previous list: ○ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ○ Distribute them securely ✓ Inner token covered the entire way ○ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ○ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ○ Detect unauthorized access
  • 75.
    Access Detection ● Candetect unauthorized access of “real” authentication token due to time and use limit ● Application: ○ Reads inner token: success ○ Reads storage but no inner token found: log error and recover (e.g. fail job) ○ Invalid token? Check current time vs. issue time + TTL: ■ Expired? YELLOW FLAG: investigate, but probably just bad timing ■ Not expired? RED FLAG: Unauthorized use
  • 76.
    Response Wrapping ● Thisparadigm is so useful that it’s available for nearly any Vault call and known as response wrapping ● Mechanism not container- or authentication-specific ○ CM tool drops wrapped secret on EC2 instance ○ File with wrapped value injected into chroot ○ Etc.
  • 77.
    Job-Finish Revocation ● Wrappedresponses containing tokens also return the token’s accessor (reference) ● Scheduling agent/system can correlate accessor with job ○ Revoke token / child leases immediately on job finish
  • 78.
    Integration ● Nomad integrationuses a variation of this scheme but basic concept is the same ○ User-supplied token at job submission time checked for access to avoid privilege escalation via job submission ○ Agent can unwrap secrets and mount into ramdisk ○ Etc.
  • 79.
    Integration ● HashiStack toolsfollow the Unix model; we set out to tackle this problem in a generic way ● Any scheduler can implement Nomad’s approach (or a variation) ○ Primitives to do so are there in Vault ○ Kubernetes (https://github.com/kelseyhightower/vault-controller) ○ Mesos (https://github.com/ChannelMeter/vault-gatekeeper-mesos)
  • 80.
    Wrap-Up ● Don’t ignorerisk, minimize it ○ Understand your circles and chains of trust ● Use Vault, or some other SM tool ● Orchestration tools are great facilitators of secure introduction
  • 81.