SlideShare a Scribd company logo
Overview of Secret
management Solutions and
Architectures
Yuechuan Chen
About this talk
Three goals:
1. Raise awareness of good secret management practices, what it is and why it’s important
2. Identify characteristics of a good solution
3. Overview of solutions on AWS, Azure and Kubernetes
Daniel Summerfield’s talk “Turtles all the way down”
What’s a secret
- A secret is anything that you want to control access to, such as API keys, passwords, certificates,
and more.
- More services = more authentication = more secrets
- People tend to take insecure shortcuts: hardcoding secrets in source code, container images,
configuration files. Credentials are shared via email, slack, and shared folders.
What is secret management
Secret management concerns with:
1. Storage
2. Secret lifecycle (creation, modification, distribution, destruction,
auditing - tracking secrets back in time)
3. Recovery and remediation
What’s an ideal secret management solution
- Security
- Encryption ( at rest/ in transit)
- Fine grained access control
- Good access logs
- Easy to manage and cheap to operate
- Central location to operate on secrets
- No more remembering where secrets are kept
- Easy to integrate, scalable
Version control & orchestrator
tools
SM with Version
Control
- Pros:
1. Easy to get started
2. Encryption at rest + in transit
3. Some compartmentalization
- Cons:
1. No access history
2. Difficult to rotate secret
3. Difficult to rotate encryption key
4. Require key management
5. Require additional protection
against tampering the repository
Bottom line: Only good for small projects
Orchestrator based solutions
Pro:
- No code change necessary, apps access secrets same way
as before.
- No need to provision decryption keys to individual nodes
- Can offload key management to KMS services, e.g. K8S
offers KMS plugin API since v1.10.0 [1]
- Access to the secrets can be audited
- Single source of truth
Cons:
- Trust between components need to be bootstrapped
- Orchestrator lock in, different tools offers different
features.
AWS Sec. Management
> Parameter Store
> Secrets Manager
> KMS
Parameter Store
- Strong Encryption
- Strong Access Control via
IAM
- No secret zero required
- Logging integrated with
CloudTrail possible SIEM
integrations
AWS Parameter Store with IAM role
Using IAM roles with Parameter Store is nice because it does not require
maintaining additional authentication tokens.
{
"Sid": "",
"Effect": "Allow",
"Action": "ssm:GetParameters",
"Resource": [
"arn:aws:ssm:*:*:parameter/SERVICE_N
AME/*",
]
},
● Secrets are namespace separated by `/`
● Grant access to a particular namespace:
"arn:aws:ssm:*:*:parameter/SERVICE_N
AME/db/*"
AWS Parameter Store pro and cons
Pro:
- Secure and scalable with no single point of failure.
- No server to manage
- Secrets are stored under paths, can grant permission to all secrets under a path
- Integrate with many services: EC2, ECS, Lambda, CloudFormation, CodeDeploy etc.
- Integrates with CloudWatch Events and Lambda trigger ( allows an event driven workflow)
- Secret rotation can be implemented as separate lambda functions
Con:
- 10k parameters per account and 4kbyte per secret
- Restricted by AWS KMS limits
- No rotation out of the box
AWS Secrets Manager
- New service
- Encrypted by default
- Support secret rotation via Lambda
- RDS secret rotation is supported by default
- Encryption backed by Keys in KMS
- Promotes programmatic retrieval of secrets
- Access control via IAM
AWS Secrets Manager
Pro:
- Application pull secrets at runtime
- Full automated RDS rotation
- Powerful access control with IAM policies
- Enforce TLS in transit and use KMS keys for encryption at rest
- Much cheaper than managing a Highly Available Hashicorp Vault cluster
Con:
- Application is locked into AWS ecosystem by having a dependency on ASM
- Limited auditing capabilities: CloudTrail only captures secret management events but not data
access events
- Does not offer much extra compare to parameter store
AWS KMS
- Backed by HSM
- More control over the key type
and storage
- No limit on key size and number
of secrets
- Require a lot of work
Recommendation:
Use Parameter store in most scenarios:
- Deploy to AWS
- Integrated services
- Does not mind the 10k secret and 4k size limit
Use Secrets Manager when:
- Working primarily with RDS databases ( credential rotation comes out of the box)
Azure Sec. Management
- Azure Key Vault Secret
Azure key Vault Secret
- Similar to Parameter store, Key Vault is the hosted secret management alternative in Azure. It’s
roughly equivalent to Parameter store + ACM + AKS in AWS
- Key Vault Secret can be encrypted by two types of keys: Software keys and Hardware keys.
- Integrates with many Azure services
- Max 25k bytes per secret
- Warning from Microsoft to keep highly sensitive data out of Key Vault ??!
Azure VM, Function and App Service
Same procedure for Azure VM, Function and App
Services
Associated services
- Azure Key Vault Secret
- Service Principal (SP) and Managed Service
Identity (MSI)
Example with Azure Function is shown below, VM and
App service also work similarly
Using Key Vault Secret with external apps and
services
Create Service Principal:
1. Create a Service Principal via Azure Active Directory > App registrations > New Application
Registration
2. Provide the app name, and an sign-on url to create the application.
3. Note down the application ID and create a new password
1
2
34. Grant permission to Key Vault
const Azure = require('azure'); // require the Azure SDK
const MsRest = require('ms-rest-azure');
MsRest.loginWithServicePrincipalSecret(
'7d5f93e7-b528-490d-925f-d80778538a8a', //app id
'ZVaIui1QaM+5oAT4iZIEv7mRLU+vIecLgTu3M41jly0=', // should be obtained dynamically
'motorolasolutions.microsoft.com', //app domain
(err, credentials) => {
if (err) throw err
let client = new KeyVault.KeyVaultClient(cred)
return client.getSecret('https://xxx.vault.azure.net/', 'secret', '')
})
.then( secret => { /*use secret */} )
Kubernetes Sec. Management
- K8S Secret
- H Vault integration using open source
projects
- H Vault integration using K8S Auth
Method
Kubernetes Overview
Master node is responsible for
coordinating the cluster, usually has the
following components:
- API Server
- Scheduler
- Controller
- ETCD Key-Value DB
Slave nodes runs containers.
Deploy on AWS with Kops on Azure with
AKS, Kubeadm. Minikube
Secret management solutions
Ways to manage and inject secrets to containers:
- Kubernetes Secrets
- Hashicorp Vault + Secret Initialization Container (kubernetes-vault, qubite implementation)
- Storing secrets in a secret object file is safer and more flexible than putting in a pod definition.
Kubernetes Secrets flow
1. Admin creates a secret via kubectl, that
makes create secret request to
the API Server
2. Secrets are written to database
3. Secrets are provisioned to the slave
node that’s running the container
4. Secrets are mounted as volume or
injected to the environment variable of
the target container
* detail here, example
Kubernetes secrets and some gotchas
- Secrets can be provisioned to a container or a namespace, containers under the namespace have
access to the secrets under the same NS.
- Secrets are written to a tempFS which are deleted on pod terminition.
- Secrets are size limited to 1Mb
- Make sure all secrets are created before referencing in containers, otherwise the Pod will hang
because container has trouble mounting secret volume
- Only possible to mount one secret per directory. Mounting a secret will mask the content of the
directory.
Some considerations
- Lock down API Server via access control (RBAC) mechanism from pods and human admins.
- By default, any user who can access the API Server can read all secrets
- More on on “controlling access to API Server”
- Use TLS for all API Server access
- ETCD database:
- Write access to ETCD is equivalent to gaining root on the kubernetes cluster.
- Secrets are, by default, stored as plaintext in etcd. enable encryption on etcd. *how-to
- Manage the symmetric encryption key by leveraging Azure Key Vault, AWS Parameter store. Etc.
- Enforce TLS between etcd cluster and API Server
- Restrict access to etcd
- Lock down access to the slave nodes.
- Anyone with root access on a node can read secret from the API Server by impersonating the kubelet.
- Lock down Kubelet: disable https-anonymous-auth, possible attack scenario
- Unless you specify some flags on Kubelet, it’s default mode of operation is to accept unauthenticated API
requests.
- Version control kubernetes configurations and store them securely ( git-secret or git-crypt for
example)
Kubernetes Secrets summary
- Secret auditing with Kubernetes Audit
- Revocation and rotation can be done by deleting and recreating secrets
- Easy to use and tightly integrated to kubernetes
Hashicorp
Vault
K8S-Vault*
Link to example
Link to project
General Recommendations
Use kubernetes Secret if:
- Secrets does not change often and are used exclusively within kubernetes
Use Vault with K8S Authentication method if:
- secrets need to be used outside of kubernetes containers
Solution comparison chart
AWS Parameter
store
AWS Secret
Manager
AWS KMS Azure Key Vault
Secret
K8S Secret Vault with K8S Vault with K8S
Auth method
Do we need to
provision secret
zero to our
app/cluster in order
to bootstrap trust?
No, IAM task role is
used to obtain a
STS token used to
transparently
authenticate to
parameter store1
No, IAM task role
does the wonder.
No, IAM task role is
used to obtain a
STS token used to
transparently
authenticate to
KMS
No, Managed
Service Identity
(MSI) is used to
obtain a secure
token to
transparently
authenticate to Key
Vault 1
Yes, database
encryption key and
tls certs need to be
provisioned to
setup the K8S
cluster via other
means
K8S Vault
Controller need to
be authenticated
with Vault.
Authentication
need to be setup
between Vault and
K8S.
Is it cloud provider
agnostic?
No, limited
advantage outside
of AWS
No No, limited
advantage outside
of AWS
No, limited
advantage outside
of Azure
Yes, but limited to
app running in
containers
managed by K8S
Yes, but relies on
Hashicorp Vault
Relies on
Hashicorp Vault
Amount of effort to
integrate into
application
Little Little A lot Little Little, K8S cluster
need to be
secured
Moderate, require
Vault and
Controllers
Little
Recommended
scenarios
Best for
deployments in an
AWS only
environment (
lambda, EC2
instances, ECS...)
RDS heavy use
cases
This approach is
generally not
recommended.
Useful for: A large
number/size of
secrets are
required. Scenario
where self
managed secret
and keys are
required.
Best for
deployments in an
Azure only
environment (VM,
Functions, App
Services, VSTS
etc.)
Good for
containers
orchestrated by
Kubernetes.
Secrets are not
used anywhere
else. Require other
means to provision
database
encryption key.
Good for secrets
required to be
shared across
platforms.
Integration with
K8S is possible via
open source
projects.
Best for secrets
required to be
shared across
platforms.
Simple integration
makes this the
best way to
manage secrets
on K8S
Thank you

More Related Content

What's hot

Introducing Vault
Introducing VaultIntroducing Vault
Introducing Vault
Ramit Surana
 
Vault
VaultVault
Vault
dawnlua
 
Vault
VaultVault
CKA Certified Kubernetes Administrator Notes
CKA Certified Kubernetes Administrator Notes CKA Certified Kubernetes Administrator Notes
CKA Certified Kubernetes Administrator Notes
Adnan Rashid
 
Vault 101
Vault 101Vault 101
Vault 101
Hazzim Anaya
 
Kubernetes security
Kubernetes securityKubernetes security
Kubernetes security
Thomas Fricke
 
Secret Management Architectures
Secret Management Architectures Secret Management Architectures
Secret Management Architectures
Stenio Ferreira
 
Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018
Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018
Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018
HashiCorp
 
Designing security & governance via AWS Control Tower & Organizations - SEC30...
Designing security & governance via AWS Control Tower & Organizations - SEC30...Designing security & governance via AWS Control Tower & Organizations - SEC30...
Designing security & governance via AWS Control Tower & Organizations - SEC30...
Amazon Web Services
 
Kubernetes - Security Journey
Kubernetes - Security JourneyKubernetes - Security Journey
Kubernetes - Security Journey
Jerry Jalava
 
Introduction to AWS Secrets Manager
Introduction to AWS Secrets ManagerIntroduction to AWS Secrets Manager
Introduction to AWS Secrets Manager
Amazon Web Services
 
Introduction to Amazon EKS
Introduction to Amazon EKSIntroduction to Amazon EKS
Introduction to Amazon EKS
Amazon Web Services
 
Managing secrets at scale
Managing secrets at scaleManaging secrets at scale
Managing secrets at scale
Alex Schoof
 
Zero trust deck 2020
Zero trust deck 2020Zero trust deck 2020
Zero trust deck 2020
Guido Marchetti
 
Automating AWS security and compliance
Automating AWS security and compliance Automating AWS security and compliance
Automating AWS security and compliance
John Varghese
 
Using Vault to decouple MySQL Secrets
Using Vault to decouple MySQL SecretsUsing Vault to decouple MySQL Secrets
Using Vault to decouple MySQL Secrets
Derek Downey
 
Advanced Container Security
Advanced Container Security Advanced Container Security
Advanced Container Security
Amazon Web Services
 
Data Protection in Transit and at Rest
Data Protection in Transit and at RestData Protection in Transit and at Rest
Data Protection in Transit and at Rest
Amazon Web Services
 
Understanding container security
Understanding container securityUnderstanding container security
Understanding container security
John Kinsella
 
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Amazon Web Services
 

What's hot (20)

Introducing Vault
Introducing VaultIntroducing Vault
Introducing Vault
 
Vault
VaultVault
Vault
 
Vault
VaultVault
Vault
 
CKA Certified Kubernetes Administrator Notes
CKA Certified Kubernetes Administrator Notes CKA Certified Kubernetes Administrator Notes
CKA Certified Kubernetes Administrator Notes
 
Vault 101
Vault 101Vault 101
Vault 101
 
Kubernetes security
Kubernetes securityKubernetes security
Kubernetes security
 
Secret Management Architectures
Secret Management Architectures Secret Management Architectures
Secret Management Architectures
 
Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018
Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018
Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018
 
Designing security & governance via AWS Control Tower & Organizations - SEC30...
Designing security & governance via AWS Control Tower & Organizations - SEC30...Designing security & governance via AWS Control Tower & Organizations - SEC30...
Designing security & governance via AWS Control Tower & Organizations - SEC30...
 
Kubernetes - Security Journey
Kubernetes - Security JourneyKubernetes - Security Journey
Kubernetes - Security Journey
 
Introduction to AWS Secrets Manager
Introduction to AWS Secrets ManagerIntroduction to AWS Secrets Manager
Introduction to AWS Secrets Manager
 
Introduction to Amazon EKS
Introduction to Amazon EKSIntroduction to Amazon EKS
Introduction to Amazon EKS
 
Managing secrets at scale
Managing secrets at scaleManaging secrets at scale
Managing secrets at scale
 
Zero trust deck 2020
Zero trust deck 2020Zero trust deck 2020
Zero trust deck 2020
 
Automating AWS security and compliance
Automating AWS security and compliance Automating AWS security and compliance
Automating AWS security and compliance
 
Using Vault to decouple MySQL Secrets
Using Vault to decouple MySQL SecretsUsing Vault to decouple MySQL Secrets
Using Vault to decouple MySQL Secrets
 
Advanced Container Security
Advanced Container Security Advanced Container Security
Advanced Container Security
 
Data Protection in Transit and at Rest
Data Protection in Transit and at RestData Protection in Transit and at Rest
Data Protection in Transit and at Rest
 
Understanding container security
Understanding container securityUnderstanding container security
Understanding container security
 
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
 

Similar to Overview of secret management solutions and architecture

MySQL Security on AWS Rds
MySQL Security on AWS RdsMySQL Security on AWS Rds
MySQL Security on AWS Rds
Mydbops
 
AWS Security Best Practices (March 2017)
AWS Security Best Practices (March 2017)AWS Security Best Practices (March 2017)
AWS Security Best Practices (March 2017)
Julien SIMON
 
Security best practices on AWS - Pop-up Loft TLV 2017
Security best practices on AWS - Pop-up Loft TLV 2017Security best practices on AWS - Pop-up Loft TLV 2017
Security best practices on AWS - Pop-up Loft TLV 2017
Amazon Web Services
 
Secrets acrosscloudk8s
Secrets acrosscloudk8sSecrets acrosscloudk8s
Secrets acrosscloudk8s
Jhonnatan Gil
 
Vijayanirmala a_community_builders_guidebook_for_securing_your_secrets
Vijayanirmala a_community_builders_guidebook_for_securing_your_secretsVijayanirmala a_community_builders_guidebook_for_securing_your_secrets
Vijayanirmala a_community_builders_guidebook_for_securing_your_secrets
VijayaNirmalaGopal
 
Deep Dive: AWS CloudHSM (Classic)
Deep Dive: AWS CloudHSM (Classic)Deep Dive: AWS CloudHSM (Classic)
Deep Dive: AWS CloudHSM (Classic)
Amazon Web Services
 
Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...
Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...
Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...
Stenio Ferreira
 
Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...
Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...
Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...
AWS Chicago
 
Managing your secrets in a cloud environment
Managing your secrets in a cloud environmentManaging your secrets in a cloud environment
Managing your secrets in a cloud environment
Taswar Bhatti
 
Well-Architected for Security: Advanced Session
Well-Architected for Security: Advanced SessionWell-Architected for Security: Advanced Session
Well-Architected for Security: Advanced Session
Amazon Web Services
 
How to implement data encryption at rest in compliance with enterprise requir...
How to implement data encryption at rest in compliance with enterprise requir...How to implement data encryption at rest in compliance with enterprise requir...
How to implement data encryption at rest in compliance with enterprise requir...
Steffen Mazanek
 
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key VaultAzure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Tom Kerkhove
 
Protecting Your Data in AWS
Protecting Your Data in AWSProtecting Your Data in AWS
Protecting Your Data in AWS
Amazon Web Services
 
Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...
Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...
Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...
Amazon Web Services
 
12 Ways Not to get 'Hacked' your Kubernetes Cluster
12 Ways Not to get 'Hacked' your Kubernetes Cluster12 Ways Not to get 'Hacked' your Kubernetes Cluster
12 Ways Not to get 'Hacked' your Kubernetes Cluster
Suman Chakraborty
 
LASCON 2013 - AWS CLoud HSM
LASCON 2013 - AWS CLoud HSM LASCON 2013 - AWS CLoud HSM
LASCON 2013 - AWS CLoud HSM
Oleg Gryb
 
Sharing secret keys in Docker containers and K8s
Sharing secret keys in Docker containers and K8sSharing secret keys in Docker containers and K8s
Sharing secret keys in Docker containers and K8s
Jose Manuel Ortega Candel
 
Crypto Options in AWS
Crypto Options in AWSCrypto Options in AWS
Crypto Options in AWS
Amazon Web Services
 
Attacking and Defending Kubernetes - Nithin Jois
Attacking and Defending Kubernetes - Nithin JoisAttacking and Defending Kubernetes - Nithin Jois
Attacking and Defending Kubernetes - Nithin Jois
OWASP Hacker Thursday
 
AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...
AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...
AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...
Amazon Web Services
 

Similar to Overview of secret management solutions and architecture (20)

MySQL Security on AWS Rds
MySQL Security on AWS RdsMySQL Security on AWS Rds
MySQL Security on AWS Rds
 
AWS Security Best Practices (March 2017)
AWS Security Best Practices (March 2017)AWS Security Best Practices (March 2017)
AWS Security Best Practices (March 2017)
 
Security best practices on AWS - Pop-up Loft TLV 2017
Security best practices on AWS - Pop-up Loft TLV 2017Security best practices on AWS - Pop-up Loft TLV 2017
Security best practices on AWS - Pop-up Loft TLV 2017
 
Secrets acrosscloudk8s
Secrets acrosscloudk8sSecrets acrosscloudk8s
Secrets acrosscloudk8s
 
Vijayanirmala a_community_builders_guidebook_for_securing_your_secrets
Vijayanirmala a_community_builders_guidebook_for_securing_your_secretsVijayanirmala a_community_builders_guidebook_for_securing_your_secrets
Vijayanirmala a_community_builders_guidebook_for_securing_your_secrets
 
Deep Dive: AWS CloudHSM (Classic)
Deep Dive: AWS CloudHSM (Classic)Deep Dive: AWS CloudHSM (Classic)
Deep Dive: AWS CloudHSM (Classic)
 
Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...
Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...
Hashicorp Chicago HUG - Secure and Automated Workflows in Azure with Vault an...
 
Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...
Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...
Mike Allen's AWS + OWASP talk "AWS secret manager for protecting and rotating...
 
Managing your secrets in a cloud environment
Managing your secrets in a cloud environmentManaging your secrets in a cloud environment
Managing your secrets in a cloud environment
 
Well-Architected for Security: Advanced Session
Well-Architected for Security: Advanced SessionWell-Architected for Security: Advanced Session
Well-Architected for Security: Advanced Session
 
How to implement data encryption at rest in compliance with enterprise requir...
How to implement data encryption at rest in compliance with enterprise requir...How to implement data encryption at rest in compliance with enterprise requir...
How to implement data encryption at rest in compliance with enterprise requir...
 
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key VaultAzure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
 
Protecting Your Data in AWS
Protecting Your Data in AWSProtecting Your Data in AWS
Protecting Your Data in AWS
 
Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...
Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...
Keeping Secrets: Securing Your Data with AWS Cryptography (SEC353-R1) - AWS r...
 
12 Ways Not to get 'Hacked' your Kubernetes Cluster
12 Ways Not to get 'Hacked' your Kubernetes Cluster12 Ways Not to get 'Hacked' your Kubernetes Cluster
12 Ways Not to get 'Hacked' your Kubernetes Cluster
 
LASCON 2013 - AWS CLoud HSM
LASCON 2013 - AWS CLoud HSM LASCON 2013 - AWS CLoud HSM
LASCON 2013 - AWS CLoud HSM
 
Sharing secret keys in Docker containers and K8s
Sharing secret keys in Docker containers and K8sSharing secret keys in Docker containers and K8s
Sharing secret keys in Docker containers and K8s
 
Crypto Options in AWS
Crypto Options in AWSCrypto Options in AWS
Crypto Options in AWS
 
Attacking and Defending Kubernetes - Nithin Jois
Attacking and Defending Kubernetes - Nithin JoisAttacking and Defending Kubernetes - Nithin Jois
Attacking and Defending Kubernetes - Nithin Jois
 
AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...
AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...
AWS re:Invent 2016: Get the Most from AWS KMS: Architecting Applications for ...
 

Recently uploaded

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Globus
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
SOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
Tier1 app
 
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...
Shahin Sheidaei
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
takuyayamamoto1800
 
Advanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowAdvanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should Know
Peter Caitens
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
Jelle | Nordend
 
Lecture 1 Introduction to games development
Lecture 1 Introduction to games developmentLecture 1 Introduction to games development
Lecture 1 Introduction to games development
abdulrafaychaudhry
 
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
informapgpstrackings
 
Visitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.appVisitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.app
NaapbooksPrivateLimi
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Globus
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Hivelance Technology
 
Strategies for Successful Data Migration Tools.pptx
Strategies for Successful Data Migration Tools.pptxStrategies for Successful Data Migration Tools.pptx
Strategies for Successful Data Migration Tools.pptx
varshanayak241
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
wottaspaceseo
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
Globus
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownload
vrstrong314
 
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
Juraj Vysvader
 

Recently uploaded (20)

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
SOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBroker
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
 
Advanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowAdvanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should Know
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
 
Lecture 1 Introduction to games development
Lecture 1 Introduction to games developmentLecture 1 Introduction to games development
Lecture 1 Introduction to games development
 
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
 
Visitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.appVisitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.app
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
 
Strategies for Successful Data Migration Tools.pptx
Strategies for Successful Data Migration Tools.pptxStrategies for Successful Data Migration Tools.pptx
Strategies for Successful Data Migration Tools.pptx
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownload
 
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
 

Overview of secret management solutions and architecture

  • 1. Overview of Secret management Solutions and Architectures Yuechuan Chen
  • 2. About this talk Three goals: 1. Raise awareness of good secret management practices, what it is and why it’s important 2. Identify characteristics of a good solution 3. Overview of solutions on AWS, Azure and Kubernetes Daniel Summerfield’s talk “Turtles all the way down”
  • 3. What’s a secret - A secret is anything that you want to control access to, such as API keys, passwords, certificates, and more. - More services = more authentication = more secrets - People tend to take insecure shortcuts: hardcoding secrets in source code, container images, configuration files. Credentials are shared via email, slack, and shared folders.
  • 4. What is secret management Secret management concerns with: 1. Storage 2. Secret lifecycle (creation, modification, distribution, destruction, auditing - tracking secrets back in time) 3. Recovery and remediation
  • 5. What’s an ideal secret management solution - Security - Encryption ( at rest/ in transit) - Fine grained access control - Good access logs - Easy to manage and cheap to operate - Central location to operate on secrets - No more remembering where secrets are kept - Easy to integrate, scalable
  • 6. Version control & orchestrator tools
  • 7. SM with Version Control - Pros: 1. Easy to get started 2. Encryption at rest + in transit 3. Some compartmentalization - Cons: 1. No access history 2. Difficult to rotate secret 3. Difficult to rotate encryption key 4. Require key management 5. Require additional protection against tampering the repository Bottom line: Only good for small projects
  • 8. Orchestrator based solutions Pro: - No code change necessary, apps access secrets same way as before. - No need to provision decryption keys to individual nodes - Can offload key management to KMS services, e.g. K8S offers KMS plugin API since v1.10.0 [1] - Access to the secrets can be audited - Single source of truth Cons: - Trust between components need to be bootstrapped - Orchestrator lock in, different tools offers different features.
  • 9. AWS Sec. Management > Parameter Store > Secrets Manager > KMS
  • 10. Parameter Store - Strong Encryption - Strong Access Control via IAM - No secret zero required - Logging integrated with CloudTrail possible SIEM integrations
  • 11. AWS Parameter Store with IAM role Using IAM roles with Parameter Store is nice because it does not require maintaining additional authentication tokens. { "Sid": "", "Effect": "Allow", "Action": "ssm:GetParameters", "Resource": [ "arn:aws:ssm:*:*:parameter/SERVICE_N AME/*", ] }, ● Secrets are namespace separated by `/` ● Grant access to a particular namespace: "arn:aws:ssm:*:*:parameter/SERVICE_N AME/db/*"
  • 12. AWS Parameter Store pro and cons Pro: - Secure and scalable with no single point of failure. - No server to manage - Secrets are stored under paths, can grant permission to all secrets under a path - Integrate with many services: EC2, ECS, Lambda, CloudFormation, CodeDeploy etc. - Integrates with CloudWatch Events and Lambda trigger ( allows an event driven workflow) - Secret rotation can be implemented as separate lambda functions Con: - 10k parameters per account and 4kbyte per secret - Restricted by AWS KMS limits - No rotation out of the box
  • 13. AWS Secrets Manager - New service - Encrypted by default - Support secret rotation via Lambda - RDS secret rotation is supported by default - Encryption backed by Keys in KMS - Promotes programmatic retrieval of secrets - Access control via IAM
  • 14. AWS Secrets Manager Pro: - Application pull secrets at runtime - Full automated RDS rotation - Powerful access control with IAM policies - Enforce TLS in transit and use KMS keys for encryption at rest - Much cheaper than managing a Highly Available Hashicorp Vault cluster Con: - Application is locked into AWS ecosystem by having a dependency on ASM - Limited auditing capabilities: CloudTrail only captures secret management events but not data access events - Does not offer much extra compare to parameter store
  • 15. AWS KMS - Backed by HSM - More control over the key type and storage - No limit on key size and number of secrets - Require a lot of work
  • 16. Recommendation: Use Parameter store in most scenarios: - Deploy to AWS - Integrated services - Does not mind the 10k secret and 4k size limit Use Secrets Manager when: - Working primarily with RDS databases ( credential rotation comes out of the box)
  • 17. Azure Sec. Management - Azure Key Vault Secret
  • 18. Azure key Vault Secret - Similar to Parameter store, Key Vault is the hosted secret management alternative in Azure. It’s roughly equivalent to Parameter store + ACM + AKS in AWS - Key Vault Secret can be encrypted by two types of keys: Software keys and Hardware keys. - Integrates with many Azure services - Max 25k bytes per secret - Warning from Microsoft to keep highly sensitive data out of Key Vault ??!
  • 19. Azure VM, Function and App Service Same procedure for Azure VM, Function and App Services Associated services - Azure Key Vault Secret - Service Principal (SP) and Managed Service Identity (MSI) Example with Azure Function is shown below, VM and App service also work similarly
  • 20. Using Key Vault Secret with external apps and services Create Service Principal: 1. Create a Service Principal via Azure Active Directory > App registrations > New Application Registration 2. Provide the app name, and an sign-on url to create the application. 3. Note down the application ID and create a new password 1 2 34. Grant permission to Key Vault
  • 21. const Azure = require('azure'); // require the Azure SDK const MsRest = require('ms-rest-azure'); MsRest.loginWithServicePrincipalSecret( '7d5f93e7-b528-490d-925f-d80778538a8a', //app id 'ZVaIui1QaM+5oAT4iZIEv7mRLU+vIecLgTu3M41jly0=', // should be obtained dynamically 'motorolasolutions.microsoft.com', //app domain (err, credentials) => { if (err) throw err let client = new KeyVault.KeyVaultClient(cred) return client.getSecret('https://xxx.vault.azure.net/', 'secret', '') }) .then( secret => { /*use secret */} )
  • 22. Kubernetes Sec. Management - K8S Secret - H Vault integration using open source projects - H Vault integration using K8S Auth Method
  • 23. Kubernetes Overview Master node is responsible for coordinating the cluster, usually has the following components: - API Server - Scheduler - Controller - ETCD Key-Value DB Slave nodes runs containers. Deploy on AWS with Kops on Azure with AKS, Kubeadm. Minikube
  • 24. Secret management solutions Ways to manage and inject secrets to containers: - Kubernetes Secrets - Hashicorp Vault + Secret Initialization Container (kubernetes-vault, qubite implementation) - Storing secrets in a secret object file is safer and more flexible than putting in a pod definition.
  • 25. Kubernetes Secrets flow 1. Admin creates a secret via kubectl, that makes create secret request to the API Server 2. Secrets are written to database 3. Secrets are provisioned to the slave node that’s running the container 4. Secrets are mounted as volume or injected to the environment variable of the target container * detail here, example
  • 26. Kubernetes secrets and some gotchas - Secrets can be provisioned to a container or a namespace, containers under the namespace have access to the secrets under the same NS. - Secrets are written to a tempFS which are deleted on pod terminition. - Secrets are size limited to 1Mb - Make sure all secrets are created before referencing in containers, otherwise the Pod will hang because container has trouble mounting secret volume - Only possible to mount one secret per directory. Mounting a secret will mask the content of the directory.
  • 27.
  • 28. Some considerations - Lock down API Server via access control (RBAC) mechanism from pods and human admins. - By default, any user who can access the API Server can read all secrets - More on on “controlling access to API Server” - Use TLS for all API Server access - ETCD database: - Write access to ETCD is equivalent to gaining root on the kubernetes cluster. - Secrets are, by default, stored as plaintext in etcd. enable encryption on etcd. *how-to - Manage the symmetric encryption key by leveraging Azure Key Vault, AWS Parameter store. Etc. - Enforce TLS between etcd cluster and API Server - Restrict access to etcd - Lock down access to the slave nodes. - Anyone with root access on a node can read secret from the API Server by impersonating the kubelet. - Lock down Kubelet: disable https-anonymous-auth, possible attack scenario - Unless you specify some flags on Kubelet, it’s default mode of operation is to accept unauthenticated API requests. - Version control kubernetes configurations and store them securely ( git-secret or git-crypt for example)
  • 29. Kubernetes Secrets summary - Secret auditing with Kubernetes Audit - Revocation and rotation can be done by deleting and recreating secrets - Easy to use and tightly integrated to kubernetes
  • 32. General Recommendations Use kubernetes Secret if: - Secrets does not change often and are used exclusively within kubernetes Use Vault with K8S Authentication method if: - secrets need to be used outside of kubernetes containers
  • 33. Solution comparison chart AWS Parameter store AWS Secret Manager AWS KMS Azure Key Vault Secret K8S Secret Vault with K8S Vault with K8S Auth method Do we need to provision secret zero to our app/cluster in order to bootstrap trust? No, IAM task role is used to obtain a STS token used to transparently authenticate to parameter store1 No, IAM task role does the wonder. No, IAM task role is used to obtain a STS token used to transparently authenticate to KMS No, Managed Service Identity (MSI) is used to obtain a secure token to transparently authenticate to Key Vault 1 Yes, database encryption key and tls certs need to be provisioned to setup the K8S cluster via other means K8S Vault Controller need to be authenticated with Vault. Authentication need to be setup between Vault and K8S. Is it cloud provider agnostic? No, limited advantage outside of AWS No No, limited advantage outside of AWS No, limited advantage outside of Azure Yes, but limited to app running in containers managed by K8S Yes, but relies on Hashicorp Vault Relies on Hashicorp Vault Amount of effort to integrate into application Little Little A lot Little Little, K8S cluster need to be secured Moderate, require Vault and Controllers Little Recommended scenarios Best for deployments in an AWS only environment ( lambda, EC2 instances, ECS...) RDS heavy use cases This approach is generally not recommended. Useful for: A large number/size of secrets are required. Scenario where self managed secret and keys are required. Best for deployments in an Azure only environment (VM, Functions, App Services, VSTS etc.) Good for containers orchestrated by Kubernetes. Secrets are not used anywhere else. Require other means to provision database encryption key. Good for secrets required to be shared across platforms. Integration with K8S is possible via open source projects. Best for secrets required to be shared across platforms. Simple integration makes this the best way to manage secrets on K8S