Eliminating Secret
Sprawl in the Cloud
Copyright © 2018 HashiCorp
Software Engineer
Vault Ecosystem
Introductions
Becca Petrin
Chris Kent
Agenda
1. What is “Secrets Sprawl”?
2. How does Vault address secrets sprawl?
3. How does Vault work?
4. How do you implement Vault organizationally?
5. What can you do next?
What is “Secrets Sprawl”?
What is A Secret?
A credential: Anything that grants you access to a system, or
authorization within one.
● Username & password
● TLS certificates
● API tokens
● Database credentials
Can’t Data Be A Secret Too?
Yes! Stay tuned, we’ll talk about protecting that too.
What Is Secrets Sprawl?
Questions your team must answer:
● How will I share credentials with new people?
● How will I share credentials with automated services?
Common Ways Secrets are Stored
There are many ways to answer:
● Send secrets like passwords and PGP keys through email
● Put them directly in source code
● Keep them in configuration management tools like Chef, Puppet,
or Ansible
● Store shared credentials somewhere like Amazon S3
● May end up in a VCS
● Often stored in plain text
● Anyone who can log in can see them
What’s Wrong With This?
● Who really has access to all of these things?
● Who has actually accessed these things?
● Are these secrets still only in this place?
● How do we rotate credentials?
How does Vault address
secrets sprawl?
Username: Instance1234
Password: Open Sesame
Username: InstanceABCD
Password: Voila
Username: Instance5678
Password: It’s A Secret
Demo
>> Password:Open Sesame
Demo
>> Password:ws0CxqC3hR/GAECaNbG8AG==
Demo
>> Password:Open Sesame
Shamir’s secret sharing
Key Shares
Storage Engines
● Azure
● Consul (supported)
● Etcd
● Filesystem
● Google Cloud Storage
● Google Spanner
● In-Memory
● MySQL
● PostgreSQL
● Cassandra
● S3
● Zookeeper
● Many more
● Roll your own
Access Control
Authentication
● AppRole
● AWS
● Azure
● Google Cloud
● Kubernetes
● Github
● LDAP
● Okta
● RADIUS
● TLS Certificates
● Tokens
● Username & Password
● Roll your own
Example
path "secret/training_*" {
capabilities = ["create", "read"]
}
base.hcl
Audit Trail
● Audit logs are in a JSON format
● Easily consumable by many log aggregators, including ELK
● Or, if you’re technical, you could directly grep the logs and pipe
the output to jq
Is Our Secret Safe?
● What if an application logs it?
● What if it gets logged in a stacktrace?
● What if someone’s debugging a memory
problem in their app and they see it in a
memory dump?
● What if it gets sent as part of an automated
monitoring report?
The long life of a secret
The longer a secret lives, the less we know.
Short-Lived
Credentials
Secret Engines
● Active Directory
● AWS
● Consul
● Cubbyhole
● Multiple databases
● Google Cloud
● Key/Value
● Identity
● Nomad
● PKI (Certificates)
● RabbitMQ
● SSH
● TOTP
● Transit
● Roll your own
Unique Credentials
Suppose you have 6 web servers...
Unique Credentials
One of them gets compromised and leaks its
credentials.
Password:
Open Sesame
Unique Credentials
You can immediately identify who leaked their
credentials and revoke credentials immediately
without impact to the other 5 servers.
Protecting Data
● Name
● Credit card
● Home address
● Phone number
● Email address
● User password
(in a large-scale system)
● SSN
Becca Petrin
4367 2243 7467 9123, exp. 11/20
19790 Grey Sky Rd., Portland, Oregon
503-425-7235
becca@hashicorp.com
grumpycat
364-89-2635
Protecting Data
● Name
● Credit card
● Home address
● Phone number
● Email address
● User password
(in a large-scale system)
● SSN
as5Zhcfk4D18p+bOywU2zsvA==
VgQCU5Q8xihOFpAmfgnALtqNMofbNHW8VM799fss
XH5+rpK5GT0EGz/vJPwC5+wa0CsMai0qvk9g6BxKq
+l1uUqnN04iKTQeJYcn+JQ==
Cc0WmtxM91/VnMu87WmZ3s/CCkimiOisb7s11QWz2
oOUV4pRnS5lu3/P3XKQSGA==
40TGd8ycs0JzM2GZDbG0ww==
Encryption As A Service
Encrypt/Decrypt
Sign/Verify
Recap
● Vault centrally locates your secrets
● Offers fine-grained access control to individuals and groups
● Provides an audit log
● Creates unique, short-lived secrets
● Encrypts sensitive data
How do I implement Vault?
Starting Vault
$ vault server -dev
Terminal
How and Where Will Vault Be Set Up?
● Run it locally or start a shared dev server
● Who will hold the Shamir keys?
● What secrets will live in Vault?
○ Most valuable first and work your way down?
● Where will the audit logs live and be sent?
● Will I have just one cluster?
Who Will Be Involved with Vault
● An “operator”, observed
● Production hardening guide
Next Steps
Next Steps
● https://www.hashicorp.com/products/vault
● https://www.vaultproject.io
● https://github.com/hashicorp/vault
● Follow-up email with recording and contact info for questions
Q&A
How will clients prove their identity and communicate with Vault?
Clients can use one of our many Authentication Backends to login to vault and get a Vault token. That token is scoped to a set of
policies that authorize it to use parts of the API. For a list of all the Auth Backends checkout
https://www.vaultproject.io/docs/auth/index.html
How is the encryption key protected that vault itself is using? Where is it stored?
The root of trust for vault is the shamir sharing algorithm, so no one person has the full encryption keys. The shamir shares
encrypt a key ring, which then encrypts everything else in Vault’s storage, including the keys used for Transit/EaaS
They are stored in Vault’s encrypted storage.
Vault Enterprise offers AutoUnseals which replace the shamir shares with a HSM, Azure Key Vault, Google Cloud KMS, or Amazon
KMS.
How do you enable aws iam auth method, does it need root credentials to aws as well as roles?
To use the iam auth method you would need to make sure the credentials given to vault has the permissions described here:
https://www.vaultproject.io/docs/auth/aws.html#recommended-vault-iam-policy
Regarding your example with 6 webworkers, how will a web-server authenticate itself against a database? Does vault
add all those credentials to mysql database on the fly?
Yes it does, when the web worker asks Vault for database credentials Vault will create the user on MySQL and return the user/pass
back to the worker. Vault will also revoke these credentials in MySQL once the configured TTL has passed.
Q&A Continued
Does Vault work natively with Kubernetes?
Vault has a built in Authentication Backend that allows applications running in kubernetes to use service account tokens to
authenticate to vault.
Does vault works with Azure Managed Service Identity for authentication?
Yes, the Azure Auth Backend can be used to authenticate a Service Principal on Azure to Vault.
https://www.vaultproject.io/docs/auth/azure.html
We're starting to use Vault at our company and are interested in dynamic secrets. Do you folks have any
recommendations for credential renewal for production services?
Since the secrets like certs and passwords are rotating, how does Vault guarantee NO break in service?
Can I have vault sign and timestamp files, like binaries, if I give it a signing certificate?
The Transit backend can be used to encrypt or sign data that is passed to it. See
https://www.vaultproject.io/docs/secrets/transit/index.html for more information.
How to store keys generated from the initialization process?
Keeping unseal keys safely stored depends greatly on the individual users. Vault provides a way to automatically pgp encrypt the
unseal keys on init. We also see users store them in a non-shared password manager like 1Password. But there are many other
options as well. We recommend revoking the root token that is created as soon as you are done using it to initially configure vault.
Another one can always be created with the unseal keys if needed.

Eliminating Secret Sprawl in the Cloud with HashiCorp Vault - 07.11.2018

  • 1.
    Eliminating Secret Sprawl inthe Cloud Copyright © 2018 HashiCorp
  • 2.
  • 3.
    Agenda 1. What is“Secrets Sprawl”? 2. How does Vault address secrets sprawl? 3. How does Vault work? 4. How do you implement Vault organizationally? 5. What can you do next?
  • 4.
  • 5.
    What is ASecret? A credential: Anything that grants you access to a system, or authorization within one. ● Username & password ● TLS certificates ● API tokens ● Database credentials
  • 6.
    Can’t Data BeA Secret Too? Yes! Stay tuned, we’ll talk about protecting that too.
  • 7.
    What Is SecretsSprawl? Questions your team must answer: ● How will I share credentials with new people? ● How will I share credentials with automated services?
  • 8.
    Common Ways Secretsare Stored There are many ways to answer: ● Send secrets like passwords and PGP keys through email ● Put them directly in source code ● Keep them in configuration management tools like Chef, Puppet, or Ansible ● Store shared credentials somewhere like Amazon S3 ● May end up in a VCS ● Often stored in plain text ● Anyone who can log in can see them
  • 9.
    What’s Wrong WithThis? ● Who really has access to all of these things? ● Who has actually accessed these things? ● Are these secrets still only in this place? ● How do we rotate credentials?
  • 10.
    How does Vaultaddress secrets sprawl?
  • 11.
    Username: Instance1234 Password: OpenSesame Username: InstanceABCD Password: Voila Username: Instance5678 Password: It’s A Secret
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
    Storage Engines ● Azure ●Consul (supported) ● Etcd ● Filesystem ● Google Cloud Storage ● Google Spanner ● In-Memory ● MySQL ● PostgreSQL ● Cassandra ● S3 ● Zookeeper ● Many more ● Roll your own
  • 17.
  • 18.
    Authentication ● AppRole ● AWS ●Azure ● Google Cloud ● Kubernetes ● Github ● LDAP ● Okta ● RADIUS ● TLS Certificates ● Tokens ● Username & Password ● Roll your own
  • 19.
    Example path "secret/training_*" { capabilities= ["create", "read"] } base.hcl
  • 20.
    Audit Trail ● Auditlogs are in a JSON format ● Easily consumable by many log aggregators, including ELK ● Or, if you’re technical, you could directly grep the logs and pipe the output to jq
  • 21.
    Is Our SecretSafe? ● What if an application logs it? ● What if it gets logged in a stacktrace? ● What if someone’s debugging a memory problem in their app and they see it in a memory dump? ● What if it gets sent as part of an automated monitoring report?
  • 22.
    The long lifeof a secret The longer a secret lives, the less we know.
  • 23.
  • 24.
    Secret Engines ● ActiveDirectory ● AWS ● Consul ● Cubbyhole ● Multiple databases ● Google Cloud ● Key/Value ● Identity ● Nomad ● PKI (Certificates) ● RabbitMQ ● SSH ● TOTP ● Transit ● Roll your own
  • 25.
    Unique Credentials Suppose youhave 6 web servers...
  • 26.
    Unique Credentials One ofthem gets compromised and leaks its credentials. Password: Open Sesame
  • 27.
    Unique Credentials You canimmediately identify who leaked their credentials and revoke credentials immediately without impact to the other 5 servers.
  • 28.
    Protecting Data ● Name ●Credit card ● Home address ● Phone number ● Email address ● User password (in a large-scale system) ● SSN Becca Petrin 4367 2243 7467 9123, exp. 11/20 19790 Grey Sky Rd., Portland, Oregon 503-425-7235 becca@hashicorp.com grumpycat 364-89-2635
  • 29.
    Protecting Data ● Name ●Credit card ● Home address ● Phone number ● Email address ● User password (in a large-scale system) ● SSN as5Zhcfk4D18p+bOywU2zsvA== VgQCU5Q8xihOFpAmfgnALtqNMofbNHW8VM799fss XH5+rpK5GT0EGz/vJPwC5+wa0CsMai0qvk9g6BxKq +l1uUqnN04iKTQeJYcn+JQ== Cc0WmtxM91/VnMu87WmZ3s/CCkimiOisb7s11QWz2 oOUV4pRnS5lu3/P3XKQSGA== 40TGd8ycs0JzM2GZDbG0ww==
  • 30.
    Encryption As AService Encrypt/Decrypt Sign/Verify
  • 31.
    Recap ● Vault centrallylocates your secrets ● Offers fine-grained access control to individuals and groups ● Provides an audit log ● Creates unique, short-lived secrets ● Encrypts sensitive data
  • 32.
    How do Iimplement Vault?
  • 33.
    Starting Vault $ vaultserver -dev Terminal
  • 34.
    How and WhereWill Vault Be Set Up? ● Run it locally or start a shared dev server ● Who will hold the Shamir keys? ● What secrets will live in Vault? ○ Most valuable first and work your way down? ● Where will the audit logs live and be sent? ● Will I have just one cluster?
  • 35.
    Who Will BeInvolved with Vault ● An “operator”, observed ● Production hardening guide
  • 36.
  • 37.
    Next Steps ● https://www.hashicorp.com/products/vault ●https://www.vaultproject.io ● https://github.com/hashicorp/vault ● Follow-up email with recording and contact info for questions
  • 38.
    Q&A How will clientsprove their identity and communicate with Vault? Clients can use one of our many Authentication Backends to login to vault and get a Vault token. That token is scoped to a set of policies that authorize it to use parts of the API. For a list of all the Auth Backends checkout https://www.vaultproject.io/docs/auth/index.html How is the encryption key protected that vault itself is using? Where is it stored? The root of trust for vault is the shamir sharing algorithm, so no one person has the full encryption keys. The shamir shares encrypt a key ring, which then encrypts everything else in Vault’s storage, including the keys used for Transit/EaaS They are stored in Vault’s encrypted storage. Vault Enterprise offers AutoUnseals which replace the shamir shares with a HSM, Azure Key Vault, Google Cloud KMS, or Amazon KMS. How do you enable aws iam auth method, does it need root credentials to aws as well as roles? To use the iam auth method you would need to make sure the credentials given to vault has the permissions described here: https://www.vaultproject.io/docs/auth/aws.html#recommended-vault-iam-policy Regarding your example with 6 webworkers, how will a web-server authenticate itself against a database? Does vault add all those credentials to mysql database on the fly? Yes it does, when the web worker asks Vault for database credentials Vault will create the user on MySQL and return the user/pass back to the worker. Vault will also revoke these credentials in MySQL once the configured TTL has passed.
  • 39.
    Q&A Continued Does Vaultwork natively with Kubernetes? Vault has a built in Authentication Backend that allows applications running in kubernetes to use service account tokens to authenticate to vault. Does vault works with Azure Managed Service Identity for authentication? Yes, the Azure Auth Backend can be used to authenticate a Service Principal on Azure to Vault. https://www.vaultproject.io/docs/auth/azure.html We're starting to use Vault at our company and are interested in dynamic secrets. Do you folks have any recommendations for credential renewal for production services? Since the secrets like certs and passwords are rotating, how does Vault guarantee NO break in service? Can I have vault sign and timestamp files, like binaries, if I give it a signing certificate? The Transit backend can be used to encrypt or sign data that is passed to it. See https://www.vaultproject.io/docs/secrets/transit/index.html for more information. How to store keys generated from the initialization process? Keeping unseal keys safely stored depends greatly on the individual users. Vault provides a way to automatically pgp encrypt the unseal keys on init. We also see users store them in a non-shared password manager like 1Password. But there are many other options as well. We recommend revoking the root token that is created as soon as you are done using it to initially configure vault. Another one can always be created with the unseal keys if needed.