This presentation was given at Percona Live Europe 2017 in Dublin. It shows how one can generate dynamic user accounts in mysql using hashicorp vault

1. Issuing temporary
credentials for MySQL using
Hashicorp Vault
Walter Heck - CTO at OlinData
Percona Live Europe 2017

2. Sounds familiar?
“Hey, Jane Doe from that department you’ve never heard of wants to do some
analysis and she ‘needs’ direct access to our production database. Can you set
that up in the next 30 minutes please, I know you have nothing better to do
anyway.
Cheers,
not-your-manager.”

3. Suuuuureeee….

4. What if...
● They could self-service?
● They could request read-only credentials by authenticating against {LDAP,
GitHub, AWS IAM, etc}
● Their credentials would automatically expire in 24 hours

6. ● AWS
○ 3 az’s, 1 region
● HashiCorp Consul
● HashiCorp Vault
○ Secrets backend: consul
○ Auth backend -> github
● MySQL
○ (any flavor/version > 5.0)
Demo Architecture

7. Vault
Vault is a tool for securely accessing secrets. A secret is
anything that you want to tightly control access to, such as
API keys, passwords, certificates, and more. Vault
provides a unified interface to any secret, while providing
tight access control and recording a detailed audit log.
● Vault is Open Source
● Enterprise support available
● The data stored with Vault is encrypted
using 256-bit AES in GCM mode with a
randomly generated nonce.
The key features of Vault are:
● Secure Secret Storage
● Dynamic Secrets
● Data Encryption
● Leasing and Renewal
● Revocation

8. Workflow (vault/mysql admin)
● Setup Consul cluster
● Setup MySQL
● Setup Vault and point it at the Consul cluster
● vault init
● Unseal the vault
● Setup GitHub auth
● Configure database secret backend
● Create one or more roles
Terraform
(Infra as Code)
Manual
(One time operations)

9. Consul
● “A fancy Key/value store”
● Backend for our Vault cluster
○ Officially supported by Hashicorp
root@ip-10-1-103-8:~# consul members
Node Address Status Type Build Protocol DC Segment
ip-10-1-103-104 10.1.103.104:8301 alive server 0.9.3 2 dc1 <all>
ip-10-1-104-233 10.1.104.233:8301 alive server 0.9.3 2 dc1 <all>
ip-10-1-105-147 10.1.105.147:8301 alive server 0.9.3 2 dc1 <all>
ip-10-1-103-8 10.1.103.8:8301 alive client 0.9.3 2 dc1 <default>

10. Vault init
● Initialises the vault
● Seals the vault
● Hands out unseal keys
○ By default 5, need 3 minimum to
unseal (Shamir's Secret Sharing)
○ Don’t lose them, you lose everything
○ Use gpg init for easier distribution
(https://www.vaultproject.io/docs/con
cepts/pgp-gpg-keybase.html)
● Hands out root auth token
root@ip-10-1-103-8:~# vault init
Unseal Key 1: p5Luba1DNJcFSvThese2rj/fJ4iJQMA8bUBG5fuvIsS
Unseal Key 2: 3+M+ajrPVCS96fKeysxUOEfM4JxsT40sosMVHfq1bqA
Unseal Key 3: aW1qaxI2H7u57YAreG26Fuchao0XEaWq/f79dljE3iLA
Unseal Key 4: tNSeeA6WWkAMK5Notjs/gEqf+8KbqQ32ypcfh3oecsfu
Unseal Key 5: nkbtNRGOUxiXPiRealtNBTai9bzVaMmkkbCVRzbaoFn8
Initial Root Token: d57d945b-yoaa-f476-5660-3f6645692555
Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.
Vault does not store the master key. Without at least 3 keys,
your vault will remain permanently sealed.

11. Sealing/Unsealing the vault
● A vault starts sealed
● If there’s ever any reason, a single
vault seal will seal the vault
● Unsealing needs majority 3 out of 5
keys by default
● Sealing requires authentication first
root@ip-10-1-103-8:~# vault unseal
Key (will be hidden):
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Unseal Nonce: d1747cd1-a850-bf79-9175-7ac1aaffdddd
root@ip-10-1-103-8:~# vault unseal
Key (will be hidden):
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
Unseal Nonce: d1747cd1-a850-bf79-9175-7ac1aaffdddd
root@ip-10-1-103-8:~# vault unseal
Key (will be hidden):
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
root@ip-10-1-103-8:~# vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.8.2
Cluster Name: vault-cluster-d2bd39fc
Cluster ID: d6957662-bc13-826e-5a10-1effde41a718
High-Availability Enabled: true
Mode: standby
Leader Cluster Address: https://10.1.104.220:8201
2
1
3
✓
root@ip-10-1-103-8:~# vault seal
Error sealing: Error making API request.
URL: PUT https://127.0.0.1:8200/v1/sys/seal
Code: 500. Errors:
* 1 error occurred:
* missing client token
root@ip-10-1-103-8:~# vault auth
Token (will be hidden):
Successfully authenticated! You are now logged in.
token: d57d945b-b0aa-f476-5660-3f6645692555
token_duration: 0
token_policies: [root]
root@ip-10-1-103-8:~# vault seal
Vault is now sealed.

12. Policies
● Policies provide a declarative way to
grant or forbid access to certain
paths and operations
cat <<EOF | vault policy-write core-policy /dev/stdin
path "sys/*" {
policy = "deny"
}
path "database/creds/readonly" {
policy = "read"
capabilities = ["list", "sudo"]
}
path "database/creds/demodb_admin" {
policy = "read"
capabilities = ["list", "sudo"]
}
path "database/roles/*" {
policy = "read"
capabilities = ["read", "list"]
}
EOF

13. Enable GitHub auth
● One of many auth mechanisms
○ AWS, Gcloud, LDAP, Radius, Okta and
more available
● Doesn’t use oauth but personal
tokens
○ Beware! Losing a personal token is a
security risk
● Access your Personal Access Tokens
in https://github.com/settings/tokens.
○ Generate a new Token that has the scope
read:org.
root@ip-10-1-103-8:~# vault auth-enable github
Successfully enabled 'github' at 'github'!
root@ip-10-1-103-8:~# vault auth -methods
Path Type Accessor Default TTL Max TTL
Replication Behavior Description
github/ github auth_github_db842730 system system
replicated
token/ token auth_token_84532020 system system
replicated token based credentials
root@ip-10-1-103-8:~# vault write auth/github/config
organization=olindata
Success! Data written to: auth/github/config
root@ip-10-1-103-8:~# vault write auth/github/map/teams/core
value=core-policy
Success! Data written to: auth/github/map/teams/core
root@ip-10-1-103-8:~# vault auth -method=github
token=10a8acd3f4ec0b2399146abb0ba6b70211bb6990
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: 58b52458-4e25-6642-f1f1-a24eda37913d
token_duration: 2764799
token_policies: [core-policy default]

14. 1
2
3

15. ● Secrets backends are mounts in
the tree
● The database backend is generic
for a number of database engines
○ Postgres, mongo, oracle, MS
SQL server
● The creation_statements
argument for the role is flexible
and can contain whatever SQL
statement you want
● Also see revocation_statements,
max_open_connections and
others
Enable MySQL secrets backend
root@ip-10-1-103-194:~# vault mount database
Successfully mounted 'database' at 'database'!
root@ip-10-1-103-8:~# vault write database/config/mysql
> plugin_name=mysql-database-plugin
> connection_url="user:mypwd@tcp(perconalive.olindata.local:3306)/"
> allowed_roles="readonly"
The following warnings were returned from the Vault server:
* Read access to this endpoint should be controlled via ACLs as it will
return the connection details as is, including passwords, if any.
root@ip-10-1-103-8:~# vault write database/roles/readonly
> db_name=mysql
> creation_statements="CREATE USER '{{name}}'@'%'
> IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';"
> default_ttl="1h"
> max_ttl="24h"
Success! Data written to: database/roles/readonly

16. Demo 1

17. Workflow (Getting creds)
1. User auths against Vault
2. User asks vault for creds
3. Vault creates records in consul and issues a grant statement to MySQL
4. Vault returns username+password back to user
… user does their thing
5. After X amount of time, vault removes grant from MySQL

18. ● Authentication on command line is not
really useful in prod
○ Use HTTP API instead
GitHub Auth & Get mysql creds
root@ip-10-1-103-8:~# vault auth -method=github
token=a8acd3f4ec0b2399146abb0ba6b70211bb699010
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You
do not
need to "vault auth" again with the token.
token: 76ee8b56-61e5-cbcd-2710-7f7d08668568
token_duration: 2764799
token_policies: [core-policy default]
root@ip-10-1-103-8:~# vault read
database/creds/readonly
Key Value
--- -----
lease_id
database/creds/readonly/1b889400-092d-e634-6444-d1217d
c93690
lease_duration 1h0m0s
lease_renewable true
password A1a-77r41spp13x57vy5
username v-github-wal-readonly-23q9t9vxx2

19. Demo 2

20. What’s next?
● Audit backend
● Vault in HA mode
● Check other integrations
○ AWS, LDAP, Kerberos, SSH, etc.

21. More reading..
Vault on AWS
https://gist.github.com/chris-moreton/f523650c1863f0181e22e2020d0f2268
Consul Cluster ASG on AWS
https://github.com/dwmkerr/terraform-consul-cluster
Vault with MySQL
https://www.percona.com/blog/2016/11/14/using-vault-mysql/