Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dynamic Database Credentials: Security Contingency Planning

1,190 views

Published on

In a dynamic infrastructure world, let's stop pretending credentials aren't public knowledge in an organization and just assume that they have already been leaked, now what?

Published in: Internet
  • Login to see the comments

Dynamic Database Credentials: Security Contingency Planning

  1. 1. Dynamic Database Credentials:
 Security Contingency Planning
  2. 2. Sean Chittenden Engineering, HashiCorp @SeanChittenden sean@hashicorp.com https://keybase.io/seanc
  3. 3. Vault
  4. 4. Vault Manages Secure Information
  5. 5. Risk Assessment
  6. 6. Security is an Operational Concern Loss Aversion byproduct of Security Real security threats often differ from theoretical security. Practical security implies loss aversion. Against what threat? How do you bound exposure?
  7. 7. Types of Security Risk Leaked secret Tightly guarded master keys leak Inability to audit access Inappropriate access to secrets or data leakage Human-scale response to compromise (slow mitigation)
  8. 8. Opportunity Assessment
  9. 9. Benefits of Low Friction Security Low friction Security Systems imply: • Security Automation • Workflow Integration • Secure by Default • Principle of Least Privilege
  10. 10. PostgreSQL Security Friction
  11. 11. Sticking Points Statically Managed Defense in Depth • pg_hba.conf: network, auth, username, database, transport • ROLE: passwords, expirations, connection limits • Data Control Language: GRANT, RLS
  12. 12. Configuration Management To The Rescue… ?
  13. 13. Keys to the Kingdom • Centrally Stored • Eventually Consistent • No Access Control • No Auditing • No Revocation
  14. 14. Meatspace Operational Concerns • How do applications get secrets? • How do humans acquire secrets? • How are secrets updated? • How is a secret revoked?
  15. 15. Why not use PostgreSQL for secrets? • Not designed for dynamic secrets • Typically plaintext storage by default • Limited auditing capabilities • No revocation abilities • Homegrown RLS access controls
  16. 16. What if…
  17. 17. Embraced Requirements • Centralized key management • Grant temporary leases to secured resources • Trust memory (not disk) • Embrace automation (and the necessary APIs) • Assume cyphered data at rest is trustworthy • Decoupled an HA storage backend from the secrets management
  18. 18. Secure like a Bank…
  19. 19. …convenient like a drive thru…
  20. 20. …and built like a tank!
  21. 21. Secrets Lifecycle in Dynamic Environments
  22. 22. Vault Architecture Audit Broker Audit Backend Audit Backend Credential Backend Secret Backend System Backend Path Routing Rollback Mgr. Expiration Mgr. Token Store Policy Store HTTP API Storage Backend Core Barrier
  23. 23. Glossary
  24. 24. Glossary Storage backend The storage backend is responsible for durable storage of encrypted data. There is only one storage backend per Vault cluster. Data is encrypted in transit and at rest with 256bit AES. Examples: in-mem, file, consul, and postgresql
  25. 25. Glossary Secret backend A secret backend is responsible for managing secrets. Some secret backends behave like encrypted key-value stores, while others dynamically generate secrets when queried. There can be multiple secret backends in a Vault cluster. Examples: generic, transit, postgresql
  26. 26. Glossary Secret backend Secret backends can perform almost any function, not just return static data or hand out credentials. PKI – Acts as a full CA, leveraging Vault’s auth Transit – Allows round-tripping data through Vault for "encryption as a service", without ever divulging the key
  27. 27. Glossary Auth backend An auth backend is a credential-based backend that can be used as a way to authenticate humans or machines against Vault. Machine-oriented: approle, tls, tokens
 Operator-oriented:  github, ldap, userpass
  28. 28. Glossary Vault token A vault token is a conceptually similar to a session cookie on a website. Once a user authenticates via an auth backend, Vault returns a token which is to be used for future requests.
  29. 29. Glossary Secret A secret is anything stored or returned by Vault that contains confidential material. A secret is anything that, if acquired by an unauthorized party, would cause political, financial, or appearance harm to an organization.
  30. 30. Glossary Server The Vault server provides an HTTP API which clients interact with and manages the interaction between all the backends, ACL enforcement, and secret lease revocation.
  31. 31. Vault Architecture Audit Broker Audit Backend Audit Backend Credential Backend Secret Backend System Backend Path Routing Rollback Mgr. Expiration Mgr. Token Store Policy Store HTTP API Storage Backend Core Barrier
  32. 32. Initializing Vault
  33. 33. Meatspace Integration Key Shares Master Key Encryption Key
  34. 34. Terminal $ vault init -key-shares=5 -key-threshold=2 -pgp-keys="keybase:seanc,keybase:jefferai,keybase:vishalnayak,alice.asc,bob.asc" ... Key 1: c1c04c03d5f43b6432ea77f3010800... Key 2: 612b611295f255baa2eb702a5e254f... Key 3: ebfd78302325e2631bcc21e11cae00... ...
  35. 35. Obtaining a Vault Token
  36. 36. Hypothetical Model User 1) userpass auth Server PostgreSQL app1 (or pgbouncer)
  37. 37. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN Server PostgreSQL app1 (or pgbouncer)
  38. 38. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy Server PostgreSQL app1 (or pgbouncer)
  39. 39. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy 4) policy-scoped VAULT_TOKEN Server PostgreSQL app1 (or pgbouncer)
  40. 40. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy 4) policy-scoped VAULT_TOKEN Server PostgreSQL app1 (or pgbouncer)
  41. 41. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy 4) policy-scoped VAULT_TOKEN Server 5) VAULT_TOKEN + cred/read/app1 PostgreSQL app1 (or pgbouncer)
  42. 42. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy 4) policy-scoped VAULT_TOKEN Server 5) VAULT_TOKEN + cred/read/app1 PostgreSQL app1 (or pgbouncer) 6) CREATE ROLE…
 GRANT…
  43. 43. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy 4) policy-scoped VAULT_TOKEN Server 5) VAULT_TOKEN + cred/read/app1 7) Dyn PG Creds PostgreSQL app1 (or pgbouncer) 6) CREATE ROLE…
 GRANT…
  44. 44. Hypothetical Model User 1) userpass auth 2) user VAULT_TOKEN 3) VAULT_TOKEN + token-create + policy 4) policy-scoped VAULT_TOKEN Server 5) VAULT_TOKEN + cred/read/app1 7) Dyn PG Creds PostgreSQL 8) PG user/pass app1 (or pgbouncer) 6) CREATE ROLE…
 GRANT…
  45. 45. Terminal pgopen1 % psql -d postgres postgres=# CREATE DATABASE app1; postgres=# c app1 postgres=# CREATE SCHEMA my_app1; postgres=# CREATE TABLE my_app1.myfoo (i INT); postgres=# INSERT INTO my_app1.myfoo VALUES (6*9);
  46. 46. Terminal pgopen1 % vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! pgopen1 % vault path-help postgresql/ | head -8 ## DESCRIPTION The PostgreSQL backend dynamically generates database users. After mounting this backend, configure it using the endpoints within the "config/" path. ## PATHS pgopen1 % vault path-help postgresql/ | grep / the "config/" path. ^config/connection$ ^config/lease$ ^creds/(?P<name>w[w-.]+w)$ ^roles/(?P<name>w[w-.]+w)$ ^roles/?$
  47. 47. Terminal pgopen1 % vault path-help postgresql/config/connection | head -15 Request: config/connection Matching Route: ^config/connection$ Configure the connection string to talk to PostgreSQL. ## PARAMETERS connection_url (string) DB connection string max_idle_connections (int) Maximum number of idle connections to the database; a zero uses the value of max_open_connections and a negative value disables idle connections. If larger than max_open_connections it will be
  48. 48. Terminal pgopen1 % vault write postgresql/config/connection 
 connection_url="postgresql://postgres@127.0.0.1/app1?sslmode=disable" Success! Data written to: postgresql/config/connection pgopen1 % vault write postgresql/config/lease lease=1h lease_max=24h Success! Data written to: postgresql/config/lease
  49. 49. Terminal pgopen1 % vault write postgresql/roles/app1 sql="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'; GRANT USAGE ON SCHEMA my_app1 TO "{{name}}";
 GRANT SELECT ON ALL TABLES IN SCHEMA my_app1 TO "{{name}}"; GRANT SELECT,USAGE ON ALL SEQUENCES IN SCHEMA my_app1 TO "{{name}}";" Success! Data written to: postgresql/roles/app1
  50. 50. Terminal pgopen1 % vault write postgresql/roles/app1 sql="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'; GRANT USAGE ON SCHEMA my_app1 TO "{{name}}";
 GRANT SELECT ON ALL TABLES IN SCHEMA my_app1 TO "{{name}}"; GRANT SELECT,USAGE ON ALL SEQUENCES IN SCHEMA my_app1 TO "{{name}}";" Success! Data written to: postgresql/roles/app1 pgopen1 % vault read postgresql/creds/app1 Key Value lease_id postgresql/creds/app1/6b614cec-ff9c-1fe2-968a-be300f8bf434 lease_duration 3600 lease_renewable true password 773104c8-aa09-88e9-6a33-e86750239dd3 username userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575
  51. 51. Terminal pgopen1 % vault read postgresql/creds/app1 Key Value lease_id postgresql/creds/app1/6b614cec-ff9c-1fe2-968a-be300f8bf434 lease_duration 3600 lease_renewable true password 773104c8-aa09-88e9-6a33-e86750239dd3 username userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575 pgopen1 % psql -q -d app1 app1=# x app1=# dn+ List of schemas -[ RECORD 1 ]-----+------------------------------------------------------------- Name | my_app1 Owner | pgsql Access privileges | pgsql=UC/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=U/pgsql Description |
  52. 52. Terminal pgopen1 % psql -q -d app1 app1=# x app1=# dn+ List of schemas -[ RECORD 1 ]-----+------------------------------------------------------------- Name | my_app1 Owner | pgsql Access privileges | pgsql=UC/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=U/pgsql Description | app1=# dp+ Access privileges -[ RECORD 1 ]-----+------------------------------------------------------------- Schema | my_app1 Name | myfoo Type | table Access privileges | pgsql=arwdDxt/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=r/pgsql Column privileges | Policies |
  53. 53. Terminal app1=# CREATE ROLE my_app1_web_tier; app1=# GRANT USAGE ON SCHEMA my_app1 TO my_app1_web_tier; app1=# GRANT SELECT ON ALL TABLES IN SCHEMA my_app1 TO my_app1_web_tier; app1=# dp+ Access privileges -[ RECORD 1 ]-----+------------------------- Schema | my_app1 Name | myfoo Type | table Access privileges | pgsql=arwdDxt/pgsql + | my_app1_web_tier=r/pgsql Column privileges | Policies |
  54. 54. Terminal pgopen1 % vault write postgresql/roles/app1 sql="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'; ALTER GROUP my_app1_web_tier ADD USER "{{name}}";" Success! Data written to: postgresql/roles/app1 pgopen1 % vault read postgresql/creds/app1 Key Value lease_id postgresql/creds/app1/b4d81724-68e8-b31c-4a81-2f1afde5ace8 lease_duration 3600 lease_renewable true password cf6627de-0ef9-1d2c-4968-b5fa703e6af2 username userpass-sean-0bf4abe8-5193-7d6c-ed7a-766d705dda66
  55. 55. Terminal pgopen1 % psql -q -U userpass-sean-0bf4abe8-5193-7d6c-ed7a-766d705dda66 app1 app1=> SET search_path = my_app1; app1=> dn+ List of schemas -[ RECORD 1 ]-----+------------------------------------------------------------- Name | my_app1 Owner | pgsql Access privileges | pgsql=UC/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=U/pgsql+ | my_app1_web_tier=U/pgsql Description | app1=> dp+ Access privileges -[ RECORD 1 ]-----+------------------------- Schema | my_app1 Name | myfoo Type | table Access privileges | pgsql=arwdDxt/pgsql + | my_app1_web_tier=r/pgsql Column privileges | Policies |
  56. 56. Terminal app1=> SELECT * FROM my_app1.myfoo ; -[ RECORD 1 ] i | 1
  57. 57. Terminal pgopen1 % # Revoke one lease pgopen1 % vault revoke postgresql/creds/app1/b4d81724-68e8-b31c-4a81-2f1afde5ace8 Key revoked with ID 'postgresql/creds/app1/b4d81724-68e8-b31c-4a81-2f1afde5ace8'.
  58. 58. Terminal pgopen1 % # Revoke all of postgresql/ ’s leases pgopen1 % vault revoke -prefix postgresql/creds app1=# du List of roles -[ RECORD 1 ]---------------------------------------------------------- Role name | my_app1_web_tier Attributes | Cannot login Member of | {} -[ RECORD 2 ]---------------------------------------------------------- Role name | pgsql Attributes | Superuser, Create role, Create DB, Replication, Bypass RLS Member of | {}
  59. 59. Terminal pgopen1 % # Create some new creds from a child token pgopen1 % vault token-create Key Value token fc20f365-4250-e840-739e-2e658dba8678 token_accessor 046f5be2-a98d-4608-4398-4c232b3afca2 token_duration 0 token_renewable true token_policies [root] $ env VAULT_TOKEN=fc20f365-4250-e840-739e-2e658dba8678 vault read postgresql/creds/app1 Key Value lease_id postgresql/creds/app1/d6e01c35-ff11-365f-5bce-d78dbe9fd995 lease_duration 2592000 lease_renewable true password 2828a500-f813-b785-5d49-0da565de2938 username token-8921e3d4-9a0a-302c-e6c2-cfb86d1107a4 $ env VAULT_TOKEN=fc20f365-4250-e840-739e-2e658dba8678 vault read postgresql/creds/app1 Key Value lease_id postgresql/creds/app1/7532550e-e5cb-501d-a8e5-3578a71a40c3 lease_duration 2592000 lease_renewable true password 2a9f3169-c49d-f83f-e0d7-b261c4aadcc9 username token-8ef907c8-e619-4b8f-01f4-372dbcac51ca
  60. 60. Terminal postgres=# SELECT usename FROM pg_catalog.pg_user; usename -------------------------------------------- pgsql token-8921e3d4-9a0a-302c-e6c2-cfb86d1107a4 token-8ef907c8-e619-4b8f-01f4-372dbcac51ca (3 rows) pgopen1 % vault token-revoke fc20f365-4250-e840-739e-2e658dba8678 Revocation successful. postgres=# SELECT usename FROM pg_catalog.pg_user; usename ------------------------------------------- pgsql (1 rows)
  61. 61. How 'bout that API? Audit Broker Audit Backend Audit Backend Credential Backend Secret Backend System Backend Path Routing Rollback Mgr. Expiration Mgr. Token Store Policy Store HTTP API Storage Backend Core Barrier
  62. 62. Terminal pgopen1 % curl --request GET 
 —header 'X-Vault-Token: 35173305-8c33-9d5d-633d-8b31fc8127d5' http://127.0.0.1:8200/v1/postgresql/creds/app1 | jq . { "lease_id": "postgresql/creds/app1/798c7488-917f-c12f-6b95-38d06cd9b689", "renewable": true, "lease_duration": 3600, "data": { "password": "eb4d64f5-4bf4-56d7-1ecd-e5f10ed008f4", "username": "userpass-sean-d558fa25-e1ed-10cc-1b00-927ac25ef3d9" }, "warnings": null, "auth": null }
  63. 63. Integrations: consul-template "Process-manager" renders templates out using consul or Vault as data sources, executes or signals child process dynamically. $ consul-template -template "in.ctmpl:out.txt:command" https://github.com/hashicorp/consul-template
  64. 64. Integrations: envconsul envconsul: See consul-template, but limited to environment variables https://github.com/hashicorp/envconsul
  65. 65. Integrations: fabio fabio: HTTP load balancer. Uses consul service discovery, can pull SSL certs from Vault. TCP-SNI support experimental (hellooo pq+TLS!) https://github.com/eBay/fabio/wiki/Certificate-Stores
  66. 66. Questions? @SeanChittenden sean@hashicorp.com https://keybase.io/seanc https://github.com/sean-/pgopen16-dyn-creds

×