Managing Secrets at Scale Alex Schoof, Fugue - Velocity EU 2015
Secrets are EVERYWHERE
Mishandling secrets can be very bad
We’re really bad at managing secrets
We tend to manage secrets the same way we tell people not to manage software
Secret management should be core infrastructure
Lots of new secret management systems have been released in 2015
There is no one-size-fits-all solution
You have to understand your Threat Model and Trust Boundaries
What are we going to talk about? ● Core principles of modern secret management ● How a modern secret management system wor...
Core Principles
Principle Minimal Privilege The set of principals (actors) able to perform an action on a secret should be the smallest se...
Implications ● We have authentication and authorization on usage of secrets ● We have enforceable access control policy fo...
Principle Secrets Expire Secrets expire for a variety of reasons: built-in expiration in protocols, third party policy, in...
Implications ● Deployment of any given secret is not a one-time event ● We must be able to easily rotate secrets ● A deplo...
Secrets should never be in source control
Corollary: Changing a secret should not require a full deployment
Principle It should be easier to do the right thing than the wrong thing If operations on secrets require human interventi...
Implications ● We need to provide easy-to-use programmatic operations on secrets ○ CRUD operations (Create, Read, Update, ...
Principle The Weakest Link The security of a secret is only as strong as the weakest access control mechanism on the stora...
Implications ● We need strong controls on every secret operation ● Secrets should NEVER be persisted in plain text ● Prope...
Principle Secrets must be highly available For many applications, access to secrets is a requirement for basic functionali...
Implications ● Tier-0 service ● Need good operational and deployment practices ● Be aware of circular dependencies and boo...
Principles of Secret Management ● Minimal Privilege ● Secrets Expire ● It should be easier to do the right thing than the ...
So, we need a system that: ● Supports easy-to-use programmatic CRUD operations on secrets ● Provides an audit trail ● Prot...
Building a Secret Management System
Quick Caveat ● DON’T invent your own crypto ● DON’T implement someone else’s crypto ● Use standard libraries and battle-te...
Secret Service
Secret Service Secret Store
ID NAME VALUE 1 prod.db.password pass123$ 2 dev.db.password qC$qwf#$FQ3f44q#$ 3 prod.some-api.key foobar
ID NAME VALUE 1 prod.db.password Ek (pass123$) 2 dev.db.password Ek (qC$qwf#$FQ3f44q#$) 3 prod.some-api.key Ek (foobar)
How do we store the key?
Secure key storage
Hardware ● Hardware Security Module (HSM) ● Trusted Platform Module ● Smartcard Provide hardware key storage and usually h...
Key Management Services ● Presents an HSM-like interface, typically over the network ● available in public cloud environme...
Or, we could not store the key...
Quorum Systems ● Shamir’s Secret Sharing ○ Make a k-1-degree polynomial with a y-intercept of the secret value ○ Take n po...
● Other secret servers that are up verify that the new server is trusted, and then transfer the secret to it, directly or ...
Secret Service Master Key Storage Secret Store
Key Wrapping ● Secure key storage is expensive ● We might want arbitrary-sized secrets ● Have one (or more) master key(s) ...
ID NAME VALUE DATA_KEY 1 prod.db.password Ek1 (pass123$) Em (k1) 2 dev.db.password Ek2 (qC$..Q3f44q#$) Em (k2) 3 prod.some...
Secret Service Master Key Storage Secret Store
Secret Servicelog Master Key Storage Secret Store
Secure logging ● Append-only ● Signed logs to detect tampering ○ Signed logs are really, REALLY hard to get right ○ Use an...
Secret Servicelog Master Key Storage Secret Store
Secret Servicelog Client Master Key Storage Secret Store
Authenticating the service
Secret Servicelog Client TLS Master Key Storage Secret Store
Authenticating the Client
Client Authentication ● Either API key to sign requests to the secret service, or SSL client certificate ● Created at prov...
Protecting the authentication token ● Agent-based systems ○ Can let you do cool things like FUSE drivers that use POSIX au...
Secret Servicelog Client TLS ✅ ✅ Master Key Storage Secret Store
Basic Operations
CREATE(name, value) ● Access control check ● Log the request ● Wrap value and store in secret store ● Set any initial acce...
READ(name) ● Perform access control check for name ● Log the request ● Unwrap value for name and send it back
UPDATE(name, value) ● Access control check ● Log the request ● Wrap value, overwrite old value
UPDATE(name, value) ● Access control check ● Log the request ● Wrap value, overwrite old value ● Generate new version of v...
In general, secrets should be versioned and immutable
Operations with versions ● CREATE(name, value) = UPDATE(name, value, 1) ● READ(name, version=`latest`) ● UPDATE(name, valu...
ID NAME VERSION VALUE DATA_KEY 1 prod.db.password 1 Ek1 (pass123$) Em (k1) 2 dev.db.password 1 Ek2 (qC$..Q3f44q#$) Em (k2)...
DELETE(name, value, version) ● Access control check ● Log the request ● Distinguish between actual delete and logical dele...
LIST(name?, version?) ● Access control check ● Log the request ● Do a scan or prefix search
PUT_POLICY(name, value, policy) ● This is a privilege escalation avenue ● Need to require higher-level interface ● Makes s...
Secret Servicelog Client TLS ✅ ✅ CREATE READ UPDATE DELETE LIST Admin Interface ✅ TLS PUT_POLICY ✅ Master Key Storage Secr...
Considerations for Interfaces and Libraries ● Make sure you don’t leak secrets outside your trust boundary ○ Logs ○ Core d...
Availability
High-Availability Secret Services ● Run in multiple Availability Zones or Data Centers ● Want secret service to be close t...
Secret management is Tier-0 ● Operational metrics ● Canaries ● Integration tests ● Code reviews ● ACL on commits/deployment
Client-side caching ● Publish/Subscribe can be a useful semantic here ● Helps with network partition from the secret servi...
Secret Servicelog Client TLS ✅ ✅ CREATE READ UPDATE DELETE LIST Admin Interface ✅ ✅ TLS PUT_POLICY Master Key Storage cach...
Some existing tools
Vault ● Client/Server system for secret management ● Pluggable backend for secret storage (disk or Consul) ● Uses secret s...
Keywhiz ● Client/Server secret management system ● Uses relational DB for secret storage ● Derivation key(s) stored in mem...
CredStash ● Lightweight Python utility for managing secrets on AWS ● Uses DynamoDB for secret store ● Uses KMS to hold mas...
Sneaker ● Golang utility for managing secrets on AWS ● Uses S3 for secret store ● Uses KMS to hold master key(s) ● IAM and...
Most of this is very new
You have to understand your Threat Model and Trust Boundaries
Secret management should be core infrastructure
Q & A
Slides: http://bit.ly/1kXSTaa
Further reading ● Threat modeling: https://msdn.microsoft.com/en-us/library/ff648644.aspx ● Remote attestation: https://co...
Upcoming SlideShare
Loading in …5
×

Managing secrets at scale

3,169 views

Published on

Slides from "Managing Secrets at scale" at Velocity EU 2015

Secrets come in many shapes and sizes: database API keys, database passwords, private keys. Distributing and managing these secrets is usually an afterthought. It's hard to get right, and can be very expensive if you get it wrong. In this session, we'll look at the core operations and properties that make up a good secret management system, and how these principals can be implemented

Published in: Software
no profile picture user

  • Be the first to comment

Show More

Managing secrets at scale

  1. 1. Managing Secrets at Scale Alex Schoof, Fugue - Velocity EU 2015
  2. 2. Secrets are EVERYWHERE
  3. 3. Mishandling secrets can be very bad
  4. 4. We’re really bad at managing secrets
  5. 5. We tend to manage secrets the same way we tell people not to manage software
  6. 6. Secret management should be core infrastructure
  7. 7. Lots of new secret management systems have been released in 2015
  8. 8. There is no one-size-fits-all solution
  9. 9. You have to understand your Threat Model and Trust Boundaries
  10. 10. What are we going to talk about? ● Core principles of modern secret management ● How a modern secret management system works ● Some existing open source secret management systems
  11. 11. Core Principles
  12. 12. Principle Minimal Privilege The set of principals (actors) able to perform an action on a secret should be the smallest set required.
  13. 13. Implications ● We have authentication and authorization on usage of secrets ● We have enforceable access control policy for operations on secrets ● We have an audit trail to make sure we are doing it right
  14. 14. Principle Secrets Expire Secrets expire for a variety of reasons: built-in expiration in protocols, third party policy, internal policy, compromise, etc.
  15. 15. Implications ● Deployment of any given secret is not a one-time event ● We must be able to easily rotate secrets ● A deployment artifact should not be the authoritative source of a secret
  16. 16. Secrets should never be in source control
  17. 17. Corollary: Changing a secret should not require a full deployment
  18. 18. Principle It should be easier to do the right thing than the wrong thing If operations on secrets require human intervention, long waits, or complex ceremony, people will do other things, like hardcode secrets
  19. 19. Implications ● We need to provide easy-to-use programmatic operations on secrets ○ CRUD operations (Create, Read, Update, Delete) ○ List ○ Audit
  20. 20. Principle The Weakest Link The security of a secret is only as strong as the weakest access control mechanism on the storage, transmission, or use of that secret
  21. 21. Implications ● We need strong controls on every secret operation ● Secrets should NEVER be persisted in plain text ● Proper root secret storage is very important ○ Operator laptops are now part of your trust boundary for secrets they can access
  22. 22. Principle Secrets must be highly available For many applications, access to secrets is a requirement for basic functionality
  23. 23. Implications ● Tier-0 service ● Need good operational and deployment practices ● Be aware of circular dependencies and bootstrapping problems
  24. 24. Principles of Secret Management ● Minimal Privilege ● Secrets Expire ● It should be easier to do the right thing than the wrong thing ● Weakest Link ● Secrets must be highly available
  25. 25. So, we need a system that: ● Supports easy-to-use programmatic CRUD operations on secrets ● Provides an audit trail ● Protects secrets at every stage of their lifecycle ● Permits access control to be defined by policy ● Is highly available
  26. 26. Building a Secret Management System
  27. 27. Quick Caveat ● DON’T invent your own crypto ● DON’T implement someone else’s crypto ● Use standard libraries and battle-tested algorithms
  28. 28. Secret Service
  29. 29. Secret Service Secret Store
  30. 30. ID NAME VALUE 1 prod.db.password pass123$ 2 dev.db.password qC$qwf#$FQ3f44q#$ 3 prod.some-api.key foobar
  31. 31. ID NAME VALUE 1 prod.db.password Ek (pass123$) 2 dev.db.password Ek (qC$qwf#$FQ3f44q#$) 3 prod.some-api.key Ek (foobar)
  32. 32. How do we store the key?
  33. 33. Secure key storage
  34. 34. Hardware ● Hardware Security Module (HSM) ● Trusted Platform Module ● Smartcard Provide hardware key storage and usually have hardware-accelerated cryptography All use PKCS#11-style interface Can generate or import keys, can use keys, cannot export keys Generally requires a certificate or password to use
  35. 35. Key Management Services ● Presents an HSM-like interface, typically over the network ● available in public cloud environments (i.e. AWS KMS) ● much less expensive than dedicated HSMs Still need some credential to access the KMS
  36. 36. Or, we could not store the key...
  37. 37. Quorum Systems ● Shamir’s Secret Sharing ○ Make a k-1-degree polynomial with a y-intercept of the secret value ○ Take n points from the polynomial ○ Any k of the n points can recreate the original polynomial via interpolation ● Can keep shares on smartcards, in safes, etc. ● Introduces a collusion requirement
  38. 38. ● Other secret servers that are up verify that the new server is trusted, and then transfer the secret to it, directly or via secret sharing ● Need secure boot/remote attestation to do safely ● Completely hands-off, but very hard to do right Peer-to-peer induction of secret servers
  39. 39. Secret Service Master Key Storage Secret Store
  40. 40. Key Wrapping ● Secure key storage is expensive ● We might want arbitrary-sized secrets ● Have one (or more) master key(s) on an HSM, or in a KMS ● For each secret in the secret-store, generate a unique data-encryption key ● encrypt the secret with the data encryption key ● encrypt the data-encryption key with the master key ● store the encrypted (wrapped) data-encryption key next to the encrypted data ● Add message authentication code (MAC) or use AES-GCM to prevent tampering and chosen-ciphertext attacks
  41. 41. ID NAME VALUE DATA_KEY 1 prod.db.password Ek1 (pass123$) Em (k1) 2 dev.db.password Ek2 (qC$..Q3f44q#$) Em (k2) 3 prod.some-api.key Ek3 (foobar) Em (k3)
  42. 42. Secret Service Master Key Storage Secret Store
  43. 43. Secret Servicelog Master Key Storage Secret Store
  44. 44. Secure logging ● Append-only ● Signed logs to detect tampering ○ Signed logs are really, REALLY hard to get right ○ Use an existing secure logging implementation
  45. 45. Secret Servicelog Master Key Storage Secret Store
  46. 46. Secret Servicelog Client Master Key Storage Secret Store
  47. 47. Authenticating the service
  48. 48. Secret Servicelog Client TLS Master Key Storage Secret Store
  49. 49. Authenticating the Client
  50. 50. Client Authentication ● Either API key to sign requests to the secret service, or SSL client certificate ● Created at provision or launch of the instance ○ Can use this token as a root of trust for giving other tokens to the client ● Should try to use some out-of-band mechanism to verify the host ● Examples: ○ IAM Roles for EC2 ○ Chef client certificates ○ Generate CSR at boot and send to CA that can list running instances for verification ○ Keys generated in hardware
  51. 51. Protecting the authentication token ● Agent-based systems ○ Can let you do cool things like FUSE drivers that use POSIX auth ● Control access to certificate or API keys using standard access control mechanisms ○ Filesystem permissions ○ Mandatory Access Control systems (SELinux, AppArmor, etc) ○ Kernel Key Retention Service
  52. 52. Secret Servicelog Client TLS ✅ ✅ Master Key Storage Secret Store
  53. 53. Basic Operations
  54. 54. CREATE(name, value) ● Access control check ● Log the request ● Wrap value and store in secret store ● Set any initial access control policy (i.e. set owner)
  55. 55. READ(name) ● Perform access control check for name ● Log the request ● Unwrap value for name and send it back
  56. 56. UPDATE(name, value) ● Access control check ● Log the request ● Wrap value, overwrite old value
  57. 57. UPDATE(name, value) ● Access control check ● Log the request ● Wrap value, overwrite old value ● Generate new version of value
  58. 58. In general, secrets should be versioned and immutable
  59. 59. Operations with versions ● CREATE(name, value) = UPDATE(name, value, 1) ● READ(name, version=`latest`) ● UPDATE(name, value, version)
  60. 60. ID NAME VERSION VALUE DATA_KEY 1 prod.db.password 1 Ek1 (pass123$) Em (k1) 2 dev.db.password 1 Ek2 (qC$..Q3f44q#$) Em (k2) 3 prod.some-api.key 1 Ek3 (foobar) Em (k3) 4 prod.db.password 2 Ek4 (b3tterPassword!) Em (k4) 5 prod.db.password 3 Ek5 (ce8hq7fq9&^Ae$) Em (k5)
  61. 61. DELETE(name, value, version) ● Access control check ● Log the request ● Distinguish between actual delete and logical delete ○ Will you need to decrypt old data encrypted with a deleted key? ● Revoke asynchronous public keys
  62. 62. LIST(name?, version?) ● Access control check ● Log the request ● Do a scan or prefix search
  63. 63. PUT_POLICY(name, value, policy) ● This is a privilege escalation avenue ● Need to require higher-level interface ● Makes sense to use separate channel or interface ● Policy can range from complex documents to a set of flags ○ IAM policy ○ POSIX-style ACL ○ Booleans for each user
  64. 64. Secret Servicelog Client TLS ✅ ✅ CREATE READ UPDATE DELETE LIST Admin Interface ✅ TLS PUT_POLICY ✅ Master Key Storage Secret Store
  65. 65. Considerations for Interfaces and Libraries ● Make sure you don’t leak secrets outside your trust boundary ○ Logs ○ Core dumps ○ Swap ● Easy-to-use verbs help usability ○ `get_secret(“prod-password”, version=3)`
  66. 66. Availability
  67. 67. High-Availability Secret Services ● Run in multiple Availability Zones or Data Centers ● Want secret service to be close to clients (physically and topologically) ● Data store needs to be highly available ● Need to be able to boot/induct new instances of the secret service ● Spend time hardening your secret servers!
  68. 68. Secret management is Tier-0 ● Operational metrics ● Canaries ● Integration tests ● Code reviews ● ACL on commits/deployment
  69. 69. Client-side caching ● Publish/Subscribe can be a useful semantic here ● Helps with network partition from the secret service ● Can also reduce read-load on the secret service ● Make sure the cache never writes plaintext to disk ○ Or swap, or core dump, etc. ● May need to perform access control check on the client ○ Unless security boundary ~= instance boundary
  70. 70. Secret Servicelog Client TLS ✅ ✅ CREATE READ UPDATE DELETE LIST Admin Interface ✅ ✅ TLS PUT_POLICY Master Key Storage cache Secret Store
  71. 71. Some existing tools
  72. 72. Vault ● Client/Server system for secret management ● Pluggable backend for secret storage (disk or Consul) ● Uses secret splitting (quorum system) to protect master key(s) ● Symmetric tokens or TLS certs for authentication ● HCL (Hashicorp Config Language -- JSON-like language) used for writing authorization policy ● Secret management operations available via CLI or HTTP API ● Nifty “dynamic secret” system https://www.vaultproject.io/
  73. 73. Keywhiz ● Client/Server secret management system ● Uses relational DB for secret storage ● Derivation key(s) stored in memory or an HSM, data keys derived from derivation keys using HKDF (HMAC-based key derivation -- see RFC5869) ● Client TLS certificates used to authenticate for CRUD operations ● LDAP or passwords used to authenticate for admin operations (like ACL management) ● Group-based ACLs ● CLI, API, or sweet FUSE driver https://square.github.io/keywhiz/
  74. 74. CredStash ● Lightweight Python utility for managing secrets on AWS ● Uses DynamoDB for secret store ● Uses KMS to hold master key(s) ● Uses IAM policies for access control ● Uses IAM Roles for EC2 for instance identity ● Secret management operations available via CLI, Python library, or Puppet/Ansible modules https://github.com/fugue/credstash
  75. 75. Sneaker ● Golang utility for managing secrets on AWS ● Uses S3 for secret store ● Uses KMS to hold master key(s) ● IAM and S3 ACLs for access control policy ● Operates on files, rather than values ● Secret management operations available via CLI https://github.com/codahale/sneaker
  76. 76. Most of this is very new
  77. 77. You have to understand your Threat Model and Trust Boundaries
  78. 78. Secret management should be core infrastructure
  79. 79. Q & A
  80. 80. Slides: http://bit.ly/1kXSTaa
  81. 81. Further reading ● Threat modeling: https://msdn.microsoft.com/en-us/library/ff648644.aspx ● Remote attestation: https://courses.cs.washington. edu/courses/csep590/06wi/finalprojects/bare.pdf ● KMS: https://docs.aws.amazon.com/kms/latest/developerguide/overview.html ● Signed syslog: https://tools.ietf.org/html/rfc5848 ● Kernel Key Retention service: https://www.kernel. org/doc/Documentation/security/keys.txt ● Shamir’s Secret Sharing: https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing ● PKCS#11: http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2. 40-os.html

×