The BlackBox project
Safely storing secrets and credentials in Git
for use by Puppet
Tom Limoncelli, SRE, StackExchange.co...
125+ Q&A Communities
ServerFault.com
StackOverflow.com
(We <3 Puppet!)
StackExchange.com
What are secrets?
Anything you don’t want exposed externally.
● SSL Certificates (the private bits)
● Passwords
● API keys
Puppet manages secrets
If you store
secrets in git,
you’re gonna
have a bad
time.
● Laptops get stolen.
● Workstations have guest accounts
● “Circle of Trust” now includes:
○ Everyone with admin access to...
You have 3 bad options:
1. Deny git access. (Hurts collaboration)
2. Permit git access. (Hurts security)
3. Email individu...
Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before
checking into Git.
● Need to edit a secret?
○ ...
What about Puppet master?
● After “git pull”, decrypt all files.
○ Automate this as part of CI.
● Files are unencrypted “a...
Easy, right?
Decrypt:
gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:
gpg --yes --trust-model=always --encrypt
-o s...
Easy, right?
Decrypt:
gpg -q --decrypt -o secret.crt secret.crt.gpg
Encrypt:
gpg --yes --trust-model=always --encrypt
-o s...
Security is 1% technology plus 99% following
the procedures correctly.
Any process with more than 1 step probably
won't be...
Therefore…. we automate
Introducing: Blackbox
Scripts for keeping Puppet secrets in git/hg.
User commands:
Decrypt for editing:
blackbox_edit_start.sh file
Encrypt when done:
blackbox_edit_end.sh file
First time a file is encrypted:
Enroll a file into the system:
blackbox_register_new_file.sh file
Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)
blackbox_postdeploy.sh
Re-encrypt all fi...
Everyone has their own key
This doesn’t use “symmetric encryption” where
there is one passphrase to decrypt/encrypt all
fi...
Indoctrinate a new user:
1. New user does this:
● Create GPG key.
● Add their username@host to blackbox-admins.
txt
● git ...
Indoctrinate a new user:
2. Existing admin does this:
$ gpg --import keyrings/live/pubring.gpg
$ blackbox_update_all_files...
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Code is open source as of TODAY
● Entirely written in bash.
● MIT License.
● Download it now:
○ https://github.com/StackEx...
In the project’s first 9 months:
StackExchange/ServerFault has eliminated
plaintext secrets in our Puppet git repo.
● 7 SR...
Future plans
❏ Open source scripts.
❏ More usability enhancements.
❏ Better setup documentation.
Join the open source project
http://github.com/StackExchange/blackbox
Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com
Shameless plug
Pre-order now! Save 35%
Ships in September.
informit.com/TPOSA
Discount code TPOSA35
Read “rough cuts” toda...
Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com
informit.com/TPOSA (code TPOSA35)
● Easier transition. No Puppet code changes
for big files like SSL certs.
● Faster. Zero run-time performance impact
on ma...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate)...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate)...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate)...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate)...
Upcoming SlideShare
Loading in...5
×

Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange

3,507

Published on

"Safely Storing Secrets and Credentials in Git for use by
Puppet: The BlackBox Project" presented by Thomas A. Limoncelli, Stack Exchange at Puppet Camp NYC 2014

0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,507
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
23
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project (Intermediate) - Thomas A. Limoncelli, Stack Exchange

  1. 1. The BlackBox project Safely storing secrets and credentials in Git for use by Puppet Tom Limoncelli, SRE, StackExchange.com Blog: EverythingSysadmin.com
  2. 2. 125+ Q&A Communities ServerFault.com StackOverflow.com (We <3 Puppet!) StackExchange.com
  3. 3. What are secrets? Anything you don’t want exposed externally. ● SSL Certificates (the private bits) ● Passwords ● API keys
  4. 4. Puppet manages secrets
  5. 5. If you store secrets in git, you’re gonna have a bad time.
  6. 6. ● Laptops get stolen. ● Workstations have guest accounts ● “Circle of Trust” now includes: ○ Everyone with admin access to workstations. ■ Your desktop support people? ○ Everyone with admin access to your git server: ■ Server team, storage team, backup team ○ Everyone you collaborate with that wants read-only access to Puppet manifests.
  7. 7. You have 3 bad options: 1. Deny git access. (Hurts collaboration) 2. Permit git access. (Hurts security) 3. Email individual files. (Hurts… just hurts)
  8. 8. Option 4: Encrypt secret parts ● If a file contains secrets, encrypt before checking into Git. ● Need to edit a secret? ○ Decrypt - Edit - Encrypt
  9. 9. What about Puppet master? ● After “git pull”, decrypt all files. ○ Automate this as part of CI. ● Files are unencrypted “at rest”. ● This does not decrease security: ○ No worse than what we were doing before. ○ If you can break into root or puppet on the master, you’ve already won.
  10. 10. Easy, right? Decrypt: gpg -q --decrypt -o secret.crt secret.crt.gpg Encrypt: gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt
  11. 11. Easy, right? Decrypt: gpg -q --decrypt -o secret.crt secret.crt.gpg Encrypt: gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt ● ...and don’t make any typos when entering the command ● ...and don't accidentally check in the unencrypted version
  12. 12. Security is 1% technology plus 99% following the procedures correctly. Any process with more than 1 step probably won't be followed consistently most of the time. Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0”, Alma Whitten", Usenix Security 1999
  13. 13. Therefore…. we automate Introducing: Blackbox Scripts for keeping Puppet secrets in git/hg.
  14. 14. User commands: Decrypt for editing: blackbox_edit_start.sh file Encrypt when done: blackbox_edit_end.sh file
  15. 15. First time a file is encrypted: Enroll a file into the system: blackbox_register_new_file.sh file
  16. 16. Commands that act on all GPG files: Decrypt all files: (for use on puppet master) blackbox_postdeploy.sh Re-encrypt all files: (after new users added) blackbox_update_all_files.sh
  17. 17. Everyone has their own key This doesn’t use “symmetric encryption” where there is one passphrase to decrypt/encrypt all files. We maintain a keyring of: ● Each person that should have access. ● A key for the Puppet master.
  18. 18. Indoctrinate a new user: 1. New user does this: ● Create GPG key. ● Add their username@host to blackbox-admins. txt ● git commit -a (Currently a doc, not a script. Patches gladly accepted.)
  19. 19. Indoctrinate a new user: 2. Existing admin does this: $ gpg --import keyrings/live/pubring.gpg $ blackbox_update_all_files.sh $ git commit -a
  20. 20. Demo: Edit a file
  21. 21. Demo: Edit a file
  22. 22. Demo: Edit a file
  23. 23. Demo: Edit a file
  24. 24. Demo: Edit a file
  25. 25. Demo: Edit a file
  26. 26. Demo: Edit a file
  27. 27. Code is open source as of TODAY ● Entirely written in bash. ● MIT License. ● Download it now: ○ https://github.com/StackExchange/blackbox
  28. 28. In the project’s first 9 months: StackExchange/ServerFault has eliminated plaintext secrets in our Puppet git repo. ● 7 SREs+Devs sharing the repo securely. ● 50+ files now stored encrypted. ○ Mostly SSL certs and SSH private keys. ● 40+ individual passwords/API keys: ○ Everything from SNMP communities, SaaS API keys, and many many passwords.
  29. 29. Future plans ❏ Open source scripts. ❏ More usability enhancements. ❏ Better setup documentation.
  30. 30. Join the open source project http://github.com/StackExchange/blackbox
  31. 31. Q&A URLs from this talk: https://github.com/StackExchange/blackbox EverythingSysadmin.com
  32. 32. Shameless plug Pre-order now! Save 35% Ships in September. informit.com/TPOSA Discount code TPOSA35 Read “rough cuts” today: safaribooksonline.com
  33. 33. Q&A URLs from this talk: https://github.com/StackExchange/blackbox EverythingSysadmin.com informit.com/TPOSA (code TPOSA35)
  34. 34. ● Easier transition. No Puppet code changes for big files like SSL certs. ● Faster. Zero run-time performance impact on master. ● eyaml didn’t exist when we started. Why didn’t we use eyaml?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×