SlideShare a Scribd company logo
1 of 38
Download to read offline
The BlackBox project 
Safely storing secrets and credentials in 
Git/Mg (mostly for use by Puppet) 
Tom Limoncelli, SRE, StackExchange.com 
Blog: EverythingSysadmin.com 
My new book! the-cloud-book.com
StackExchange.com 
125+ Q&A Communities 
ServerFault.com 
StackOverflow.com 
(We <3 Puppet!)
What are secrets? 
Anything you don’t want exposed externally. 
● SSL Certificates (the private bits) 
● Passwords 
● API keys
Puppet manages secrets
If you store 
secrets in git, 
you’re gonna 
have a bad 
time.
● Laptops get stolen. 
● Workstations have guest accounts 
● Git server “Circle of Trust” includes: 
○ Everyone with admin access to workstations. 
■ Your desktop support people? 
○ Everyone with admin access to your git server: 
■ Server team, storage team, backup team 
○ Everyone you collaborate with that wants read-only 
access to Puppet manifests.
You have 3 bad options: 
1. Deny git access. (Hurts collaboration) 
2. Permit git access. (Hurts security) 
3. Email individual files. (Hurts… just hurts)
Option 4: Encrypt secret parts 
● If a file contains secrets, encrypt before 
checking into Git. 
● Need to edit a secret? 
○ Decrypt - Edit - Encrypt
What about Puppet master? 
● After “git pull”, decrypt all files. 
○ Automate this as part of CI. 
● Files are unencrypted “at rest”. 
● This does not decrease security: 
○ No worse than what we were doing before. 
○ If you can break into root or puppet on the master, 
you’ve already won.
Easy, right? 
Decrypt: 
Encrypt:
Easy, right? 
Decrypt: 
Encrypt: 
● ...and don’t make any typos when entering the command 
● ...and don't accidentally check in the unencrypted version
Security is 1% technology plus 99% following 
the procedures correctly. 
Any process with more than 1 step probably 
won't be followed consistently most of the time. 
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 
5.0”, Alma Whitten", Usenix Security 1999
Therefore…. we automate 
Introducing: Blackbox 
Scripts for keeping Puppet secrets in git/hg.
User commands: 
Decrypt for editing: 
Encrypt when done:
First time a file is encrypted: 
Enroll a file into the system:
Commands that act on all GPG files: 
Decrypt all files: (for use on puppet master) 
Re-encrypt all files: (after new users added)
Everyone has their own key 
This doesn’t use “symmetric encryption” where 
there is one passphrase to decrypt/encrypt all 
files. 
We maintain a keyring of: 
● Each person that should have access. 
● A key for the Puppet master.
Indoctrinate a new user: 
1. New user does this: 
●● 
● 
(Currently a doc, not a script. Patches gladly accepted.)
Indoctrinate a new user: 
2. Existing admin does this:
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Demo: Edit a file
Code is open source as of 7/2014 
● Entirely written in bash. 
● MIT License. 
● Download it now: 
○ https://github.com/StackExchange/blackbox
In the project’s first 9 months: 
StackExchange/ServerFault has eliminated 
plaintext secrets in our Puppet git repo. 
● 7 SREs+Devs sharing the repo securely. 
● 50+ files now stored encrypted. 
○ Mostly SSL certs and SSH private keys. 
● 40+ individual passwords/API keys: 
○ Everything from SNMP communities, SaaS API 
keys, and many many passwords.
Future plans 
❏ Open source scripts. 
❏ More usability enhancements. 
❏ Better setup documentation.
Join the open source project 
http://github.com/StackExchange/blackbox
Q&A 
URLs from this talk: 
https://github.com/StackExchange/blackbox 
EverythingSysadmin.com
Shameless plug 
Pre-order now! Save 35% 
Ships in September. 
informit.com/TPOSA 
Discount code TPOSA35 
Read “rough cuts” today: 
safaribooksonline.com
Q&A 
URLs from this talk: 
https://github.com/StackExchange/blackbox 
EverythingSysadmin.com 
the-cloud-book.com 
informit.com/TPOSA (code TPOSA35)
Why didn’t we use eyaml? 
● Easier transition. No Puppet code changes 
for big files like SSL certs. 
● Faster. Zero run-time performance impact 
on master. 
● eyaml didn’t exist when we started.
The BlackBox Project: Safely store secrets in Git/Mercurial (originally for Puppet)

More Related Content

What's hot

Plny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesPlny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practices
Dimas Prasetyo
 
Awx user guide
Awx user guideAwx user guide
Awx user guide
mspirko
 

What's hot (20)

What Is Helm
 What Is Helm What Is Helm
What Is Helm
 
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
High Availability and Disaster Recovery in PostgreSQL - EQUNIXHigh Availability and Disaster Recovery in PostgreSQL - EQUNIX
High Availability and Disaster Recovery in PostgreSQL - EQUNIX
 
Azure Database for MySQL PostgreSQLを使って運用の手間を省きませんか?
Azure Database for MySQL PostgreSQLを使って運用の手間を省きませんか?Azure Database for MySQL PostgreSQLを使って運用の手間を省きませんか?
Azure Database for MySQL PostgreSQLを使って運用の手間を省きませんか?
 
Azure API Management 俺的マニュアル
Azure API Management 俺的マニュアルAzure API Management 俺的マニュアル
Azure API Management 俺的マニュアル
 
サポート エンジニアが Azure Networking をじっくりたっぷり語りつくす会
サポート エンジニアが Azure Networking をじっくりたっぷり語りつくす会サポート エンジニアが Azure Networking をじっくりたっぷり語りつくす会
サポート エンジニアが Azure Networking をじっくりたっぷり語りつくす会
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
 
Scouter와 influx db – grafana 연동 가이드
Scouter와 influx db – grafana 연동 가이드Scouter와 influx db – grafana 연동 가이드
Scouter와 influx db – grafana 연동 가이드
 
MySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaMySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & Grafana
 
Plny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesPlny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practices
 
Getting started with Ansible
Getting started with AnsibleGetting started with Ansible
Getting started with Ansible
 
Room 3 - 7 - Nguyễn Như Phúc Huy - Vitastor: a fast and simple Ceph-like bloc...
Room 3 - 7 - Nguyễn Như Phúc Huy - Vitastor: a fast and simple Ceph-like bloc...Room 3 - 7 - Nguyễn Như Phúc Huy - Vitastor: a fast and simple Ceph-like bloc...
Room 3 - 7 - Nguyễn Như Phúc Huy - Vitastor: a fast and simple Ceph-like bloc...
 
AnsibleによるInfrastructure as code入門
AnsibleによるInfrastructure as code入門AnsibleによるInfrastructure as code入門
AnsibleによるInfrastructure as code入門
 
How to be Successful with Scylla
How to be Successful with ScyllaHow to be Successful with Scylla
How to be Successful with Scylla
 
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication CheatsheetPostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication Cheatsheet
 
Ansible - Hands on Training
Ansible - Hands on TrainingAnsible - Hands on Training
Ansible - Hands on Training
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and Containers
 
Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!
 
What you need to know about ceph
What you need to know about cephWhat you need to know about ceph
What you need to know about ceph
 
Metasploit El Kitabı
Metasploit El KitabıMetasploit El Kitabı
Metasploit El Kitabı
 
Awx user guide
Awx user guideAwx user guide
Awx user guide
 

Viewers also liked

Blackbox marketing 2011 v2pdf
Blackbox marketing 2011 v2pdfBlackbox marketing 2011 v2pdf
Blackbox marketing 2011 v2pdf
Meher Khan
 
LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)
LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)
LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)
ASHOKKUMAR RAMAR
 
ECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDED
ECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDEDECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDED
ECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDED
ASHOKKUMAR RAMAR
 
Packet sniffers
Packet sniffersPacket sniffers
Packet sniffers
Wanwen Wen
 
Blackbox security white paper april 27, 2012
Blackbox security white paper april 27, 2012Blackbox security white paper april 27, 2012
Blackbox security white paper april 27, 2012
Grapeshot
 

Viewers also liked (20)

Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
 
Blackbox marketing 2011 v2pdf
Blackbox marketing 2011 v2pdfBlackbox marketing 2011 v2pdf
Blackbox marketing 2011 v2pdf
 
Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers
Stealing the Best Ideas from DevOps: A Guide for Sysadmins without DevelopersStealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers
Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers
 
Black Box for Machine Tools; Based on Open CNC Architecture Control Systems
Black Box for Machine Tools; Based on Open CNC Architecture Control SystemsBlack Box for Machine Tools; Based on Open CNC Architecture Control Systems
Black Box for Machine Tools; Based on Open CNC Architecture Control Systems
 
Blackbox(englisch)
Blackbox(englisch)Blackbox(englisch)
Blackbox(englisch)
 
LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)
LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)
LATEST EMBEDDED PROJECTS TITLES-COCKPIT WHITE BOX AUTOMATION(ECE/EEE/E&I)
 
ECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDED
ECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDEDECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDED
ECE PROJECTS ABSTRACT-COCKPIT WHITE BOX AUTOMATION SYSTEMS-AVIONICS/EMBEDDED
 
Packet sniffers
Packet sniffersPacket sniffers
Packet sniffers
 
Ceh v8 labs module 08 sniffers
Ceh v8 labs module 08 sniffersCeh v8 labs module 08 sniffers
Ceh v8 labs module 08 sniffers
 
"Black Box for a Car" report
"Black Box for a Car" report"Black Box for a Car" report
"Black Box for a Car" report
 
Cellonics-Seminar-Report[1]
Cellonics-Seminar-Report[1]Cellonics-Seminar-Report[1]
Cellonics-Seminar-Report[1]
 
Pia Intership Report
Pia Intership ReportPia Intership Report
Pia Intership Report
 
Blackbox security white paper april 27, 2012
Blackbox security white paper april 27, 2012Blackbox security white paper april 27, 2012
Blackbox security white paper april 27, 2012
 
HAAPS Report
HAAPS ReportHAAPS Report
HAAPS Report
 
Black box
Black boxBlack box
Black box
 
Application of Composite Material in Aerospace Industry
Application of Composite Material in Aerospace IndustryApplication of Composite Material in Aerospace Industry
Application of Composite Material in Aerospace Industry
 
HAAPS Technology
HAAPS TechnologyHAAPS Technology
HAAPS Technology
 
AIRCRAFT PROPULSION SYSTEM seminar report
AIRCRAFT PROPULSION SYSTEM seminar reportAIRCRAFT PROPULSION SYSTEM seminar report
AIRCRAFT PROPULSION SYSTEM seminar report
 
Sniffer for Detecting Lost Mobile
Sniffer for Detecting Lost MobileSniffer for Detecting Lost Mobile
Sniffer for Detecting Lost Mobile
 
Black box of Aircraft
Black box of AircraftBlack box of Aircraft
Black box of Aircraft
 

Similar to The BlackBox Project: Safely store secrets in Git/Mercurial (originally for Puppet)

The Listening: Email Client Backdoor
The Listening: Email Client BackdoorThe Listening: Email Client Backdoor
The Listening: Email Client Backdoor
Michael Scovetta
 
Can Kubernetes Keep a Secret?
Can Kubernetes Keep a Secret?Can Kubernetes Keep a Secret?
Can Kubernetes Keep a Secret?
Soluto
 
A "Box" Full of Tools and Distros
A "Box" Full of Tools and DistrosA "Box" Full of Tools and Distros
A "Box" Full of Tools and Distros
Dario Faggioli
 

Similar to The BlackBox Project: Safely store secrets in Git/Mercurial (originally for Puppet) (20)

The Listening: Email Client Backdoor
The Listening: Email Client BackdoorThe Listening: Email Client Backdoor
The Listening: Email Client Backdoor
 
Pentester++
Pentester++Pentester++
Pentester++
 
Git ongithub
Git ongithubGit ongithub
Git ongithub
 
Linux Security and How Web Browser Sandboxes Really Work (NDC Oslo 2017)
Linux Security  and How Web Browser Sandboxes Really Work (NDC Oslo 2017)Linux Security  and How Web Browser Sandboxes Really Work (NDC Oslo 2017)
Linux Security and How Web Browser Sandboxes Really Work (NDC Oslo 2017)
 
12 tricks to avoid hackers breaks your CI / CD
12 tricks to avoid hackers breaks your  CI / CD12 tricks to avoid hackers breaks your  CI / CD
12 tricks to avoid hackers breaks your CI / CD
 
Us 17-krug-hacking-severless-runtimes
Us 17-krug-hacking-severless-runtimesUs 17-krug-hacking-severless-runtimes
Us 17-krug-hacking-severless-runtimes
 
Can Kubernetes Keep a Secret?
Can Kubernetes Keep a Secret?Can Kubernetes Keep a Secret?
Can Kubernetes Keep a Secret?
 
Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)
 
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
 
Defense in Depth: Securing your new Kubernetes cluster from the challenges th...
Defense in Depth: Securing your new Kubernetes cluster from the challenges th...Defense in Depth: Securing your new Kubernetes cluster from the challenges th...
Defense in Depth: Securing your new Kubernetes cluster from the challenges th...
 
HKG18-203 - Overview of Linaro DRM
HKG18-203 - Overview of Linaro DRMHKG18-203 - Overview of Linaro DRM
HKG18-203 - Overview of Linaro DRM
 
Introduction to Exploitation
Introduction to ExploitationIntroduction to Exploitation
Introduction to Exploitation
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
 
Chapter 4 access control fundamental ii
Chapter 4   access control fundamental iiChapter 4   access control fundamental ii
Chapter 4 access control fundamental ii
 
A "Box" Full of Tools and Distros
A "Box" Full of Tools and DistrosA "Box" Full of Tools and Distros
A "Box" Full of Tools and Distros
 
X page developer
X page developerX page developer
X page developer
 
When the internet bleeded : RootConf 2014
When the internet bleeded : RootConf 2014When the internet bleeded : RootConf 2014
When the internet bleeded : RootConf 2014
 
Git slides
Git slidesGit slides
Git slides
 
Remote file path traversal attacks for fun and profit
Remote file path traversal attacks for fun and profitRemote file path traversal attacks for fun and profit
Remote file path traversal attacks for fun and profit
 
Sonatype DevSecOps Leadership forum 2020
Sonatype DevSecOps Leadership forum 2020Sonatype DevSecOps Leadership forum 2020
Sonatype DevSecOps Leadership forum 2020
 

Recently uploaded

Recently uploaded (20)

Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024
 
Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...
Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...
Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test Automation
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and Prevention
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
Sourcing Success - How to Find a Clothing Manufacturer
Sourcing Success - How to Find a Clothing ManufacturerSourcing Success - How to Find a Clothing Manufacturer
Sourcing Success - How to Find a Clothing Manufacturer
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 

The BlackBox Project: Safely store secrets in Git/Mercurial (originally for Puppet)

  • 1. The BlackBox project Safely storing secrets and credentials in Git/Mg (mostly for use by Puppet) Tom Limoncelli, SRE, StackExchange.com Blog: EverythingSysadmin.com My new book! the-cloud-book.com
  • 2. StackExchange.com 125+ Q&A Communities ServerFault.com StackOverflow.com (We <3 Puppet!)
  • 3. What are secrets? Anything you don’t want exposed externally. ● SSL Certificates (the private bits) ● Passwords ● API keys
  • 5.
  • 6. If you store secrets in git, you’re gonna have a bad time.
  • 7. ● Laptops get stolen. ● Workstations have guest accounts ● Git server “Circle of Trust” includes: ○ Everyone with admin access to workstations. ■ Your desktop support people? ○ Everyone with admin access to your git server: ■ Server team, storage team, backup team ○ Everyone you collaborate with that wants read-only access to Puppet manifests.
  • 8. You have 3 bad options: 1. Deny git access. (Hurts collaboration) 2. Permit git access. (Hurts security) 3. Email individual files. (Hurts… just hurts)
  • 9. Option 4: Encrypt secret parts ● If a file contains secrets, encrypt before checking into Git. ● Need to edit a secret? ○ Decrypt - Edit - Encrypt
  • 10. What about Puppet master? ● After “git pull”, decrypt all files. ○ Automate this as part of CI. ● Files are unencrypted “at rest”. ● This does not decrease security: ○ No worse than what we were doing before. ○ If you can break into root or puppet on the master, you’ve already won.
  • 12. Easy, right? Decrypt: Encrypt: ● ...and don’t make any typos when entering the command ● ...and don't accidentally check in the unencrypted version
  • 13. Security is 1% technology plus 99% following the procedures correctly. Any process with more than 1 step probably won't be followed consistently most of the time. Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0”, Alma Whitten", Usenix Security 1999
  • 14. Therefore…. we automate Introducing: Blackbox Scripts for keeping Puppet secrets in git/hg.
  • 15. User commands: Decrypt for editing: Encrypt when done:
  • 16. First time a file is encrypted: Enroll a file into the system:
  • 17. Commands that act on all GPG files: Decrypt all files: (for use on puppet master) Re-encrypt all files: (after new users added)
  • 18. Everyone has their own key This doesn’t use “symmetric encryption” where there is one passphrase to decrypt/encrypt all files. We maintain a keyring of: ● Each person that should have access. ● A key for the Puppet master.
  • 19. Indoctrinate a new user: 1. New user does this: ●● ● (Currently a doc, not a script. Patches gladly accepted.)
  • 20. Indoctrinate a new user: 2. Existing admin does this:
  • 21. Demo: Edit a file
  • 22. Demo: Edit a file
  • 23. Demo: Edit a file
  • 24. Demo: Edit a file
  • 25. Demo: Edit a file
  • 26. Demo: Edit a file
  • 27. Demo: Edit a file
  • 28.
  • 29.
  • 30. Code is open source as of 7/2014 ● Entirely written in bash. ● MIT License. ● Download it now: ○ https://github.com/StackExchange/blackbox
  • 31. In the project’s first 9 months: StackExchange/ServerFault has eliminated plaintext secrets in our Puppet git repo. ● 7 SREs+Devs sharing the repo securely. ● 50+ files now stored encrypted. ○ Mostly SSL certs and SSH private keys. ● 40+ individual passwords/API keys: ○ Everything from SNMP communities, SaaS API keys, and many many passwords.
  • 32. Future plans ❏ Open source scripts. ❏ More usability enhancements. ❏ Better setup documentation.
  • 33. Join the open source project http://github.com/StackExchange/blackbox
  • 34. Q&A URLs from this talk: https://github.com/StackExchange/blackbox EverythingSysadmin.com
  • 35. Shameless plug Pre-order now! Save 35% Ships in September. informit.com/TPOSA Discount code TPOSA35 Read “rough cuts” today: safaribooksonline.com
  • 36. Q&A URLs from this talk: https://github.com/StackExchange/blackbox EverythingSysadmin.com the-cloud-book.com informit.com/TPOSA (code TPOSA35)
  • 37. Why didn’t we use eyaml? ● Easier transition. No Puppet code changes for big files like SSL certs. ● Faster. Zero run-time performance impact on master. ● eyaml didn’t exist when we started.