Discussion of secrets management for DevOps and a recipe for success.

1. Recipe for Good Secrets Management
Boston Chef Meetup - August 2014

2. Secrets Management
A strength when done right
Otherwise…

3. First the bad news
◁ CodeSpaces
◁ AWS Credential leaks in Github
◁ HIPAA breach violations
◁ Heartbleed update woes
⊃ Was tricky even for the experts
⊃ Required BOTH openssl library update AND certificate rotation

4. “I don’t care about compliance”
Really? Would your company ever like to get
6-figure checks from large enterprises?

5. Traditional approach
Secrets
Configuration
Management
(CM)
Artifacts
Orchestratio
n
New Machine
“Secrets has always been kind of
a hacky bit; like GPG encrypt a
piece of data and stick it in a
YAML file”
- Anonymous CM Technologist

6. Secrets Management is pretty hard

7. Requirement : separation of duties
One single actor should not be able to do everything.
Use separate roles for:
a) Loading credentials
b) Retrieving and using credentials

8. Requirement : least privilege
Only give each actor as much power
as is necessary to get the job done.

9. Requirement : leak-resistance
Don’t leave secrets lying around on:
◁ unencrypted persistent disks
◁ backups
◁ snapshots

10. Requirement : audit
Record changes:
◁ policies which govern access
◁ each time a secret is changed
◁ each time a secret is fetch and used

11. Requirement : rotation
A secret (e.g. database password, cloud credential)
Should be changed regularly

12. Here’s a recipe for good secrets management

16. can
admin
can
admin
can
update
can
update
can
fetch
can
fetch
“break glass”
account
Environment A : “stage” Environment B : “production”

17. can
admin
can
admin
“break glass”
account
Have an emergency access credential

18. Separate security environments
Environment A : “stage” Environment B : “production”

19. can
update
Design for “robot” actors
can
fetch
can
fetch
Environment A : “stage” Environment B : “production”

20. Minimize human access to production
can
update
can
fetch
Environment B : “production”

21. Rotate credentials with privileged robots
can
update
Environment B : “production”

22. Automatically record system activity
Environment A : “stage” Environment B : “production”

23. Where do Configuration Management
and Orchestration fit in?

24. Provisioning robots into the system
Environment A : “stage” Environment B : “production”

25. Bootstrapping machine identity
Orchestration
Server
Launch Script
or
Console
New
Machine
Robot
Identity Server
● New machine calls to Orchestration Server for
identity
● Orchestration passes a credential (token) to
Robot Identity Server
● Robot Identity assigns robot identity
● Orchestration / CM installs identity on the new
machine

26. A new machine is impotent
until identity is acquired

27. Fetching secrets
password = secrets_manager.secret([node.
chef_environment,
'mysql/server_root_password'].join('/')
mysql_database 'phpapp' do
connection (host: 'localhost', username:
'root', password: password)
action :create
end
http://gettingstartedwithchef.com/first-steps-with-chef.html
● Replace sensitive
attribute data with
secrets from the
secrets manager
● Use the environment
name to separate
secrets into
permissions
namespaces

28. Keep secrets separate from data
General strategy for Linux - install secrets to /dev/shm
template '/dev/shm/mysql.conf' do
…
end
link '/dev/shm/mysql.conf' do
to '/etc/mysql.conf'
end
General strategy for ec2 - install secrets to /mnt
template '/mnt/etc/mysql.conf' do
…
end
link '/mnt/etc/mysql.conf' do
to '/etc/mysql.conf'
end

29. Keeping secrets separate from
data helps to satisfy important
compliance and security standards
such as PCI and HIPAA