You hate certificates? Struggling with the Puppet PKI? You'd prefer to get rid of security just to avoid having such trouble? Or no problems as you enjoy the benefits of Puppet Enterprise, but still curious to find out what's going on behind the scenes? This speech wants to invite you to dive into the beautiful world of X.509 PKI infrastructures. Certificates are like pets. They are cute and lovely as long as you care about them. And grumpy as soon as they get the feeling that you don't. So let's find out what your pets need to feel comfortable. After a jumpstart introduction into the X.509 wilderness we are going to inspect different ways of handling your whole Puppet (and MCollective) certificate lifecycle. Security matters!

1. Puppet CA: Certificates explained
Thomas Gelf - PuppetCamp Düsseldorf 2014

2. Thomas Gelf, nice to meet you!
joined NETWAYS in 2010
formerly more than ten years of...
web (application) development
routing/switching: bank/ISP backbones
ISP: Mail, Hosting, SIP-Carrier, IPv6...

3. Origins
nationality: Italian
mother tongue: German
kind of.
SOUTH TYROLEAN!!!

4. Me and Puppet
first Puppet steps with 0.24
talks, articles, blog posts
trainer, consultant
over-certified

5. Me @ PuppetConf 2014
Had a great time, the conference was awesome!
PuppetConf 2015 will be in Portland - see you there :)

6. NETWAYS

7. Netways and Puppet
German Puppet Labs Training Partner
Trainings
Consulting
Workshops

8. Puppet Trainings
http://www.netways.de/training

9. What this talk is all about
certificates
puppet certificates
REST API
distributed environments
security issues and their consequences
certificate lifecyle

10. WHY SHOULD I CARE?

11. Running Puppet Enterprise?

12. CERTIFICATES

13. Public Key Infrastructure - PKI
everybody has it's own private key
signs or encrypts a message
verification/decryption uses public key
algorithms: RSA, DSA...

14. PKI - Wikipedia

15. X.509
describes how our Puppet PKI works
https:// - you use it every day
ITU-T standard
defines a strict hierarchy
a tree instead of a "web of trust"
X509v3: allows extensions

16. Certificate structure
(distinguished) name
serial number
algorythm
issuer
validity: FROM - TO
...

17. The distinguished name: DN
just a string
often a DNS name
could also be "CA: puppet master"
something you should care about!

18. The revocation list
allows to invalidate certificates
does so based on serial numbers
important if you "loose" certificates

19. Filename extensions
.csr: certificate signing request, Base64
-----BEGIN CERTIFICATE REQUEST-----
...
-----END CERTIFICATE REQUEST-----
.pem: a certificate, Base64
-----BEGIN CERTIFICATE-----
Puppet uses .pem also for private keys:
-----BEGIN RSA PRIVATE KEY-----

20. PUPPET CERTIFICATES

21. Puppet certificates: archeology
Want to see a fresh new Puppet CA? Try it out!
mkdir /tmp/ssltest
puppet master --no-daemonize --verbose
--ssldir /tmp/ssltest
--certname test.example.com

22. Puppet certificates: archeology
A fresh new Puppet CA!

23. Puppet certificates: archeology
ls -l /tmp/ssltest

24. Same thing for the agent
puppet agent --test
--ssldir /tmp/sslagent
--certname test.example.com

25. We all know the basics
puppet cert list
puppet cert list --all
puppet cert sign test.example.com
puppet cert revoke test.example.com
puppet cert clean test.example.com
find ./ -name 'test.example.com*' --delete

26. SSL directories
puppet master --configprint ssldir
puppet agent --configprint ssldir
manual configuration makes sense
think about user permissions
~/.puppet, /var/lib/puppet
master and agent on the same host
passenger VS debug (--no-daemonize)

27. Let's dump a certificate
openssl x509 -in testexample.com.pem -noout -text
puppet cert print test.example.com

28. Custom data in your certificates
https://docs.puppetlabs.com/puppet/latest/reference
/ssl_attributes_extensions.html
/etc/puppet/csr_attributes.yaml
custom attributes in your CSR

29. MCollective

30. Study security guidelines!
Study security guidelines!
Study security guidelines!
STUDY SECURITY GUIDELINES!
puppetlabs.com/mcollective/security-overview

31. Get inspired by existing modules
make sure you understood them
or write your own ones
re-use Puppet certificates
read about trust
and STUDY THE SECURITY GUIDELINES!

32. THE REST API

33. It's a web application!
<VirtualHost *:8140>
SSLEngine on
SSLProtocol ALL -SSLv2 -SSLv3
SSLCipherSuite EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+...
SSLHonorCipherOrder on
SSLCertificateFile $ssldir/certs/$fqdn.pem
SSLCertificateKeyFile $ssldir/private_keys/$fqdn.pem
SSLCertificateChainFile $ssldir/ca/ca_crt.pem
SSLCACertificateFile $ssldir/ca/ca_crt.pem
SSLCARevocationFile $ssldir/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData

34. The Rest API
# http://docs.puppetlabs.com/guides/rest_api.html
https://master:8140/{environment}/{resource}/{key}
available on puppet master
and on VERY ancient agents (listen=true)

35. Puppet REST API URI examples
GET /{environment}/catalog/{node certificate name}
GET /{environment}/file_bucket_file/md5/{checksum}
GET /{environment}/facts/{node certname}

36. Permissions
http://docs.puppetlabs.com/guides/rest_auth_conf.html
# auth.conf
# allow all nodes to store their own reports
path ~ ^/report/([^/]+)$
method save
allow $1

37. SSL-enabled curl example
Use your certificates and discover the API:
curl
--cert /var/lib/puppet/ssl/certs/host.pem
--key /var/lib/puppet/ssl/private_keys/host.pem
--cacert /var/lib/puppet/ssl/ca/ca_crt.pem
-k -H "Accept: yaml"
https://master:8140/production/facts/somehostname

38. DISTRIBUTED SETUP

39. Puppet Advanced* Training

40. Configuration for such a setup
One CA is more than enough:
[master]
ca = false
[agent]
ca_server = ca.example.com
Optionally, still experimental: DNS SRV records

41. Chain of trust
Since 3.2.1 you can use intermediate CAs to delegate trust
# http://docs.puppetlabs.com/puppet/3/reference
# /config_ssl_external_ca.html
[agent]
ssl_client_ca_auth = $certdir/issuer.pem
Tell Apache about your chain:
SSLCertificateChainFile "/path/to/ca_bundle.pem"

42. It could look like this
+------------------------+
| Root self-signed CA |
+------+----------+------+
| |
+----------+ +------------+
| |
v v
+-----------------+ +----------------+
| Master CA | | Agent CA |
+--------+--------+ +--------+-------+
| |
v v
+-----------------+ +----------------+
| Master SSL Cert | | Agent SSL Cert |
+-----------------+ +----------------+

43. SSL Professional?
integrate it in your existing hierarchy
use your own toolchain
ship signed certificates (carefully)

44. SECURITY

45. Puppet and security issues
Read Security Disclosures!
https://puppetlabs.com/security/

46. Thank Heartbleed!
docs.puppetlabs.com/trouble_remediate _heartbleed_overview.html
docs.puppetlabs.com/latest/reference /ssl_regenerate_certificates.html

47. A specific security problem
Very interesting and worth to read: CVE-2011-3872
"In versions prior to 2.6.12 and 2.7.6, the Puppet CA will
improperly insert any certdnsnames values into agent
certificates as well as master certificates. This bug was
introduced in Puppet 0.24.0."
puppet master --configprint certdnsnames
puppet, puppet.example.com

48. Study it!
http://links.puppetlabs.com/cve20113872_remediation
Have a look at the remediation toolkit
And to be on the safe side, check your agent certs:
openssl x509 -in test.example.com.pem -noout -text |
grep 'Subject Alt' -A 1
X509v3 Subject Alternative Name:
DNS:test.example.com, DNS:puppet, DNS:puppet.example.com

49. WARNING
"upgrading" doesn't fix a mess like this
old certificates would remain valid
you have to switch to a new CA...
...and this leads us to the next topic

50. CA LIFECYCLE MANAGEMENT

51. Bad news
Puppet should allow for automatic resigning of SSL certs
http://projects.puppetlabs.com/issues/7272
There is no such thing in Puppet
"...will be available with Puppet Sites"

52. YOU ARE ON YOUR OWN

53. One way of replacing a CA
stop all agents
throw away their certificates
create a new CA with a new name
start your agents
sign their new CSRs

54. CA...
master: rm -rf $(puppet master --configprint ssldir)
agents: rm -rf $(puppet agent --configprint ssldir)
# default ca_name: "Puppet CA: <master certname>"
CERTNAME=$(puppet master --configprint certname)
TS=$(date +%Y-%m-%d)
puppet cert --generate
--ca_name "Puppet CA: $CERTNAME <$TS>" $CERTNAME
--dns_alt_names puppet,puppet.example.com
puppet cert --allow-dns-alt-names sign $CERTNAME

55. You could also
get inspired by the remediation kits
write your own SSH loop
fix it with MCollective (carefully!)
open new feature requests

56. Don't like trouble?
Before generating your CA:
[master]
ca_ttl = 20y
Leave your company in time
NB: expiration > 2038-01-19 == bad idea

57. BTW: WE ARE HIRING ;-)

58. Thank you for your attention!

59. Questions?
class puppetcamp {
package { 'questions':
ensure => answered
}
}