Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PuppetConf 2016: Nice and Secure: Good OpSec Hygiene With Puppet! – Peter Souter, Puppet


Published on

Here are the slides from Peter Souter's PuppetConf 2016 presentation called Nice and Secure: Good OpSec Hygiene With Puppet!. Watch the videos at

Published in: Technology
  • Be the first to comment

PuppetConf 2016: Nice and Secure: Good OpSec Hygiene With Puppet! – Peter Souter, Puppet

  1. 1. Nice and Secure: Good OpSec Hygiene with Puppet! Peter Souter Professional Services Engineer | Puppet @petersouter
  2. 2. @petersouter 2 Who am I? @petersouter Professional Services Engineer 5 years using Puppet 2 years @ Puppet Inc Help customers deploy Puppet Enterprise Teach Puppet classes petems IRC/Slack/GitHub
  3. 3. @petersouter Warning: I speak quickly And I have a different accent... 3
  4. 4. @petersouter My feelings on Q&A ● Tweet me @petersouter ● Come up after this talk ● Meet me in the hallway 4 When will this QA be over so I can leave? We’ve got lots to cover - No Q&A!
  5. 5. @petersouter So, why are we here? (This room specifically, listening to this talk...) 5
  6. 6. @petersouter 6 Every time someone uses this picture, Pete Cheslock gets his wings!
  7. 7. @petersouter Show of hands in the room Let’s take the temperature of security here 7
  8. 8. @petersouter Why is Puppet good for security? Infrastructure as code RBAC Auditing Enforcement 8
  9. 9. @petersouter Don’t let Puppet be the attack vector! aka. How do we make sure we’re not pushing the problem elsewhere? 9
  10. 10. @petersouter What is OPSEC? Before we talk about something we should define it 10
  11. 11. @petersouter “Operations Security, or OPSEC, is the process by which we protect unclassified information that can be used against us. OPSEC challenges us to look at ourselves through the eyes of an adversary (individuals, groups, countries, organizations). Essentially, anyone who can harm people, resources, or mission is an adversary.” 11 Department of Defense Education Activity
  12. 12. @petersouter 12 ● Keeping your code clear of sensitive information ● Approaches to secrets management with the Puppet toolchain ● Making sure security is part of your workflow, rather than an afterthought What are we going to cover?
  13. 13. @petersouter Let’s start with secrets... 13 We’ve all got them...
  14. 14. @petersouter 14 What are secrets in IT? Radioactive Consequences are dire from a leak Examples Passwords, API Keys, SSH Keys, SSL Certs... Small A few kb at most Required The infrastructure won't work without them!
  15. 15. @petersouter Easiest to hardest ● Avoid exposing secrets in logs ● Remove data from code and into the data layer (hiera) ● Encryption 15 How do we avoid exposing secrets in Puppet?
  16. 16. @petersouter Don’t expose secrets in logs Keep your secrets hidden 16
  17. 17. @petersouter show_diff The first place for leaks 17
  18. 18. @petersouter 18 root@homebox:~# puppet agent --show_diff Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds Notice: /Stage[main]/Main/File[/etc/sensitive]/content: --- /etc/sensitive 2016-08-14 23:01:37.036863915 +0100 +++ /tmp/puppet-file20160814-24654-ak1ywd 2016-08-14 23:01:56.852882307 +0100 @@ -1 +1 @@ -Not Secret No newline at end of file +SECRET-CONTENT No newline at end of file Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed '{md5}2ab96390c7dbe3439de74d0c9b0b1767' to '{md5}44c7be48226ebad5dca8216674cad62b' Notice: Applied catalog in 0.20 seconds How it looks...
  19. 19. @petersouter Anywhere reports go: ● syslog ● interactive terminal output ● PE Console ● ENC ● report processors 19 Where does the information from show_diff go?
  20. 20. @petersouter 20 file { ‘/etc/secrets.txt’: ensure => 'file', owner => 'root', mode => '0600', content => 'hunter2', show_diff => false, } Setting show_diff to false at the resource level
  21. 21. @petersouter 21 An example from a Supported Module: mysql file { "${::root_home}/.my.cnf": content => template('mysql/my.cnf.pass.erb'), owner => 'root', mode => '0600', } # show_diff was added with puppet 3.0 if versioncmp($::puppetversion, '3.0') >= 0 { File["${::root_home}/.my.cnf"] { show_diff => false } }
  22. 22. @petersouter Setting show_diff to false at resource scope show_diff = false 22 root@homebox:~# puppet apply secret.pp Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed '{md5}d3b07384d113edec49eaa6238ad5ff00' to '{md5}44c7be48226ebad5dca8216674cad62b' Notice: Applied catalog in 0.19 seconds
  23. 23. @petersouter There’s a balance... Hiding diffs reduces visibility of change... 23
  24. 24. @petersouter inifile module to hide changes Allows you to only hide the sensitive fields 24
  25. 25. @petersouter 25 ini_file now has show_diff from the 1.5.0 release
  26. 26. @petersouter Setting show_diff on individual sensitive fields show_diff = false 26 ini_setting { 'ACME App Timezone': section => 'TimeDate', setting => 'TimeZone', value => $acme_app_time_zone, } ini_setting { 'ACME App Password: section => 'Settings', setting => 'Password', value => $acme_app_password, show_diff => false, }
  27. 27. @petersouter Sensitive type New for the Puppet 4.6+ release 27
  28. 28. @petersouter 28 file { '/etc/sensitive': ensure => 'present', owner => 'root', group => 'root', content => Sensitive('hunter2'), } root@homebox:~# puppet apply secret.pp Notice: /Stage[main]/Main/File[/etc/sensitive]/ensure: changed [redacted] to [redacted] Notice: Applied catalog in 0.18 seconds Ability to redact strings with the new Sensitive Type
  29. 29. @petersouter 29 $secret = Sensitive(‘Unwrapped’) $unwrapped = $secret.unwrap |$sensitive| { $sensitive } notice("Unwrapped: ${unwrapped}") $secret.unwrap |$sensitive| { notice("Lambda: ${sensitive}") } Unwrapping the secrets
  30. 30. @petersouter 30 You can use a dedicated redacted resource Still on < 4.6? 30
  31. 31. @petersouter 31 newproperty(:value, :array_matching => :all) do desc 'The value of the setting to be defined.' munge do |value| value = value.to_s.strip value.capitalize! if value =~ /^(true|false)$/i value end newvalues(/^[S ]*$/) def is_to_s( currentvalue ) if resource.secret? return '[old secret redacted]' else return currentvalue end end def should_to_s( newvalue ) if resource.secret? return '[new secret redacted]' else return newvalue end end
  32. 32. @petersouter 3232 Encrypt secrets on a node by node basis Try binford2k-node_encrypt 32
  33. 33. @petersouter 33 ● Master encrypts secrets for each node using their own certificate ● Secret can only be decrypted with the node's private key ● Uses built-in Puppet CA, so the base case is zero-config node_encrypt::file {'/etc/company_app/credentials': ensure => file, owner => 'root', content => 'hunter2', # transparently encrypted } How does it work?
  34. 34. @petersouter node_encrypt 34 $ puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for master.puppetlabs.vm Info: Applying configuration version '1450109738' Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/ensure: created Notice: Applied catalog in 9.33 seconds $ echo blah > /tmp/foo $ puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for master.puppetlabs.vm Info: Applying configuration version '1450109821' Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/content: content changed '<<encrypted>>' to '<<encrypted>>' Notice: Applied catalog in 7.61 seconds
  35. 35. @petersouter So data is no longer exposed in logs But the data is still visible in the code! 35
  36. 36. @petersouter Remove data from code Especially organisation specific data 36
  37. 37. @petersouter Bad! Don’t do this... 37
  38. 38. @petersouter 38 class example_company_app { if $::fqdn == '' { class {'company_app': ensure => 'present', password => 'hunter2', ssl_enable => true, } } else { class {'company_app': ensure => 'present', password => 'example123', ssl_enable => false, } } }
  39. 39. @petersouter Good! Do this... 39
  40. 40. @petersouter 40 class profile::example_company_app { $app_password = hiera('profile::example_company_app::password') $app_ssl_enable_password = hiera('profile::example_company_app::password') class {'company_app': ensure => 'present', password => $app_password, ssl_enable => $ssl_enable, } }
  41. 41. @petersouter Roles and profiles help a lot Abstracting implementation specifics away 41
  42. 42. @petersouter Allows you to set organisational defaults in your roles and profiles 42 ● Keep organisational specific data in hiera ● Move organisational specific setup into role and profile wrappers Advantage: Not only more secure, cleaner code that’s more reusable!
  43. 43. @petersouter Default parameters are important! Abstracting implementation specifics away 43
  44. 44. @petersouter Storytime: Openstack The perils of bad defaults... 44
  45. 45. @petersouter 45 The open source OpenStack project infrastructure Fully public Puppet
  46. 46. @petersouter The more abstracted your control-repo, the less chance of leaking or compromising of secrets... 46
  47. 47. @petersouter Another gotcha: hierarchy lookups Aka. Why trusted facts are good! 47
  48. 48. @petersouter Take a look at this hiera config... 48 # hiera.yaml --- :hierarchy: - "node/%{fqdn}" - "common" :backends: - yaml :datadir: '/etc/puppet/environments/%{environment}/hieradata'
  49. 49. @petersouter Facts are spoofable! 49 [root@testbox]# facter fqdn pe-201620-master.puppetdebug.vlan [root@testbox]# facter fqdn Facts are spoofable
  50. 50. @petersouter Trusted facts got your back! 50 Locked in from the certificate request
  51. 51. @petersouter Trusted facts are stamped on Node creation 51 Trusted facts are stamped on Node creation
  52. 52. @petersouter We have a bunch of OIDs for this also... 52
  53. 53. @petersouter New, better hiera hierarchy... 53 # hiera.yaml --- :hierarchy: - "node/%{trusted.certname}" - "common" :backends: - yaml :datadir: '/etc/puppet/environments/%{environment}/hieradata'
  54. 54. @petersouter Theoretically, you should be able to release most of the code you write publically without any sort of security issues 54
  55. 55. @petersouter 55 This is actually a tenet of 12 Factor Apps... Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not. A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials. Note that this definition of “config” does not include internal application config, such as config/routes.rb in Rails, or how code modules are connected in Spring. This type of config does not vary between deploys, and so is best done in the code.
  56. 56. @petersouter Example: GDS Government Digital Service, UK 56
  57. 57. @petersouter 57 Meeting the Digital Service Standard To meet point 8 (understand security and privacy issues) you must: ● Make all new source code open and reusable ● Publish code under an appropriate licence ● Explain your reasoning for any code you haven’t made open You’ll have to explain how you did this at your service assessments.
  58. 58. @petersouter 58 Meeting the Digital Service Standard When GOV.UK was first set up we were unable to publish our Puppet repository because our code and secrets were tied together. This goes against patterns like the 12-factor app which “requires strict separation of config from code” This wasn’t true for our Puppet repository, but we gradually moved our credentials into a separate repository (rotating them as we did so). “A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.”
  59. 59. @petersouter 59 $ strings modules/**/*.pp | tr ' ' 'n' | sort -n | uniq | view - Check code for unique strings that look secret-y Note: Requires zsh for the strings function!
  60. 60. @petersouter It’s not just Puppet code! Git commits can sustain sensitive data! 60
  61. 61. @petersouter 61 $ git commit -a -m "Changed the password to password1"
  62. 62. @petersouter $ while read line; do echo $line; git --no-pager log -p -S $line; done < puppet_search 62 Manually searching through git commits for sensitive information...
  63. 63. @petersouter Opening GOV.UK’s Puppet Repository -gov-uks-puppet-repository/ Git Repo Want to know more? 63
  64. 64. @petersouter Your data is now separated. Hooray! But it’s still plaintext in Hiera. Boo! :-( 64
  65. 65. @petersouter Encryption Told you we’d come back to it! 65
  66. 66. @petersouter Bad! Don’t do this! 66
  67. 67. @petersouter 67
  68. 68. @petersouter Good! Do this! 68
  69. 69. @petersouter 69 Preso title goes here. To update, go to File > Page Setup > Header/Footer, paste title, Apply All
  70. 70. @petersouter 70 Hiera eyaml
  71. 71. @petersouter hiera-eyaml is probably the best method for internal data encryption with Puppet It’s widely used, and has a number of plugins 71
  72. 72. @petersouter eyaml plugins 72 ● ● ● ● ●
  73. 73. @petersouter The idea is that Puppet will natively support encrypted data in the future Follow this ticket for the roadmap view: PUP-1974 73
  74. 74. @petersouter ● Transcrypt Git-Crypt Blackbox ● Turtles All The Way Down: Storing Secrets in the Cloud and in the Data Center behind Closed Doors VCS based encryption 74
  75. 75. @petersouter Dedicated secret devices Going deeper... 75
  76. 76. @petersouter Why use a secret server? 76 ● Dynamic secrets ● ACL (Access control policies) ● Leasing and renewal ● Revocation ● Encryption ● Auditing ● Supportability
  77. 77. @petersouter 77 Conjur, Vault, Keywhiz, Amazon KMS, Confidant
  78. 78. @petersouter 78 Hiera will plug into any secret service app with a little bit of Ruby glue Hiera is just key/value lookup A hiera backend to basically any secret server setup is possible
  79. 79. @petersouter 79 $planet = conjur_variable('planet') file { '/etc/hello.txt': content => "Hello ${planet}!n" } conjurize_file { '/etc/hello.txt': variable_map => { planet => ‘!var puppetdemo/planet’ } } Conjur and Puppet
  80. 80. @petersouter 80 hiera-vault
  81. 81. @petersouter If you want to know more about Puppet + Vault, Seth Vargo from Hashicorp is presenting tomorrow 81
  82. 82. @petersouter Behind Closed Doors - Managing Passwords in a Dangerous World by Noah Kantrowitz ● Really great in-depth presentation ● ● O-5-RA ● Great breakdown of secret management, advantages and disadvantages of approaches and tooling Want to know more about secrets? 82
  83. 83. @petersouter Cleaning up the current codebase 83 How to find secrets currently exposed
  84. 84. @petersouter Manual Grepping 84 $ git grep -i -e "(api|key|username|user|pw|password|pass|email|mail )" -- `git ls-files | grep -v .html` | cat
  85. 85. @petersouter -------------------------------------------------------------------- gittyleaks' Bot Detective at work ... -------------------------------------------------------------------- file: site/profiles/templates/rhn/RHN-ORG-TRUSTED-SSL-CERT.erb what: Key value: (2048 match: Public-Key: (2048 bit) num_of_revisions: 59 Gittyleaks 85
  86. 86. @petersouter Scumblr 86
  87. 87. @petersouter Unfortunately, there’s no silver bullet to detect leaked credentials or proper usage of encryption... 87
  88. 88. @petersouter A lot of it is about process, gating and reviews 88
  89. 89. @petersouter 89 Sometimes the job is too big for a repository, and it’s better to migrate to a cleaner repo
  90. 90. @petersouter Ensuring it stays clean 90 After cleanup, ensuring it stays clean
  91. 91. @petersouter 91 At a minimum, you want to make sure that what you’re doing with Puppet isn’t making things worse!
  92. 92. @petersouter 92 It’s largely a people and process problem
  93. 93. @petersouter Making sure security is part of your workflow, rather than an afterthought “Shift security left” 93
  94. 94. @petersouter 94 “To keep up with the pace of Continuous Delivery, security must “shift left,” earlier into design and coding and into the automated test cycles, instead of waiting until the system is designed and built and then trying to fit some security checks just before release. In DevOps, security must fit into the way that engineers think and work: more iterative and incremental, and automated in ways that are efficient, repeatable, and easy to use.” - DevOpsSec: Delivering Secure Software Through Continuous Delivery, Jim Bird Shifting left!
  95. 95. @petersouter 95 How do we check things aren’t getting worse? ● Game days and internal evil attempt teams ● Continuous security integration (Gittyleaks/code-review) ● Dedicated security stories for sprints ○ Evil users or (mis)use cases ● Embedded security team members ● Dedicated audits on sensitive apps/stacks from external firms
  96. 96. @petersouter 96 Game Day example: Agent spoofing Let's say someone gets access to an agent. What’s the worst they can do?
  97. 97. @petersouter As soon as security becomes a blocker, you’ve lost! Security has to be automated where possible, otherwise we’re back to the throw-over-the-wall problems of pre-DevOps! 97
  98. 98. @petersouter Summary What have we learnt? 98
  99. 99. @petersouter Remove sensitive data from your logs Use the new sensitive type or write custom providers 99
  100. 100. @petersouter Use the roles and profiles pattern for original defaults Separate secrets, reduce the surface area for problematic setups and always read the docs before using Puppet modules 100
  101. 101. @petersouter Use Trusted Facts in your hiera hierarchy Make sure hiera lookups can’t spoofed 101
  102. 102. @petersouter Encrypt the separated data Hiera-eyaml or a dedicated secret server 102
  103. 103. @petersouter Ensure your code stays clean People, processes and automated testing 103
  104. 104. @petersouter Move security left Make it a part of your process, rather than an afterthought 104
  105. 105. @petersouter