#hashtalks presentation

1. Puppet and the
HashiCorp stack
Bram Vogelaar
HashiTalks 2019

2. So your boss wants a cloudSo your boss wants a cloud
Lets build a self-service thingieLets build a self-service thingie

3. All of IaCAll of IaC

4. Base ImageBase Image
You cant PXE boot a cloudYou cant PXE boot a cloud

5. PackerPacker
● Open Source tool to make OS imagesOpen Source tool to make OS images
● Supports Cloud Providers, Docker, Vbox, …Supports Cloud Providers, Docker, Vbox, …
(builders)(builders)
● Has hooks to provision the base imagesHas hooks to provision the base images
(provisioners)(provisioners)
● Create artifacts (Post-Processors)Create artifacts (Post-Processors)

6. PackerPacker
{{
"builders": ["builders": [
{{
"type": "azure-arm","type": "azure-arm",
......
}}
],],
"provisioners": ["provisioners": [
{{
"scripts": ["scripts": [
"scripts/common/puppet.sh""scripts/common/puppet.sh"
],],
"type": "shell","type": "shell",
},},
{{
"type": "puppet-server","type": "puppet-server",
"puppet_server": "puppet""puppet_server": "puppet"
}}
]]
}}
$ packer build template.json$ packer build template.json

7. Packer & PuppetPacker & Puppet
https://github.com/petems/puppet-install-shellhttps://github.com/petems/puppet-install-shell
$ ./puppet.sh [-p] [-v version] …$ ./puppet.sh [-p] [-v version] …

8. VagrantVagrant
● Open Source tool to bootstrap vmsOpen Source tool to bootstrap vms
● Supports many vm Providers, Docker, Vbox, …Supports many vm Providers, Docker, Vbox, …
● Has hooks to provision the base imagesHas hooks to provision the base images
(provisioners), Puppet, Chef, Ansible(provisioners), Puppet, Chef, Ansible

9. VagrantVagrant
ifif nodenode[["hiera_path""hiera_path"]]
srvsrv.vm.synced_folder.vm.synced_folder nodenode[["hiera_path""hiera_path"],],
"/etc/puppetlabs/code/environments/"/etc/puppetlabs/code/environments/#{environment}#{environment}/hieradata"/hieradata"
elseelse
srvsrv.vm.synced_folder.vm.synced_folder ""#{environment}#{environment}/hieradata"/hieradata",,
"/etc/puppetlabs/code/environments/"/etc/puppetlabs/code/environments/#{environment}#{environment}/hieradata"/hieradata"
endend
srvsrv.vm.provision :puppet.vm.provision :puppet dodo ||puppetpuppet||
puppetpuppet.environment =.environment = ""#{environment}#{environment}""
ifif nodenode[["environment_path""environment_path"]]
puppetpuppet.environment_path =.environment_path = nodenode[["environment_path""environment_path"]]
elseelse
puppetpuppet.environment_path =.environment_path = ".""."
endend
ifif nodenode[["hiera_config_path""hiera_config_path"]]
puppetpuppet.hiera_config_path =.hiera_config_path = nodenode[["hiera_config_path""hiera_config_path"]]
elseelse
puppetpuppet.hiera_config_path =.hiera_config_path = ""#{environment}#{environment}/hiera.yaml"/hiera.yaml"
endend
endend
endend

10. VagrantVagrant
casecase nodenode[["provision_type""provision_type"]]
whenwhen 'puppet_agent''puppet_agent'
ifif nodenode[["puppetmaster""puppetmaster"]]
puppetmaster =puppetmaster = nodenode[["puppetmaster""puppetmaster"]]
elseelse
puppetmaster =puppetmaster = "puppetmaster."puppetmaster.#{environment}#{environment}.vagrant".vagrant"
endend
srvsrv.vm.provision.vm.provision "puppet_server" do"puppet_server" do ||puppetpuppet||
puppetpuppet.options =.options = "-t --environment"-t --environment #{environment}#{environment}""
puppetpuppet.puppet_server = puppetmaster.puppet_server = puppetmaster
endend
srvsrv.trigger.after :destroy.trigger.after :destroy dodo ||triggertrigger||
triggertrigger.name =.name = "Cleaning puppet certificate""Cleaning puppet certificate"
triggertrigger.run = {inline:.run = {inline: "vagrant ssh puppetmaster -c 'sudo"vagrant ssh puppetmaster -c 'sudo
/opt/puppetlabs/bin/puppet cert clean/opt/puppetlabs/bin/puppet cert clean #{#{nodenode[["hostname""hostname"]}]}'"'"}}
endend

11. TerraformTerraform
● Open Source Automation ToolOpen Source Automation Tool
● ““cloud” orientedcloud” oriented
● Cloud are API’sCloud are API’s
● API’s orientedAPI’s oriented
Terraform is an open source automation toolTerraform is an open source automation tool
which can deal with any kind of CRUD api’s –which can deal with any kind of CRUD api’s –
including major cloud providersincluding major cloud providers

12. The Terraform modelThe Terraform model
● You model your infrastructureYou model your infrastructure
● You make a planYou make a plan
● If ok, you apply that planIf ok, you apply that plan
● Current state is saved for future changesCurrent state is saved for future changes

13. HCLHCL
● Hashicorp Configuration LanguageHashicorp Configuration Language
● Yet another cfgmgmt DSLYet another cfgmgmt DSL
● Desired stateDesired state
● Used by multiple hashicorp tool but also 3rdUsed by multiple hashicorp tool but also 3rd
party toolsparty tools

14. Lets magic a nodeLets magic a node
resourceresource "azurerm_virtual_machine" "puppetmaster" {"azurerm_virtual_machine" "puppetmaster" {
name = "vm-${var.node_name}"name = "vm-${var.node_name}"
location = "${var.azure_region_name}"location = "${var.azure_region_name}"
resource_group_name = "${var.resource_group}"resource_group_name = "${var.resource_group}"
network_interface_ids = ["$network_interface_ids = ["$
{azurerm_network_interface.puppetmaster.id}"]{azurerm_network_interface.puppetmaster.id}"]
vm_size = "${var.vm_size}"vm_size = "${var.vm_size}"
storage_image_referencestorage_image_reference {{
id = "${var.image_id}"id = "${var.image_id}"
}}
storage_os_diskstorage_os_disk {{
......
}}
os_profileos_profile {{
......
}}
}}

15. Bootstrap yourBootstrap your
puppetmasterpuppetmaster
#!/usr/bin/env bash#!/usr/bin/env bash
apt-get install puppetserverapt-get install puppetserver
/opt/puppetlabs/bin/puppet agent -t --environment=mgmt/opt/puppetlabs/bin/puppet agent -t --environment=mgmt
/opt/puppetlabs/bin/puppet cert clean --all/opt/puppetlabs/bin/puppet cert clean --all
/opt/puppetlabs/bin/puppet cert generate $(hostname --fqdn)/opt/puppetlabs/bin/puppet cert generate $(hostname --fqdn)
--dns_alt_names ${dns_alt_names}--dns_alt_names ${dns_alt_names}
systemctl restart puppetserver.servicesystemctl restart puppetserver.service

16. Chicken and Egg ProblemChicken and Egg Problem

17. Its a DNS problemIts a DNS problem

18. ConsulConsul
● Open Source Service Discovery ToolOpen Source Service Discovery Tool
● Service Mesh toolService Mesh tool
● Build-in KV storeBuild-in KV store

19. ConsulConsul
classclass { '::consul':{ '::consul':
config_hash => $config,config_hash => $config,
extra_options => $options,extra_options => $options,
version => $version,version => $version,
}}
::consul::check { 'puppetmaster_tcp':::consul::check { 'puppetmaster_tcp':
interval => '60',interval => '60',
tcp => 'localhost:8140',tcp => 'localhost:8140',
notes => 'Puppetmasters listen on port 8140'notes => 'Puppetmasters listen on port 8140'
service_id => 'puppetmaster',service_id => 'puppetmaster',
}}
::consul::service { 'puppetmaster':::consul::service { 'puppetmaster':
port => 8140,port => 8140,
}}
https://forge.puppet.com/KyleAnderson/consulhttps://forge.puppet.com/KyleAnderson/consul

20. ConsulConsul
Dig @127.0.0.1 -p 8600 puppetmaster.service.consul ANY

21. Geolocation StuffGeolocation Stuff
consul_prepared_query { 'mysql':consul_prepared_query { 'mysql':
ensure => 'present',ensure => 'present',
service_name => 'mysql',service_name => 'mysql',
service_failover_n => 1,service_failover_n => 1,
service_failover_dcs => [ 'dc1', 'dc2' ],service_failover_dcs => [ 'dc1', 'dc2' ],
service_only_passing => true,service_only_passing => true,
service_tags => [ 'tag1', 'tag2' ],service_tags => [ 'tag1', 'tag2' ],
ttl => 10,ttl => 10,
}}

22. Fast ExportedFast Exported
ResourcesResources
::consul::watch { 'detect_backend_changes':::consul::watch { 'detect_backend_changes':
type => 'service',type => 'service',
handler => '/usr/bin/runpuppet.sh',handler => '/usr/bin/runpuppet.sh',
service => 'Application1',service => 'Application1',
passingonly => true,passingonly => true,
require => File['/usr/bin/runpuppet.sh'],require => File['/usr/bin/runpuppet.sh'],
}}

23. Lets magic a nodeLets magic a node
resourceresource "azurerm_virtual_machine" "puppetmaster" {"azurerm_virtual_machine" "puppetmaster" {
......
os_profileos_profile {{
custom_data = "${data.template_file.bootstrap_sh.rendered}"custom_data = "${data.template_file.bootstrap_sh.rendered}"
}}
}}

24. cloud-initcloud-init
#!/usr/bin/env bash#!/usr/bin/env bash
consul agent -retry-join "provider=azure consul agent -retry-join "provider=azure
tag_name=consul tag_name=consul
tenant_id=${tid} tenant_id=${tid}
client_id=${cid} client_id=${cid}
subscription_id=$cid} subscription_id=$cid}
secret_access_key=${key}"secret_access_key=${key}"
/opt/puppetlabs/bin/puppet agent -t /opt/puppetlabs/bin/puppet agent -t
--environment=${environment} --environment=${environment}
--server=puppet.service.consul--server=puppet.service.consul

25. So your boss wants a cloudSo your boss wants a cloud
Lets build a self-service cloudLets build a self-service cloud
We are getting closeWe are getting close

26. So your boss wants a cloudSo your boss wants a cloud
Lets build a self-service cloudLets build a self-service cloud
We are getting closeWe are getting close
We forgot about that self-service thingWe forgot about that self-service thing

27. Puppet FactsPuppet Facts
● Facter is very coolFacter is very cool
● Facts are very usefulFacts are very useful
● Facts are also very very insecureFacts are also very very insecure
ifif versioncmp($::puppetversion, '4.0.0') <= 0 {versioncmp($::puppetversion, '4.0.0') <= 0 {
file { 'hiera.yaml':file { 'hiera.yaml':
mode => '0644',mode => '0644',
owner => 'puppet',owner => 'puppet',
group => 'puppet',group => 'puppet',
content => template('profiles/hiera.yaml.erb'),content => template('profiles/hiera.yaml.erb'),
path => '/etc/puppetlabs/puppet/hiera.yaml',path => '/etc/puppetlabs/puppet/hiera.yaml',
}}
Class['::puppet']Class['::puppet']
-> File['hiera.yaml']-> File['hiera.yaml']
}}

28. Puppet Trusted FactsPuppet Trusted Facts
cat << EOF > /etc/puppetlabs/puppet/csr_attributes.yamlcat << EOF > /etc/puppetlabs/puppet/csr_attributes.yaml
------
extension_requests:extension_requests:
pp_environment: ${environment}pp_environment: ${environment}
pp_network: ${network}pp_network: ${network}
pp_provisioner: terraformpp_provisioner: terraform
pp_region: ${region}pp_region: ${region}
pp_role: ${role}pp_role: ${role}
EOFEOF

29. Puppet Trusted FactsPuppet Trusted Facts
cat default.ppcat default.pp
if $trusted['extensions']['pp_role'] {if $trusted['extensions']['pp_role'] {
include "roles::${trusted['extensions']['pp_role']}"include "roles::${trusted['extensions']['pp_role']}"
} else {} else {
fail("No role is set for this node.")fail("No role is set for this node.")
}}

30. cloud-initcloud-init
#!/usr/bin/env bash#!/usr/bin/env bash
consul agent -retry-join "provider=azure consul agent -retry-join "provider=azure
tag_name=consul tag_name=consul
tenant_id=${tid} tenant_id=${tid}
client_id=${cid} client_id=${cid}
subscription_id=$cid} subscription_id=$cid}
secret_access_key=${key}"secret_access_key=${key}"
cat << EOF > /etc/puppetlabs/puppet/csr_attributes.yamlcat << EOF > /etc/puppetlabs/puppet/csr_attributes.yaml
------
custom_attributes:custom_attributes:
1.2.840.113549.1.9.7: mySuperAwesomePassword1.2.840.113549.1.9.7: mySuperAwesomePassword
extension_requests:extension_requests:
pp_preshared_key: ${otp}pp_preshared_key: ${otp}
pp_environment: ${environment}pp_environment: ${environment}
pp_network: ${network}pp_network: ${network}
pp_provisioner: terraformpp_provisioner: terraform
pp_region: ${region}pp_region: ${region}
pp_role: ${role}pp_role: ${role}
EOFEOF
/opt/puppetlabs/bin/puppet agent -t /opt/puppetlabs/bin/puppet agent -t
--environment=${environment} --environment=${environment}
--server=puppet.service.consul--server=puppet.service.consul

31. policy-basedpolicy-based
autosigningautosigning
chmod a+x /usr/local/bin/autosign-otpchmod a+x /usr/local/bin/autosign-otp
puppet config set --section master autosignpuppet config set --section master autosign
/usr/local/bin/autosign-otp/usr/local/bin/autosign-otp
#! /bin/bash#! /bin/bash
PSK_DIR=/etc/puppet/autosign/pskPSK_DIR=/etc/puppet/autosign/psk
csr=$(< /dev/stdin)csr=$(< /dev/stdin)
certname=$1certname=$1
function extension {function extension {
echo "$csr" | openssl req -noout -text | fgrep -A1 "$1" | tail -necho "$csr" | openssl req -noout -text | fgrep -A1 "$1" | tail -n
1 1
| sed -e 's/^ *//;s/ *$//'| sed -e 's/^ *//;s/ *$//'
}}
otp=$(extension '1.3.6.1.4.1.34380.1.1.4')otp=$(extension '1.3.6.1.4.1.34380.1.1.4')

32. Pesky PasswordsPesky Passwords
classclass { '::vault':{ '::vault':
install_method => 'repo',install_method => 'repo',
}}
$ vault write kv/my-secret value="s3c(eT"$ vault write kv/my-secret value="s3c(eT"
https://forge.puppet.com/jsok/vaulthttps://forge.puppet.com/jsok/vault

33. HieraHiera
cat hiera.yamlcat hiera.yaml
------
version: 5version: 5
hierarchy:hierarchy:
- name: "Hiera-vault lookup"- name: "Hiera-vault lookup"
lookup_key: hiera_vaultlookup_key: hiera_vault
options:options:
confine_to_keys:confine_to_keys:
- '^sensitive_.*'- '^sensitive_.*'
- '^password.*'- '^password.*'
address: http://vault:8200address: http://vault:8200
mounts:mounts:
generic:generic:
- secret/puppet/%{::trusted.certname}/- secret/puppet/%{::trusted.certname}/
- secret/puppet/common/- secret/puppet/common/
https://github.com/petems/hiera_backend_vaulthttps://github.com/petems/hiera_backend_vault

34. PuppetPuppet
classclass profiles::database::mysql (profiles::database::mysql (
Sensitive $root_password,Sensitive $root_password,
) {) {
classclass { '::mysql::server':{ '::mysql::server':
root_password => $root_password.unwrap,root_password => $root_password.unwrap,
}}
notice($root_password)notice($root_password)
}}
Notice: Scope(Class[main]): Sensitive [value redacted]Notice: Scope(Class[main]): Sensitive [value redacted]

35. Puppet 6Puppet 6
$ vault auth enable cert$ vault auth enable cert
$ vault write auth/cert/certs/puppetserver $ vault write auth/cert/certs/puppetserver
display_name=puppet display_name=puppet
policies=prod,test policies=prod,test
certificate=@/path/to/puppetserver/ca.pem certificate=@/path/to/puppetserver/ca.pem
Ttl=3600Ttl=3600
https://forge.puppet.com/puppet/vault_lookuphttps://forge.puppet.com/puppet/vault_lookup

36. Puppet 6Puppet 6
$root_password = Deferred('vault_lookup::lookup',$root_password = Deferred('vault_lookup::lookup',
["password/mysql", 'https://vault:8200'])["password/mysql", 'https://vault:8200'])
classclass { '::mysql::server':{ '::mysql::server':
root_password => $root_password,root_password => $root_password,
}}

37. ConclusionConclusion
● Packer and Terraform are great tools to build aPacker and Terraform are great tools to build a
cloud infracloud infra
● Adding Puppet on top of it enables you toAdding Puppet on top of it enables you to
configure and run the nodes in a well-knownconfigure and run the nodes in a well-known
way (cloud without bash :) )way (cloud without bash :) )
● Make the right tradeoffs between puppet inMake the right tradeoffs between puppet in
base image vs puppet on machine bootbase image vs puppet on machine boot