Neil Millard describes the components required to run puppet in a masterless configuration. This allows you to boot immutable servers quickly, and the option of tearing them down and replacing/rebuilding quickly. This was presented at Puppetcamp London 2017

1. Short Lived, Immutable Servers
with Masterless Puppet?
Presented by: Neil Millard
www.millowgroup.co.uk

2. whoami
• Always programming
• IT professional since 1995

3. whoami
• Devops / Infrastructure as Code
• Helicopter pilot

4. whoami
• Educate
• Automate
• Have FUN

5. Infrastructure
Application
Monitoring & Logging
Application Deployments
Automation

6. agenda
• Masterless puppet
• Bootstrap
• Modules
• Hieradata
• Roles and Profiles
• Ordering
• Stages

7. What is Masterless
• Puppet without a master
• Without a master, the automation needs to ‘provide’
all the configuration the
Master gives us
• The master and node
in one

8. Why Masterless puppet
Pets
http://www.slideshare.net/gmccance/cern-data-centre-evolution - attrib Bill Baker/Microsoft
https://www.iha.com
Scale UP

9. Why Masterless puppet
Cattle
http://www.slideshare.net/gmccance/cern-data-centre-evolution - attrib Bill Baker/Microsoft
Scale OUT

10. Why Masterless puppet
• Puppet Masters
• Pets
• Maintenance
• Care
• Sometimes fragile

11. Why Masterless puppet
• Rebuild
rather than
reconfigure
- Immutable
• Temporary,
worker,
short lived
• Build -> work-> destroy server

12. Masterless puppet
• Bootstrapping
• Cloud-init
• userdata
- Environment
- Role

13. Masterless puppet (baking)
• Everything included
• Lots of maintenance required

14. Masterless puppet (baking)
• Bare shell
• Build according to configuration
(version controlled)

15. Bootstrapping
• Puppet Prerequisites
• Ruby
• Git
• Gems (puppet, hiera)
• Modules

16. Bootstrapping
• Preparing to run puppet
Tru-strap https://github.com/MSMFG/tru-strap
Args: Provisioning or configuration repository
Role + Environment Facts, for hiera lookup
echo “init_role=${value}" > /etc/facter/facts.d/init-role.txt

17. Components
• Hieradata
• Roles and Profiles
• Modules

18. Masterless puppet
• Fetch configuration (infrastructure as code)
• Puppet
• hieradata
• Puppetfiles
• manifests
• modules
• profiles
• roles
• https://github.com/neilmillard/puppet-dockerhost

19. hieradata
• Separation of data from code
docker::run { $container:
image => $image,
command => $command,
memory_limit => $memory_limit,
cpuset => $cpuset,
ports => $ports,
expose => $expose,
volumes => $volumes,
links => $links,
use_name => $use_name,
running => $running,
volumes_from => $volumes_from,
net => $net,
username => $username,
hostname => $hostname,
env => $env,
dns => $dns,
dns_search => $dns_search,
lxc_conf => $lxc_conf,
restart_service => $restart_service,
disable_network => $disable_network,
privileged => $privileged,
detach => $detach,
extra_parameters => $extra_parameters,
pull_on_start => $pull_on_start,
depends => $depends,
tty => $tty,
require => $requires,
}
eyamld:
image: "nginx"
ports:
- "80:80"
env:
- NGINX_HOST=foobar.com
- NGINX_PORT=80
use_name: true
docker::run { ‘eyamld’:
image => "nginx",
command => undef,
memory_limit => 0b,
cpuset => [],
ports => ["80:80“],
expose => [],
volumes => [],
links => [],
use_name => true,
running => true,
volumes_from => [],
net => 'bridge',
username => false,
hostname => false,
env => [NGINX_HOST=foobar.com,
NGINX_PORT=80
],
dns => [],
dns_search => [],
lxc_conf => [],
restart_service => true,
disable_network => false,
privileged => false,
detach => true,
extra_parameters => undef,
pull_on_start => false,
depends => [],
tty => false,
require => [],
}
+ =

20. hieradata
• Separation of data from code
• Code reuse with lookups
profile::docker_containers::containers:
eyamld:
image: "nginx"
ports:
- "80:80"
env:
- NGINX_HOST=foobar.com
- NGINX_PORT=80
use_name: true
# profile::docker_containers
class profile::docker_containers
($containers={}) {
create_resources (
'profile::docker_container', $containers )
}

21. hieradata
• Separation of data from code
Like pilot from the helicopter
• Code reuse with lookups
• Automatic parameter lookup
# In this example, $parameter's value gets set
# when `myclass` is eventually declared.
# Class definition:
class myclass ($parameter_one = "default text") {
file {'/tmp/foo':
ensure => file,
content => $parameter_one,
}
}

22. hieradata
• Hiera.yaml – configuration
---
:backends:
- eyaml
- yaml
:eyaml:
:datadir: /etc/puppetlabs/puppet/hieradata
:pkcs7_private_key: /etc/puppet/secure/keys/private_key.pkcs7.pem
:pkcs7_public_key: /etc/puppet/secure/keys/public_key.pkcs7.pem
:yaml:
:datadir: /etc/puppetlabs/puppet/hieradata
:hierarchy:
- "%{::init_env}/%{::init_role}"
- "%{::init_role}"
- "%{::init_env}"
- common

23. Roles and Profiles
• Business Layer (Roles)
only includes profiles
no logic
one server, one role
• Implementation Layer (Profiles)
Includes classes
Modules and Resources
create_resources{}
Craig Dunn - http://www.slideshare.net/PuppetLabs/roles-talk
Puppet - https://docs.puppet.com/pe/2016.2/r_n_p_intro.html
http://www.slideshare.net/DaeHyung/learning-puppet-basic-thing #64

24. Modules
• Puppetforge or Git (Github)

25. Modules
Librarian or r10k from Puppetfile
forge "https://forgeapi.puppetlabs.com"
# Base modules
mod "saz/timezone", "3.0.1"
mod "saz/rsyslog", "4.0.2"

26. Masterless puppet
• Puppet catalog compilation
• Puppet catalog apply
http://www.slideshare.net/bernstein_aaron/puppet-introduction-26593192 #25
Data
Hieradata

27. Puppet apply
• Site.pp
• Classes
• Hieradata
• Roles
• Manifests
node default {
hiera_include('classes')
}

28. hieradata
• Hiera.yaml – configuration
---
:backends:
- eyaml
- yaml
:eyaml:
:datadir: /etc/puppetlabs/puppet/hieradata
:pkcs7_private_key: /etc/puppet/secure/keys/private_key.pkcs7.pem
:pkcs7_public_key: /etc/puppet/secure/keys/public_key.pkcs7.pem
:yaml:
:datadir: /etc/puppetlabs/puppet/hieradata
:hierarchy:
- "%{::init_env}/%{::init_role}"
- "%{::init_role}"
- "%{::init_env}"
- common
:yaml:
:datadir: /etc/puppetlabs/puppet/hieradata
:hierarchy:
- "%{::init_env}/%{::init_role}"
- "%{::init_role}"
- "%{::init_env}"
- common

29. Hiera Lookups
• Under the Hieradata folder
• ::init_role == dockerhost
• Example code
• https://github.com/neilmillard/puppet-dockerhost
echo “init_role=${value}" > /etc/facter/facts.d/init-role.txt

30. Classes and role
• Classes:
# hieradata/dockerhost.yaml
---
classes:
- role::dockerhost

31. Roles and Profiles
• Defined as classes within either Roles Module or Profiles Module
• Roles contain Profiles
• Use include, require or class
class role::dockerhost {
include ::profile::base
include ::profile::os_limits
include ::profile::docker_base
include ::profile::docker_containers
class { '::profile::swapfile':
before => Class['profile::base']
}
}
# profile::docker_containers
class profile::docker_containers
($containers={}) {
create_resources (
'profile::docker_container', $containers )
}

32. Data file flow summary
Role fact
hieradata
site.pp
role.yaml classes
Role::
Profile::
Profile::

33. Components
• Hieradata
Colours and size
• Roles and Profiles
Helicopter made up of
components
• Modules

34. ordering
• Puppet execution is in parallel
• Dependencies need order
anchor { 'ntp::begin': } ->
class { '::ntp::install': } ->
class { '::ntp::config': } ~>
class { '::ntp::service': } ->
anchor { 'ntp::end': }

35. ordering
• before, require
class { ‘install-ssl':
installdir => "$installdir",
require => Exec['unarchive-source'],
before => File['copy-init-file'],
}

36. ordering
• ->
• Everything else is attempted at the same time
->
class { ‘install-ssl':
installdir => "$installdir",
}
->

37. Ordering - stages
• main
• WARNING – use with care.
Can cause dependency cycles

38. Ordering - stages
• Catalog compiles
• Runs each stage based on order
• Ordering across stages WILL break
Notice: Compiled catalog for ip-10-96-4-130.internal in environment production in 2.10
Notice: /Stage[swapfile]/Profile::Swapfile/Exec[Create swap file /mnt/swap.1]/returns:
Notice: /Stage[swapfile]/Profile::Swapfile/File[/mnt/swap.1]/mode: mode changed '0644'
...
Notice: /Stage[main]/Profile::Docker_base/Exec[yum install -y docker-io]/returns:
Notice: /Stage[main]/Docker::Service/File[/etc/sysconfig/docker]/content: content

39. Ordering - stages
• Simple manifests best
• swapfile stage
stage { 'swapfile':
before => Stage['main'],
}
class { '::profile::swapfile':
stage => swapfile
before => Class['profile::base']
}

40. What we covered
• Masterless puppet
• Bootstrap
• Modules
• Hieradata
• Roles and Profiles
• Ordering
• Stages

41. Neil Millard
Blog: www.neilmillard.com
Info: devops.neilmillard.com
github.com/neilmillard
e: neil.millard@millowgroup.co.uk