Webinar - Managing Files with Puppet

Managing Files
with Puppet
An introduction to configuration management
OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files

Who am I?
• Walter Heck, Software engineer turned DBA, turned
Sysadmin, turned entrepreneur
• Founder of OlinData (http://www.olindata.com)
o Puppet Labs training partner for all of Asia and part of
Europe
o Node.JS, OpenStack, Linux Foundation training
o MySQL consulting

Overview
• What is puppet (for those not aware)?
• Which modules to use for file management?
• Built-in puppet resources for working with files
• How to deal with cross-resource type conflicts
• Exported resources and the concat module
• Questions

What is Puppet and why do we care?
• Configuration management software
• Scales well (1-200k+ nodes)
• Multi-platform (windows, *nix, Mac OS, BSD)
• Commercially supported Open Source
• Infrastructure as code

What options do we have?
1. source and content attributes of the file resource type
2. the concat module
3. the file_line resource type from the stdlib module
4. the inifile module
5. the augeas tool

The file resource type
• used to manage whole files (and directories and symlinks)
• content of the file is managed in two mutually exclusive ways:
o source => ‘puppet:///modules/apache/httpd.conf’
o content => template(apache/httpd.conf.erb’)
• in heavy or large environments, split off file serving to a
separate puppetmaster

The file resource type - source
• Source attribute can use
o local file on the agent
• syntax: source => ‘/usr/local/myfile.txt’
o remote file on the master
• syntax: source => ‘puppet://<puppetmaster>/modules/apache/httpd.conf’
• source attribute copies content from source non-modifiable
o great for static files eg. init.d scripts

The file resource (source example)
$master: cat mymodule/manifests/bar.pp
class mymodule::bar {
file { '/tmp/file01':
ensure => file,
owner => 'root',
group => 'root',
mode => '0644',
source => 'puppet:///modules/mymodule/file01',
}
file { '/tmp/file02':
ensure => file,
owner => 'root',
group => 'root',
mode => '0644',
source => '/opt/local/file02',
}
}
$master: cat mymodule/files/file01
This is a static file
$agent: cat /opt/local/file02
This is a locally sourced static file
$agent: puppet apply -e ‘include mymodule::bar’
$agent: cat /tmp/file01
This is a static file
$agent: cat /tmp/file02
This is a locally sourced static file

The file resource type - content
• Content attribute assigns a static string to the content of the
file
o use the template() function to parse an erb template and
assign the result.
• eg. content => template(‘apache/httpd.conf’)
• erb templates: standard ruby embedded templates
• https://docs.puppetlabs.com/guides/templating.html

The file resource (content example)
$ cat mymodule/manifests/foo.pp
class mymodule::foo {
$say_hello_to = 'dude'
$myname = 'file03'
file { "/tmp/$myname":
ensure => file,
content => template('mymodule/polite-file.erb'),
}
}
$ cat mymodule/templates/polite-file.erb
<% if @say_hello_to -%>
Hello <%= @say_hello_to %>,
<% end -%>
I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.
$ puppet apply -e ‘include mymodule::foo’
$ cat /tmp/file03
Hello dude,
I'm file03, on a Ubuntu system, nice to meet you.
Note: the @operatingsystem variable (replaced here by “Ubuntu”) value is provided by Facter

The concat module
• puppet module developed by R.I.Pienaar originally in 2010
o adopted by PuppetLabs:
https://github.com/puppetlabs/puppetlabs-concat
• Allows to concatenate separate sections of files
o eg. puppet.conf on your puppet master
• Uses a local nifty trick with a shell script
o check out: files/concatfragments.sh
• Bonus: use exported resources on fragments to collect for
instance haproxy loadbalancer backends

The concat module (example)
class motd {
$motd = '/etc/motd'
concat { $motd:
owner => 'root',
group => 'root',
mode => '0644'
}
concat::fragment{ 'motd_header':
target => $motd,
content => "nPuppet modules on this server:nn",
order => '01'
}
# local users on the machine can append to motd by just creating
# /etc/motd.local
concat::fragment{ 'motd_local':
target => $motd,
source => '/etc/motd.local',
order => '15'
}
}
#Note: concat::fragment resources for a single target file can appear anywhere in your repository, across
modules and classes

The stdlib module - file_line
• used for managing single lines in a file you otherwise don’t
care about
o supports regex matching
o caution: managing the same file with file_line and a file
resource can cause flip-flopping file content
• part of the puppetlabs/stdlib module
o https://github.com/puppetlabs/puppetlabs-stdlib
• implemented as custom resource type
o lib/puppet/provider/file_line/ruby.rb

file_line (example)
class foo::bar {
file_line { 'sudo_rule':
path => '/etc/sudoers',
line => '%sudo ALL=(ALL) ALL',
}
file_line { 'change_mount':
path => '/etc/fstab',
line => '10.0.0.1:/vol/data /opt/data nfs defaults 0 0',
match => '^172.16.17.2:/vol/old',
}
}

The inifile module
• used for managing inifile style files
o created by Chris Price, maintained by PuppetLabs
o https://github.com/puppetlabs/puppetlabs-inifile
• most important resource type is ini_setting
o Auto-creates ini section headers

The inifile module - example
class foo::baz {
ini_setting { "sample setting":
ensure => present,
path => '/tmp/foo.ini',
section => 'foo',
setting => 'foosetting',
value => 'FOO!',
}
}
$ puppet apply -e ‘include foo::baz’
$ cat /tmp/foo.ini
[foo]
foosetting = FOO!

Augeas
• Advanced parsing of config files
o http://augeas.net/
• Uses so-called lenses that define the grammar of known
config files. Many available for well known software
(http://augeas.net/stock_lenses.html)
o Creating your own lense possible, but tough!
• After parsing, a configuration file will be available in tree
format, with read/write capabilities to each node in the tree
• Works regardless of order of lines in the file

Augeas (example)
walter@web02:~$ cat -n /etc/hosts
1 # HEADER: This file was autogenerated at Sun
May 12 07:31:47 +0200 2013
2 # HEADER: by puppet. While it can still be
managed manually, it
3 # HEADER: is definitely not recommended.
4 ### OlinData installimage
5 # nameserver config
6 # IPv4
7 127.0.0.1 localhost
8 1.2.3.4 web02.olindata.com
9 1.2.3.5 db01.olindata.com
10 1.2.3.6 test01.olindata.com
11 192.168.100.100 db01
12 192.168.100.101 web02
13 1.2.3.7 mail01.olindata.com puppet
[..SNIP..]
walter@web02:~$ augtool
augtool> ls /files/etc/hosts/7
ipaddr = 1.2.3.7
canonical = mail01.olindata.com
alias = puppet
ipaddr = 1.2.3.7
canonical = mail01.olindata.com
alias = puppet
augtool> match /files/etc/hosts/*/ipaddr
1.2.3.4
/files/etc/hosts/2/ipaddr
augtool> print /files/etc/hosts/*/ipaddr
/files/etc/hosts/1/ipaddr = "127.0.0.1"

Trivia - what happens?
root@web02 /home/walter # cat test.pp
class test {
file { '/tmp/motd':
ensure => present,
content => "foo nbar n"
}
file_line { 'test line':
path => '/tmp/motd',
line => 'baz',
match => '^ba.',
}
}
include test
root@web02 /home/walter # puppet apply test.pp
Notice: Compiled catalog for web02.olindata.com in environment dev in 0.06 seconds
Notice: /Stage[main]/Test/File[/tmp/motd]/ensure: created
Notice: /Stage[main]/Test/File_line[test line]/ensure: created
Notice: Finished catalog run in 0.71 seconds

Trivia - what happens?
root@web02 /home/walter # cat test.pp
class test {
file { '/tmp/motd':
ensure => present,
content => "foo nbar n"
}
file_line { 'test line':
path => '/tmp/motd',
line => 'baz',
match => '^ba.',
}
}
root@web02 /home/walter # cat /tmp/motd
foo
baz
include test
root@web02 /home/walter # puppet apply test.pp
Notice: Compiled catalog for web02.olindata.com in environment dev in 0.06 seconds
Notice: /Stage[main]/Test/File[/tmp/motd]/ensure: created
Notice: /Stage[main]/Test/File_line[test line]/ensure: created
Notice: Finished catalog run in 0.71 seconds

Exported resources
PuppetDB
3. Store in PuppetDB 5. Retrieve from PuppetDB
Puppet
Master
WEB01 DB01
@@concat::fragment{ ‘test’:
ensure => present,
target => ‘/etc/motd’
}
concat {‘/etc/motd’:
ensure => present,
}
Concat::Fragment <<| |>>
1. Puppet agent run
6. Send to node
2. Export to PM
4. Collect on DB01

Exported resources
node /^webd{3}.olindata.vm$/ {
include role::haproxy::backend
}
node 'proxy01.olindata.vm' {
include role::haproxy
}
class role::haproxy {
concat {‘/etc/haproxy.cfg’:
ensure => present,
}
Concat::Fragment <<| |>>
}
class role::haproxy::backend {
@@concat::fragment { $::fqdn:
ensure => 'present',
target => '/etc/haproxy.cfg'
}
}

Upcoming training
• Puppet Fundamentals Training, Vienna
Monday, November 17, 2014
• Puppet Fundamentals Training, Barcelona
• Puppet Fundamentals Training, Hyderabad
• Puppet Fundamentals Training, Pune
Monday, December 1, 2014
• Puppet Architect Training, Singapore
Wednesday, December 17, 2014

We’re hiring!
EU and Asia based
trainers
jobs@olindata.com

Questions?
@walterheck / @olindata
http://www.olindata.com
walterheck@olindata.com
http://github.com/olindata

Webinar - Managing Files with Puppet

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Webinar - Managing Files with Puppet

Similar to Webinar - Managing Files with Puppet (20)

More from OlinData

More from OlinData (20)

Recently uploaded

Recently uploaded (20)

Webinar - Managing Files with Puppet