Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
. 
AAuuggeeaass 
SSwwiissss--kknniiffee rreessoouurrcceess ffoorr yyoouurr ppuuppppeett ttrreeee 
Julien Pivotto 
Belgian ...
. 
wwhhooaammii 
JJuulliieenn PPiivvoottttoo 
• Open-Source consultant at inuits.eu 
• FOSS defender since 2004 
• DevOps ...
. 
. 
ınuits.eu
. 
. 
SSyyssaaddmmiinn 110011 
CC BY-SA 2.0 https://www.flickr.com/photos/arthur-caranta/2926332140
. 
SSeettttiinngg uupp aa sseerrvviiccee 
• Install the package 
• Change the configuration 
• Start the daemon
. 
3 steps. 
What can go wrong?
. 
PPaacckkaaggiinngg 
• Where is the package? 
• Which version do we need? 
• Does it conflict with something else?
. 
. 
DDeeppeennddeenncciieess HHeellll 
CC BY-SA 2.0 https://www.flickr.com/photos/coconinonationalforest/4587053982
. 
CCoonnffiigguurraattiioonn 
• Where is the file? 
• How many files? 
• Configuration is in the database? 
• The file is...
. 
SSttaarrttiinngg tthhee sseerrvviiccee 
• Does not start 
▶ Bad config file 
▶ Stale lock file 
▶ Data corruption 
• Hi...
. 
LLeett''ss ttaallkk aabboouutt PPuuppppeett aanndd ffiilleess 
• Classical approach: File[] resource 
• Advanced approa...
. 
HHiiddddeenn wwaayyss ttoommaannaaggee ffiilleess 
• Ssh_authorized_key[] 
• Nagios_* 
• To purge or not to purge
. 
. 
TThhee FFiillee[[]] rreessoouurrccee 
CC BY 2.0 https://www.flickr.com/photos/80497449@N04/10567875696/
. 
FFiillee 
• Built-in puppet resource 
• Most used 
• Works with a lot of usecases 
• Text files, binary files
. 
. 
file{"${::icinga::confdir_server}/cgi.cfg": 
ensure => present, 
content => template('icinga/redhat/cgi.cfg.erb'), 
...
. 
CCoonntteenntt ooff aa ffiillee 
• content => String, template(), file() 
• source => puppet:///, /local/file
. 
FFiillee[[]] bbeehhaavviioouurr 
• Array as "source": Puppet will pick the first 
available one 
• Multiple arguments t...
. 
DDoowwnnssiiddee ooff FFiillee[[]] 
• You can only have at one "content" 
• That resource describe the whole file 
• Wo...
. 
. 
concat 
Public Domain http://commons.wikimedia.org/wiki/File:Adhesive_tapes_clear.JPG
. 
CCoonnccaatt 
• A "reference" puppet module: 
puppetlabs/concat 
• https://github.com/puppetlabs/puppetlabs-concat 
• P...
. 
CCoonnccaatt?? 
• Concat takes a bunch of snippets 
• Assemble them info a file 
• Each snippet is a define 
• The fina...
. 
. 
concat { '/tmp/file': 
ensure => present, 
} 
concat::fragment { 'tmpfile': 
target => '/tmp/file', 
content => 'tes...
. 
BBaassee aanndd ffrraaggmmeennttss 
• Concat[] defines owner, ensure, mode of the 
file 
• Concat::Fragment[] defines t...
. 
AAddvvaannttaaggeess ooff ccoonnccaatt 
• More flexibility 
▶ if 
▶ virtual resource 
▶ exported resources 
▶ create_re...
. 
DDiissaaddvvaannttaaggeess ooff ccoonnccaatt 
• External Puppet module 
• Concat[] is the whole file 
• Performances
. 
. 
Exec{sed: onlyif => grep} 
CC BY-SA 3.0 http://commons.wikimedia.org/wiki/File:Ca%C3%AFn_par_Henri_Vidal.jpg
. 
. 
https://github.com/search?o=desc&q=exec+sed+onlyif+grep+language%3APuppet
. 
eexxeecc[[sseedd]] iiss bbrr00kkeenn 
• Which options to pass to sed and grep? 
• You should use as few Exec[] as possi...
. 
AAnnootthheerr aalltteerrnnaattiivvee:: ccoonnff..dd 
• Some services support conf.d directories 
• But it is hard to c...
. 
. 
Augeas 
CC BY-SA 3.0 http://commons.wikimedia.org/wiki/File:Students_assisting_surgery.JPG
. 
AAuuggeeaass 
• Configuration editing tool 
• First release in 2007 
• API coded in C 
• Command-line tools 
• bindings...
. 
CCoonnffiigguurraattiioonn eeddiittiinngg ttooooll 
• Parsing the configuration files 
• Turning them into a tree 
• Ed...
. 
. 
$ cat /etc/nsswitch.conf 
# /etc/nsswitch.conf 
## 
Example configuration 
# 
passwd: db files 
group: db files 
ini...
. 
. 
augtool> ls /files/etc/nsswitch.conf/ 
#comment[1] = /etc/nsswitch.conf 
#comment[2] = Example configuration 
databa...
. 
. 
augtool> ls /files/etc/nsswitch.conf/database[1]/ 
service[1] = db 
service[2] = files 
.
. 
NNaattiivvee ffoorrmmaatt -->> ttrreeee 
• Augeas understand comments 
• Augeas does not care about empty lines 
• The ...
. 
. 
augtool> set /files/etc/nsswitch.conf/database[1]/ 
service[last()+1] ldap 
augtool> save 
Saved 1 file(s) 
.
. 
. 
$ cat /etc/nsswitch.conf 
# /etc/nsswitch.conf 
## 
Example configuration 
# 
passwd: db files ldap 
group: db files...
. 
. 
augtool> match /files/etc/nsswitch.conf/*/* ldap 
/files/etc/nsswitch.conf/database[1]/service[3] 
augtool> print /f...
. 
. 
augtool> rm /files/etc/nsswitch.conf/database[1]/service[3] 
rm : /files/etc/nsswitch.conf/database[1]/service[3] 1 ...
. 
OOnnee AAPPII ttoo eeddiitt tthheemmaallll 
• Can talk XML, ini, named, nginx, … 
• Only change what is needed 
• Ensur...
. 
AAuuggeeaass LLeennsseess 
• Lenses are files that explain how to edit files 
• It contains paths and syntax 
• There a...
. 
”This brings the total number of lenses to 
178. […] It’s depressing to think that 
Linux/Unix systems have managed to ...
. 
117788 lleennsseess 
activemq_conf activemq_xml aliases aptconf 
apt_update_manager backuppchosts bbhosts bootconf buil...
. 
AA sshhoorrtt lleennssee 
. 
module Hostname = 
autoload xfm 
(* View: lns *) 
let lns = [ label "hostname" . store Rx....
. 
PPuuppppeett <<33 aauuggeeaass 
• Native "augeas" resource 
• Support for pluginsync 
• Helpers available
. 
PPuuppppeett eexxaammppllee 
. 
augeas { $name: 
context => "/files${fstab::variables::fstab_file}", 
changes => [ 
"rm...
. 
RReeaall uusseeccaasseess 
• Change grub options 
• Modify /etc/hosts 
• Modify XML's (puppetlabs-tomcat) 
• Configure ...
. 
PPlluuggiinnssyynncc 
• Puppet has pluginsync support for Augeas 
• Drop your lenses in your modules 
• lib/augeas/lens...
. 
PPuuppppeett eexxaammppllee 
. 
augeas{"jboss_conf": 
. 
context => "/files/etc/jbossas", 
changes => [ 
"set jbossas.c...
. 
AAuuggeeaass ccoommmmaannddss 
set rm mv clear insert …
. 
AAuuggeeaass ccoommppaarraattoorrss ((oonnllyyiiff)) 
match get
. 
AAuuggeeaasspprroovviiddeerrss 
• Helpers around augeas 
• Puppet modules 
• No augeas knowledge needed
. 
aappaacchhee 
. 
apache_setenv { "SPECIAL_PATH": 
ensure => present, 
value => "/foo/bin", 
} 
.
. 
kkeerrnneell__ppaarraammeetteerr 
. 
kernel_parameter { "quiet": 
ensure => present, 
bootmode => "normal", 
} 
.
. 
Conclusion
. 
DDiissaaddvvaannttaaggeess 
• Learning required 
• Library to install 
• Writing lenses is hard
. 
AAddvvaannttaaggeess 
• Augeas is a mature tool 
• Preserves comments in files 
• It fails (if needed) 
• Only changes ...
. 
FFiinnaall nnoottee 
Most of the time, File[] resources are the way 
to go. Augeas can help when you need to 
change fi...
. 
RReeaaddiinnggss 
• http://augeas.net/ 
• http://augeasproviders.com/ 
• https://docs.puppetlabs.com/
. 
TThhaannkk yyoouu 
Any question? 
Thanks to @raphink
. 
CCoonnttaacctt 
Julien Pivotto 
julien@inuits.eu 
@roidelapluie 
INUITS bvba 
Belgium 
+32 473 441 636 
https://inuits....
Upcoming SlideShare
Loading in …5
×

Augeas, swiss knife resources for your puppet tree

3,426 views

Published on

Talk given for the Belgian Puppet User Group meetup at Inuits office on the 12th November 2014.

Published in: Technology
  • Be the first to comment

Augeas, swiss knife resources for your puppet tree

  1. 1. . AAuuggeeaass SSwwiissss--kknniiffee rreessoouurrcceess ffoorr yyoouurr ppuuppppeett ttrreeee Julien Pivotto Belgian Puppet User Group Holiday is over Meetup!!! - November 12th, 2014
  2. 2. . wwhhooaammii JJuulliieenn PPiivvoottttoo • Open-Source consultant at inuits.eu • FOSS defender since 2004 • DevOps believer and evangelist • Puppet User since 2011 • @roidelapluie on twitter/github
  3. 3. . . ınuits.eu
  4. 4. . . SSyyssaaddmmiinn 110011 CC BY-SA 2.0 https://www.flickr.com/photos/arthur-caranta/2926332140
  5. 5. . SSeettttiinngg uupp aa sseerrvviiccee • Install the package • Change the configuration • Start the daemon
  6. 6. . 3 steps. What can go wrong?
  7. 7. . PPaacckkaaggiinngg • Where is the package? • Which version do we need? • Does it conflict with something else?
  8. 8. . . DDeeppeennddeenncciieess HHeellll CC BY-SA 2.0 https://www.flickr.com/photos/coconinonationalforest/4587053982
  9. 9. . CCoonnffiigguurraattiioonn • Where is the file? • How many files? • Configuration is in the database? • The file is *huge*
  10. 10. . SSttaarrttiinngg tthhee sseerrvviiccee • Does not start ▶ Bad config file ▶ Stale lock file ▶ Data corruption • High Availability • Replication
  11. 11. . LLeett''ss ttaallkk aabboouutt PPuuppppeett aanndd ffiilleess • Classical approach: File[] resource • Advanced approach: Concat[] define • Broken approach: Exec[sed] resource • Surgical approach: Augeas[] resource
  12. 12. . HHiiddddeenn wwaayyss ttoommaannaaggee ffiilleess • Ssh_authorized_key[] • Nagios_* • To purge or not to purge
  13. 13. . . TThhee FFiillee[[]] rreessoouurrccee CC BY 2.0 https://www.flickr.com/photos/80497449@N04/10567875696/
  14. 14. . FFiillee • Built-in puppet resource • Most used • Works with a lot of usecases • Text files, binary files
  15. 15. . . file{"${::icinga::confdir_server}/cgi.cfg": ensure => present, content => template('icinga/redhat/cgi.cfg.erb'), owner => $::icinga::server_user, group => $::icinga::server_group, require => Class['icinga::config'], notify => [ Service[$::icinga::service_client], Service[$::icinga::service_server], Exec['fix_collected_permissions'] ], } .
  16. 16. . CCoonntteenntt ooff aa ffiillee • content => String, template(), file() • source => puppet:///, /local/file
  17. 17. . FFiillee[[]] bbeehhaavviioouurr • Array as "source": Puppet will pick the first available one • Multiple arguments to template(): Puppet will concatenate them all • File[/foo/bar] will autorequire File[/foo]
  18. 18. . DDoowwnnssiiddee ooff FFiillee[[]] • You can only have at one "content" • That resource describe the whole file • Works in almost every situation
  19. 19. . . concat Public Domain http://commons.wikimedia.org/wiki/File:Adhesive_tapes_clear.JPG
  20. 20. . CCoonnccaatt • A "reference" puppet module: puppetlabs/concat • https://github.com/puppetlabs/puppetlabs-concat • Provides definitions to manage file • Alternative modules: ▶ onyxpoint/pupmod-concat ▶ theforeman/puppet-concat (fork of onyxpoint)
  21. 21. . CCoonnccaatt?? • Concat takes a bunch of snippets • Assemble them info a file • Each snippet is a define • The final file is a define
  22. 22. . . concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target => '/tmp/file', content => 'test contents', order => '01' } .
  23. 23. . BBaassee aanndd ffrraaggmmeennttss • Concat[] defines owner, ensure, mode of the file • Concat::Fragment[] defines the contents of the file • One Concat[] has multiple Concat::Fragment[]
  24. 24. . AAddvvaannttaaggeess ooff ccoonnccaatt • More flexibility ▶ if ▶ virtual resource ▶ exported resources ▶ create_resources • Mix templates and files
  25. 25. . DDiissaaddvvaannttaaggeess ooff ccoonnccaatt • External Puppet module • Concat[] is the whole file • Performances
  26. 26. . . Exec{sed: onlyif => grep} CC BY-SA 3.0 http://commons.wikimedia.org/wiki/File:Ca%C3%AFn_par_Henri_Vidal.jpg
  27. 27. . . https://github.com/search?o=desc&q=exec+sed+onlyif+grep+language%3APuppet
  28. 28. . eexxeecc[[sseedd]] iiss bbrr00kkeenn • Which options to pass to sed and grep? • You should use as few Exec[] as possible • grep .... • Escape, regexes…
  29. 29. . AAnnootthheerr aalltteerrnnaattiivvee:: ccoonnff..dd • Some services support conf.d directories • But it is hard to change existing parameters • In which order are the files read? • Don't forget to purge
  30. 30. . . Augeas CC BY-SA 3.0 http://commons.wikimedia.org/wiki/File:Students_assisting_surgery.JPG
  31. 31. . AAuuggeeaass • Configuration editing tool • First release in 2007 • API coded in C • Command-line tools • bindings for different languages
  32. 32. . CCoonnffiigguurraattiioonn eeddiittiinngg ttooooll • Parsing the configuration files • Turning them into a tree • Edit the tree & save the configuration
  33. 33. . . $ cat /etc/nsswitch.conf # /etc/nsswitch.conf ## Example configuration # passwd: db files group: db files initgroups: db [SUCCESS=continue] files shadow: db files gshadow: files .
  34. 34. . . augtool> ls /files/etc/nsswitch.conf/ #comment[1] = /etc/nsswitch.conf #comment[2] = Example configuration database[1]/ = passwd database[2]/ = group database[3]/ = initgroups database[4]/ = shadow database[5]/ = gshadow .
  35. 35. . . augtool> ls /files/etc/nsswitch.conf/database[1]/ service[1] = db service[2] = files .
  36. 36. . NNaattiivvee ffoorrmmaatt -->> ttrreeee • Augeas understand comments • Augeas does not care about empty lines • The cli tool (augtool) has autocomplete • It recognize a lot of formats
  37. 37. . . augtool> set /files/etc/nsswitch.conf/database[1]/ service[last()+1] ldap augtool> save Saved 1 file(s) .
  38. 38. . . $ cat /etc/nsswitch.conf # /etc/nsswitch.conf ## Example configuration # passwd: db files ldap group: db files initgroups: db [SUCCESS=continue] files shadow: db files gshadow: files .
  39. 39. . . augtool> match /files/etc/nsswitch.conf/*/* ldap /files/etc/nsswitch.conf/database[1]/service[3] augtool> print /files/etc/nsswitch.conf/database[1] /files/etc/nsswitch.conf/database[1] = "passwd" /files/etc/nsswitch.conf/database[1]/service[1] = "db" /files/etc/nsswitch.conf/database[1]/service[2] = "files" /files/etc/nsswitch.conf/database[1]/service[3] = "ldap" .
  40. 40. . . augtool> rm /files/etc/nsswitch.conf/database[1]/service[3] rm : /files/etc/nsswitch.conf/database[1]/service[3] 1 augtool> print /files/etc/nsswitch.conf/database[1] /files/etc/nsswitch.conf/database[1] = "passwd" /files/etc/nsswitch.conf/database[1]/service[1] = "db" /files/etc/nsswitch.conf/database[1]/service[2] = "files" augtool> save Saved 1 file(s) .
  41. 41. . OOnnee AAPPII ttoo eeddiitt tthheemmaallll • Can talk XML, ini, named, nginx, … • Only change what is needed • Ensure the syntax is right
  42. 42. . AAuuggeeaass LLeennsseess • Lenses are files that explain how to edit files • It contains paths and syntax • There are a lot of them available • You can write your own lenses
  43. 43. . ”This brings the total number of lenses to 178. […] It’s depressing to think that Linux/Unix systems have managed to grow this many special snowflake formats.” David Lutterkort, main developer about Augeas 1.3.0
  44. 44. . 117788 lleennsseess activemq_conf activemq_xml aliases aptconf apt_update_manager backuppchosts bbhosts bootconf build carbon cgrules channels cobblermodules cobblersettings collectd crypttab cyrus_imapd darkice debctrl desktop device_map dhcpd dnsmasq dovecot dpkg dput ethers exports fai_diskconfig fonts fuse gdm grub gshadow hostname inetd inputrc interfaces iproute2 iptables jaas jmxaccess keepalived known_hosts koji krb5 ldif limits login_defs logrotate mcollective memcached mke2fs mongodbserver mysql nagioscfg nagiosobjects netmasks nginx ntp ntpd odbc openshift_config openshift_http openvpn pam passwd pbuilder postfix_main postfix_transport postfix_virtual puppet_auth qpid rabbitmq resolv rmt securetty sep services shells shellvars_list sip_conf slapd smbusers squid sshd stunnel subversion sudoers sysconfig systemd thttpd up2date vfstab
  45. 45. . AA sshhoorrtt lleennssee . module Hostname = autoload xfm (* View: lns *) let lns = [ label "hostname" . store Rx.word . Util.eol ] (* View: filter *) let filter = incl "/etc/hostname" . incl "/etc/mailname" let xfm = transform lns filter .
  46. 46. . PPuuppppeett <<33 aauuggeeaass • Native "augeas" resource • Support for pluginsync • Helpers available
  47. 47. . PPuuppppeett eexxaammppllee . augeas { $name: context => "/files${fstab::variables::fstab_file}", changes => [ "rm ${fstab_match_line}", ], onlyif => "match ${fstab_match_line} size > 0" } .
  48. 48. . RReeaall uusseeccaasseess • Change grub options • Modify /etc/hosts • Modify XML's (puppetlabs-tomcat) • Configure Jenkins
  49. 49. . PPlluuggiinnssyynncc • Puppet has pluginsync support for Augeas • Drop your lenses in your modules • lib/augeas/lenses • Use the "lens" parameter of the augeas resource
  50. 50. . PPuuppppeett eexxaammppllee . augeas{"jboss_conf": . context => "/files/etc/jbossas", changes => [ "set jbossas.conf/JBOSS_IP $ipaddress", "set jbossas.conf/JAVA_HOME /usr", ], lens => "Jboss.aug", }
  51. 51. . AAuuggeeaass ccoommmmaannddss set rm mv clear insert …
  52. 52. . AAuuggeeaass ccoommppaarraattoorrss ((oonnllyyiiff)) match get
  53. 53. . AAuuggeeaasspprroovviiddeerrss • Helpers around augeas • Puppet modules • No augeas knowledge needed
  54. 54. . aappaacchhee . apache_setenv { "SPECIAL_PATH": ensure => present, value => "/foo/bin", } .
  55. 55. . kkeerrnneell__ppaarraammeetteerr . kernel_parameter { "quiet": ensure => present, bootmode => "normal", } .
  56. 56. . Conclusion
  57. 57. . DDiissaaddvvaannttaaggeess • Learning required • Library to install • Writing lenses is hard
  58. 58. . AAddvvaannttaaggeess • Augeas is a mature tool • Preserves comments in files • It fails (if needed) • Only changes what is needed • A lot of lenses available • Puppet integration • Helpers available
  59. 59. . FFiinnaall nnoottee Most of the time, File[] resources are the way to go. Augeas can help when you need to change files generated by an application or that you can not manage entirely.
  60. 60. . RReeaaddiinnggss • http://augeas.net/ • http://augeasproviders.com/ • https://docs.puppetlabs.com/
  61. 61. . TThhaannkk yyoouu Any question? Thanks to @raphink
  62. 62. . CCoonnttaacctt Julien Pivotto julien@inuits.eu @roidelapluie INUITS bvba Belgium +32 473 441 636 https://inuits.eu

×