Troubleshooting Puppet talk I gave at PuppetCamp Seattle 2014.

1. Troubleshooting
Puppet
Thomas Uphill
thomas@narrabilis.com

2. Latest version
http://goo.gl/b2NISc
If you see something,
say something!
Google Comments enabled

3. Watch in Presentation Mode!
Seriously, Trust Me
...Animations Are Good things

4. Me
PuppetConf 2013
Mastering Puppet / Puppet Cookbook 4*
PUGS
/r/dadjokes

5. Outline
network
rest api
certificates
catalog
problem workers
compiling
printing
scope

7. Failures
- can't find puppet
- can't connect to puppet
- can't get certificate
- can't get/compile a catalog
- can't apply a catalog
- can't upgrade puppet

8. can't connect to puppet...

9. can't find puppet
puppet
puppet.domain
[main/agent]
server
ca_server
DNS
nsswitch.conf
/etc/hosts

10. can't connect to puppet
node
puppet
master
network
pixies
8140
masterport

11. can't connect to puppet
● ping
● mtr (--port 8140)
● netcat (nc)

12. can't connect to puppet
$ ping puppet
ping: unknown host puppet
$ mtr puppet My traceroute [v0.75]
cookbook.example.com (0.0.0.0) Sat Oct 25 01:26:31 2014
Keys: Help Display mode Restart statistics Order of fields quit
$ nc -v puppet 8140
Packets Pings
Ncat: Version 6.45 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.122.100:8140.
Host Loss% Snt Last Avg Best Wrst StDev
1. puppet.example.com 0.0% 157 0.7 0.5 0.2 1.6 0.2

13. can't get certificate
● already signed (clean)
● dates off - expired CA, expired cert
puppet cert clean host.example.com
rm /var/lib/puppet/ssl/*/hostname*
openssl x509 -in cert.pem -text
ntpq -p

14. can't get certificate
● don't know your own name
● basic unix permissions
● something else...
root@puppet:~# sudo -iu puppet
puppet@puppet:~$ cd /etc/puppet/environments/
-bash: cd: /etc/puppet/environments/: Permission denied
SELinux

15. REST API

16. Rest API
node
puppet
master
8140
GET https://puppet:8140/production/certificate/ca HTTP/1.1
-----BEGIN CERTIFICATE -----
MIIFXjCCA0agAwIBAgIBATANBgkqhkiG9w0BAqsFADAcMRowGAYDVQQDDBFQdXdb
w
FiyvryAxxuETs9KORMwoThDIMd4bHLDbqtAz0q0cJ7W/8w==
-----END CERTIFICATE-----

17. Rest API
ca_server/server
environment
GET https://puppet:8140/production/certificate/ca HTTP/1.1
ca_port/masterport
resource
key

18. Rest API
GET https://puppet:8140/production/certificate/ca HTTP/1.1
resource
key
resource
certificate
file_metadata
file_content
node
catalog

19. Rest API
● wget
● curl
# diff puppet-ca.pem /var/lib/puppet/ssl/ca.pem
# echo $?
0
# curl --insecure https://puppet:8140/production/certificate/ca >puppet-ca.pem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
101 1923 101 1923 0 0 25506 0 --:--:-- --:--:-- --:--:-- 312k
# openssl x509 -in puppet-ca.pem -fingerprint
SHA1 Fingerprint=5A:C2:03:7B:40:44:2C:81:45:81:07:11:D3:AC:29:FB:A5:EC:E3:55

20. Rest API
# curl --cacert puppet-ca.pem
https://puppet:8140/production/certificate/lisa.example.com
Not Found: Could not find certificate lisa.example.com
# curl --cacert puppet-ca.pem
https://puppet:8140/production/certificate/cookbook.example.com
-----BEGIN CERTIFICATE-----
MIIFcTCCA1mgAwIBAgIBCjANBgkqhkiG9w0BAQsFADAcMRowGAYDVQQDDBFQdXBw
ZXQgQ0E6IHB1cHBldDAeFw0xNDA5MzAwNDM1MDdaFw0xOTA5MzAwNDM1MDdaMB8x
# openssl x5.0.9. -in cookbook.pem -text
…
Issuer: CN=Puppet CA: puppet
Validity
Not Before: Sep 30 04:35:07 2014 GMT
Not After : Sep 30 04:35:07 2019 GMT
Subject: CN=cookbook.example.com
…

21. Catalog
# curl --cacert puppet-ca.pem
--cert /var/lib/puppet/ssl/certs/cookbook.example.com.pem
--key /var/lib/puppet/ssl/private_keys/cookbook.example.com.pem
-H 'Accept: yaml'
https://puppet:8140/production/catalog/cookbook.example.com > cookbook.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 26099 100 26099 0 0 17434 0 0:00:01 0:00:01 --:--:-- 18642

22. Tools (gnutls-cli or openssl s_client)
$ gnutls-cli --insecure --port 8140 puppet.example.com
GET /production/certificate/ca HTTP/1.0
Accept: s
HTTP/1.1 200
Server: nginx/1.6.1
Date: Thu, 23 Oct 2014 21:03:58 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 875
Connection: close
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.11
Status: 200

23. can't get/compile a
catalog

24. Problem workers
node
puppet
worker
8140
puppet
worker
8140
mod_proxy_balancer
puppet
master
8140
devel
puppet
worker
8140
problem
production

25. problem/bugfixes branches
git branches
● per user
● per problem
● per ticket

26. Problem workers
puppet
worker
8140
--logdest /var/log/puppet/problem.log
--debug
--profile
logrotate

27. Compiling
$ sudo puppet master
Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding
file resource 'bucketdir':
'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet",
:ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false,
:mode=>"750", :path=>"/var/lib/puppet/bucket"}'
Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding
file resource 'publickeydir':
'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug,
:group=>"puppet", :ensure=>:directory, :links=>:follow,
:owner=>"puppet", :backup=>false, :mode=>"755",
:path=>"/var/lib/puppet/ssl/public_keys"}'
Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding
file resource 'plugindest':
'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory,
:links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}'
Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding
file resource 'fileserverconfig':
'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file,
:links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}'
--compile problem.example.com
--debug --trace
--logdest /tmp/problem.puppet.log
--environment sandbox
{
"data": {
"resources": [
{
"title": "main",
"exported": false,
"tags": ["stage"],
"type": "Stage",
"parameters": {
JSON
"before": "Stage[post]",

28. Apply
$ sudo puppet apply
--debug --trace
--environment sandbox code.pp
JSON
$date = "+%S"
exec {'epoch':
command => "echo $((`date $date` / 86400))",
path => '/bin:/usr/bin:/sbin:/usr/sbin',
}
Info: Applying configuration version '1415729233'
Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` /
86400))'
Debug: Executing 'echo $((`date +%S` / 86400))'
Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully

29. Catalog
YAML - read it
JSON - jq

30. Catalog
catalog classes.txt
jq '.data.classes[]' < catalog

31. Catalog
fails to compile
● duplicate resource
● modulepath/bad module name
fails to apply
● unpredicable exec
● bad/broken service
● bad/missing variable

32. fails to compile

33. duplicate resource
separate into subclass ( package {'httpd'} )
virtual resources ( @user, @package, @service)
modulepath
puppet config print modulepath
root@puppet:~# puppet config print modulepath --environment production
/etc/puppet/environments/production/public:/etc/puppet/environments/producti
on/modules
root@puppet:~# puppet config print modulepath --environment master
/etc/puppet/modules:/usr/share/puppet/modules

34. fails to apply

35. Unpredictable exec
#!/bin/bash
echo $JAVA_HOME
/home/javadev/.bashrc
JAVA_HOME=/your/face

36. Broken Service
service provider
hasstatus => true
/sbin/service $service status
/etc/init.d/$service status
/usr/bin/systemctl is-active $service

37. Bad/Missing Variable
$one = "1"
file {"lisaone":
Info: Caching catalog for node1.example.com
Error: path Failed => "/to tmp/apply lisa$catalog: one",
Cannot alias File[lisa1] to
["/ensure tmp/lisa1"] => 'directory',
at
/}
etc/puppet/environments/production/manifests/site.pp:34;
resource file {"lisa1":
["File", "/tmp/lisa1"] already declared at
/etc/puppet/environments/production/manifests/site.pp:30
path => "/tmp/lisa1",
ensure => 'file',
}

38. Bad/Missing Variable
lisa {'one':
place => "/tmp/$LISA",
type => "directory",
Info: Caching catalog for node1.example.com
Error: Failed to apply catalog: Cannot alias File[two] to
["/tmp"] at
/etc/puppet/environments/production/modules/lisa/manifests/i
nit.pp:5; resource ["File", "/tmp"] already declared at
/etc/puppet/environments/production/modules/lisa/manifests/i
nit.pp:5
}
lisa {'two':
place => "/tmp/$LISA",
type => "file",
}
define lisa ($place,$type) {
file {"$title":
path => $place,
ensure => $type,
}
}

39. Printing - Notify
notify {"$variable": }

40. chaining
notify {'something':
}->exec{'thingthatfails':
}->notify{'after': }

41. Checking
exec{'before resolv.conf':
command => '/usr/local/bin/puppet-debug before resolv.conf',
require => Class['debug']
} -> file { '/etc/resolv.conf':
source => template("dns/resolv.conf"),
noop => true,
}
class debug {
file {'puppet-debug':
path => '/usr/local/bin/puppet-debug',
source => 'puppet:///modules/debug/puppet-debug',
mode => 0755,
}
}

42. Debug Script… just an example
#!/bin/bash
LOG=$(mktemp /tmp/puppet-debug.XXXXXX)
echo Puppet Debug -- $@ -- $(date) | tee $LOG
echo "-- Disk --" | tee -a $LOG
df -h |tee -a $LOG
df -i |tee -a $LOG
echo "-- Mem --" | tee -a $LOG
free | tee -a $LOG
echo "-- Files --" | tee -a $LOG
PUPPET=$(pgrep puppet)
for proc in $PUPPET
do
lsof -p $proc |tee -a $LOG
done
Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014
-- Disk --
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
6.7G 2.5G 3.9G 39% /
tmpfs 246M 0 246M 0% /dev/shm
/dev/vda1 485M 80M 380M 18% /boot
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
440640 79253 361387 18% /
tmpfs 62783 1 62782 1% /dev/shm
/dev/vda1 128016 50 127966 1% /boot
-- Mem --
total used free shared buffers cached
Mem: 502268 415488 86780 0 22176 172036
-/+ buffers/cache: 221276 280992
Swap: 835580 0 835580
-- Files --
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
puppet 2058 root cwd DIR 253,0 4096 14 /root
puppet 2058 root rtd DIR 253,0 4096 2 /
puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/ruby
puppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.so
puppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so

43. Printing - Template
- scope.to_hash
- reject a few
file { "/tmp/puppet-debug.txt":
content => inline_template("<% vars =
- sort
scope.to_hash.reject { |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%=
v %>n<% end %>"),
- print, one per line
}
vars = scope.to_hash.reject
{ |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) };
vars.sort.each do |k,v|
k=vn
end

44. _timestamp=2014-10-23 22:29:52 -0700
architecture=x86_64
augeasversion=1.0.0
bios_release_date=01/01/2011
bios_vendor=Bochs
bios_version=Bochs
blockdevice_vda_size=8589934592
blockdevice_vda_vendor=6900
blockdevices=vda
caller_module_name=
clientcert=cookbook.example.com
clientnoop=false
clientversion=3.7.1
concat_basedir=/var/lib/puppet/concat
domain=example.com
environment=production
facterversion=2.2.0
filesystems=ext4,iso9660
fqdn=cookbook.example.com
gid=root
hardwareisa=x86_64
hardwaremodel=x86_64
hostname=cookbook
id=root
interfaces=eth0,lo
Printing - Template
- scope.to_hash
- reject a few
file { "/tmp/puppet-debug.txt":
content => inline_template("<% vars =
- sort
scope.to_hash.reject { |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%=
v %>n<% end %>"),
- print, one per line
}
vars = scope.to_hash.reject
{ |k,v| !( k.is_a?(String) && v.is_a?(String) ) };
vars.sort.each do |k,v|
k=vn
end

45. Scope
The scene:
class ntp {
include ntp::server
● roles and profiles
● ntp server
class role::ntp {
include ntp
}
}

46. Scope
The solution:
class ntp {
include ntp::server
● fully scope everything
● remember scope
class role::ntp {
include ::ntp
}
}

47. Summary
learn some networking
remember the REST api
read up on SSL / x509
use --trace
make a debug class
remember scope

48. Questions/Comments?
http://goo.gl/b2NISc
http://ramblings.narrabilis.com