PuppetConf 2016: Puppet Troubleshooting – Thomas Uphill, Wells Fargo
•
3 likes•1,440 views
Here are the slides from Thomas Uphill's presentation called Puppet Troubleshooting. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to PuppetConf 2016: Puppet Troubleshooting – Thomas Uphill, Wells Fargo
Similar to PuppetConf 2016: Puppet Troubleshooting – Thomas Uphill, Wells Fargo (20)
More from Puppet
More from Puppet (20)
Recently uploaded
Recently uploaded (20)
PuppetConf 2016: Puppet Troubleshooting – Thomas Uphill, Wells Fargo
- 2. https://goo.gl/8LyZzN Mastering Puppet / Puppet Cookbook Packt Pub
- 3. https://goo.gl/8LyZzN LOPSA / SASAG / PUGS
- 4. When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth Sir Arthur Conan Doyle 4
- 5. https://goo.gl/8LyZzN Troubleshooting 101 ● document current state ● discover recent changes ○ audit everything, even things you don't touch ○ never assume it's magic ● change one thing at a time ○ if it doesn't fix the problem, revert it 5 1 une uno um unis odin jeden jedward yksi 一 एक אחד واﺣد
- 6. https://goo.gl/8LyZzN Troubleshooting 101 ● don't be afraid of diving deep ○ you can't fix what you don't understand ● use low level tools 6
- 7. https://goo.gl/8LyZzN TAKE HOME puppet is an HTTPS service 7
- 9. https://goo.gl/8LyZzN can't find puppet $ puppet config print server yourface $ puppet config print confdir /home/thomas/.puppetlabs/etc/puppet # puppet config print server puppet # puppet config print ca_server puppet # puppet config print confdir /etc/puppetlabs/puppet 9
- 10. puppet agent Runs as root… puppetserver Runs as puppet. 10 Basic UNIX permissions
- 12. https://goo.gl/8LyZzN can't find puppet dns files /etc/hosts nsswitch.conf # host puppet Host puppet not found: 3(NXDOMAIN) # nslookup puppet Server: 10.0.2.3 Address: 10.0.2.3#53 ** server can't find puppet: NXDOMAIN # dig puppet # getent hosts puppet 127.0.0.1 puppet.example.com puppet localhost localhost.localdomain localhost4 localhost4.localdomain4 # ping puppet PING puppet.example.com (127.0.0.1) 56(84) bytes of data. 64 bytes from puppet.example.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms 12
- 13. https://goo.gl/8LyZzN can't connect to puppet node puppet server network pixies 8140 server masterport 13
- 14. https://goo.gl/8LyZzN can't connect to puppet Verify you can connect: # nc -v puppet.example.com 8140 Ncat: Version 7.12 ( https://nmap.org/ncat ) Ncat: Connected to 192.168.1.1:8140 14
- 15. https://goo.gl/8LyZzN can't connect to puppet Q: How can you know you are talking to puppet? A: Use the REST API 15
- 16. https://goo.gl/8LyZzN can't connect to puppet $ curl -k https://puppet.example.com:8140/puppet-ca/v1/certificate/ca?environment=production -----BEGIN CERTIFICATE----- MIIFhDCCA2ygAwIBAgIBATANBgkqhkiG9w0BAQsFADAoMSYwJAYDVQQDDB1QdXBw ZXQgQ0E6IHB1cHBldC5leGFtcGxlLmNvbTAeFw0xNjEwMDgyMDEyNDRaFw0yMTEw … qsHOzPEQwJS5q1yPciBQ5jsKCB5vXJEdnPMhA5GBx9rp91hBsiWURlyWt0DMqMTq VqcUEGNc267LrCzKJsFBHCNO53e8z2I0 -----END CERTIFICATE----- masterportca_server puppet or puppet-ca api endpoint key environment https://docs.puppet.com/puppet/latest/reference/http_api/http_api_index.html 16
- 17. https://goo.gl/8LyZzN can't connect to puppet $ gnutls-cli --port 8140 puppet.example.com --x509cafile=ca.pem $ openssl s_client -connect puppet.example.com:8140 GET /path/to/thing/you/want Accept: PSON/YAML/Text/??? puppet master (puppetserver) nodeHTTPS/8140 17 lsof -i :8140 mtr traceroute ping
- 18. https://goo.gl/8LyZzN in order to authenticate... you must first authenticate. 18
- 19. https://goo.gl/8LyZzN can't connect to puppet $ curl --cert certs/cottage.pem --key private_keys/cottage.pem.orig --cacert certs/ca.pem https://puppet.example.com:8140/puppet/v3/environments | jq … "environments": { "production": { "settings": { "modulepath": [ "/etc/puppetlabs/code/environments/production/modules", "/etc/puppetlabs/code/modules", "/opt/puppetlabs/puppet/modules" ], "manifest": "/etc/puppetlabs/code/environments/production/manifests", "environment_timeout": 0, "config_version": "" 19
- 20. https://goo.gl/8LyZzN API FTW $ curl --cacert certs/ca.pem --cert certs/cottage.pem --key private_keys/cottage.pem -H 'Accept: pson' https://puppet.example.com:8140/puppet/v3/catalog/cottage?environment=production > cottage.pson 20 catalog
- 21. https://goo.gl/8LyZzN PSON Puppet JSON → PSON $ jq <pson> $ curl … | jq JQ? sed for JSON $ jq .environment catalog.pson $ jq .classes catalog.pson $ jq '.resources | length' catalog.pson $ jq '.resources[] | select(.type == "Stage")' catalog.pson 21
- 22. https://goo.gl/8LyZzN PSON Puppet JSON → PSON $ jq <pson> $ curl … | jq JQ? sed for JSON $ jq .environment catalog.pson $ jq .classes catalog.pson $ jq '.resources | length' catalog.pson $ jq '.resources[] | select(.type == "Stage")' catalog.pson Sometimes it's good to go... 22
- 24. https://goo.gl/8LyZzN can't get certificate already signed (clean) dates off - expired CA, expired cert openssl x509 -in cert.pem -text puppet cert fingerprint host.example.com puppet cert print host.example.com puppet cert clean host.example.com ntpq -p chronyc sources rm /etc/puppetlabs/puppet/ssl/*/hostname* 24
- 28. https://goo.gl/8LyZzN x.509 Public Key Infrastructure Certificate Authority Trust https://www.flickr.com/photos/genvessel/431100596 28
- 29. https://goo.gl/8LyZzN ca.pem x.509 puppet master CERTIFICATE node CA? ca.pem server ca_server 29
- 31. https://goo.gl/8LyZzN puppet.pem ca.pem x.509 puppet master CERTIFICATE node server ca_server 31
- 32. https://goo.gl/8LyZzN OpenSSL View Certificate: openssl x509 -in ca.pem -text -noout |less Validity X509v3 Key Usage View Fingerprint: openssl x509 -in ca.pem -fingerprint -noout openssl x509 -in ca.pem -fingerprint -noout -sha256 SHA1 (default) Verify Certificate: openssl verify -CAfile ca.pem puppet.pem 32
- 33. https://goo.gl/8LyZzN OpenSSL (Puppet shortcuts) View Certificate: puppet cert print ca View Fingerprint: puppet cert fingerprint ca List Certificates: puppet cert list -a 33
- 35. https://goo.gl/8LyZzN CRL # openssl crl -in crl.pem -text -noout |grep Serial Serial Number: 02 # openssl x509 -in cert.pem -text -noout |grep Serial Serial Number: 3 (0x3) 35
- 36. https://goo.gl/8LyZzN CRL # cat ca_crt.pem ca_crl.pem >combined.pem # openssl verify -CAfile combined.pem -crl_check cottage.pem cottage.pem: OK # puppet cert clean cottage Notice: Revoked certificate with serial 3 Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/ca/signed/cottage.pem' Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/certs/cottage.pem' # cat ca_crl.pem ca_crt.pem >combined.pem again? # openssl verify -CAfile combined.pem -crl_check cottage.pem cottage.pem: CN = cottage error 23 at 0 depth lookup:certificate revoked 36
- 37. https://goo.gl/8LyZzN Modulus n = pq OpenSSL # openssl rsa -noout -modulus -in ca_key.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 # openssl x509 -noout -modulus -in ca_crt.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 BIG *RSE PRIME 37
- 38. https://goo.gl/8LyZzN Modulus n = pq OpenSSL # openssl rsa -noout -modulus -in ca_key.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 # openssl x509 -noout -modulus -in ca_crt.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 BIG *RSE PRIME $ puppet agent -t Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key. Certificate fingerprint: D4:D3:76:F1:6B:51:83:3C:4B:72:69:BF:BC:B0:80:94:79:75:1A:3B:D8:29:F5:EF:81:2C:44:3 5:21:93:CE:FD To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certificate. On the master: puppet cert clean cottage On the agent: 1a. On most platforms: find /home/thomas/.puppetlabs/etc/puppet/ssl -name cottage.pem -delete 1b. On Windows: del "homethomas.puppetlabsetcpuppetsslcertscottage.pem" /f 2. puppet agent -t 38 The certificate retrieved from the master does not match the agent's private key.
- 39. https://goo.gl/8LyZzN OpenSSL recap ● x509 view certificate check expiration check serial number ● crl revoked cert serial# ● verify verify cert with CA and/or CRL ● modulus how the cert was encrypted 39
- 42. https://goo.gl/8LyZzN problem/bugfixes environments configure r10k to make environments automatically environment.conf - modulepath (debug module) ← what is that? 42
- 43. https://goo.gl/8LyZzN Compiling $ sudo puppet master --compile problem.example.com --debug --trace --logdest /tmp/problem.puppet.log --environment sandbox { "data": { "resources": [ { "title": "main", "exported": false, "tags": ["stage"], "type": "Stage", "parameters": { "before": "Stage[post]", JSON Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'bucketdir': 'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/bucket"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'fileserverconfig': 'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}' 43
- 44. https://goo.gl/8LyZzN Apply $ sudo puppet apply --debug --trace --environment sandbox code.pp $date = "+%S" exec {'epoch': command => "echo $((`date $date` / 86400))", path => '/bin:/usr/bin:/sbin:/usr/sbin', } Info: Applying configuration version '1415729233' Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` / 86400))' Debug: Executing 'echo $((`date +%S` / 86400))' Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully 44
- 47. catalog fails to compile duplicate resource modulepath/bad module name fails to apply unpredictable exec bad/broken service bad/missing variable 47
- 49. https://goo.gl/8LyZzN help yourself function verb() { erb -P -x -T '-' $1 | ruby -c is_ok $? } function vyaml() { ruby -ryaml -e "YAML.load_file '$1'" is_ok $? } function ppv() { puppet parser validate $* is_ok $? } 49 function vepp() { puppet epp validate $* is_ok $? } function is_ok() { if [ $1 == 0 ]; then if [[ "$TERM" == *"xterm"* ]] || [[ "$TERM" == *"vt100"* ]]; then echo -e "033[1mSyntax 033[32;1mOk!033[0m" else echo "Syntax Ok!" fi else return $1 fi }
- 50. https://goo.gl/8LyZzN duplicate resource separate into subclass ( package {'httpd'} ) virtual resources ( @user, @package, @service) modulepath puppet config print modulepath root@puppet:~# puppet config print modulepath --environment production /etc/puppet/environments/production/public:/etc/puppet/environments/production /modules root@puppet:~# puppet config print modulepath --environment master /etc/puppet/modules:/usr/share/puppet/modules 50
- 53. https://goo.gl/8LyZzN Unpredictable exec #!/bin/bash echo $JAVA_HOME /home/javadev/.bashrc JAVA_HOME=/your/face It works for me! puppet runs as puppet 53
- 54. https://goo.gl/8LyZzN Broken Service service provider hasstatus => true /sbin/service $service status /etc/init.d/$service status /usr/bin/systemctl is-active $service 54
- 55. https://goo.gl/8LyZzN Bad/Missing Variable $one = "1" file {"pcone": path => "/tmp/pc$one", ensure => 'directory', } file {"pc1": path => "/tmp/pc1", ensure => 'file', } Info: Caching catalog for puppet.example.com Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[pc1] to ["/tmp/pc1"] at /root/pc.pp:6; resource ["File", "/tmp/pc1"] already declared at /root/pc.pp:2 at /root/pc.pp:6:3 on node puppet.example.com 55
- 56. https://goo.gl/8LyZzN Bad/Missing Variable $PC = 'puppetconf' pc {'one': place => "/tmp/$PC", type => "directory", } pc {'two': place => "/tmp/$PC", type => "file", } define pc ( String $place, String $type, ) { file {"$title": path => $place, ensure => $type, } } Info: Caching catalog for puppet.example.com Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[two] to ["/tmp/puppetconf"] at /root/define.pp:5; resource ["File", "/tmp/puppetconf"] already declared at /root/define.pp:5 at /root/define.pp:5:2 at /root/define.pp:15 on node puppet.example.com 56
- 57. https://goo.gl/8LyZzN Printing - Notify notify {"$variable": } 57
- 59. https://goo.gl/8LyZzN Checking exec{'before resolv.conf': command => '/usr/local/bin/puppet-debug before resolv.conf', require => Class['debug'] } -> file { '/etc/resolv.conf': source => template("dns/resolv.conf"), noop => true, } class debug { file {'puppet-debug': path => '/usr/local/bin/puppet-debug', source => 'puppet:///modules/debug/puppet-debug', mode => 0755, } } 59
- 60. https://goo.gl/8LyZzN Debug Script… just an example #!/bin/bash LOG=$(mktemp /tmp/puppet-debug.XXXXXX) echo Puppet Debug -- $@ -- $(date) | tee $LOG echo "-- Disk --" | tee -a $LOG df -h |tee -a $LOG df -i |tee -a $LOG echo "-- Mem --" | tee -a $LOG free | tee -a $LOG echo "-- Files --" | tee -a $LOG PUPPET=$(pgrep puppet) for proc in $PUPPET do lsof -p $proc |tee -a $LOG done Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014 -- Disk -- Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 6.7G 2.5G 3.9G 39% / tmpfs 246M 0 246M 0% /dev/shm /dev/vda1 485M 80M 380M 18% /boot Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VolGroup-lv_root 440640 79253 361387 18% / tmpfs 62783 1 62782 1% /dev/shm /dev/vda1 128016 50 127966 1% /boot -- Mem -- total used free shared buffers cached Mem: 502268 415488 86780 0 22176 172036 -/+ buffers/cache: 221276 280992 Swap: 835580 0 835580 -- Files -- COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME puppet 2058 root cwd DIR 253,0 4096 14 /root puppet 2058 root rtd DIR 253,0 4096 2 / puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/ruby puppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.so puppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so 60
- 61. https://goo.gl/8LyZzN Printing - Template - scope.to_hash - reject a few - sort - print, one per line file { "/tmp/puppet-debug.txt": content => inline_template("<% vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>n<% end %>"), } vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| k=vn end 61
- 62. https://goo.gl/8LyZzN Printing - Template _timestamp=2014-10-23 22:29:52 -0700 architecture=x86_64 augeasversion=1.0.0 bios_release_date=01/01/2011 bios_vendor=Bochs bios_version=Bochs blockdevice_vda_size=8589934592 blockdevice_vda_vendor=6900 blockdevices=vda caller_module_name= clientcert=cookbook.example.com clientnoop=false clientversion=3.7.1 concat_basedir=/var/lib/puppet/concat domain=example.com environment=production facterversion=2.2.0 filesystems=ext4,iso9660 fqdn=cookbook.example.com gid=root hardwareisa=x86_64 hardwaremodel=x86_64 hostname=cookbook id=root interfaces=eth0,lo 62
- 64. https://goo.gl/8LyZzN Scope The scene: roles and profiles ntp server class role::ntp { include ntp } class ntp { include ntp::server } 64
- 65. https://goo.gl/8LyZzN Scope The solution: fully scope everything remember scope class role::ntp { include ::ntp } class ntp { include ntp::server } 65
- 67. https://goo.gl/8LyZzN pry IRB replacement REPL available at runtime 67 pryrepl.org require 'pry' … # amazing code here # wow, much amaze … binding.pry …
- 68. https://goo.gl/8LyZzN pry demo 68 module Puppet::Parser::Functions newfunction(:pry) do |args| require 'pry' binding.pry end end modules/pry/lib/puppet/parser/functions/pry.rb node default { … pry() … } manifests/site.pp
- 69. https://goo.gl/8LyZzN pry demo # 69 # … From: /etc/puppetlabs/code/environments/production/modules/pry /lib/puppet/parser/functions/pry.rb @ line 4 #<Module:0xfb588d1>#real_function_pry: 2: newfunction(:pry) do |args| 3: require 'pry' => 4: binding.pry 5: end [1] pry(#<Puppet::Parser::Scope>)> puppet agent -tpuppetserver foreground … Puppet Server has successfully started and is now ready to handle requests exit … Puppet Compiled Catalog for xxx.example.com in y.z seconds Info: Caching catalog for xxx.example.com Info: Applying configuration version 'XXX'
- 70. https://goo.gl/8LyZzN where to go for help ● IRC #puppet / #puppet-dev ● slack puppetcommunity.slack.com #pug #puppet ● google group / mail list https://groups.google.com/forum/#!forum/puppet-users ● PUG https://www.meetup.com/Seattle-Puppet-Meetup/ 70
- 71. https://goo.gl/8LyZzN Summary Puppet is an HTTPS service End-to-end (gethostbyname, nc mtr) OpenSSL is your friend (x509,crl, verify, s_client make a debug class remember scope basic UNIX permissions 71
- 72. 72