Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PuppetConf 2016: Puppet Troubleshooting – Thomas Uphill, Wells Fargo

639 views

Published on

Here are the slides from Thomas Uphill's presentation called Puppet Troubleshooting. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Published in: Technology
  • Be the first to comment

PuppetConf 2016: Puppet Troubleshooting – Thomas Uphill, Wells Fargo

  1. 1. https://goo.gl/8LyZzN puppet troubleshooting Thomas Uphill <thomas@narrabilis.com>
  2. 2. https://goo.gl/8LyZzN Mastering Puppet / Puppet Cookbook Packt Pub
  3. 3. https://goo.gl/8LyZzN LOPSA / SASAG / PUGS
  4. 4. When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth Sir Arthur Conan Doyle 4
  5. 5. https://goo.gl/8LyZzN Troubleshooting 101 ● document current state ● discover recent changes ○ audit everything, even things you don't touch ○ never assume it's magic ● change one thing at a time ○ if it doesn't fix the problem, revert it 5 1 une uno um unis odin jeden jedward yksi 一 एक ‫אחד‬ ‫واﺣد‬
  6. 6. https://goo.gl/8LyZzN Troubleshooting 101 ● don't be afraid of diving deep ○ you can't fix what you don't understand ● use low level tools 6
  7. 7. https://goo.gl/8LyZzN TAKE HOME puppet is an HTTPS service 7
  8. 8. problems connection certificate catalog compilation catalog application scope pry 8
  9. 9. https://goo.gl/8LyZzN can't find puppet $ puppet config print server yourface $ puppet config print confdir /home/thomas/.puppetlabs/etc/puppet # puppet config print server puppet # puppet config print ca_server puppet # puppet config print confdir /etc/puppetlabs/puppet 9
  10. 10. puppet agent Runs as root… puppetserver Runs as puppet. 10 Basic UNIX permissions
  11. 11. https://goo.gl/8LyZzN 11
  12. 12. https://goo.gl/8LyZzN can't find puppet dns files /etc/hosts nsswitch.conf # host puppet Host puppet not found: 3(NXDOMAIN) # nslookup puppet Server: 10.0.2.3 Address: 10.0.2.3#53 ** server can't find puppet: NXDOMAIN # dig puppet # getent hosts puppet 127.0.0.1 puppet.example.com puppet localhost localhost.localdomain localhost4 localhost4.localdomain4 # ping puppet PING puppet.example.com (127.0.0.1) 56(84) bytes of data. 64 bytes from puppet.example.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms 12
  13. 13. https://goo.gl/8LyZzN can't connect to puppet node puppet server network pixies 8140 server masterport 13
  14. 14. https://goo.gl/8LyZzN can't connect to puppet Verify you can connect: # nc -v puppet.example.com 8140 Ncat: Version 7.12 ( https://nmap.org/ncat ) Ncat: Connected to 192.168.1.1:8140 14
  15. 15. https://goo.gl/8LyZzN can't connect to puppet Q: How can you know you are talking to puppet? A: Use the REST API 15
  16. 16. https://goo.gl/8LyZzN can't connect to puppet $ curl -k https://puppet.example.com:8140/puppet-ca/v1/certificate/ca?environment=production -----BEGIN CERTIFICATE----- MIIFhDCCA2ygAwIBAgIBATANBgkqhkiG9w0BAQsFADAoMSYwJAYDVQQDDB1QdXBw ZXQgQ0E6IHB1cHBldC5leGFtcGxlLmNvbTAeFw0xNjEwMDgyMDEyNDRaFw0yMTEw … qsHOzPEQwJS5q1yPciBQ5jsKCB5vXJEdnPMhA5GBx9rp91hBsiWURlyWt0DMqMTq VqcUEGNc267LrCzKJsFBHCNO53e8z2I0 -----END CERTIFICATE----- masterportca_server puppet or puppet-ca api endpoint key environment https://docs.puppet.com/puppet/latest/reference/http_api/http_api_index.html 16
  17. 17. https://goo.gl/8LyZzN can't connect to puppet $ gnutls-cli --port 8140 puppet.example.com --x509cafile=ca.pem $ openssl s_client -connect puppet.example.com:8140 GET /path/to/thing/you/want Accept: PSON/YAML/Text/??? puppet master (puppetserver) nodeHTTPS/8140 17 lsof -i :8140 mtr traceroute ping
  18. 18. https://goo.gl/8LyZzN in order to authenticate... you must first authenticate. 18
  19. 19. https://goo.gl/8LyZzN can't connect to puppet $ curl --cert certs/cottage.pem --key private_keys/cottage.pem.orig --cacert certs/ca.pem https://puppet.example.com:8140/puppet/v3/environments | jq … "environments": { "production": { "settings": { "modulepath": [ "/etc/puppetlabs/code/environments/production/modules", "/etc/puppetlabs/code/modules", "/opt/puppetlabs/puppet/modules" ], "manifest": "/etc/puppetlabs/code/environments/production/manifests", "environment_timeout": 0, "config_version": "" 19
  20. 20. https://goo.gl/8LyZzN API FTW $ curl --cacert certs/ca.pem --cert certs/cottage.pem --key private_keys/cottage.pem -H 'Accept: pson' https://puppet.example.com:8140/puppet/v3/catalog/cottage?environment=production > cottage.pson 20 catalog
  21. 21. https://goo.gl/8LyZzN PSON Puppet JSON → PSON $ jq <pson> $ curl … | jq JQ? sed for JSON $ jq .environment catalog.pson $ jq .classes catalog.pson $ jq '.resources | length' catalog.pson $ jq '.resources[] | select(.type == "Stage")' catalog.pson 21
  22. 22. https://goo.gl/8LyZzN PSON Puppet JSON → PSON $ jq <pson> $ curl … | jq JQ? sed for JSON $ jq .environment catalog.pson $ jq .classes catalog.pson $ jq '.resources | length' catalog.pson $ jq '.resources[] | select(.type == "Stage")' catalog.pson Sometimes it's good to go... 22
  23. 23. problems connection certificate catalog compilation catalog application scope pry 23
  24. 24. https://goo.gl/8LyZzN can't get certificate already signed (clean) dates off - expired CA, expired cert openssl x509 -in cert.pem -text puppet cert fingerprint host.example.com puppet cert print host.example.com puppet cert clean host.example.com ntpq -p chronyc sources rm /etc/puppetlabs/puppet/ssl/*/hostname* 24
  25. 25. Intermission 25 x.509 the crash course
  26. 26. https://goo.gl/8LyZzN x.509 PKI CA SSL TLS CRL REQ RSA DSA DH PKCS7 PKCS8 PKCS12 MD2 MD5 SHA1 SHA256 SHA384 SHA512 PEM DER OID RFC KS TS BOB ALICE "ENCRYPTION""ENCRYPTION" 26
  27. 27. https://goo.gl/8LyZzN x.509 PKI CA SSL TLS CRL REQ RSA DSA DH PKCS7 PKCS8 PKCS12 MD2 MD5 SHA1 SHA256 SHA384 SHA512 PEM DER OID RFC KS TS BOB ALICE 27
  28. 28. https://goo.gl/8LyZzN x.509 Public Key Infrastructure Certificate Authority Trust https://www.flickr.com/photos/genvessel/431100596 28
  29. 29. https://goo.gl/8LyZzN ca.pem x.509 puppet master CERTIFICATE node CA? ca.pem server ca_server 29
  30. 30. https://goo.gl/8LyZzN node.crt ca.crt x.509 puppet master CERTIFICATE node node.csr node.crt server ca_server 30 cert me pls?
  31. 31. https://goo.gl/8LyZzN puppet.pem ca.pem x.509 puppet master CERTIFICATE node server ca_server 31
  32. 32. https://goo.gl/8LyZzN OpenSSL View Certificate: openssl x509 -in ca.pem -text -noout |less Validity X509v3 Key Usage View Fingerprint: openssl x509 -in ca.pem -fingerprint -noout openssl x509 -in ca.pem -fingerprint -noout -sha256 SHA1 (default) Verify Certificate: openssl verify -CAfile ca.pem puppet.pem 32
  33. 33. https://goo.gl/8LyZzN OpenSSL (Puppet shortcuts) View Certificate: puppet cert print ca View Fingerprint: puppet cert fingerprint ca List Certificates: puppet cert list -a 33
  34. 34. https://goo.gl/8LyZzN node.crt ca.crt CRL puppet master CERTIFICATE node crl? crl 34 crl
  35. 35. https://goo.gl/8LyZzN CRL # openssl crl -in crl.pem -text -noout |grep Serial Serial Number: 02 # openssl x509 -in cert.pem -text -noout |grep Serial Serial Number: 3 (0x3) 35
  36. 36. https://goo.gl/8LyZzN CRL # cat ca_crt.pem ca_crl.pem >combined.pem # openssl verify -CAfile combined.pem -crl_check cottage.pem cottage.pem: OK # puppet cert clean cottage Notice: Revoked certificate with serial 3 Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/ca/signed/cottage.pem' Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/certs/cottage.pem' # cat ca_crl.pem ca_crt.pem >combined.pem again? # openssl verify -CAfile combined.pem -crl_check cottage.pem cottage.pem: CN = cottage error 23 at 0 depth lookup:certificate revoked 36
  37. 37. https://goo.gl/8LyZzN Modulus n = pq OpenSSL # openssl rsa -noout -modulus -in ca_key.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 # openssl x509 -noout -modulus -in ca_crt.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 BIG *RSE PRIME 37
  38. 38. https://goo.gl/8LyZzN Modulus n = pq OpenSSL # openssl rsa -noout -modulus -in ca_key.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 # openssl x509 -noout -modulus -in ca_crt.pem |sha256sum 69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38 BIG *RSE PRIME $ puppet agent -t Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key. Certificate fingerprint: D4:D3:76:F1:6B:51:83:3C:4B:72:69:BF:BC:B0:80:94:79:75:1A:3B:D8:29:F5:EF:81:2C:44:3 5:21:93:CE:FD To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certificate. On the master: puppet cert clean cottage On the agent: 1a. On most platforms: find /home/thomas/.puppetlabs/etc/puppet/ssl -name cottage.pem -delete 1b. On Windows: del "homethomas.puppetlabsetcpuppetsslcertscottage.pem" /f 2. puppet agent -t 38 The certificate retrieved from the master does not match the agent's private key.
  39. 39. https://goo.gl/8LyZzN OpenSSL recap ● x509 view certificate check expiration check serial number ● crl revoked cert serial# ● verify verify cert with CA and/or CRL ● modulus how the cert was encrypted 39
  40. 40. https://goo.gl/8LyZzN Problem workers node puppet server puppet server load balancer devel problem puppet server problem production mod_proxy_balancer 40
  41. 41. https://goo.gl/8LyZzN Problem workers puppet server --logdest /var/log/puppetlabs/puppetserver/problem.log --debug --profile logrotate 41
  42. 42. https://goo.gl/8LyZzN problem/bugfixes environments configure r10k to make environments automatically environment.conf - modulepath (debug module) ← what is that? 42
  43. 43. https://goo.gl/8LyZzN Compiling $ sudo puppet master --compile problem.example.com --debug --trace --logdest /tmp/problem.puppet.log --environment sandbox { "data": { "resources": [ { "title": "main", "exported": false, "tags": ["stage"], "type": "Stage", "parameters": { "before": "Stage[post]", JSON Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'bucketdir': 'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/bucket"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'fileserverconfig': 'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}' 43
  44. 44. https://goo.gl/8LyZzN Apply $ sudo puppet apply --debug --trace --environment sandbox code.pp $date = "+%S" exec {'epoch': command => "echo $((`date $date` / 86400))", path => '/bin:/usr/bin:/sbin:/usr/sbin', } Info: Applying configuration version '1415729233' Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` / 86400))' Debug: Executing 'echo $((`date +%S` / 86400))' Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully 44
  45. 45. problems connection certificate catalog compilation catalog application scope pry 45
  46. 46. https://goo.gl/8LyZzN catalogs 46
  47. 47. catalog fails to compile duplicate resource modulepath/bad module name fails to apply unpredictable exec bad/broken service bad/missing variable 47
  48. 48. https://goo.gl/8LyZzN fails to compile 48
  49. 49. https://goo.gl/8LyZzN help yourself function verb() { erb -P -x -T '-' $1 | ruby -c is_ok $? } function vyaml() { ruby -ryaml -e "YAML.load_file '$1'" is_ok $? } function ppv() { puppet parser validate $* is_ok $? } 49 function vepp() { puppet epp validate $* is_ok $? } function is_ok() { if [ $1 == 0 ]; then if [[ "$TERM" == *"xterm"* ]] || [[ "$TERM" == *"vt100"* ]]; then echo -e "033[1mSyntax 033[32;1mOk!033[0m" else echo "Syntax Ok!" fi else return $1 fi }
  50. 50. https://goo.gl/8LyZzN duplicate resource separate into subclass ( package {'httpd'} ) virtual resources ( @user, @package, @service) modulepath puppet config print modulepath root@puppet:~# puppet config print modulepath --environment production /etc/puppet/environments/production/public:/etc/puppet/environments/production /modules root@puppet:~# puppet config print modulepath --environment master /etc/puppet/modules:/usr/share/puppet/modules 50
  51. 51. problems connection certificate catalog compilation catalog application scope pry 51
  52. 52. https://goo.gl/8LyZzN fails to apply 52
  53. 53. https://goo.gl/8LyZzN Unpredictable exec #!/bin/bash echo $JAVA_HOME /home/javadev/.bashrc JAVA_HOME=/your/face It works for me! puppet runs as puppet 53
  54. 54. https://goo.gl/8LyZzN Broken Service service provider hasstatus => true /sbin/service $service status /etc/init.d/$service status /usr/bin/systemctl is-active $service 54
  55. 55. https://goo.gl/8LyZzN Bad/Missing Variable $one = "1" file {"pcone": path => "/tmp/pc$one", ensure => 'directory', } file {"pc1": path => "/tmp/pc1", ensure => 'file', } Info: Caching catalog for puppet.example.com Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[pc1] to ["/tmp/pc1"] at /root/pc.pp:6; resource ["File", "/tmp/pc1"] already declared at /root/pc.pp:2 at /root/pc.pp:6:3 on node puppet.example.com 55
  56. 56. https://goo.gl/8LyZzN Bad/Missing Variable $PC = 'puppetconf' pc {'one': place => "/tmp/$PC", type => "directory", } pc {'two': place => "/tmp/$PC", type => "file", } define pc ( String $place, String $type, ) { file {"$title": path => $place, ensure => $type, } } Info: Caching catalog for puppet.example.com Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[two] to ["/tmp/puppetconf"] at /root/define.pp:5; resource ["File", "/tmp/puppetconf"] already declared at /root/define.pp:5 at /root/define.pp:5:2 at /root/define.pp:15 on node puppet.example.com 56
  57. 57. https://goo.gl/8LyZzN Printing - Notify notify {"$variable": } 57
  58. 58. https://goo.gl/8LyZzN chaining notify {'something': }->exec{'thingthatfails': }->notify{'after': } 58
  59. 59. https://goo.gl/8LyZzN Checking exec{'before resolv.conf': command => '/usr/local/bin/puppet-debug before resolv.conf', require => Class['debug'] } -> file { '/etc/resolv.conf': source => template("dns/resolv.conf"), noop => true, } class debug { file {'puppet-debug': path => '/usr/local/bin/puppet-debug', source => 'puppet:///modules/debug/puppet-debug', mode => 0755, } } 59
  60. 60. https://goo.gl/8LyZzN Debug Script… just an example #!/bin/bash LOG=$(mktemp /tmp/puppet-debug.XXXXXX) echo Puppet Debug -- $@ -- $(date) | tee $LOG echo "-- Disk --" | tee -a $LOG df -h |tee -a $LOG df -i |tee -a $LOG echo "-- Mem --" | tee -a $LOG free | tee -a $LOG echo "-- Files --" | tee -a $LOG PUPPET=$(pgrep puppet) for proc in $PUPPET do lsof -p $proc |tee -a $LOG done Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014 -- Disk -- Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 6.7G 2.5G 3.9G 39% / tmpfs 246M 0 246M 0% /dev/shm /dev/vda1 485M 80M 380M 18% /boot Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VolGroup-lv_root 440640 79253 361387 18% / tmpfs 62783 1 62782 1% /dev/shm /dev/vda1 128016 50 127966 1% /boot -- Mem -- total used free shared buffers cached Mem: 502268 415488 86780 0 22176 172036 -/+ buffers/cache: 221276 280992 Swap: 835580 0 835580 -- Files -- COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME puppet 2058 root cwd DIR 253,0 4096 14 /root puppet 2058 root rtd DIR 253,0 4096 2 / puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/ruby puppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.so puppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so 60
  61. 61. https://goo.gl/8LyZzN Printing - Template - scope.to_hash - reject a few - sort - print, one per line file { "/tmp/puppet-debug.txt": content => inline_template("<% vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>n<% end %>"), } vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| k=vn end 61
  62. 62. https://goo.gl/8LyZzN Printing - Template _timestamp=2014-10-23 22:29:52 -0700 architecture=x86_64 augeasversion=1.0.0 bios_release_date=01/01/2011 bios_vendor=Bochs bios_version=Bochs blockdevice_vda_size=8589934592 blockdevice_vda_vendor=6900 blockdevices=vda caller_module_name= clientcert=cookbook.example.com clientnoop=false clientversion=3.7.1 concat_basedir=/var/lib/puppet/concat domain=example.com environment=production facterversion=2.2.0 filesystems=ext4,iso9660 fqdn=cookbook.example.com gid=root hardwareisa=x86_64 hardwaremodel=x86_64 hostname=cookbook id=root interfaces=eth0,lo 62
  63. 63. problems connection certificate catalog compilation catalog application scope pry 63
  64. 64. https://goo.gl/8LyZzN Scope The scene: roles and profiles ntp server class role::ntp { include ntp } class ntp { include ntp::server } 64
  65. 65. https://goo.gl/8LyZzN Scope The solution: fully scope everything remember scope class role::ntp { include ::ntp } class ntp { include ntp::server } 65
  66. 66. problems connection certificate catalog compilation catalog application scope pry 66
  67. 67. https://goo.gl/8LyZzN pry IRB replacement REPL available at runtime 67 pryrepl.org require 'pry' … # amazing code here # wow, much amaze … binding.pry …
  68. 68. https://goo.gl/8LyZzN pry demo 68 module Puppet::Parser::Functions newfunction(:pry) do |args| require 'pry' binding.pry end end modules/pry/lib/puppet/parser/functions/pry.rb node default { … pry() … } manifests/site.pp
  69. 69. https://goo.gl/8LyZzN pry demo # 69 # … From: /etc/puppetlabs/code/environments/production/modules/pry /lib/puppet/parser/functions/pry.rb @ line 4 #<Module:0xfb588d1>#real_function_pry: 2: newfunction(:pry) do |args| 3: require 'pry' => 4: binding.pry 5: end [1] pry(#<Puppet::Parser::Scope>)> puppet agent -tpuppetserver foreground … Puppet Server has successfully started and is now ready to handle requests exit … Puppet Compiled Catalog for xxx.example.com in y.z seconds Info: Caching catalog for xxx.example.com Info: Applying configuration version 'XXX'
  70. 70. https://goo.gl/8LyZzN where to go for help ● IRC #puppet / #puppet-dev ● slack puppetcommunity.slack.com #pug #puppet ● google group / mail list https://groups.google.com/forum/#!forum/puppet-users ● PUG https://www.meetup.com/Seattle-Puppet-Meetup/ 70
  71. 71. https://goo.gl/8LyZzN Summary Puppet is an HTTPS service End-to-end (gethostbyname, nc mtr) OpenSSL is your friend (x509,crl, verify, s_client make a debug class remember scope basic UNIX permissions 71
  72. 72. 72

×