Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Puppet Performance Profiling - CM Camp 2015

This talk will cover the basic life cycle of a Puppet Catalog from compilation request to report processing. It will explore the performance of some of the life cycle steps and show how you might instrument these steps using tools Puppet make available.

Along the way it will provide hints and tips on how to write performant facts and manifests.

  • Be the first to comment

  • Be the first to like this

Puppet Performance Profiling - CM Camp 2015

  1. 1. R.I.Pienaar February 2015 Puppet Performance Profiling
  2. 2. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Who am I? • Puppet user since 0.22.x • Blog at http://devco.net • Tweets at @ripienaar • Volcane on IRC
  3. 3. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Why? • Unit and Integration testing • Cloud based scaling • Maintenance windows and fast feedback • Resource usage on the master
  4. 4. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Agent Life Cycle • Agent gather facts using facter * • Agent submit facts and request catalog • Master compiles catalog, sends to agent * • Agent stores catalog • Agent applies catalog * • Agent submits report • Master processes report
  5. 5. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Facter
  6. 6. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Shows per fact timings $ facter —timing lsbdistid: 56.06ms operatingsystem: 57.21ms osfamily: 58.54ms macaddress: 0.29ms ipaddress6: 18.36ms ipaddress6_lo: 31.41ms mtu_dummy0: 29.82ms vlans: 0.20ms selinux: 11.91ms mtu_ip_vti0: 29.09ms Timing Facts
  7. 7. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Shows per fact timings…but only the setcode part FREEGEOURL = “http://freegeoip.net/json/" geoipinfo = JSON.parse(open(FREEGEOURL).read) geoipinfo.each do |k, v| next if k == "ip" Facter.add("geo_#{k.downcase}") { setcode { v } } end Timing Facts geo_latitude: 0.02ms geo_region_name: 0.04ms geo_metro_code: 0.04ms geo_time_zone: 0.03ms geo_country_code: 0.02ms geo_country_name: 0.02ms geo_zip_code: 0.02ms geo_region_code: 0.04ms geo_city: 0.04ms geo_longitude: 0.03ms
  8. 8. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Restructuring fact to do the slow work in setcode geoipinfo = nil ["country_code", “country_name”…].each do |key| Facter.add(“geo_%s” % key.downcase) do setcode do geoipinfo ||= JSON.parse(open(FREEGEOURL).read) ["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key] end end end Timing Facts geo_zipcode: 7174.49ms geo_metro_code: 0.04ms geo_country_code: 0.03ms geo_region_name: 0.04ms geo_latitude: 0.05ms geo_area_code: 0.04ms geo_country_name: 0.03ms geo_city: 0.04ms geo_longitude: 0.04ms geo_region_code: 0.04ms Fetch once per run and re-use
  9. 9. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Add a local cache, now only the first invoke is slow def geoip_cached_fetch store = PStore.new(File.join(Dir.tmpdir, “fgeoip_fact.pstore")) store.transaction do store[:freegeoip] ||= JSON.parse(open(FREEGEOURL).read) end end geoipinfo = nil ["country_code", “country_name”…].each do |key| Facter.add(“geo_%s” % key.downcase) do setcode do geoipinfo ||= geoip_cached_fetch ["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key] end end end Timing Facts
  10. 10. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Catalog Compilation
  11. 11. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar class apache($version=present) { validate_string($version) class{“::apache::package”: version => $version } } Compilation
  12. 12. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version) class{“::apache::package”: version => $version } } Compilation Needs to find the file
  13. 13. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true } Compilation $ strace puppet parser validate test.pp 2>&1 |grep stat|wc -l 10546 $ strace puppet parser validate test.pp 2>&1 |egrep '((stat|open). +modules)'|wc -l 217
  14. 14. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version) class{“::apache::package”: version => $version } } Compilation hiera(“apache::version”, “present”)
  15. 15. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar :hierarchy: - 01_Nodes/%{::fqdn} - 02_Domain/%{::domain} - 03_Perimeter/%{::perimeter}/%{::datacenter} - 03_Perimeter/%{::perimeter} - 04_DC/%{::datacenter} - 05_OS/%{::operatingsystem}/%{::hardwareisa} - 05_OS/%{::operatingsystem} - 06_Manufacturer/%{::manufacturer} - 07_RealTime/%{::real_time} - 09_Mco_Teams/%{::team} - 50_Teams/%{team}/01_Nodes/%{::fqdn} - 50_Teams/HPCE/%{::hpce_env}/%{::hpce_type} - 50_Teams/HPCE/%{::hpce_env}/common - 50_Teams/%{::team}/common - common Compilation
  16. 16. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true } Compilation $ puppet apply test.pp 2>&1|grep "Looking for data source"|wc -l 3562
  17. 17. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version) class{“::apache::package”: version => $version } } Compilation ruby code, similar issues as the fact
  18. 18. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version) class{“::apache::package”: version => $version } } Compilation also speeds up finding the class internally
  19. 19. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # /etc/puppet/modules/apache/init.pp class apache($version=present) { validate_string($version) class{“::apache::package”: version => $version } } Compilation No hiera lookup, authoritative value supplied
  20. 20. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true } Profiling Compilation $ puppet apply test.pp —profile
  21. 21. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar importing ‘..epel/manifests/init.pp’ importing ‘..epel/manifests/params.pp’ importing ‘..epel/manifests/rpm_gpg_key.pp’ importing ‘..puppetdb/manifests/init.pp’ importing ‘..puppetdb/manifests/params.pp’ PROFILE [apply] 2.3.1 Called defined: took 0.0001 seconds PROFILE [apply] 2.3.2 Called downcase: took 0.0000 seconds PROFILE [apply] 2.3.3 Called validate_re: took 0.0000 seconds PROFILE [apply] 2.3.4 Called downcase: took 0.0000 seconds PROFILE [apply] 2.3.5 Called validate_re: took 0.0000 seconds PROFILE [apply] 2.3.6 Called downcase: took 0.0001 seconds PROFILE [apply] 2.3.7 Called validate_re: took 0.0000 seconds importing ‘..puppetdb/manifests/server.pp’ in environment production PROFILE [apply] 2.3.8 Called downcase: took 0.0000 seconds PROFILE [apply] 2.3.9 Called validate_re: took 0.0000 seconds Profiling Compilation class {'epel': } -> class {'puppetdb': } -> class {'puppet::master': storeconfigs => true }
  22. 22. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Scope(Apache::Vhost[default]): Retrieving template apache/ vhost.conf.erb template[…vhost.conf.erb]: Bound template variables for … vhost.conf.erb in 0.00 seconds template[…vhost.conf.erb]: Interpolated template …vhost.conf.erb in 0.05 seconds Profiling Compilation define apache::vhost(…) { file { “${priority_real}-${filename}.conf": content => template(‘apache/vhost.conf.erb’) } }
  23. 23. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar AGGREGATE PROFILING RESULTS: compiler -> compile: 5.800916 ms (1 calls) compiler -> evaluate_resource: 1.53646 ms (79 calls) compiler -> evaluate_resource -> Apache::Vhost[…]: 0.148923 ms functions: 1.892293 ms (562 calls) functions -> include: 1.091945 ms (41 calls) functions -> template: 0.754388 ms (137 calls) Profiling Compilation
  24. 24. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Catalog apply phase
  25. 25. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # puppet agent —test —evaltrace Class[Postgresql::Server::Reload]: Starting to evaluate Class[Postgresql::Server::Reload]: Evaluated in 0.00 seconds Class[Postgresql::Server::Config]: Starting to evaluate Class[Postgresql::Server::Config]: Evaluated in 0.00 seconds Class[Postgresql::Server::Service]: Starting to evaluate Class[Postgresql::Server::Service]: Evaluated in 0.00 seconds Apache::Vhost[puppet-localhost]: Starting to evaluate Apache::Vhost[puppet-localhost]: Evaluated in 0.00 seconds Package[mailcap]: Starting to evaluate Package[mailcap]/ensure: created Package[mailcap]: Evaluated in 65.36 seconds Profiling the Agent Tracing the catalog apply phase
  26. 26. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar # report_print.rb —count 5 —report 201411101110.yaml Report for localhost at Wed Nov 10 11:10:54 +0000 2014 Report File: /var/…/last_run_report.yaml Report Kind: apply Puppet Version: 3.7.3 Report Format: 4 Configuration Version: 1416395447 UUID: d4a9fac9-…-7db61b0dfa89 Log Lines: 2 (show with —log) Profiling the Agent Processing reports for performance metrics
  27. 27. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Report Metrics: Changes: Total: 2 Events: Total: 2 Success: 2 Failure: 0 Resources: Total: 213 Out of sync: 2 Changed: 2 Scheduled: 0 Skipped: 0 Failed to restart: 0 Failed: 0 Restarted: 0 Profiling the Agent Processing reports for performance metrics
  28. 28. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Time: Total: 28.44 Package: 15.55 Config retrieval: 7.71 File: 1.43 Exec: 1.33 Puppetdb conn validator: 0.87 Service: 0.73 Postgresql psql: 0.66 Augeas: 0.07 Ini setting: 0.04 Profiling the Agent Processing reports for performance metrics
  29. 29. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Resources by resource type: 107 File 38 Ini_setting 13 Exec 12 Postgresql_psql 9 Package 6 Yumrepo 6 Anchor 6 Schedule 4 Service 3 Postgresql_conf Profiling the Agent Processing reports for performance metrics
  30. 30. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Slowest 5 resources by evaluation time: 15.55 Package[mailcap] 0.88 Puppetdb_conn_validator[puppetdb_conn] 0.66 File[/usr…/validate_postgresql_connection.sh] 0.31 Exec[validate postgres connection] 0.30 Service[puppetmaster] Profiling the Agent Processing reports for performance metrics
  31. 31. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar 5 largest managed files (only those that are readable) 3.85 KB /var/…/bin/concatfragments.sh 1.65 KB /etc/puppet/rack/config.ru 1.61 KB /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6 1.23 KB /etc/puppet/puppet.conf 849.00 B /etc/httpd/conf/httpd.conf Profiling the Agent Processing reports for performance metrics
  32. 32. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar Questions? twitter: @ripienaar email: rip@devco.net blog: www.devco.net github: ripienaar freenode: Volcane geo fact: http://srt.ly/hx report processor: http://srt.ly/gq

×