Puppet Performance Profiling

Puppet Performance
Profiling
NLUUG Fall Conference 2014
R.I.Pienaar

Who am I?
• Puppet user since 0.22.x
• Architect of MCollective
• Author of Extlookup and Hiera
• Independant Consultant
• Blog at http://devco.net
• Tweets at @ripienaar
• Volcane on IRC
R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar

Why?
• Unit and Integration testing
• Cloud based scaling
• Maintenance windows and fast feedback
• Resource usage on the master

Agent Life Cycle
• Agent gather facts using facter *
• Agent submit facts and request catalog
• Master compiles catalog, sends to agent *
• Agent stores catalog
• Agent applies catalog *
• Agent submits report
• Master processes report

Facter

Timing Facts
$ facter —timing
lsbdistid: 56.06ms
operatingsystem: 57.21ms
osfamily: 58.54ms
macaddress: 0.29ms
ipaddress6: 18.36ms
ipaddress6_lo: 31.41ms
mtu_dummy0: 29.82ms
vlans: 0.20ms
selinux: 11.91ms
mtu_ip_vti0: 29.09ms
Shows per fact timings

Timing Facts
FREEGEOURL = “http://freegeoip.net/json/"
geoipinfo = JSON.parse(open(FREEGEOURL).read)
geoipinfo.each do |k, v|
next if k == "ip"
Facter.add("geo_#{k.downcase}") { setcode { v } }
end
geo_latitude: 0.02ms
geo_region_name: 0.04ms
geo_metro_code: 0.04ms
geo_time_zone: 0.03ms
geo_country_code: 0.02ms
geo_country_name: 0.02ms
geo_zip_code: 0.02ms
geo_region_code: 0.04ms
geo_city: 0.04ms
geo_longitude: 0.03ms
Shows per fact timings…but only the setcode part

Timing Facts
geoipinfo = nil
["country_code", “country_name”…].each do |key|
Facter.add(“geo_%s” % key.downcase) do
setcode do
geoipinfo ||= JSON.parse(open(FREEGEOURL).read)
["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key]
end
end
end
geo_zipcode: 7174.49ms
geo_metro_code: 0.04ms
geo_country_code: 0.03ms
geo_region_name: 0.04ms
geo_latitude: 0.05ms
geo_area_code: 0.04ms
geo_country_name: 0.03ms
geo_city: 0.04ms
geo_longitude: 0.04ms
geo_region_code: 0.04ms
Fetch once per run
and re-use
Restructuring fact to do the slow work in setcode

Timing Facts
def geoip_cached_fetch
store = PStore.new(File.join(Dir.tmpdir, “fgeoip_fact.pstore"))
store.transaction do
store[:freegeoip] ||= JSON.parse(open(FREEGEOURL).read)
end
end
geoipinfo = nil
["country_code", “country_name”…].each do |key|
Facter.add(“geo_%s” % key.downcase) do
setcode do
geoipinfo ||= geoip_cached_fetch
["", nil].include?(geoipinfo[key]) ? "unknown" : geoipinfo[key]
end
end
end
Add a local cache, now only the first invoke is slow

Catalog Compilation

Compilation
class apache($version=present) {
validate_string($version)
class{“::apache::package”:
version => $version
}
}

Compilation
# /etc/puppet/modules/apache/init.pp
version => $version
}
}
Needs to find the file

Compilation
class {'epel': } ->
class {'puppetdb': } ->
class {'puppet::master':
storeconfigs => true
}
$ strace puppet parser validate test.pp 2>&1 |
grep stat|wc -l
10546
$ strace puppet parser validate test.pp 2>&1 |
egrep '((stat|open).+modules)'|wc -l
217

Compilation
version => $version
}
}
hiera(“apache::version”, “present”)

Compilation
:hierarchy:
- 01_Nodes/%{::fqdn}
- 02_Domain/%{::domain}
- 03_Perimeter/%{::perimeter}/%{::datacenter}
- 03_Perimeter/%{::perimeter}
- 04_DC/%{::datacenter}
- 05_OS/%{::operatingsystem}/%{::hardwareisa}
- 05_OS/%{::operatingsystem}
- 06_Manufacturer/%{::manufacturer}
- 07_RealTime/%{::real_time}
- 09_Mco_Teams/%{::team}
- 50_Teams/%{team}/01_Nodes/%{::fqdn}
- 50_Teams/HPCE/%{::hpce_env}/%{::hpce_type}
- 50_Teams/HPCE/%{::hpce_env}/common
- 50_Teams/%{::team}/common
- common

Compilation
class {'epel': } ->
}
$ puppet apply test.pp 2>&1|grep "Looking
for data source"|wc -l
3562

Compilation
version => $version
}
}
ruby code, similar issues as the fact

Compilation
version => $version
}
}
also speeds up finding
the class internally

Compilation
version => $version
}
}
No hiera lookup,
authoritative value supplied

Profiling Compilation
class {'epel': } ->
}
$ puppet apply test.pp —profile

class {'epel': } ->
}
importing ‘..epel/manifests/init.pp’
importing ‘..epel/manifests/params.pp’
importing ‘..epel/manifests/rpm_gpg_key.pp’
importing ‘..puppetdb/manifests/init.pp’
importing ‘..puppetdb/manifests/params.pp’
PROFILE [apply] 2.3.1 Called defined: took 0.0001 seconds
PROFILE [apply] 2.3.2 Called downcase: took 0.0000 seconds
PROFILE [apply] 2.3.3 Called validate_re: took 0.0000 seconds
importing ‘..puppetdb/manifests/server.pp’ in environment production

define apache::vhost(…) {
file { “${priority_real}-${filename}.conf":
content => template(‘apache/vhost.conf.erb’)
}
}
Scope(Apache::Vhost[default]): Retrieving template apache/vhost.conf.erb
template[…vhost.conf.erb]: Bound template variables for …vhost.conf.erb in 0.00
seconds
template[…vhost.conf.erb]: Interpolated template …vhost.conf.erb in 0.05 seconds

AGGREGATE PROFILING RESULTS:
compiler -> compile: 5.800916 ms (1 calls)
compiler -> evaluate_resource: 1.53646 ms (79 calls)
compiler -> evaluate_resource -> Apache::Vhost[…]: 0.148923 ms
functions: 1.892293 ms (562 calls)
functions -> include: 1.091945 ms (41 calls)
functions -> template: 0.754388 ms (137 calls)

Catalog apply phase

Profiling the Agent
# puppet agent —test —evaltrace
Class[Postgresql::Server::Reload]: Starting to evaluate
Class[Postgresql::Server::Reload]: Evaluated in 0.00 seconds
Class[Postgresql::Server::Config]: Starting to evaluate
Class[Postgresql::Server::Config]: Evaluated in 0.00 seconds
Class[Postgresql::Server::Service]: Starting to evaluate
Class[Postgresql::Server::Service]: Evaluated in 0.00 seconds
Apache::Vhost[puppet-localhost]: Starting to evaluate
Apache::Vhost[puppet-localhost]: Evaluated in 0.00 seconds
Package[mailcap]: Starting to evaluate
Package[mailcap]/ensure: created
Package[mailcap]: Evaluated in 65.36 seconds
Tracing the catalog apply phase

Profiling the Agent
# report_print.rb —count 5 —report 201411101110.yaml
Report for localhost at Wed Nov 10 11:10:54 +0000 2014
Report File: /var/…/last_run_report.yaml
Report Kind: apply
Puppet Version: 3.7.3
Report Format: 4
Configuration Version: 1416395447
UUID: d4a9fac9-…-7db61b0dfa89
Log Lines: 2 (show with —log)
Processing reports for performance metrics

Profiling the Agent
Report Metrics:
Changes:
Total: 2
Events:
Total: 2
Success: 2
Failure: 0
Resources:
Total: 213
Out of sync: 2
Changed: 2
Scheduled: 0
Skipped: 0
Failed to restart: 0
Failed: 0
Restarted: 0

Profiling the Agent
Time:
Total: 28.44
Package: 15.55
Config retrieval: 7.71
File: 1.43
Exec: 1.33
Puppetdb conn validator: 0.87
Service: 0.73
Postgresql psql: 0.66
Augeas: 0.07
Ini setting: 0.04

Profiling the Agent
Resources by resource type:
107 File
38 Ini_setting
13 Exec
12 Postgresql_psql
9 Package
6 Yumrepo
6 Anchor
6 Schedule
4 Service
3 Postgresql_conf

Profiling the Agent
Slowest 5 resources by evaluation time:
15.55 Package[mailcap]
0.88 Puppetdb_conn_validator[puppetdb_conn]
0.66 File[/usr…/validate_postgresql_connection.sh]
0.31 Exec[validate postgres connection]
0.30 Service[puppetmaster]

Profiling the Agent
5 largest managed files (only those that are readable)
3.85 KB /var/…/bin/concatfragments.sh
1.65 KB /etc/puppet/rack/config.ru
1.61 KB /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
1.23 KB /etc/puppet/puppet.conf
849.00 B /etc/httpd/conf/httpd.conf

Questions?
geo fact: http://srt.ly/hx
report processor: http://srt.ly/gq
twitter: @ripienaar
email: rip@devco.net
blog: www.devco.net
github: ripienaar
freenode: Volcane

Puppet Performance Profiling

More Related Content

What's hot

Similar to Puppet Performance Profiling

Recently uploaded

Puppet Performance Profiling