Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014

2,535 views

Published on

Performance Tuning Your Puppet Infrastructure - Nic Benders, New Relic

Published in: Technology
  • Be the first to comment

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014

  1. 1. Performance Tuning Your (Puppet) Infrastructure Nic Benders OWNING http://nicbenders.com/presentations/puppetconf-2014
  2. 2. I work at New Relic. ! This is a story about treating your infrastructure the same way you treat your applications. ! This is also a story about how we use New Relic, at New Relic. http://nicbenders.com/presentations/puppetconf-2014
  3. 3. Managing Our Applications • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  4. 4. Managing Our Infrastructure • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  5. 5. Source Control
  6. 6. Continuous Deployment
  7. 7. Performance Monitoring
  8. 8. DON’T PANIC • These example are for Puppet Enterprise 3.1, but this will work with Puppet Open Source, or with newer versions. You’ll just need to make a few changes here and there. • If you don’t have a New Relic account, you can sign-up for a free “Lite” account. • All of the examples are up in a more useful format at the URL below. (The URL will also be shown at the end of the presentation.) http://nicbenders.com/presentations/puppetconf-2014
  9. 9. Instrumenting the Puppet Master • Install newrelic_rpm using the Puppet’s Ruby interpreter • Modify the config.ru in the Puppet Master root dir • Create a newrelic.yml in the same directory # /var/opt/lib/pe-­‐puppetmaster/config.ru ! # Setup the New Relic Ruby Agent before anything else require 'rubygems' require 'newrelic_rpm' # END New Relic Ruby Agent setup # /var/opt/lib/pe-­‐puppetmaster/newrelic.yml ! common: &default_settings license_key: '000-­‐your-­‐license-­‐key-­‐here-­‐000' app_name: "Puppet Master" log_level: info log_file_path: '/var/log/pe-­‐puppet/' ssl: false capture_params: true transaction_tracer: enabled: true error_collector: enabled: true ignore_errors: "ActionController::RoutingError" production: <<: *default_settings monitor_mode: true
  10. 10. Getting good transaction names • Use set_transaction_name in a Rack Middleware to tell the Ruby Agent how we are going to name the “web transactions” for the Puppet Master • Use add_custom_parameters to pass the environment and other useful data along # # We don't want every transaction to just say "call", so define a # Transaction "namer" for the Ruby Agent based on the URL structure # class CoolNamer def initialize(app, opts = {}) @app = app end def call(env) req = Rack::Request.new(env) _, environment, kind_of_thing, resource = req.path.split('/',4) NewRelic::Agent.set_transaction_name(kind_of_thing, :category => :rack) NewRelic::Agent.add_custom_parameters(:puppet_env => environment) NewRelic::Agent.add_custom_parameters(:puppet_resource => resource) @app.call(env) end end use CoolNamer # END Transaction "namer"
  11. 11. Getting more detail • The Ruby Agent’s MethodTracer allows us to get details of what is going on inside a Transaction. • By hooking into Puppet’s built-in Profiler calls, we can quickly instrument all the most important things. • This example is for Puppet 3.3.1. In newer versions it is much easier. require "puppet/util/profiler" module Puppet::Util::Profiler include ::NewRelic::Agent::MethodTracer self.profile(message, &block) if /^Processed request def / === message # Top level trace, skip for now yield else metric_name = self.message_to_metric(message) trace_execution_scoped([metric_name]) do yield end end end def self.message_to_metric(message) message.chomp! message.gsub!(/^(Compiled catalog) for .*/, '1') message.gsub!(/^(Filtered result for catalog) .*/, '1') message.gsub!(/(d+) /, '') message.gsub!(/^(Evaluated resource) /, '1/') message.gsub!(/[.*]/,'') message.gsub!(/:s/,'/') message.gsub!(/^(Called) /, '1/') "Custom/Puppet/#{message}" end end
  12. 12. Instrumenting the Puppet DB • Download the Java Agent from your “Account Settings” page. • Create an /opt/puppet/newrelic dir and place the newrelic.jar and newrelic.yaml file into it. • Modify the newrelic.yaml • Modify JAVA_ARGS # /etc/syslog/pe-­‐puppetdb ! # Enable the New Relic Java Agent by adding this # to the bottom of the file JAVA_ARGS="-­‐javaagent:/opt/puppet/newrelic/newrelic.jar ${JAVA_ARGS}" # /opt/puppet/newrelic/newrelic.yml ! common: &default_settings license_key: '000-­‐your-­‐license-­‐key-­‐here-­‐000' ! app_name: 'Puppet DB' log_level: info log_file_path: /var/log/pe-­‐puppetdb/ ssl: false transaction_tracer: enabled: true transaction_threshold: apdex_f record_sql: obfuscated stack_trace_threshold: 0.2 explain_enabled: true explain_threshold: 0.2 browser_monitoring: auto_instrument: false production: <<: *default_settings
  13. 13. Operational Analytics
  14. 14. Log Collection
  15. 15. Puppet Agent already goes into Syslog • Agent syslogs have a ton of useful information. We just need to get it to a useful place. • Our application and syslog data is collected by a Heka process on each server. It then makes its way to ElasticSearch, where it can be queried by Kibana.
  16. 16. Send run reports to the same place • Add your log system as a Report processor! • We use https://github.com/ logstash/puppet-logstash-reporter • To get the data into Heka, we had to write some Lua, but it was pretty straightforward.
  17. 17. pe_puppet::master::config_hash: reports: 'http,puppetdb,logstash' ! logstash_reporter::logstash_host: 'our-­‐heka-­‐server' # Logstash-­‐style report processer through Heka class { 'logstash_reporter': logstash_port => '22253', logstash_host => hiera('logstash_reporter::logstash_host'), }
  18. 18. Managing Our Infrastructure • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  19. 19. We’re Hiring Site Reliability Engineers MySQL Experts Engineering Managers Data Pipeline Engineers newrelic.com/about/careers
  20. 20. http://nicbenders.com/presentations/puppetconf-2014 nic@newrelic.com

×