Building Data-Driven Infrastructure with Puppet - PuppetConf 2013
Upcoming SlideShare
Loading in...5
×
 

Building Data-Driven Infrastructure with Puppet - PuppetConf 2013

on

  • 3,586 views

"Building Data-Driven Infrastructure with Puppet" by James Fryman, Operations Hacker, GitHub, Inc. ...

"Building Data-Driven Infrastructure with Puppet" by James Fryman, Operations Hacker, GitHub, Inc.

Presentation Overview: As your Puppet Infrastructure grows, so does the complexity of the Puppet codebase. The complexity of the codebase often creates a scenario where it becomes more time consuming to modify/add to the codebase. Likewise, any new addition or node still may require modifications to the Puppet database, which could include the management of many edge cases. Fortunately, the software industry has been working on developing techniques with code abstraction, refactoring, and software maturity. This talk will focus on how to write scalable modules within Puppet to be used to create Data Driven Infrastructures. In addition, this talk will demonstrate how to structure process/procedure/code to quickly and rapidly scale operations with minimal modifications to Puppet code.

Speaker Bio: James Fryman, Operations Hacker, GitHub, Inc. James Fryman is a Technologist who has been working on spreading the good word of technology via the greatest mechanism known to man: the beer fueled rant. James has been working to automate software and infrastructure for the last 10 years, and has learned quite a bit about Security, Architecture, Scaling, and Development as a result. James currently works for GitHub as an Operations Hacker.

Statistics

Views

Total Views
3,586
Views on SlideShare
1,413
Embed Views
2,173

Actions

Likes
1
Downloads
22
Comments
0

3 Embeds 2,173

http://puppetlabs.com 2171
http://analyst.test.ciradar.com 1
http://plus.url.google.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building Data-Driven Infrastructure with Puppet - PuppetConf 2013 Building Data-Driven Infrastructure with Puppet - PuppetConf 2013 Presentation Transcript

  • Friday, August 23, 13
  • DataDrive nInfrastructur e Friday, August 23, 13
  • Friday, August 23, 13
  • ? Friday, August 23, 13
  • ? http://git.io/UmUdKA Friday, August 23, 13
  • Let me Tell you a story Friday, August 23, 13
  • What is a Cloud Friday, August 23, 13
  • I don’t have to think about IT With the Cloud... Friday, August 23, 13
  • Friday, August 23, 13
  • aaSPlatform Software Infrastruct ure Friday, August 23, 13
  • Friday, August 23, 13
  • Friday, August 23, 13
  • Where is our cloud? Friday, August 23, 13
  • Friday, August 23, 13
  • UtopiaFriday, August 23, 13
  • James Fryman Friday, August 23, 13
  • Friday, August 23, 13
  • AutomationsJunkie Friday, August 23, 13
  • Are you Crazy? Friday, August 23, 13
  • Goals?Data? What Friday, August 23, 13
  • Goals?Data? What Friday, August 23, 13
  • Goals?Data? What Friday, August 23, 13
  • Machine Parsable Friday, August 23, 13
  • There is SystemOne Friday, August 23, 13
  • Feedback PuppetDB Provisioning Configuring Destroying gPanel Friday, August 23, 13
  • Parts of the System Friday, August 23, 13
  • provisioner Friday, August 23, 13
  • Home Grown Friday, August 23, 13
  • Home Grown Friday, August 23, 13
  • controller Friday, August 23, 13
  • Lots O’ Models Friday, August 23, 13
  • file { '/etc/facter/facts.d/quagga_manage_service': ensure => present, replace => false, }   if $::quagga_manage_service { $service_state[ensure] = running $service_state[enable] = true } else { $service_state[ensure] = undef $service_state[enable] = undef }   service { 'quagga': ensure => $service_state[ensure], enable => $service_state[enable], } Friday, August 23, 13
  • orchestrator Friday, August 23, 13
  • Chat Ops Friday, August 23, 13
  • Chat Ops Friday, August 23, 13
  • Feedback of the System Friday, August 23, 13
  • Metrics &Monitoring Friday, August 23, 13
  • begin JSON.parse(RestClient.get(url)).each do |cache| data["#{cache['target']}"] = 0 count = 0 cache["datapoints"].each do |point| unless (point[0].nil?) data["#{cache['target']}"] += point[0] count += 1 end end data["#{cache['target']}"] /= count data["total"] += data["#{cache['target']}"] end rescue ZeroDivisionError => e puts "UNKNOWN metric not in graphite!" exit EXIT_UNKNOWN rescue => e puts e.message exit EXIT_CRITICAL end Friday, August 23, 13
  • begin JSON.parse(RestClient.get(url)).each do |cache| data["#{cache['target']}"] = 0 count = 0 cache["datapoints"].each do |point| unless (point[0].nil?) data["#{cache['target']}"] += point[0] count += 1 end end data["#{cache['target']}"] /= count data["total"] += data["#{cache['target']}"] end rescue ZeroDivisionError => e puts "UNKNOWN metric not in graphite!" exit EXIT_UNKNOWN rescue => e puts e.message exit EXIT_CRITICAL end Friday, August 23, 13
  • event Friday, August 23, 13
  • event log Friday, August 23, 13
  • event alertlog Friday, August 23, 13
  • event alertlog error Friday, August 23, 13
  • Must self-correct Friday, August 23, 13
  • Feedback PuppetDB Provisioning Configuring Destroying gPanel Friday, August 23, 13
  • Friday, August 23, 13
  • Deployable using text files Friday, August 23, 13
  • Modularity Friday, August 23, 13
  • haproxy::proxy { $es_proxy_name: proxy => 'listen', mode => 'http', ip => $::ipaddress_lo, port => '9200', config => { balance => 'roundrobin', }, } Friday, August 23, 13
  • haproxy::proxy { $es_proxy_name: proxy => 'listen', mode => 'http', ip => $::ipaddress_lo, port => '9200', config => { balance => 'roundrobin', }, } Friday, August 23, 13
  • haproxy::proxy::member { $es_proxy_name: hostname => $::ec2_local_ipv4, port => '9200', param => [ 'weight 1', 'maxconn 1000', 'check', ], } Friday, August 23, 13
  • collectd::plugin { [ 'cpu', 'load', 'memory', 'swap', 'irq', 'exec', 'entropy', ]: }   collectd::plugin { [ 'df', 'interface', 'protocols', 'disk', ]: config => true } Friday, August 23, 13
  • Level 4: Templates Level 5: Data Driven Friday, August 23, 13
  • Self Authoritati ve Friday, August 23, 13
  • def self.collect_resources(resource) query = [ "and", ["=", "type", "File"], ["=", "exported", true], ["=", "tag", "nagios::object::#{resource}"], ["=", ["node", "active"], true] ] self.search('/v2/resources', query.to_json) end Friday, August 23, 13
  • def self.collect_resources(resource) query = [ "and", ["=", "type", "File"], ["=", "exported", true], ["=", "tag", "nagios::object::#{resource}"], ["=", ["node", "active"], true] ] self.search('/v2/resources', query.to_json) end Friday, August 23, 13
  • # Seed the initial file with false, and enable # on next run with gh-nagios enable_pager <hostname> file { '/etc/facter/facts.d/enable_pager.txt': ensure => file, owner => 'root', group => 'root', mode => '0644', content => 'enable_pager=false', replace => false, } Friday, August 23, 13
  • @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg": content => template('nagios/etc/nagios/object.erb'), mode => '0444', group => 'nagios', notify => Service['nagios'], } Friday, August 23, 13
  • Tips Friday, August 23, 13
  • Refactoring Friday, August 23, 13
  • Refactoring Friday, August 23, 13
  • least to most specific Friday, August 23, 13
  • frymanet.com mysqlnginx rails rubycommon admin package repos Friday, August 23, 13
  • graduate to params Friday, August 23, 13
  • class ntp::params {   $defaults = { package => { version => ‘latest’, }, config => { servers => [‘pool.ntp.org’], }, } } Friday, August 23, 13
  • externalize Friday, August 23, 13
  • class ntp::params {   $defaults = { package => { version => hiera(‘ntp_package_version’), }, config => { servers => hiera(‘ntp_servers’), }, } } Friday, August 23, 13
  • CloudFormation Friday, August 23, 13
  • CloudFormation Friday, August 23, 13
  • "Resources": { "RendererServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "LaunchConfigurationName": { "Ref": "LaunchConfig" }, "MinSize": "2", "MaxSize": "16", "Tags": [ { "Key": "Environment", "Value": "Production", "PropagateAtLaunch": "true" }, { "Key": "Role", "Value": "renderer", "PropagateAtLaunch": "true" } ] } }, Friday, August 23, 13
  • Autoloading Friday, August 23, 13
  • # autoloader.pp class nagios::autoload_helpers { $helpers = get_nagios_helpers() case $::puppetversion { /^3/: { include $helpers } default: { nagios::autoload_helpers::import_shim { $helpers: } } } } Friday, August 23, 13
  • # autoloader.pp class nagios::autoload_helpers { $helpers = get_nagios_helpers() case $::puppetversion { /^3/: { include $helpers } default: { nagios::autoload_helpers::import_shim { $helpers: } } } } Friday, August 23, 13
  • module Puppet::Parser::Functions newfunction(:get_nagios_helpers, :type => :rvalue, :doc => "Grab all modules that have nagios helpers for import") do module_path = File.expand_path('..', Puppet::Module.find('nagios',compiler.environment.to_s).path) helpers = Dir["#{module_path}/**/nagios/helpers.pp"].map do |d| "#{d.split('/')[-4]}::nagios::helpers" done helpers end end Friday, August 23, 13
  • Modeling Friday, August 23, 13
  • Be Dynamic Friday, August 23, 13
  • Be Dynamic Friday, August 23, 13
  • <%- if @comment -%> ### <%= @comment %> <%- end -%> <%= @type %> <% if @label %><%= @label %><% end %> { <%- @config.sort.reverse.each do | key,value| -%> <%- if value.class == Array -%> <%- value.each do |element| - %> <%= key %>(<%= element %>); <%- end -%> <%- else -%> <%= key %>(<%= value %>); <%- end -%> <%- end -%> }; Friday, August 23, 13
  • Fencing Resources Friday, August 23, 13
  • # Seed the initial file with false, and enable # on next run with gh-nagios enable_pager <hostname> file { '/etc/facter/facts.d/enable_pager.txt': ensure => file, owner => 'root', group => 'root', mode => '0644', content => 'enable_pager=false', replace => false, } Friday, August 23, 13
  • if $::enable_pager { @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg": content => template('nagios/etc/nagios/object.erb'), mode => '0444', group => 'nagios', notify => Service['nagios'], } } Friday, August 23, 13
  • Buy it? Friday, August 23, 13
  • Buy it? Friday, August 23, 13
  • It’s About Friday, August 23, 13
  • Missing? What’s Friday, August 23, 13
  • Language Friday, August 23, 13
  • Predictive Analysis Friday, August 23, 13
  • Coming Home Friday, August 23, 13
  • Goals?Data? What Friday, August 23, 13
  • Systems Thinking Recap Friday, August 23, 13
  • There is SystemOne Machine Parsable Must be a Know n Quantity Self Authoritati ve Must self-correctFriday, August 23, 13
  • Friday, August 23, 13
  • jfryman fryman@github.com Friday, August 23, 13
  • jfryman fryman@github.com Friday, August 23, 13
  • what’s next? Friday, August 23, 13
  • what’s next? Friday, August 23, 13