Puppet Data Mining

Puppet Data Mining
Report processors, stored conﬁgs and more

PuppetCamp 22nd March 2012

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/map408/2412123378

Gareth Rushgrove @garethr

gareth rushgrove | morethanseven.net

Blog at morethanseven.net


Curate devopsweekly.com


Text

Work at UK Government Digital Service


Serious Government Business


- Why do we want it?
- How do we get our hands on it
- What can we build with it

Puppet data

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/iancarroll/5027441664

Single source of truth

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/joeshlabotnik/4132307070/

- Spreadsheets
- Wiki
- Deployment scripts
- Application code
- SSH configs
- More deployment scripts
- Monitoring configuration

Example: find a list of hostnames


Operational reports

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/nettsu/5276327339

Company hardware

Example: hardware spreadsheet


Hooks for tooling

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/nickstone333/3135318558

Example: Nagios integration


Example: Rundeck integration


- Report processors
- Stored conﬁgurations
- Puppet internal APIs

Where to ﬁnd the data


Report processors

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/nathaninsandiego/3757033518

[agent]
report = true
pluginsync = true

[master]
reports = store
pluginsync = true

Conﬁguration


[agent]
report = true
pluginsync = true

[master]
reports = store,http,log,sample
pluginsync = true

Adding reports


Built in


Store the YAML report on disk. Each host sends
its report as a YAML dump and this just stores
the ﬁle on disk, in the reportdir directory.

Store


Send all received logs to the local log
destinations. Usually the log destination is syslog.

Log


Send report information via HTTP to the
reporturl. Each host sends its report as a YAML
dump and this sends this YAML to a client via
HTTP POST.

HTTP


This report sends speciﬁc log messages to
speciﬁc email addresses based on the tags in
the log messages.

Tagmail


Graph all available data about hosts using
the RRD library.

RRDgraph


.
└── sample-report
├── lib
│   └── puppet
│   └── reports
│   └── sample.rb
└── manifests
└── init.pp

Simple report structure


require 'puppet'
require 'yaml'
require 'logger'

LOG = Logger.new('/tmp/puppet.log')

Puppet::Reports.register_report(:sample) do
def process
message = "Puppet run for #{self.host} #{self.status}"
Puppet.debug "Hello from sample report processor"
LOG.info message
end
end

Simple report processor


require 'puppet'
require 'yaml'
require 'logger'


def process
LOG.info message
end
end

Register report


require 'puppet'
require 'yaml'
require 'logger'


def process
LOG.info message
end
end

Do something with the data in self


Open source examples


IRC/Campﬁre


Ganglia


Graphs!


Zendesk


James Turnbull will have got there ﬁrst


Stored conﬁguration database


[master]
storeconfigs = true
dbadapter = mysql
dbuser = puppet
dbpassword = password
dbserver = localhost
dbsocket = /var/run/mysql.sock

Conﬁguration


SQL


mysql> SELECT filename,updated_at FROM source_files;
+-----------------------------------------------------+---------------------+
| filename | updated_at |
+-----------------------------------------------------+---------------------+
| NULL | 2011-10-11 10:34:36 |
| /etc/puppet/modules/users/manifests/init.pp | 2011-10-12 14:13:49 |
| /etc/puppet/modules/nagios/manifests/init.pp | 2011-10-12 14:13:49 |
| /etc/puppet/modules/ntp/manifests/init.pp | 2011-10-12 14:13:50 |
| /etc/puppet/modules/base_packages/manifests/init.pp | 2011-10-12 14:13:50 |
| /etc/puppet/modules/sudo/manifests/init.pp | 2011-10-12 14:13:50 |
| /etc/puppet/modules/apt/manifests/init.pp | 2011-10-12 14:13:50 |
| /etc/puppet/modules/logrotate/manifests/init.pp | 2011-10-12 14:13:51 |
| /etc/puppet/modules/ganglia/manifests/init.pp | 2011-10-12 14:13:52 |
| /etc/puppet/modules/motd/manifests/init.pp | 2011-10-12 14:13:57 |
| /etc/puppet/manifests/classes.pp | 2011-10-12 14:17:25 |
| /etc/puppet/modules/mysql/manifests/init.pp | 2011-10-12 14:17:25 |
| /etc/puppet/modules/apache2/manifests/init.pp | 2011-10-12 14:17:25 |
| /etc/puppet/modules/passenger/manifests/init.pp | 2011-10-12 14:17:26 |
+-----------------------------------------------------+---------------------+

Other interesting data too


Active Record

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/simplebitsdan/480696708

#!/usr/bin/env ruby

require 'puppet/rails'

Puppet[:config] = '/etc/puppet/puppet.conf'
Puppet.parse_config
config = Puppet.settings.instance_variable_get(:@values)
master_config = config[:master]

Load Puppet master conﬁguration


adapter = master_conf[:dbadapter]
args = {:adapter => adapter,
:log_level => master_conf[:rails_loglevel]}

args[:host] = master_conf[:dbserver]
args[:username] = master_conf[:dbuser]
args[:password] = master_conf[:dbpassword]
args[:database] = master_conf[:dbname]
args[:database] = "puppet" unless not args[:database].to_s.empty?
args[:port] = master_conf[:dbport]
socket = master_conf[:dbsocket]
args[:socket] = socket unless socket.to_s.empty?

Map database connection settings


ActiveRecord::Base.establish_connection(args)

Puppet::Rails::Host.all.each { |h|
puts "#{h.name}"
}

Make an active record query


⚡ ./list.rb
ubuntu.localdomain

Example run


Code spelunking


#!/usr/bin/env ruby

require 'puppet'
Puppet[:config] = "/etc/puppet/puppet.conf"
Puppet.parse_config
Puppet[:clientyamldir] = Puppet[:yamldir]
Puppet::Node.indirection.terminus_class = :yaml

Load Puppet master conﬁguration


nodes = Puppet::Node.indirection.search("*")
nodes.each do |n|
facts = Puppet::Node::Facts.indirection.find(n.name)
tags = Puppet::Resource::Catalog.indirection.find(n.name).tags
puts "#{n.name} - #{tags.join(', ')}"
end

Make a query using the internal API


⚡ sudo ./puppettags.rb
ubuntu.localdomain - settings, default, node

Needs root permissions


Dashboards


web-puppet


JSON over HTTP


capistrano-puppet


require 'capistrano-puppet'

web_puppet = CapistranoPuppet::Server.new(
'http://username:password@localhost:9295')

role :web do
web_puppet.get_servers('webserver')
end

Get hosts from puppet


Visualisation


Your imagination
?

- web-puppet - https://github.com/garethr/web-puppet
- capistrano-puppet - https://github.com/garethr/capistrano-puppet
- puppet-ganglia - https://github.com/jamtur01/puppet-ganglia
- puppet-zendesk - https://github.com/jamtur01/puppet-zendesk
- puppet-irc - https://github.com/jamtur01/puppet-irc
- puppet-campﬁre - https://github.com/jamtur01/puppet-campﬁre

Links


One more thing

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/benterrett/6852348725/

Questions?

gareth rushgrove | morethanseven.net http://ﬂickr.com/photos/psd/102332391/

Puppet Data Mining

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Puppet Data Mining

Similar to Puppet Data Mining (20)

More from Gareth Rushgrove

More from Gareth Rushgrove (16)

Recently uploaded

Recently uploaded (20)

Puppet Data Mining