PuppetDB: Higher-order Puppet

higher-order puppetPuppetDB

deepak@puppetlabs.com@grim_radical

Let’s talk about...

puppet generateslots of data

persistent data long termdata ephemeral data machlocal data meticulously straapuppet generates data frd lots of data free form dahuman readable data machdata resource data dependdata ssl certificate data ho

resources

ﬁle { “/tmp/foo”: content => “This is a test”}

{"exported": false, "ﬁle": "/puppet/site.pp", "line": 1, "parameters": { "content": "This is a test" }, "tags": [ "ﬁle", "node", “default” ], "title": "/tmp/foo", "type": "File"}

File “/tmp/foo/bar” User “deepak” Dir “/tmp/foo” Dir “/tmp”

Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”

catalogs

resourcecatalog

resource catalogcowherd

resource catalogcow crowherd murder

resource catalogcow superhero crowherd avengers murder

Group[peadmin] User[peadmin] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile]peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/sha Relationships

Group[peadmin] Group[puppet-dashboard] Class[Pe_accounts::Data] User[peadmin] User[puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/agent] File[/opt/puppet/libexec/mcollective/mcollective/security] Exec[mcollective-server-cert] File[/etc/puppetlabs/mcollective/ssl] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Pe_accounts::User[puppet-dashboard] File[/opt/puppet/share/puppet-dashboard] Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/util] File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb] File[mcollective-cert.pem] File[mcollective-private.pem] File[mcollective-public.pem] File[/etc/puppetlabs/mcollective/ssl/clients] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile] Exec[puppet-dashboard-client-cert] File[/opt/puppet/share/puppet-dashboard/.mcollective.d] File[/opt/puppet/share/puppet-dashboard/.mcollective] File[/opt/puppet/share/puppet-dashboard/.bashrc.custom] File[/opt/puppet/share/puppet-dashboard/.bashrc] File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim] File[/opt/puppet/share/puppet-dashbo/mcollective/mcollective/agent/puppetral.rb] File[/etc/puppetlabs/mcollective/server.cfg] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl] File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb] File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb] File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[peadmin-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem] File[/opt/puppet/share/puppet-dashboard/.ssh/ Service[mcollective] Relationships

Group[peadmin] User[peadmin] File[/var/lib/peadmin] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc]le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]

Catalog: all the things we manage on a node, and how they relate to each other

> facter

netmask_lo: 255.0.0.0 kernel: Linux augeasversion: 0.10.0 kernelrelease: 2.6.32-5-686 fqdn: pe-debian6.localdomain ipaddress: 172.16.245.128 manufacturer: "VMware, Inc." processor0: Intel(R) Core(TM) processorcount: "1" i7-2635QM CPU @ 2.00GHz productname: VMware Virtual lsbdistrelease: 6.0.2Platform uniqueid: 007f0101 physicalprocessorcount: 1 hardwaremodel: i686 facterversion: 1.6.7 kernelversion: 2.6.32 boardproductname: 440BX operatingsystem: DebianDesktop Reference Platform architecture: i386 kernelmajversion: "2.6" lsbdistdescription: Debian GNU/ hardwareisa: unknown Linux 6.0.2 (squeeze) timezone: PDT lsbmajdistrelease: "6" puppetversion: 2.7.12 (Puppet interfaces: "eth0,lo"Enterprise 2.5.1) ipaddress_lo: 127.0.0.1 lsbdistcodename: squeeze uptime_days: 0 is_virtual: "true" lsbdistid: Debian operatingsystemrelease: 6.0.2 rubysitedir: /opt/puppet/lib/ virtual: vmware site_ruby/1.8 type: Other rubyversion: 1.8.7 domain: localdomain osfamily: Debian hostname: pe-debian6 memorytotal: &id001 502.57 MB selinux: "false" memorysize: *id001

Catalogs:what we tell puppetabout a nodeFacts:what a node tellspuppet about itself

It’s about who controlsthe information.

“Theres a war out there,old friend. A world war.And its not about whosgot the most bullets.It’s about who controlsthe information. -- Sneakers (1992)What we see and hear,how we work, what wethink... its all about theinformation!”

every resource every parameterevery relationship every class every fact for every node

Query this data, foruse in scripts orother tools

Integration withother tools is great,but can we feed thatdata back intopuppet itself?

storeconﬁgs

Configure a nodeusing resourcesfrom other nodes

class ssh { @@sshkey { $hostname: type => dsa, key => $sshdsakey } Sshkey <<| |>>}

Every host exportsits public key, andimports the publickeys of every othernode, automatically!

How aboutmonitoring?

class nagios_target { @@nagios_host { $fqdn: ensure => present, alias => $hostname, address => $ipaddress, use => "generic-host", } @@nagios_service { "check_ping_${hostname}": check_command => "check_ping!100.0,20%!500.0,60%", use => "generic-service", host_name => "$fqdn", notiﬁcation_period => "24x7", service_description => "${hostname}_check_ping" }}

class nagios-monitor { # collect resources and # populate /etc/nagios/nagios_*.cfg Nagios_host <<||>> Nagios_service <<||>>}

Thus, you canautomatically createchecks for thingsyou’re managing

key distribution monitoring clustered servicesmaster/slave replication load balancers shared filesystems firewall rules ...

Higher-order Puppet

Using Puppet’sknowledge toimprove Puppet’sknowledge

Using Puppet’sknowledge toimprove Puppet’sknowledge Achievement unlocked YO DAWG

Using Puppet’sknowledge toimprove Puppet’sknowledge

Data is great!

Reading from the Puppet Data Library Nick Lewis 3:50P @ Meeting Room 2

Why aren’t we doingstuff like this all the damn time?

V UME OL

Every node,on every puppet run,generates data

We have customersgenerating over750GB of data a day!even storing a small subset ofthat much information adds up...

(a brief simulation)

When data storage isslow, the wholesystem slows downand it makes baby Deepak cry! :(

APIissues

Current APIs arelimited!Hard to get at the data, andperformance concerns discourageuse

We demand: Store as much data as we can! Much better queryability!Oh yeah, but: Don’t slow down the system! Don’t compromise reliability!

PuppetDB

PuppetDB Definitely Better!

Fast storageof catalogs & facts like, *way* faster!

Compatiblewith storeconfigs andinventory service you don’t have to change your Puppet code!

HTTP APIsfor resource, fact, andnode retrieval plenty of data, just a “curl” away!

Securedusing SSL client andserver certificatesthe same certificate infrastructure you’re already using!

science &secret alientechnology

Exportingresources

PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g

PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g

PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F

PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F

PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g

Resourcecollection

PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F ?

PuppetDB Server DLO DB Workers HTTP MQ ? FAgent Master Facts Catalo Resrc F g

PuppetDB Server DLO DB Workers ? F HTTP MQAgent Master Facts Catalo Resrc F g

PuppetDB Server DLO DB Workers F HTTP MQ ?Agent Master Facts Catalo Resrc F g

PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g ?

PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g

Failure

PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F

PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g

PuppetDB Server DLO F DB Workers HTTP MQAgent Master Facts Catalo Resrc g

Deployment

PuppetDB Server DLO DB Workers HTTP MQ

PuppetDB Server Workers DLO DB HTTP MQ

PuppetDB Server Workers DLOHTTP DBProxy(SSL) HTTP MQ

(another demo)

Reliability!

We work very hard topersist everything weaccept

Acknowledgements with UUIDS,Checksums,Queueing,Automatic retry,Automatic reconnect, and the Dead Letter Office if all else fails!

Anything Puppetdoes with PuppetDB,you can do, too we don’t cheat!

Query your own resources,Upload new fact sets,Create catalogs,Inspect facts, all open and documented!

#> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}

#> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, Achievement unlocked "RateUnit": "SECONDS", "StdDev": 833.6079354075728} WALL OF TEXT

#> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}

curl -H "Accept: application/json""http://puppetdb/facts/host.my.net"

curl -H "Accept: application/json""http://puppetdb/resources?query=..."

https://github.com/dalen/puppet-puppetdbquery

Transparent!

Ships with a real-time dashboard,Dozens of metrics and gauges,Correlate-able logs,Easy to monitor we care about operational visibility!

https://github.com/jasonhancock/nagios-puppetdb

Speedy!

We’ve seen huge reductions in compile times, resource collection times, time to persist catalogs and facts, etc. O_o

ONE DOES NOT SIMPLY SPEED UP PUPPET

Notable details

Posit:Hosts are not entirely unique snowflakes

Therefore: A resource oftenexists across multiple hosts

Feature: Single-instanceresource storage

Resource dedupeCompute unique hashes for resourcesWe quickly hash all the resources in a catalog,and use bulk operations to compare them tohashes stored.

Resource dedupeSignificant speed improvement!Internal to Puppet Labs, we see ~83% resourceduplication; this number is consistent with whatwe’ve seen in most customer environments.

Posit:Puppet runs frequently, but catalogs change infrequently

Therefore:We’ll often receive the same catalog for a host

Feature:Single-instancecatalog storage

Catalog dedupeCompute unique hashes for catalogsPuppet Labs sees ~88% catalog duplication, restof the planet sees even bigger numbersBig savings!

Posit:You have more than onecore, though storeconfigs is single-threaded

Therefore:Throughput is not maximized

Feature:Massively parallel operation

ParallelWe can pat our heads and rub our tummies atthe same timeDatabase operations don’t block MQ operationsdon’t block HTTP operations don’t block hashcomputation operations don’t block metriccalculations don’t block...Dozens of threads, zero locks

So anyways,

Documented athttp://docs.puppetlabs.com/puppetdb install, config, upkeep, specs, the works!

Packagedas deb and rpm forboth open source andPuppet Enterprise available in the Puppet Labs package repositories

Puppetizedusing thepuppetlabs/puppetdbmodule available now, on the Module Forge!

> puppet module install puppetlabs/puppetdb> vim site.ppnode puppetmaster { include puppetdb include puppetdb::master::config}

Open sourcehttp://github.com/puppetlabs/puppetdb same license as Puppet itself!

Production ready!

Many productiondeploymentsSmall shops with a dozen hosts,large shops with thousands ofhosts, intercontinentaldeployments... over a billion resources served!

Coming soon!

Report storageHistorical dataGrand Unified Query and of course, keep it fast!

Use it! and send us moredashboard screenshots! :)

deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode]

7 5 3 11 82 9 10

7 5 3 11 82 9 Achievement unlocked Let’s get TOPOLOGICAL! 10

7 5 3 11 82 9 10

PuppetDB: Higher-order Puppet

by grim_radical on Nov 14, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

PuppetDB: Higher-order Puppet Presentation Transcript