The life and times of
PuppetDB
Friday, August 23, 13

DEEPAK GIRIDHARAGOPAL
deepak@puppetlabs.com
@grim_radical
Friday, August 23, 13

We need to talk!
Friday, August 23, 13

Friday, August 23, 13

Friday, August 23, 13

Puppet
agent
Puppet
master
Friday, August 23, 13

Puppet
agent
Puppet
master
facts
Friday, August 23, 13

Puppet
agent
Puppet
master
facts
netmask_lo: 255.0.0.0
augeasversion: 0.10.0
fqdn: pe-debian6.localdomain
manufacturer: "VMware, Inc."
processorcount: "1"
productname: VMware Virtual
Platform
physicalprocessorcount: 1
facterversion: 1.6.7
boardproductname: 440BX Desktop
Reference Platform
kernelmajversion: "2.6"
hardwareisa: unknown
timezone: PDT
puppetversion: 2.7.12 (Puppet
Enterprise 2.5.1)
lsbdistcodename: squeeze
is_virtual: "true"
operatingsystemrelease: 6.0.2
virtual: vmware
type: Other
domain: localdomain
hostname: pe-debian6
selinux: "false"
kernel: Linux
kernelrelease: 2.6.32-5-686
ipaddress: 172.16.245.128
processor0: Intel(R) Core(TM)
i7-2635QM CPU @ 2.00GHz
lsbdistrelease: 6.0.2
uniqueid: 007f0101
hardwaremodel: i686
kernelversion: 2.6.32
operatingsystem: Debian
architecture: i386
lsbdistdescription: Debian GNU/Linux
6.0.2 (squeeze)
lsbmajdistrelease: "6"
interfaces: "eth0,lo"
ipaddress_lo: 127.0.0.1
uptime_days: 0
lsbdistid: Debian
rubysitedir: /opt/puppet/lib/site_ruby/
1.8
rubyversion: 1.8.7
osfamily: Debian
memorytotal: &id001 502.57 MB
memorysize: *id001
boardmanufacturer: Intel Corporation
path: /usr/local/sbin:/usr/local/bin:/
Friday, August 23, 13

Puppet
agent
Puppet
master
facts
Friday, August 23, 13

Puppet
agent
Puppet
master
facts
Friday, August 23, 13

Puppet
agent
Puppet
master
catalog
Friday, August 23, 13

Puppet
agent
Puppet
master
catalog
Friday, August 23, 13

file {“/tmp/foo”: content => “This is a test”}
Friday, August 23, 13

target: &id063 !ruby/object:Puppet::Resource
catalog: *id001
exported: false
file: /etc/puppetlabs/puppet/manifests/site.pp
line: 44
parameters:
!ruby/sym content: This is a test
!ruby/sym backup: main
reference: "File[/tmp/foo]"
tags:
- file
- node
- default
- class
title: /tmp/foo
type: File
file {“/tmp/foo”: content => “This is a test”}
Friday, August 23, 13

File[/var/lib/peadmin/.vim]
le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bashrc]
Group[peadmin]
User[peadmin]
Friday, August 23, 13

Relationships
File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective]
File[/opt/puppet/shapeadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
Pe_accounts::Home_dir[/var/lib/peadmin]File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Pe_accounts::User[peadmin]
Group[peadmin]
User[peadmin]
Friday, August 23, 13

Relationships
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
File[/opt/puppet/share/puppet-dashboard/.ssh/File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashbo
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Pe_accounts::User[peadmin] Pe_accounts::User[puppet-dashboard]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Friday, August 23, 13

Puppet
agent
Puppet
master
catalog
Friday, August 23, 13

Puppet
agent
Puppet
master
catalog
Friday, August 23, 13

Puppet
agent
Puppet
master
catalog
Friday, August 23, 13

Puppet
agent
Puppet
master
report
Friday, August 23, 13

Puppet
agent
Puppet
master
report
"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status
change_count: 1
changed: true
evaluation_time: 0.001869
events:
- !ruby/object:Puppet::Transaction::Event
audited: false
desired_value: !ruby/sym file
historical_value:
message: *id006
name: !ruby/sym file_created
previous_value: !ruby/sym absent
property: ensure
status: success
time: 2011-10-25 18:51:37.143970 -07:00
failed: false
file: *id007
line: 44
out_of_sync: true
out_of_sync_count: 1
resource: "File[/tmp/foo]"
resource_type: File
skipped: false
tags:
- file
- node
- default
- class
time: 2011-10-25 18:51:37.143396 -07:00
title: /tmp/foo
Friday, August 23, 13

Puppet
agent
Puppet
master
report
Friday, August 23, 13

Puppet
agent
Puppet
master
report
Friday, August 23, 13

Puppet
agent
Puppet
master
Friday, August 23, 13

Puppet
agent
Puppet
master
Friday, August 23, 13

Friday, August 23, 13

Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
catalog
facts
catalog
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
catalog
catalog
facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
catalog facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
catalog facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
report
catalog facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
report
catalog facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
report
catalog facts
Friday, August 23, 13

Puppet
agent
Puppet
master PuppetDB
report
catalog facts
Friday, August 23, 13

Friday, August 23, 13

Friday, August 23, 13

Active
Record
Puppet
master
catalog
Friday, August 23, 13

Active
Record
Puppet
master
catalogcatalogcatalogcatalogcatalogcatalog
Friday, August 23, 13

Active
Record
Puppet
master
catalogcatalogcatalogcatalogcatalog catalog
Friday, August 23, 13

Active
Record
Puppet
master
catalogcatalogcatalogcatalog catalogcatalog
Friday, August 23, 13

Active
Record
Puppet
master
catalogcatalogcatalog catalogcatalogcatalog
Friday, August 23, 13

Active
Record
Puppet
master
catalogcatalog catalogcatalogcatalogcatalog
Friday, August 23, 13

Active
Record
Puppet
master
catalog catalogcatalogcatalogcatalogcatalog
Friday, August 23, 13

Active
Record
Puppet
master
catalog catalog
Friday, August 23, 13

Puppet
master
catalog
Friday, August 23, 13

Friday, August 23, 13

Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
agent agent agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agent
Puppet
agentFriday, August 23, 13

Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Which boxes are
running nginx?
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
How many servers
are running a
vulnerable version
of rails?
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
What are the IP
addresses of my
webservers?
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Which users have
sudo access?
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
Friday, August 23, 13

Active
Record
LOLWUT
Friday, August 23, 13

Active
Record
LOLWUT
ಠ ಠ
_
Friday, August 23, 13

And now for
something
completely
different
Friday, August 23, 13

PuppetDB
Friday, August 23, 13

/resources/Service/
nginx
PuppetDB
Friday, August 23, 13

resources
/resources/Service/
nginx
PuppetDB)
O O
Friday, August 23, 13

PuppetDB
Friday, August 23, 13

/resources/Package/
rails
PuppetDB
Friday, August 23, 13

resources
/resources/Package/
rails
PuppetDB)
O O
Friday, August 23, 13

PuppetDB
Friday, August 23, 13

/nodes/foo.com/
resources/User/
deepak
PuppetDB
Friday, August 23, 13

resources
/nodes/foo.com/
resources/User/
deepak
PuppetDB)
O O
Friday, August 23, 13

(demo)
Friday, August 23, 13

We built something
quite different
Friday, August 23, 13

1. Asynchrony
Friday, August 23, 13

Storage &
Querying
Friday, August 23, 13

CQRS
Friday, August 23, 13

Command
Query
Responsibility
Separation
use a different model to update
information than the model you
use to read information
Friday, August 23, 13

CQRS
write pipeline
async, parallel, MQ-based, with
automatic retry
Friday, August 23, 13

{
:command "replace catalog"
:version 2
:payload {...}
}
Friday, August 23, 13

/commands MQ Parse
Delayed
Dead Letter
Office
Process
UUID
Friday, August 23, 13

Command
processors must be
retry-aware
expect failure, because
it *will* happen.
Friday, August 23, 13

Failures like,
oh I don't know,
a database crash?
Friday, August 23, 13

2. New runtime
Friday, August 23, 13

Fast,
Free,
Portable,
Multi-core,
Popular,
The JVM is all these things
Friday, August 23, 13

Haters gonna hate!
Friday, August 23, 13

Tons and tons of high
quality libraries
Web servers, concurrency
frameworks, databases, fast
parsing/lexing, clustering,
debugging, profiling, etc.
Friday, August 23, 13

Can ship an uberjar,
makes deployment
straightforward with
few moving pieces
Friday, August 23, 13

And it's fast.
Friday, August 23, 13

Nobody cares what
runtime we use.
Users just want stuff
to work.
Friday, August 23, 13

3. AST querying
Friday, August 23, 13

Queries
are expressed in their
own“language”
domain specific,AST-based
query language
Friday, August 23, 13

["and",
["=", "type", "User"],
["=", "title", "deepak"]]
Friday, August 23, 13

["and",
["=", ["fact", "operatingsystem"], "Debian"],
["<", ["fact", "uptime_seconds"], 10000]]
Friday, August 23, 13

["and",
["=", "name", "ipaddress"],
["in", "certname",
["extract", "certname", ["select-resources",
["and",
["=", "type", "Class"],
["=", "title", "Apache"]]]]
Friday, August 23, 13

["or",
["=", "certname", "foo.com"],
["=", "certname", "bar.com"],
["=", "certname", "baz.com"]]
Friday, August 23, 13

We walk the tree,
compiling it to
efficient SQL
Friday, August 23, 13

Haters gonna hate!
Friday, August 23, 13

AST-based API lets
users write their own
languages
ah, you’ve got to love
open source!
Friday, August 23, 13

(Package[httpd] and country=fr)
or country=us
Package["mysql-server"]
and architecture=amd64
Erik Dalén, Spotify
https://github.com/dalen/puppet-puppetdbquery
Friday, August 23, 13

AST-based API lets
us more safely
manipulate queries
Friday, August 23, 13

daenny, Puppetboard
https://github.com/nedap/puppetboard
Friday, August 23, 13

Puppet Enterprise, Event Inspector
https://puppetlabs.com
Friday, August 23, 13

Foreman Integration (CERN)
https://github.com/cernops/puppetdb_foreman
Web UI
https://github.com/dima-exe/puppetdb-db
Web UI
https://github.com/gbougeard/puppetdb-frontend
Friday, August 23, 13

Ruby
https://github.com/dalen/puppet-puppetdbquery
Ruby (DataMapper)
https://github.com/dalen/dm-puppetdb-adapter
Ruby
https://github.com/ripienaar/ruby-puppetdb
Friday, August 23, 13

Python
https://github.com/nedap/pypuppetdb
Python
https://github.com/arcus-io/puppetdb-python
Python
https://github.com/JHaals/puppetdb-grep
Friday, August 23, 13

Java
https://github.com/thallgren/puppetdb-javaclient
Go
https://github.com/nightlyone/puppetquery
Scala
https://github.com/gbougeard/puppetdb-frontend
CoffeeScript
https://gist.github.com/pmuellr/5591686
Node.js
https://github.com/nightfly19/minidb
Friday, August 23, 13

MCollective
https://github.com/ploubser/mcollective-puppetdb-
discovery
Rundeck
https://github.com/sirhopcount/puppetdb-rundeck
Rundeck
https://github.com/martin2110/puppetdb-rundeck
Friday, August 23, 13

OpenStack
https://github.com/bodepd/puppet-
openstack_puppetdb
Vagrant
https://github.com/grahamgilbert/vagrant-
puppetmaster
PowerDNS
https://github.com/evenup/evenup-pdns
Friday, August 23, 13

4. Boring technology
Friday, August 23, 13

Relational Database,
embedded or
PostgreSQL
because they’re actually pretty
fantastic at ad-hoc queries,
aggregation, windowing, etc.
while maintaining safety
Friday, August 23, 13

Friday, August 23, 13

Relational Database,
embedded or
PostgreSQL
we use arrays, recursive queries,
indexing inside complex
structures
Friday, August 23, 13

5. Weird alien
technology
Friday, August 23, 13

Friday, August 23, 13

--Jeff Gagliardi
Friday, August 23, 13

Thousands of deployments,
Hundreds of threads per install,
Zero deadlocks,
Zero bugs involving mutable state
companion Ruby code has
~10x the defect rate
Friday, August 23, 13

All with a pretty tiny codebase
Friday, August 23, 13

6. Conjectures
about performance
Friday, August 23, 13

Posit:
A resource often
exists across multiple
hosts
Friday, August 23, 13

Feature:
Single-instance
resource storage
Friday, August 23, 13

Posit:
We’ll often receive the
same catalog for a
host
Friday, August 23, 13

Feature:
Single-instance
catalog storage
Friday, August 23, 13

In the field,we
almost always see
Resource and catalog
duplication rates of
over 85%.
Friday, August 23, 13

Monitoring and
instrumentation is a
big deal.Users want
easy ways to
consume metrics and
analyze performance.
Friday, August 23, 13

Friday, August 23, 13

Nagios
https://github.com/jasonhancock/nagios-puppetdb
Nagios
https://github.com/favoretti/puppetdb-external-naginator
Munin
https://github.com/vpetersson/munin_puppetdb
Munin
https://github.com/dalen/puppetdb-muninplugins
Collectd
https://gist.github.com/mfournier/5615125
Friday, August 23, 13

Turns out, people
appreciate these
efforts
Friday, August 23, 13

(how many?)
Friday, August 23, 13

Thousands of
production
deployments
Small shops with a dozen hosts,
large shops with thousands of
hosts,standalone,clustered...
Friday, August 23, 13

There is a new
deployment of
PuppetDB every
15 minutes.
Friday, August 23, 13

So...long time since
we last spoke
Friday, August 23, 13

Availability
Friday, August 23, 13

Available in PE3
On by default,fully supported,
and the basis for upcoming
reporting and analytics features.
Friday, August 23, 13

Performance
Friday, August 23, 13

20% faster storage
Improvements to memoization
and caching,eliminate double-
serialization,nuked superfluous
indexes
Friday, August 23, 13

Much faster terminus
Better caching and data
structures.For a catalog with
10k resources,drops
serialization time from ~80s to
~6s.
Friday, August 23, 13

Resiliance
Friday, August 23, 13

Death to keystores
Can now use PEM certificates
directly,eliminating one of the
largest sources of configuration
problems.
Friday, August 23, 13

Configurable HTTPS
Can customize the set of cipher
suites and SSL protocols you'd
like to use,to match your
security needs.
Friday, August 23, 13

Automatic:
-Recovery from MQ corruption
-Compression of the DLO
-Purging of inactive node data
-DB connection recycling
Friday, August 23, 13

Backup and restore
Now integrated into the
daemon,can restore while
PuppetDB is running.
Friday, August 23, 13

Query changes
Friday, August 23, 13

V2 API
-No need to ask for only active
nodes
-Full fact queries (instead of
just a list of facts for a node)
-Node metadata
Friday, August 23, 13

Wildcard Accept
Headers
curl localhost:8080/v2/nodes
Friday, August 23, 13

Subqueries
You can now correlate data from
resource queries with fact
queries with node queries.
"Give me the IP address of all machines with
the Nginx service configured"
Friday, August 23, 13

Report storage
-Comes with a report
processing plugin
-Store report-level metadata
-Can do queries on events that
span reports
-Basis for PE's Event Inspector
Friday, August 23, 13

Streaming
queries!
Friday, August 23, 13

Streaming queries
Stream results to clients on-the-
fly,as they come in from the
database.
Massively lower latency for first
response!
Friday, August 23, 13

resourceresourceresourceresourceresourceresourceresource
PuppetDB
Friday, August 23, 13

resourceresourceresourceresourceresourceresourceresource
/v2/resources
PuppetDB
Friday, August 23, 13

resourceresourceresourceresourceresourceresourceresource
/v2/resources
PuppetDB)
O O
Friday, August 23, 13

(demo)
Friday, August 23, 13

Coming up!
Friday, August 23, 13

We will be developing tools to replicate
data from one PuppetDB daemon to
another. This will help with HA and DR.
PuppetDB
Diff &
Mirror PuppetDB
Friday, August 23, 13

By initially developing an out-of-band
mirroring tool, we can create more
interesting replication topologies:
PuppetDB
Diff &
Mirror PuppetDB
Diff &
Mirror
Friday, August 23, 13

We can also later optimize the process to
lower latency, but preserve eventual
consistency:
PuppetDB
Diff &
Mirror
PuppetDBDirect MQ connection
Friday, August 23, 13

More flexible routing is coming, allowing
for soft failures and read/write splits:
PuppetDB
Puppetmaster
PuppetDB
Replication
Catalogs,Facts,
Reports
Collection
queries
Log error and
continue
Friday, August 23, 13

So anyways,
Friday, August 23, 13

Documented at
http://
docs.puppetlabs.com
/puppetdb
install, config, upkeep, specs,
the works!
Friday, August 23, 13

Packaged
as deb and rpm for
open source,part of
Puppet Enterprise
available in the Puppet Labs
package repositories
Friday, August 23, 13

Puppetized
using the
puppetlabs/puppetdb
module
available now, on the
Module Forge!
Friday, August 23, 13

Open source
http://github.com/
puppetlabs/puppetdb
same license as Puppet itself!
Friday, August 23, 13

deepak
giridharagopal
deepak@puppetlabs.com
@grim_radical [github twitter freenode]
Friday, August 23, 13