Your SlideShare is downloading. ×
0
higher-order puppetPuppetDB
deepak@puppetlabs.com@grim_radical
Let’s talk about...
puppet generateslots of data
persistent data long termdata ephemeral data machlocal data meticulously straapuppet generates data frd lots of data free ...
resources
file { “/tmp/foo”:  content => “This is a test”}
{"exported": false, "file": "/puppet/site.pp", "line": 1, "parameters": {     "content": "This is a test" }, "tags": [     ...
File “/tmp/foo/bar”   User “deepak”   Dir “/tmp/foo”     Dir “/tmp”
Dir “/tmp”    User “deepak”      Dir “/tmp/foo”   File “/tmp/foo/bar”
Dir “/tmp”    User “deepak”      Dir “/tmp/foo”   File “/tmp/foo/bar”
catalogs
resourcecatalog
resource       catalogcowherd
resource       catalogcow               crowherd              murder
resource        catalogcow    superhero   crowherd   avengers    murder
Group[peadmin]                                                                                                            ...
Group[peadmin]                                                                                                            ...
Group[peadmin]                                                                                       User[peadmin]        ...
Catalog:    all the things we  manage on a node, and how they relate        to each other
facts
> facter
netmask_lo: 255.0.0.0             kernel: Linux augeasversion: 0.10.0             kernelrelease: 2.6.32-5-686 fqdn: pe-deb...
Catalogs:what we tell puppetabout a nodeFacts:what a node tellspuppet about itself
It’s about who controlsthe information.
“Theres a war out there,old friend. A world war.And its not about whosgot the most bullets.It’s about who controlsthe info...
every resource every parameterevery relationship       every class        every fact   for every node
Query this data, foruse in scripts orother tools
Integration withother tools is great,but can we feed thatdata back intopuppet itself?
storeconfigs
Configure a nodeusing resourcesfrom other nodes
class ssh {    @@sshkey { $hostname:      type => dsa,      key => $sshdsakey    }    Sshkey <<| |>>}
Every host exportsits public key, andimports the publickeys of every othernode, automatically!
How aboutmonitoring?
class nagios_target {    @@nagios_host { $fqdn:      ensure => present,      alias => $hostname,      address => $ipaddres...
class nagios-monitor {    # collect resources and    # populate /etc/nagios/nagios_*.cfg    Nagios_host <<||>>    Nagios_s...
Thus, you canautomatically createchecks for thingsyou’re managing
key distribution      monitoring  clustered servicesmaster/slave replication    load balancers  shared filesystems     fir...
Higher-order   Puppet
Using Puppet’sknowledge toimprove Puppet’sknowledge
Using Puppet’sknowledge toimprove Puppet’sknowledge      Achievement unlocked            YO DAWG
Using Puppet’sknowledge toimprove Puppet’sknowledge
Data is great!
Reading from the Puppet Data Library            Nick Lewis     3:50P @ Meeting Room 2
Why aren’t we doingstuff like this all the    damn time?
V    UME    OL
Every node,on every puppet run,generates data
We have customersgenerating over750GB of data a day!even storing a small subset ofthat much information adds up...
(a brief simulation)
When data storage isslow, the wholesystem slows downand it makes baby Deepak cry! :(
APIissues
Current APIs arelimited!Hard to get at the data, andperformance concerns discourageuse
We demand: Store as much data as we can!     Much better queryability!Oh yeah, but:  Don’t slow down the system!  Don’t co...
PuppetDB
PuppetDB     Definitely Better!
Fast storageof catalogs & facts          like, *way* faster!
Compatiblewith storeconfigs andinventory service     you don’t have to change           your Puppet code!
HTTP APIsfor resource, fact, andnode retrieval          plenty of data, just               a “curl” away!
Securedusing SSL client andserver certificatesthe same certificate infrastructure            you’re already using!
science     &secret alientechnology
Exportingresources
PuppetDB Server        DLO        DB        Workers        HTTP          MQAgent                              Master      ...
PuppetDB Server        DLO          DB        Workers         HTTP           MQAgent                                Master...
PuppetDB Server        DLO        DB        Workers        HTTP          MQAgent                              Master      ...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ         FAgent                              Ma...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ                      FAgent                   ...
PuppetDB Server        DLO        DB        Workers                      F        HTTP          MQAgent                   ...
PuppetDB Server        DLO            DB    Workers        F        HTTP          MQAgent                              Mas...
PuppetDB Server        DLO            DB    Workers        F        HTTP          MQAgent                              Mas...
Resourcecollection
PuppetDB Server        DLO        DB        Workers        HTTP          MQAgent                              Master      ...
PuppetDB Server        DLO          DB        Workers         HTTP           MQAgent                                Master...
PuppetDB Server        DLO        DB        Workers        HTTP          MQAgent                              Master      ...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ         FAgent                              Ma...
PuppetDB Server        DLO          DB        Workers         HTTP           MQ           ?            FAgent             ...
PuppetDB Server        DLO          DB        Workers           ?            F         HTTP           MQAgent             ...
PuppetDB Server        DLO          DB        Workers                        F         HTTP           MQ           ?Agent ...
PuppetDB Server        DLO          DB        Workers           F         HTTP           MQAgent                          ...
PuppetDB Server        DLO          DB        Workers           F         HTTP           MQAgent                          ...
PuppetDB Server        DLO          DB        Workers           F         HTTP           MQAgent                          ...
PuppetDB Server        DLO          DB        Workers           F         HTTP           MQAgent                          ...
PuppetDB Server        DLO          DB        Workers           F         HTTP           MQAgent                          ...
PuppetDB Server        DLO          DB        Workers           F         HTTP           MQAgent                          ...
PuppetDB Server        DLO             DB     Workers         F         HTTP           MQAgent                            ...
Failure
PuppetDB Server        DLO        DB        Workers        HTTP          MQAgent                              Master      ...
PuppetDB Server        DLO          DB        Workers         HTTP           MQAgent                                Master...
PuppetDB Server        DLO        DB        Workers        HTTP          MQAgent                              Master      ...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ         FAgent                              Ma...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ                      FAgent                   ...
PuppetDB Server        DLO        DB        Workers                      F        HTTP          MQAgent                   ...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ                      FAgent                   ...
PuppetDB Server        DLO        DB        Workers                      F        HTTP          MQAgent                   ...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ                      FAgent                   ...
PuppetDB Server        DLO        DB        Workers                      F        HTTP          MQAgent                   ...
PuppetDB Server        DLO        DB        Workers        HTTP          MQ                      FAgent                   ...
PuppetDB Server        DLO        DB        Workers                      F        HTTP          MQAgent                   ...
PuppetDB Server        DLO                             F        DB        Workers        HTTP          MQAgent            ...
Deployment
PuppetDB Server        DLO    DB        Workers   HTTP           MQ
PuppetDB Server  Workers    DLO                        DB   HTTP            MQ
PuppetDB Server          Workers    DLOHTTP                            DBProxy(SSL)           HTTP            MQ
(another demo)
Reliability!
We work very hard topersist everything weaccept
Acknowledgements with UUIDS,Checksums,Queueing,Automatic retry,Automatic reconnect,     and the Dead Letter Office if all ...
APIs!
Anything Puppetdoes with PuppetDB,you can do, too         we don’t cheat!
Query your own resources,Upload new fact sets,Create catalogs,Inspect facts,          all open and documented!
#> curl   -H "Accept: application/json"   "http://puppetdb/metrics/mbean/   com.puppetlabs.puppetdb.command:type=global,na...
#> curl   -H "Accept: application/json"   "http://puppetdb/metrics/mbean/   com.puppetlabs.puppetdb.command:type=global,na...
#> curl   -H "Accept: application/json"   "http://puppetdb/metrics/mbean/   com.puppetlabs.puppetdb.command:type=global,na...
curl   -H "Accept: application/json""http://puppetdb/facts/host.my.net"
curl    -H "Accept: application/json""http://puppetdb/resources?query=..."
https://github.com/dalen/puppet-puppetdbquery
Transparent!
Ships with a real-time dashboard,Dozens of metrics and gauges,Correlate-able logs,Easy to monitor we care about operationa...
https://github.com/jasonhancock/nagios-puppetdb
Speedy!
We’ve seen huge reductions in compile times,  resource collection times, time to persist           catalogs and facts, etc...
ONE DOES NOT SIMPLY  SPEED UP PUPPET
Notable details
Posit:Hosts are not entirely unique snowflakes
Therefore:  A resource oftenexists across multiple         hosts
Feature: Single-instanceresource storage
Resource dedupeCompute unique hashes for resourcesWe quickly hash all the resources in a catalog,and use bulk operations t...
Resource dedupeSignificant speed improvement!Internal to Puppet Labs, we see ~83% resourceduplication; this number is cons...
Posit:Puppet runs frequently, but catalogs change     infrequently
Therefore:We’ll often receive the same catalog for a          host
Feature:Single-instancecatalog storage
Catalog dedupeCompute unique hashes for catalogsPuppet Labs sees ~88% catalog duplication, restof the planet sees even big...
Posit:You have more than onecore, though storeconfigs   is single-threaded
Therefore:Throughput is not   maximized
Feature:Massively parallel   operation
ParallelWe can pat our heads and rub our tummies atthe same timeDatabase operations don’t block MQ operationsdon’t block H...
So anyways,
Documented athttp://docs.puppetlabs.com/puppetdb   install, config, upkeep, specs,                       the works!
Packagedas deb and rpm forboth open source andPuppet Enterprise   available in the Puppet Labs           package repositor...
Puppetizedusing thepuppetlabs/puppetdbmodule       available now, on the              Module Forge!
> puppet module install puppetlabs/puppetdb> vim site.ppnode puppetmaster {  include puppetdb  include puppetdb::master::c...
Open sourcehttp://github.com/puppetlabs/puppetdb  same license as Puppet itself!
Production ready!
Many productiondeploymentsSmall shops with a dozen hosts,large shops with thousands ofhosts, intercontinentaldeployments.....
Coming soon!
Report storageHistorical dataGrand Unified Query  and of course, keep it fast!
Use it!   and send us moredashboard screenshots! :)
deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode]
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7        5       3    11       82        9       10
7           5                 3    11               82           9      Achievement unlocked     Let’s get TOPOLOGICAL!   10
7        5       3    11       82        9       10
PuppetDB: Higher-order Puppet
PuppetDB: Higher-order Puppet
PuppetDB: Higher-order Puppet
PuppetDB: Higher-order Puppet
PuppetDB: Higher-order Puppet
Upcoming SlideShare
Loading in...5
×

PuppetDB: Higher-order Puppet

996

Published on

My talk at PuppetConf 2012 on PuppetDB, a Clojure-based centralized storage daemon for Puppet.

Video of this talk: http://www.youtube.com/watch?v=xw83cRofkpM&list=PLV86BgbREluVFB73Wwqp_tCbw5Z9TMLX1&index=4&feature=plpp_video

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
996
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
58
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • Not just in terms of volume, but also many different kinds of data\n
  • persistent, ephemeral, free form, machine readable, that&amp;#x2019;s a lot of stuff to sift through!\n\nso where to start?\n
  • \n
  • this is a resource, pictured as what you&amp;#x2019;d type\n
  • this is the same resource, only post-compilation. way more useful stuff in here!\n
  • so most of the time, you&amp;#x2019;ll have more than one resource at play on a node.\nBut you don&amp;#x2019;t want them just applied randomly; order is important!\n
  • You want the things at the top to happen first.\nInternally, Puppet represents this as a directed-graph we call...\n
  • \n
  • a collection of resources and their relationships is a catalog\n
  • a collection of resources and their relationships is a catalog\n
  • a collection of resources and their relationships is a catalog\n
  • This is the catalog for a fresh install of Puppet Enterprise\nNot too different from the one I showed earlier, just with more resources and relationships.\nBut this is actually a bit of a lie, because the _entire_ catalog...\n
  • Hard to see, but we can zoom in on a tiny area\n
  • So, same stuff you&amp;#x2019;re used to: users, groups, files, etc.\nTrying to make sense of this manually is insane, but that&amp;#x2019;s where Puppet comes in!\nInfrastructure is messy, but puppet untangles that web.\n
  • But that&amp;#x2019;s only one half of the story\n
  • \n
  • \n
  • and if that doesn&amp;#x2019;t cut it, you can always make your own\n
  • So, catalogs and facts are great, by why is having access to that stuff important?\n\n
  • Information is powerful!\n\nAs operators, our decisions are only as good as the information upon which they&amp;#x2019;re based. Tools are no different; they&amp;#x2019;re only as smart as the input data.\n
  • Information is powerful!\n\nAs operators, our decisions are only as good as the information upon which they&amp;#x2019;re based. Tools are no different; they&amp;#x2019;re only as smart as the input data.\n
  • Information is powerful!\n\nAs operators, our decisions are only as good as the information upon which they&amp;#x2019;re based. Tools are no different; they&amp;#x2019;re only as smart as the input data.\n
  • So what can you do when you&amp;#x2019;ve got all this juicy data sitting around?\n
  • Integration with monitoring systems, inventory databases, provisioning systems, auto-remediation code, ad-hoc activity\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • And it&amp;#x2019;s not just for things like key distribution and monitoring,\n
  • ...this is a generic pattern that can be applied to all manner of situations.\n\nAnything where you need information from one node to configure another\n
  • Utilizing your data in this way catapults you into the world of Higher-order puppet.\nBy which, I mean...\n
  • Delightfully meta, don&amp;#x2019;t you think? &lt;advance&gt;\n\nDoubles-down on your automation, making it even more powerful.\n
  • Delightfully meta, don&amp;#x2019;t you think? &lt;advance&gt;\n\nDoubles-down on your automation, making it even more powerful.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n...but it&amp;#x2019;s not just a question of scalability...\n
  • \n
  • ...and therein lies the problem\n
  • \n
  • A lot of people ask me what the DB stands for &lt;advance&gt;\n
  • I&amp;#x2019;ll talk more about how we make that happen in a bit\n
  • We&amp;#x2019;ve tested this on our own code, running Puppet Labs\n
  • \n
  • ...and it&amp;#x2019;s all built using...\n
  • So let&amp;#x2019;s talk about how it works, at a high-level.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • So earlier, I mentioned that we need PuppetDB to be reliable, queryable, and fast\n
  • \n
  • \n
  • In fact, let&amp;#x2019;s try an experiment...\n
  • \n
  • \n
  • for example...\n
  • &lt;explain what&amp;#x2019;s going on&gt;\n\nWow, that&amp;#x2019;s a big wall of text, isn&amp;#x2019;t it? &lt;advance&gt; But hopefully it illustrates how much instrumentation we&amp;#x2019;ve put in.\n
  • &lt;explain what&amp;#x2019;s going on&gt;\n\nWow, that&amp;#x2019;s a big wall of text, isn&amp;#x2019;t it? &lt;advance&gt; But hopefully it illustrates how much instrumentation we&amp;#x2019;ve put in.\n
  • \n
  • Query language is documented and pretty versatile.\nSuper-set of resource-collection features (like complex boolean operators).\n...and this is why people have written libraries for use with puppet.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How did we accomplish such a thing?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • And we run it ourselves at Puppet Labs!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript of "PuppetDB: Higher-order Puppet"

    1. 1. higher-order puppetPuppetDB
    2. 2. deepak@puppetlabs.com@grim_radical
    3. 3. Let’s talk about...
    4. 4. puppet generateslots of data
    5. 5. persistent data long termdata ephemeral data machlocal data meticulously straapuppet generates data frd lots of data free form dahuman readable data machdata resource data dependdata ssl certificate data ho
    6. 6. resources
    7. 7. file { “/tmp/foo”: content => “This is a test”}
    8. 8. {"exported": false, "file": "/puppet/site.pp", "line": 1, "parameters": { "content": "This is a test" }, "tags": [ "file", "node", “default” ], "title": "/tmp/foo", "type": "File"}
    9. 9. File “/tmp/foo/bar” User “deepak” Dir “/tmp/foo” Dir “/tmp”
    10. 10. Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”
    11. 11. Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”
    12. 12. catalogs
    13. 13. resourcecatalog
    14. 14. resource catalogcowherd
    15. 15. resource catalogcow crowherd murder
    16. 16. resource catalogcow superhero crowherd avengers murder
    17. 17. Group[peadmin] User[peadmin] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile]peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/sha Relationships
    18. 18. Group[peadmin] Group[puppet-dashboard] Class[Pe_accounts::Data] User[peadmin] User[puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/agent] File[/opt/puppet/libexec/mcollective/mcollective/security] Exec[mcollective-server-cert] File[/etc/puppetlabs/mcollective/ssl] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Pe_accounts::User[puppet-dashboard] File[/opt/puppet/share/puppet-dashboard] Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/util] File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb] File[mcollective-cert.pem] File[mcollective-private.pem] File[mcollective-public.pem] File[/etc/puppetlabs/mcollective/ssl/clients] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile] Exec[puppet-dashboard-client-cert] File[/opt/puppet/share/puppet-dashboard/.mcollective.d] File[/opt/puppet/share/puppet-dashboard/.mcollective] File[/opt/puppet/share/puppet-dashboard/.bashrc.custom] File[/opt/puppet/share/puppet-dashboard/.bashrc] File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim] File[/opt/puppet/share/puppet-dashbo/mcollective/mcollective/agent/puppetral.rb] File[/etc/puppetlabs/mcollective/server.cfg] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl] File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb] File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb] File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[peadmin-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem] File[/opt/puppet/share/puppet-dashboard/.ssh/ Service[mcollective] Relationships
    19. 19. Group[peadmin] User[peadmin] File[/var/lib/peadmin] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc]le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
    20. 20. Catalog: all the things we manage on a node, and how they relate to each other
    21. 21. facts
    22. 22. > facter
    23. 23. netmask_lo: 255.0.0.0 kernel: Linux augeasversion: 0.10.0 kernelrelease: 2.6.32-5-686 fqdn: pe-debian6.localdomain ipaddress: 172.16.245.128 manufacturer: "VMware, Inc." processor0: Intel(R) Core(TM) processorcount: "1" i7-2635QM CPU @ 2.00GHz productname: VMware Virtual lsbdistrelease: 6.0.2Platform uniqueid: 007f0101 physicalprocessorcount: 1 hardwaremodel: i686 facterversion: 1.6.7 kernelversion: 2.6.32 boardproductname: 440BX operatingsystem: DebianDesktop Reference Platform architecture: i386 kernelmajversion: "2.6" lsbdistdescription: Debian GNU/ hardwareisa: unknown Linux 6.0.2 (squeeze) timezone: PDT lsbmajdistrelease: "6" puppetversion: 2.7.12 (Puppet interfaces: "eth0,lo"Enterprise 2.5.1) ipaddress_lo: 127.0.0.1 lsbdistcodename: squeeze uptime_days: 0 is_virtual: "true" lsbdistid: Debian operatingsystemrelease: 6.0.2 rubysitedir: /opt/puppet/lib/ virtual: vmware site_ruby/1.8 type: Other rubyversion: 1.8.7 domain: localdomain osfamily: Debian hostname: pe-debian6 memorytotal: &id001 502.57 MB selinux: "false" memorysize: *id001
    24. 24. Catalogs:what we tell puppetabout a nodeFacts:what a node tellspuppet about itself
    25. 25. It’s about who controlsthe information.
    26. 26. “Theres a war out there,old friend. A world war.And its not about whosgot the most bullets.It’s about who controlsthe information. -- Sneakers (1992)What we see and hear,how we work, what wethink... its all about theinformation!”
    27. 27. every resource every parameterevery relationship every class every fact for every node
    28. 28. Query this data, foruse in scripts orother tools
    29. 29. Integration withother tools is great,but can we feed thatdata back intopuppet itself?
    30. 30. storeconfigs
    31. 31. Configure a nodeusing resourcesfrom other nodes
    32. 32. class ssh { @@sshkey { $hostname: type => dsa, key => $sshdsakey } Sshkey <<| |>>}
    33. 33. Every host exportsits public key, andimports the publickeys of every othernode, automatically!
    34. 34. How aboutmonitoring?
    35. 35. class nagios_target { @@nagios_host { $fqdn: ensure => present, alias => $hostname, address => $ipaddress, use => "generic-host", } @@nagios_service { "check_ping_${hostname}": check_command => "check_ping!100.0,20%!500.0,60%", use => "generic-service", host_name => "$fqdn", notification_period => "24x7", service_description => "${hostname}_check_ping" }}
    36. 36. class nagios-monitor { # collect resources and # populate /etc/nagios/nagios_*.cfg Nagios_host <<||>> Nagios_service <<||>>}
    37. 37. Thus, you canautomatically createchecks for thingsyou’re managing
    38. 38. key distribution monitoring clustered servicesmaster/slave replication load balancers shared filesystems firewall rules ...
    39. 39. Higher-order Puppet
    40. 40. Using Puppet’sknowledge toimprove Puppet’sknowledge
    41. 41. Using Puppet’sknowledge toimprove Puppet’sknowledge Achievement unlocked YO DAWG
    42. 42. Using Puppet’sknowledge toimprove Puppet’sknowledge
    43. 43. Data is great!
    44. 44. Reading from the Puppet Data Library Nick Lewis 3:50P @ Meeting Room 2
    45. 45. Why aren’t we doingstuff like this all the damn time?
    46. 46. V UME OL
    47. 47. Every node,on every puppet run,generates data
    48. 48. We have customersgenerating over750GB of data a day!even storing a small subset ofthat much information adds up...
    49. 49. (a brief simulation)
    50. 50. When data storage isslow, the wholesystem slows downand it makes baby Deepak cry! :(
    51. 51. APIissues
    52. 52. Current APIs arelimited!Hard to get at the data, andperformance concerns discourageuse
    53. 53. We demand: Store as much data as we can! Much better queryability!Oh yeah, but: Don’t slow down the system! Don’t compromise reliability!
    54. 54. PuppetDB
    55. 55. PuppetDB Definitely Better!
    56. 56. Fast storageof catalogs & facts like, *way* faster!
    57. 57. Compatiblewith storeconfigs andinventory service you don’t have to change your Puppet code!
    58. 58. HTTP APIsfor resource, fact, andnode retrieval plenty of data, just a “curl” away!
    59. 59. Securedusing SSL client andserver certificatesthe same certificate infrastructure you’re already using!
    60. 60. science &secret alientechnology
    61. 61. Exportingresources
    62. 62. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g
    63. 63. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g
    64. 64. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F
    65. 65. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
    66. 66. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
    67. 67. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    68. 68. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    69. 69. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    70. 70. Resourcecollection
    71. 71. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g
    72. 72. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g
    73. 73. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F
    74. 74. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F ?
    75. 75. PuppetDB Server DLO DB Workers HTTP MQ ? FAgent Master Facts Catalo Resrc F g
    76. 76. PuppetDB Server DLO DB Workers ? F HTTP MQAgent Master Facts Catalo Resrc F g
    77. 77. PuppetDB Server DLO DB Workers F HTTP MQ ?Agent Master Facts Catalo Resrc F g
    78. 78. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g ?
    79. 79. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g ?
    80. 80. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
    81. 81. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
    82. 82. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
    83. 83. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
    84. 84. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
    85. 85. Failure
    86. 86. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g
    87. 87. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g
    88. 88. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F
    89. 89. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
    90. 90. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
    91. 91. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    92. 92. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g
    93. 93. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    94. 94. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g
    95. 95. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    96. 96. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g
    97. 97. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
    98. 98. PuppetDB Server DLO F DB Workers HTTP MQAgent Master Facts Catalo Resrc g
    99. 99. Deployment
    100. 100. PuppetDB Server DLO DB Workers HTTP MQ
    101. 101. PuppetDB Server Workers DLO DB HTTP MQ
    102. 102. PuppetDB Server Workers DLOHTTP DBProxy(SSL) HTTP MQ
    103. 103. (another demo)
    104. 104. Reliability!
    105. 105. We work very hard topersist everything weaccept
    106. 106. Acknowledgements with UUIDS,Checksums,Queueing,Automatic retry,Automatic reconnect, and the Dead Letter Office if all else fails!
    107. 107. APIs!
    108. 108. Anything Puppetdoes with PuppetDB,you can do, too we don’t cheat!
    109. 109. Query your own resources,Upload new fact sets,Create catalogs,Inspect facts, all open and documented!
    110. 110. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}
    111. 111. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, Achievement unlocked "RateUnit": "SECONDS", "StdDev": 833.6079354075728} WALL OF TEXT
    112. 112. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}
    113. 113. curl -H "Accept: application/json""http://puppetdb/facts/host.my.net"
    114. 114. curl -H "Accept: application/json""http://puppetdb/resources?query=..."
    115. 115. https://github.com/dalen/puppet-puppetdbquery
    116. 116. Transparent!
    117. 117. Ships with a real-time dashboard,Dozens of metrics and gauges,Correlate-able logs,Easy to monitor we care about operational visibility!
    118. 118. https://github.com/jasonhancock/nagios-puppetdb
    119. 119. Speedy!
    120. 120. We’ve seen huge reductions in compile times, resource collection times, time to persist catalogs and facts, etc. O_o
    121. 121. ONE DOES NOT SIMPLY SPEED UP PUPPET
    122. 122. Notable details
    123. 123. Posit:Hosts are not entirely unique snowflakes
    124. 124. Therefore: A resource oftenexists across multiple hosts
    125. 125. Feature: Single-instanceresource storage
    126. 126. Resource dedupeCompute unique hashes for resourcesWe quickly hash all the resources in a catalog,and use bulk operations to compare them tohashes stored.
    127. 127. Resource dedupeSignificant speed improvement!Internal to Puppet Labs, we see ~83% resourceduplication; this number is consistent with whatwe’ve seen in most customer environments.
    128. 128. Posit:Puppet runs frequently, but catalogs change infrequently
    129. 129. Therefore:We’ll often receive the same catalog for a host
    130. 130. Feature:Single-instancecatalog storage
    131. 131. Catalog dedupeCompute unique hashes for catalogsPuppet Labs sees ~88% catalog duplication, restof the planet sees even bigger numbersBig savings!
    132. 132. Posit:You have more than onecore, though storeconfigs is single-threaded
    133. 133. Therefore:Throughput is not maximized
    134. 134. Feature:Massively parallel operation
    135. 135. ParallelWe can pat our heads and rub our tummies atthe same timeDatabase operations don’t block MQ operationsdon’t block HTTP operations don’t block hashcomputation operations don’t block metriccalculations don’t block...Dozens of threads, zero locks
    136. 136. So anyways,
    137. 137. Documented athttp://docs.puppetlabs.com/puppetdb install, config, upkeep, specs, the works!
    138. 138. Packagedas deb and rpm forboth open source andPuppet Enterprise available in the Puppet Labs package repositories
    139. 139. Puppetizedusing thepuppetlabs/puppetdbmodule available now, on the Module Forge!
    140. 140. > puppet module install puppetlabs/puppetdb> vim site.ppnode puppetmaster { include puppetdb include puppetdb::master::config}
    141. 141. Open sourcehttp://github.com/puppetlabs/puppetdb same license as Puppet itself!
    142. 142. Production ready!
    143. 143. Many productiondeploymentsSmall shops with a dozen hosts,large shops with thousands ofhosts, intercontinentaldeployments... over a billion resources served!
    144. 144. Coming soon!
    145. 145. Report storageHistorical dataGrand Unified Query and of course, keep it fast!
    146. 146. Use it! and send us moredashboard screenshots! :)
    147. 147. deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode]
    148. 148. 7 5 3 11 82 9 10
    149. 149. 7 5 3 11 82 9 10
    150. 150. 7 5 3 11 82 9 10
    151. 151. 7 5 3 11 82 9 10
    152. 152. 7 5 3 11 82 9 10
    153. 153. 7 5 3 11 82 9 10
    154. 154. 7 5 3 11 82 9 10
    155. 155. 7 5 3 11 82 9 10
    156. 156. 7 5 3 11 82 9 10
    157. 157. 7 5 3 11 82 9 Achievement unlocked Let’s get TOPOLOGICAL! 10
    158. 158. 7 5 3 11 82 9 10
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×