PuppetDB: Higher-order Puppet

  • 863 views
Uploaded on

My talk at PuppetConf 2012 on PuppetDB, a Clojure-based centralized storage daemon for Puppet. …

My talk at PuppetConf 2012 on PuppetDB, a Clojure-based centralized storage daemon for Puppet.

Video of this talk: http://www.youtube.com/watch?v=xw83cRofkpM&list=PLV86BgbREluVFB73Wwqp_tCbw5Z9TMLX1&index=4&feature=plpp_video

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
863
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
51
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • Not just in terms of volume, but also many different kinds of data\n
  • persistent, ephemeral, free form, machine readable, that’s a lot of stuff to sift through!\n\nso where to start?\n
  • \n
  • this is a resource, pictured as what you’d type\n
  • this is the same resource, only post-compilation. way more useful stuff in here!\n
  • so most of the time, you’ll have more than one resource at play on a node.\nBut you don’t want them just applied randomly; order is important!\n
  • You want the things at the top to happen first.\nInternally, Puppet represents this as a directed-graph we call...\n
  • \n
  • a collection of resources and their relationships is a catalog\n
  • a collection of resources and their relationships is a catalog\n
  • a collection of resources and their relationships is a catalog\n
  • This is the catalog for a fresh install of Puppet Enterprise\nNot too different from the one I showed earlier, just with more resources and relationships.\nBut this is actually a bit of a lie, because the _entire_ catalog...\n
  • Hard to see, but we can zoom in on a tiny area\n
  • So, same stuff you’re used to: users, groups, files, etc.\nTrying to make sense of this manually is insane, but that’s where Puppet comes in!\nInfrastructure is messy, but puppet untangles that web.\n
  • But that’s only one half of the story\n
  • \n
  • \n
  • and if that doesn’t cut it, you can always make your own\n
  • So, catalogs and facts are great, by why is having access to that stuff important?\n\n
  • Information is powerful!\n\nAs operators, our decisions are only as good as the information upon which they’re based. Tools are no different; they’re only as smart as the input data.\n
  • Information is powerful!\n\nAs operators, our decisions are only as good as the information upon which they’re based. Tools are no different; they’re only as smart as the input data.\n
  • Information is powerful!\n\nAs operators, our decisions are only as good as the information upon which they’re based. Tools are no different; they’re only as smart as the input data.\n
  • So what can you do when you’ve got all this juicy data sitting around?\n
  • Integration with monitoring systems, inventory databases, provisioning systems, auto-remediation code, ad-hoc activity\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • And it’s not just for things like key distribution and monitoring,\n
  • ...this is a generic pattern that can be applied to all manner of situations.\n\nAnything where you need information from one node to configure another\n
  • Utilizing your data in this way catapults you into the world of Higher-order puppet.\nBy which, I mean...\n
  • Delightfully meta, don’t you think? <advance>\n\nDoubles-down on your automation, making it even more powerful.\n
  • Delightfully meta, don’t you think? <advance>\n\nDoubles-down on your automation, making it even more powerful.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n...but it’s not just a question of scalability...\n
  • \n
  • ...and therein lies the problem\n
  • \n
  • A lot of people ask me what the DB stands for <advance>\n
  • I’ll talk more about how we make that happen in a bit\n
  • We’ve tested this on our own code, running Puppet Labs\n
  • \n
  • ...and it’s all built using...\n
  • So let’s talk about how it works, at a high-level.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • So earlier, I mentioned that we need PuppetDB to be reliable, queryable, and fast\n
  • \n
  • \n
  • In fact, let’s try an experiment...\n
  • \n
  • \n
  • for example...\n
  • <explain what’s going on>\n\nWow, that’s a big wall of text, isn’t it? <advance> But hopefully it illustrates how much instrumentation we’ve put in.\n
  • <explain what’s going on>\n\nWow, that’s a big wall of text, isn’t it? <advance> But hopefully it illustrates how much instrumentation we’ve put in.\n
  • \n
  • Query language is documented and pretty versatile.\nSuper-set of resource-collection features (like complex boolean operators).\n...and this is why people have written libraries for use with puppet.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • How did we accomplish such a thing?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • And we run it ourselves at Puppet Labs!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. higher-order puppetPuppetDB
  • 2. deepak@puppetlabs.com@grim_radical
  • 3. Let’s talk about...
  • 4. puppet generateslots of data
  • 5. persistent data long termdata ephemeral data machlocal data meticulously straapuppet generates data frd lots of data free form dahuman readable data machdata resource data dependdata ssl certificate data ho
  • 6. resources
  • 7. file { “/tmp/foo”: content => “This is a test”}
  • 8. {"exported": false, "file": "/puppet/site.pp", "line": 1, "parameters": { "content": "This is a test" }, "tags": [ "file", "node", “default” ], "title": "/tmp/foo", "type": "File"}
  • 9. File “/tmp/foo/bar” User “deepak” Dir “/tmp/foo” Dir “/tmp”
  • 10. Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”
  • 11. Dir “/tmp” User “deepak” Dir “/tmp/foo” File “/tmp/foo/bar”
  • 12. catalogs
  • 13. resourcecatalog
  • 14. resource catalogcowherd
  • 15. resource catalogcow crowherd murder
  • 16. resource catalogcow superhero crowherd avengers murder
  • 17. Group[peadmin] User[peadmin] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile]peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/sha Relationships
  • 18. Group[peadmin] Group[puppet-dashboard] Class[Pe_accounts::Data] User[peadmin] User[puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/agent] File[/opt/puppet/libexec/mcollective/mcollective/security] Exec[mcollective-server-cert] File[/etc/puppetlabs/mcollective/ssl] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Pe_accounts::User[puppet-dashboard] File[/opt/puppet/share/puppet-dashboard] Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/util] File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb] File[mcollective-cert.pem] File[mcollective-private.pem] File[mcollective-public.pem] File[/etc/puppetlabs/mcollective/ssl/clients] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile] Exec[puppet-dashboard-client-cert] File[/opt/puppet/share/puppet-dashboard/.mcollective.d] File[/opt/puppet/share/puppet-dashboard/.mcollective] File[/opt/puppet/share/puppet-dashboard/.bashrc.custom] File[/opt/puppet/share/puppet-dashboard/.bashrc] File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim] File[/opt/puppet/share/puppet-dashbo/mcollective/mcollective/agent/puppetral.rb] File[/etc/puppetlabs/mcollective/server.cfg] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl] File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb] File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb] File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[peadmin-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem] File[/opt/puppet/share/puppet-dashboard/.ssh/ Service[mcollective] Relationships
  • 19. Group[peadmin] User[peadmin] File[/var/lib/peadmin] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc]le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
  • 20. Catalog: all the things we manage on a node, and how they relate to each other
  • 21. facts
  • 22. > facter
  • 23. netmask_lo: 255.0.0.0 kernel: Linux augeasversion: 0.10.0 kernelrelease: 2.6.32-5-686 fqdn: pe-debian6.localdomain ipaddress: 172.16.245.128 manufacturer: "VMware, Inc." processor0: Intel(R) Core(TM) processorcount: "1" i7-2635QM CPU @ 2.00GHz productname: VMware Virtual lsbdistrelease: 6.0.2Platform uniqueid: 007f0101 physicalprocessorcount: 1 hardwaremodel: i686 facterversion: 1.6.7 kernelversion: 2.6.32 boardproductname: 440BX operatingsystem: DebianDesktop Reference Platform architecture: i386 kernelmajversion: "2.6" lsbdistdescription: Debian GNU/ hardwareisa: unknown Linux 6.0.2 (squeeze) timezone: PDT lsbmajdistrelease: "6" puppetversion: 2.7.12 (Puppet interfaces: "eth0,lo"Enterprise 2.5.1) ipaddress_lo: 127.0.0.1 lsbdistcodename: squeeze uptime_days: 0 is_virtual: "true" lsbdistid: Debian operatingsystemrelease: 6.0.2 rubysitedir: /opt/puppet/lib/ virtual: vmware site_ruby/1.8 type: Other rubyversion: 1.8.7 domain: localdomain osfamily: Debian hostname: pe-debian6 memorytotal: &id001 502.57 MB selinux: "false" memorysize: *id001
  • 24. Catalogs:what we tell puppetabout a nodeFacts:what a node tellspuppet about itself
  • 25. It’s about who controlsthe information.
  • 26. “Theres a war out there,old friend. A world war.And its not about whosgot the most bullets.It’s about who controlsthe information. -- Sneakers (1992)What we see and hear,how we work, what wethink... its all about theinformation!”
  • 27. every resource every parameterevery relationship every class every fact for every node
  • 28. Query this data, foruse in scripts orother tools
  • 29. Integration withother tools is great,but can we feed thatdata back intopuppet itself?
  • 30. storeconfigs
  • 31. Configure a nodeusing resourcesfrom other nodes
  • 32. class ssh { @@sshkey { $hostname: type => dsa, key => $sshdsakey } Sshkey <<| |>>}
  • 33. Every host exportsits public key, andimports the publickeys of every othernode, automatically!
  • 34. How aboutmonitoring?
  • 35. class nagios_target { @@nagios_host { $fqdn: ensure => present, alias => $hostname, address => $ipaddress, use => "generic-host", } @@nagios_service { "check_ping_${hostname}": check_command => "check_ping!100.0,20%!500.0,60%", use => "generic-service", host_name => "$fqdn", notification_period => "24x7", service_description => "${hostname}_check_ping" }}
  • 36. class nagios-monitor { # collect resources and # populate /etc/nagios/nagios_*.cfg Nagios_host <<||>> Nagios_service <<||>>}
  • 37. Thus, you canautomatically createchecks for thingsyou’re managing
  • 38. key distribution monitoring clustered servicesmaster/slave replication load balancers shared filesystems firewall rules ...
  • 39. Higher-order Puppet
  • 40. Using Puppet’sknowledge toimprove Puppet’sknowledge
  • 41. Using Puppet’sknowledge toimprove Puppet’sknowledge Achievement unlocked YO DAWG
  • 42. Using Puppet’sknowledge toimprove Puppet’sknowledge
  • 43. Data is great!
  • 44. Reading from the Puppet Data Library Nick Lewis 3:50P @ Meeting Room 2
  • 45. Why aren’t we doingstuff like this all the damn time?
  • 46. V UME OL
  • 47. Every node,on every puppet run,generates data
  • 48. We have customersgenerating over750GB of data a day!even storing a small subset ofthat much information adds up...
  • 49. (a brief simulation)
  • 50. When data storage isslow, the wholesystem slows downand it makes baby Deepak cry! :(
  • 51. APIissues
  • 52. Current APIs arelimited!Hard to get at the data, andperformance concerns discourageuse
  • 53. We demand: Store as much data as we can! Much better queryability!Oh yeah, but: Don’t slow down the system! Don’t compromise reliability!
  • 54. PuppetDB
  • 55. PuppetDB Definitely Better!
  • 56. Fast storageof catalogs & facts like, *way* faster!
  • 57. Compatiblewith storeconfigs andinventory service you don’t have to change your Puppet code!
  • 58. HTTP APIsfor resource, fact, andnode retrieval plenty of data, just a “curl” away!
  • 59. Securedusing SSL client andserver certificatesthe same certificate infrastructure you’re already using!
  • 60. science &secret alientechnology
  • 61. Exportingresources
  • 62. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g
  • 63. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g
  • 64. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F
  • 65. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
  • 66. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
  • 67. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 68. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 69. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 70. Resourcecollection
  • 71. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g
  • 72. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g
  • 73. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F
  • 74. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F ?
  • 75. PuppetDB Server DLO DB Workers HTTP MQ ? FAgent Master Facts Catalo Resrc F g
  • 76. PuppetDB Server DLO DB Workers ? F HTTP MQAgent Master Facts Catalo Resrc F g
  • 77. PuppetDB Server DLO DB Workers F HTTP MQ ?Agent Master Facts Catalo Resrc F g
  • 78. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g ?
  • 79. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g ?
  • 80. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
  • 81. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
  • 82. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
  • 83. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
  • 84. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc F g
  • 85. Failure
  • 86. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g
  • 87. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc F g
  • 88. PuppetDB Server DLO DB Workers HTTP MQAgent Master Facts Catalo Resrc g F
  • 89. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
  • 90. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g F
  • 91. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 92. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g
  • 93. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 94. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g
  • 95. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 96. PuppetDB Server DLO DB Workers HTTP MQ FAgent Master Facts Catalo Resrc g
  • 97. PuppetDB Server DLO DB Workers F HTTP MQAgent Master Facts Catalo Resrc g
  • 98. PuppetDB Server DLO F DB Workers HTTP MQAgent Master Facts Catalo Resrc g
  • 99. Deployment
  • 100. PuppetDB Server DLO DB Workers HTTP MQ
  • 101. PuppetDB Server Workers DLO DB HTTP MQ
  • 102. PuppetDB Server Workers DLOHTTP DBProxy(SSL) HTTP MQ
  • 103. (another demo)
  • 104. Reliability!
  • 105. We work very hard topersist everything weaccept
  • 106. Acknowledgements with UUIDS,Checksums,Queueing,Automatic retry,Automatic reconnect, and the Dead Letter Office if all else fails!
  • 107. APIs!
  • 108. Anything Puppetdoes with PuppetDB,you can do, too we don’t cheat!
  • 109. Query your own resources,Upload new fact sets,Create catalogs,Inspect facts, all open and documented!
  • 110. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}
  • 111. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, Achievement unlocked "RateUnit": "SECONDS", "StdDev": 833.6079354075728} WALL OF TEXT
  • 112. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}
  • 113. curl -H "Accept: application/json""http://puppetdb/facts/host.my.net"
  • 114. curl -H "Accept: application/json""http://puppetdb/resources?query=..."
  • 115. https://github.com/dalen/puppet-puppetdbquery
  • 116. Transparent!
  • 117. Ships with a real-time dashboard,Dozens of metrics and gauges,Correlate-able logs,Easy to monitor we care about operational visibility!
  • 118. https://github.com/jasonhancock/nagios-puppetdb
  • 119. Speedy!
  • 120. We’ve seen huge reductions in compile times, resource collection times, time to persist catalogs and facts, etc. O_o
  • 121. ONE DOES NOT SIMPLY SPEED UP PUPPET
  • 122. Notable details
  • 123. Posit:Hosts are not entirely unique snowflakes
  • 124. Therefore: A resource oftenexists across multiple hosts
  • 125. Feature: Single-instanceresource storage
  • 126. Resource dedupeCompute unique hashes for resourcesWe quickly hash all the resources in a catalog,and use bulk operations to compare them tohashes stored.
  • 127. Resource dedupeSignificant speed improvement!Internal to Puppet Labs, we see ~83% resourceduplication; this number is consistent with whatwe’ve seen in most customer environments.
  • 128. Posit:Puppet runs frequently, but catalogs change infrequently
  • 129. Therefore:We’ll often receive the same catalog for a host
  • 130. Feature:Single-instancecatalog storage
  • 131. Catalog dedupeCompute unique hashes for catalogsPuppet Labs sees ~88% catalog duplication, restof the planet sees even bigger numbersBig savings!
  • 132. Posit:You have more than onecore, though storeconfigs is single-threaded
  • 133. Therefore:Throughput is not maximized
  • 134. Feature:Massively parallel operation
  • 135. ParallelWe can pat our heads and rub our tummies atthe same timeDatabase operations don’t block MQ operationsdon’t block HTTP operations don’t block hashcomputation operations don’t block metriccalculations don’t block...Dozens of threads, zero locks
  • 136. So anyways,
  • 137. Documented athttp://docs.puppetlabs.com/puppetdb install, config, upkeep, specs, the works!
  • 138. Packagedas deb and rpm forboth open source andPuppet Enterprise available in the Puppet Labs package repositories
  • 139. Puppetizedusing thepuppetlabs/puppetdbmodule available now, on the Module Forge!
  • 140. > puppet module install puppetlabs/puppetdb> vim site.ppnode puppetmaster { include puppetdb include puppetdb::master::config}
  • 141. Open sourcehttp://github.com/puppetlabs/puppetdb same license as Puppet itself!
  • 142. Production ready!
  • 143. Many productiondeploymentsSmall shops with a dozen hosts,large shops with thousands ofhosts, intercontinentaldeployments... over a billion resources served!
  • 144. Coming soon!
  • 145. Report storageHistorical dataGrand Unified Query and of course, keep it fast!
  • 146. Use it! and send us moredashboard screenshots! :)
  • 147. deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode]
  • 148. 7 5 3 11 82 9 10
  • 149. 7 5 3 11 82 9 10
  • 150. 7 5 3 11 82 9 10
  • 151. 7 5 3 11 82 9 10
  • 152. 7 5 3 11 82 9 10
  • 153. 7 5 3 11 82 9 10
  • 154. 7 5 3 11 82 9 10
  • 155. 7 5 3 11 82 9 10
  • 156. 7 5 3 11 82 9 10
  • 157. 7 5 3 11 82 9 Achievement unlocked Let’s get TOPOLOGICAL! 10
  • 158. 7 5 3 11 82 9 10