0
PuppetDB                     Higher-order Puppet                                                 Deepak Giridharagopal    ...
Let’s talk                      about                       dataMonday, May 21, 12
Monday, May 21, 12
Data!         Puppet generates a lot of it, in many         delicious flavors!         Persisted, ephemeral, machine local...
Catalogs         “The Graph”         Containment edges, dependency edges,         classes, tags, resources, resource      ...
file {“/tmp/foo”: content => “This is a test”}                     target: &id063 !ruby/object:Puppet::Resource           ...
Group[peadmin]                                                                                                            ...
Group[peadmin]                                                                                                            ...
Group[peadmin]                                                                                       User[peadmin]        ...
Facts         Everything detected by facter         Facts for hardware, software, networking,         CPUs, memory, virtua...
netmask_lo: 255.0.0.0            kernelrelease: 2.6.32-5-686   augeasversion: 0.10.0            ipaddress: 172.16.245.128 ...
Reports         Catalogs say what you want, reports say         what you got.         Desired state, actual state, events,...
"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status                       change_count: 1                       change...
Why                     bother?Monday, May 21, 12
“Theres a war out there, old               friend. A world war. And its               not about whos got the most         ...
Storeconfigs         Centralized storage of the configuration of         all your nodes.         All resources, all parame...
class exporter {                       @@file {                         "/var/lib/puppet/nodes/$fqdn":                    ...
public key distribution                       monitoring checks                       clustered services                  ...
Query         Interrogation, investigation, correlation         Use Puppet-generated data in scripts or for         integr...
Higher                      order                     PuppetMonday, May 21, 12
Volume         Every node, on every puppet run, generates         data         We have customers generating over 750G of  ...
(demo)Monday, May 21, 12
Slow = :(         When data storage is slow, it makes baby         Deepak cry!         Slows down catalog compilation,    ...
API         Current APIs are limited         Hard to get at the data, and performance         concerns discourage use.    ...
Paradox         Seemingly contradictory goals         We want to store as much data as we can,         and allow for bette...
We need         An information clearinghouse         Something that evolves the Puppet Data         Library. A scalable, s...
PuppetDB                     Definitely Better!Monday, May 21, 12
GrayskullMonday, May 21, 12
PuppetDBMonday, May 21, 12
PuppetDB is         Fast storage of current catalogs and current         facts,         100% compatible with storeconfigs ...
science                          &                     secret alien                     technology!Monday, May 21, 12
Message Queue                                                     "new catalog"        "new catalog"                      ...
(export)Monday, May 21, 12
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                                DB           Workers                               HTTP       ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                                DB           Workers                                          ...
PuppetDB Server          DLO                                  DB         Workers                              F           ...
PuppetDB Server          DLO                                  DB         Workers                              F          C...
(collection)Monday, May 21, 12
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                                DB           Workers                               HTTP       ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                                DB           Workers                               HTTP       ...
PuppetDB Server          DLO                                DB           Workers                                ?         ...
PuppetDB Server          DLO                                DB           Workers                                          ...
PuppetDB Server          DLO                                DB           Workers                                F         ...
PuppetDB Server          DLO                                DB           Workers                                F         ...
PuppetDB Server          DLO                                DB           Workers                                F         ...
PuppetDB Server          DLO                                DB           Workers                                F         ...
PuppetDB Server          DLO                                DB           Workers                                F         ...
PuppetDB Server          DLO                                DB           Workers                                F         ...
PuppetDB Server          DLO                                   DB        Workers                               F        C ...
(failure)Monday, May 21, 12
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                                DB           Workers                               HTTP       ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                              DB           Workers                             HTTP           ...
PuppetDB Server          DLO                                DB           Workers                                          ...
PuppetDB Server          DLO                                DB           Workers                                          ...
PuppetDB Server          DLO                                DB           Workers                                C         ...
PuppetDB Server          DLO                                DB           Workers                                C         ...
PuppetDB Server          DLO                                DB           Workers                                C         ...
PuppetDB Server          DLO                                DB           Workers                                C         ...
PuppetDB Server          DLO                                DB           Workers                                C         ...
PuppetDB Server          DLO                                                       F                                DB    ...
PuppetDB Server          DLO                          DB           Workers                         HTTP            MQMonda...
PuppetDB Server                        Workers    DLO                                              DB                     ...
PuppetDB Server                          Workers    DLO               HTTP                             DB               Pr...
(launch)Monday, May 21, 12
Reliable!         We work very hard to persist everything we         accept         Acknowledgements with UUIDS,         C...
APIs!         We don’t cheat         Anything Puppet does with PuppetDB, you         can do to         Query your own reso...
#> curl      -H "Accept: application/json"      "http://puppetdb/metrics/mbean/      com.puppetlabs.puppetdb.command:type=...
curl                        -H "Accept: application/json"                     "http://puppetdb/facts/host.my.net"Monday, M...
curl                         -H "Accept: application/json"                     "http://puppetdb/resources?query=..."Monday...
Transparent!         We care about operational visibility         Ships with a real-time dashboard,         Dozens of metr...
Speedy!         PuppetDB is much, *much* faster than the         previous storeconfigs and inventory services         At P...
Design                     decisionsMonday, May 21, 12
Posit:                      Hosts are not                     entirely unique                       snowflakesMonday, May ...
Therefore:                     A resource often                       exists across                      multiple hostsMon...
Feature:                      Single-instance                     resource storageMonday, May 21, 12
Resource dedupe         Compute unique hashes for resources         We quickly hash all the resources in a         catalog...
Resource dedupe         Significant speed improvement!         Internal to Puppet Labs, we see ~83%         resource dupli...
Posit:                       Puppet runs                     frequently, but                     catalogs change          ...
Therefore:                  We’ll often receive                 the same catalog for                         a hostMonday,...
Feature:                     Single-instance                     catalog storageMonday, May 21, 12
Catalog dedupe         Compute unique hashes for catalogs         We use a Merkle Tree approach (hash tree)         for qu...
Posit:                     You have more than                      one core, though                        storeconfigs is...
Therefore:                     Throughput is not                        maximizedMonday, May 21, 12
Feature:                     Massively parallel                        operationMonday, May 21, 12
Parallel         We can pat our heads and rub our tummies         at the same time         Database operations don’t block...
Monday, May 21, 12
science                          &                     secret alien                     technology!Monday, May 21, 12
The FutureMonday, May 21, 12
http://github.com/                        puppetlabs/                         puppetdbMonday, May 21, 12
http://         docs.puppetlabs.com/               puppetdbMonday, May 21, 12
Use it, and tell us                         about it!Monday, May 21, 12
PuppetDB                     Thanks for your time!                                                   Deepak Giridharagopal...
Upcoming SlideShare
Loading in...5
×

Puppet DB: Higher-Order Puppet - Deepak Giridharagopal - PuppetCamp LA '12

1,818

Published on

Deepak Giridharagopal - "Puppet DB: Higher-Order Puppet". Video at http://youtu.be/fDi-_NpeUcA

PuppetCamp LA, May '12

Published in: Technology, Spiritual
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,818
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
46
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Transcript of "Puppet DB: Higher-Order Puppet - Deepak Giridharagopal - PuppetCamp LA '12"

  1. 1. PuppetDB Higher-order Puppet Deepak Giridharagopal Lead Engineer @ Puppet Labs deepak@puppetlabs.com grim_radical, #puppetMonday, May 21, 12
  2. 2. Let’s talk about dataMonday, May 21, 12
  3. 3. Monday, May 21, 12
  4. 4. Data! Puppet generates a lot of it, in many delicious flavors! Persisted, ephemeral, machine local, centralized, meticulously structured, totally free-form, human readable, machine optimized...Monday, May 21, 12
  5. 5. Catalogs “The Graph” Containment edges, dependency edges, classes, tags, resources, resource parameters, metadataMonday, May 21, 12
  6. 6. file {“/tmp/foo”: content => “This is a test”} target: &id063 !ruby/object:Puppet::Resource catalog: *id001 exported: false file: /etc/puppetlabs/puppet/manifests/site.pp line: 44 parameters: !ruby/sym content: This is a test !ruby/sym backup: main reference: "File[/tmp/foo]" tags: - file - node - default - class title: /tmp/foo type: FileMonday, May 21, 12
  7. 7. Group[peadmin] User[peadmin] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile]peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/sha Relationships Monday, May 21, 12
  8. 8. Group[peadmin] Group[puppet-dashboard] Class[Pe_accounts::Data] User[peadmin] User[puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/agent] File[/opt/puppet/libexec/mcollective/mcollective/security] Exec[mcollective-server-cert] File[/etc/puppetlabs/mcollective/ssl] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Pe_accounts::User[puppet-dashboard] File[/opt/puppet/share/puppet-dashboard] Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard] File[/opt/puppet/libexec/mcollective/mcollective/util] File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb] File[mcollective-cert.pem] File[mcollective-private.pem] File[mcollective-public.pem] File[/etc/puppetlabs/mcollective/ssl/clients] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile] Exec[puppet-dashboard-client-cert] File[/opt/puppet/share/puppet-dashboard/.mcollective.d] File[/opt/puppet/share/puppet-dashboard/.mcollective] File[/opt/puppet/share/puppet-dashboard/.bashrc.custom] File[/opt/puppet/share/puppet-dashboard/.bashrc] File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim] File[/opt/puppet/share/puppet-dashbo/mcollective/mcollective/agent/puppetral.rb] File[/etc/puppetlabs/mcollective/server.cfg] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl] File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb] File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb] File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[peadmin-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem] File[/opt/puppet/share/puppet-dashboard/.ssh/ Service[mcollective] Relationships Monday, May 21, 12
  9. 9. Group[peadmin] User[peadmin] File[/var/lib/peadmin] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc]le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] Monday, May 21, 12
  10. 10. Facts Everything detected by facter Facts for hardware, software, networking, CPUs, memory, virtualization, manufacturer info, custom facts... Coming soon: structured facts!Monday, May 21, 12
  11. 11. netmask_lo: 255.0.0.0 kernelrelease: 2.6.32-5-686 augeasversion: 0.10.0 ipaddress: 172.16.245.128 fqdn: pe-debian6.localdomain processor0: Intel(R) Core(TM) manufacturer: "VMware, Inc." i7-2635QM CPU @ 2.00GHz processorcount: "1" lsbdistrelease: 6.0.2 productname: VMware Virtual uniqueid: 007f0101 Platform hardwaremodel: i686 physicalprocessorcount: 1 kernelversion: 2.6.32 facterversion: 1.6.7 operatingsystem: Debian boardproductname: 440BX architecture: i386 Desktop Reference Platform lsbdistdescription: Debian GNU/ kernelmajversion: "2.6" Linux 6.0.2 (squeeze) hardwareisa: unknown lsbmajdistrelease: "6" timezone: PDT interfaces: "eth0,lo" puppetversion: 2.7.12 (Puppet ipaddress_lo: 127.0.0.1 Enterprise 2.5.1) uptime_days: 0 lsbdistcodename: squeeze lsbdistid: Debian is_virtual: "true" rubysitedir: /opt/puppet/lib/ operatingsystemrelease: 6.0.2 site_ruby/1.8 virtual: vmware rubyversion: 1.8.7 type: Other osfamily: Debian domain: localdomain memorytotal: &id001 502.57 MB hostname: pe-debian6 memorysize: *id001 selinux: "false" boardmanufacturer: Intel kernel: LinuxMonday, May 21, 12 Corporation
  12. 12. Reports Catalogs say what you want, reports say what you got. Desired state, actual state, events, duration, timestamps...Monday, May 21, 12
  13. 13. "File[/tmp/foo]": !ruby/object:Puppet::Resource::Status change_count: 1 changed: true evaluation_time: 0.001869 events: - !ruby/object:Puppet::Transaction::Event audited: false desired_value: !ruby/sym file historical_value: message: *id006 name: !ruby/sym file_created previous_value: !ruby/sym absent property: ensure status: success time: 2011-10-25 18:51:37.143970 -07:00 failed: false file: *id007 line: 44 out_of_sync: true out_of_sync_count: 1 resource: "File[/tmp/foo]" resource_type: File skipped: false tags: - file - node - default - class time: 2011-10-25 18:51:37.143396 -07:00 title: /tmp/fooMonday, May 21, 12
  14. 14. Why bother?Monday, May 21, 12
  15. 15. “Theres a war out there, old friend. A world war. And its not about whos got the most bullets. Its about who controls the information. What we see and hear, how we work, what we think... its all about the information!” -- SneakersMonday, May 21, 12
  16. 16. Storeconfigs Centralized storage of the configuration of all your nodes. All resources, all parameters, all classes, all tags, all stages... Enables use of exported resourcesMonday, May 21, 12
  17. 17. class exporter { @@file { "/var/lib/puppet/nodes/$fqdn": content => "$ipaddressn", tag => "ip" } } node "export1.daysofwonder.com" { include exporter } node "export2.daysofwonder.com" { include exporter } node "collector.daysofwonder.com" { File <<| tag == "ip" |>> } http://www.masterzen.fr/2009/03/08/all-about-puppet-storeconfigs/Monday, May 21, 12
  18. 18. public key distribution monitoring checks clustered services master/slave replication load balancers shared filesystems firewall rules ...Monday, May 21, 12
  19. 19. Query Interrogation, investigation, correlation Use Puppet-generated data in scripts or for integration with other toolsMonday, May 21, 12
  20. 20. Higher order PuppetMonday, May 21, 12
  21. 21. Volume Every node, on every puppet run, generates data We have customers generating over 750G of data a day. Even storing a small subset of that much information adds up...Monday, May 21, 12
  22. 22. (demo)Monday, May 21, 12
  23. 23. Slow = :( When data storage is slow, it makes baby Deepak cry! Slows down catalog compilation, More quickly saturates a Puppetmaster, Thrashes disk, Bad news!Monday, May 21, 12
  24. 24. API Current APIs are limited Hard to get at the data, and performance concerns discourage use. We need better ways of searching, filtering, and correlating data.Monday, May 21, 12
  25. 25. Paradox Seemingly contradictory goals We want to store as much data as we can, and allow for better querying, but without slowing stuff down or reducing reliability.Monday, May 21, 12
  26. 26. We need An information clearinghouse Something that evolves the Puppet Data Library. A scalable, safe place to store the information Puppet collects and generates. This is a hard problem!Monday, May 21, 12
  27. 27. PuppetDB Definitely Better!Monday, May 21, 12
  28. 28. GrayskullMonday, May 21, 12
  29. 29. PuppetDBMonday, May 21, 12
  30. 30. PuppetDB is Fast storage of current catalogs and current facts, 100% compatible with storeconfigs and inventory service, REST APIs for resource, fact, and node retrieval, ...and other things, even!Monday, May 21, 12
  31. 31. science & secret alien technology!Monday, May 21, 12
  32. 32. Message Queue "new catalog" "new catalog" "new facts" "new facts" "delete node" "delete node" Puppetmaster Compiler Command Handler Storeconfigs Parsing Transformation Validation Storeconfigs, Catalogs, Facts REST Puppet (SCF) "inventory query" Enterprise Domain "interactive query" Console objects Query handling CLI & Other ToolsMonday, May 21, 12
  33. 33. (export)Monday, May 21, 12
  34. 34. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog ResrcMonday, May 21, 12
  35. 35. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  36. 36. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  37. 37. PuppetDB Server DLO DB Workers HTTP MQ F Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  38. 38. PuppetDB Server DLO DB Workers HTTP MQ C F Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  39. 39. PuppetDB Server DLO DB Workers F HTTP MQ C Agent Master Facts Catalog Resrc CMonday, May 21, 12
  40. 40. PuppetDB Server DLO DB Workers F C HTTP MQ Agent Master Facts Catalog Resrc CMonday, May 21, 12
  41. 41. PuppetDB Server DLO DB Workers F C HTTP MQ Agent Master Facts Catalog Resrc CMonday, May 21, 12
  42. 42. (collection)Monday, May 21, 12
  43. 43. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog ResrcMonday, May 21, 12
  44. 44. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  45. 45. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  46. 46. PuppetDB Server DLO DB Workers HTTP MQ F Agent Master Facts Catalog Resrc F ?Monday, May 21, 12
  47. 47. PuppetDB Server DLO DB Workers HTTP MQ ? F Agent Master Facts Catalog Resrc FMonday, May 21, 12
  48. 48. PuppetDB Server DLO DB Workers ? F HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  49. 49. PuppetDB Server DLO DB Workers F HTTP MQ ? Agent Master Facts Catalog Resrc FMonday, May 21, 12
  50. 50. PuppetDB Server DLO DB Workers F HTTP MQ Agent Master Facts Catalog Resrc F ?Monday, May 21, 12
  51. 51. PuppetDB Server DLO DB Workers F HTTP MQ Agent Master Facts Catalog Resrc F ?Monday, May 21, 12
  52. 52. PuppetDB Server DLO DB Workers F HTTP MQ Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  53. 53. PuppetDB Server DLO DB Workers F HTTP MQ C Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  54. 54. PuppetDB Server DLO DB Workers F HTTP MQ C Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  55. 55. PuppetDB Server DLO DB Workers F C HTTP MQ Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  56. 56. PuppetDB Server DLO DB Workers F C HTTP MQ Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  57. 57. (failure)Monday, May 21, 12
  58. 58. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog ResrcMonday, May 21, 12
  59. 59. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  60. 60. PuppetDB Server DLO DB Workers HTTP MQ Agent Master Facts Catalog Resrc FMonday, May 21, 12
  61. 61. PuppetDB Server DLO DB Workers HTTP MQ F Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  62. 62. PuppetDB Server DLO DB Workers HTTP MQ C F Agent Master Facts Catalog Resrc F CMonday, May 21, 12
  63. 63. PuppetDB Server DLO DB Workers F HTTP MQ C Agent Master Facts Catalog Resrc CMonday, May 21, 12
  64. 64. PuppetDB Server DLO DB Workers C HTTP MQ F Agent Master Facts Catalog Resrc CMonday, May 21, 12
  65. 65. PuppetDB Server DLO DB Workers C F HTTP MQ Agent Master Facts Catalog Resrc CMonday, May 21, 12
  66. 66. PuppetDB Server DLO DB Workers C HTTP MQ F Agent Master Facts Catalog Resrc CMonday, May 21, 12
  67. 67. PuppetDB Server DLO DB Workers C F HTTP MQ Agent Master Facts Catalog Resrc CMonday, May 21, 12
  68. 68. PuppetDB Server DLO DB Workers C HTTP MQ F Agent Master Facts Catalog Resrc CMonday, May 21, 12
  69. 69. PuppetDB Server DLO DB Workers C F HTTP MQ Agent Master Facts Catalog Resrc CMonday, May 21, 12
  70. 70. PuppetDB Server DLO F DB Workers C HTTP MQ Agent Master Facts Catalog Resrc CMonday, May 21, 12
  71. 71. PuppetDB Server DLO DB Workers HTTP MQMonday, May 21, 12
  72. 72. PuppetDB Server Workers DLO DB HTTP MQMonday, May 21, 12
  73. 73. PuppetDB Server Workers DLO HTTP DB Proxy (SSL) HTTP MQMonday, May 21, 12
  74. 74. (launch)Monday, May 21, 12
  75. 75. Reliable! We work very hard to persist everything we accept Acknowledgements with UUIDS, Checksums, Queueing, Automatic retry and reconnect, and the Dead Letter Office if all else fails!Monday, May 21, 12
  76. 76. APIs! We don’t cheat Anything Puppet does with PuppetDB, you can do to Query your own resources, upload new fact sets, create catalogs, inspect facts...all part of the Puppet Data LibraryMonday, May 21, 12
  77. 77. #> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time" { "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728 }Monday, May 21, 12
  78. 78. curl -H "Accept: application/json" "http://puppetdb/facts/host.my.net"Monday, May 21, 12
  79. 79. curl -H "Accept: application/json" "http://puppetdb/resources?query=..."Monday, May 21, 12
  80. 80. Transparent! We care about operational visibility Ships with a real-time dashboard, Dozens of metrics and gauges, Correlate-able logs, Easy to integrate with monitoring systemsMonday, May 21, 12
  81. 81. Speedy! PuppetDB is much, *much* faster than the previous storeconfigs and inventory services At Puppet Labs, we’ve seen huge reductions in compile times, resource collection times, time to persist catalogs and facts, etc.Monday, May 21, 12
  82. 82. Design decisionsMonday, May 21, 12
  83. 83. Posit: Hosts are not entirely unique snowflakesMonday, May 21, 12
  84. 84. Therefore: A resource often exists across multiple hostsMonday, May 21, 12
  85. 85. Feature: Single-instance resource storageMonday, May 21, 12
  86. 86. Resource dedupe Compute unique hashes for resources We quickly hash all the resources in a catalog, and use bulk operations to compare them to hashes stored.Monday, May 21, 12
  87. 87. Resource dedupe Significant speed improvement! Internal to Puppet Labs, we see ~83% resource duplication; this number is consistent with what we’ve seen in most customer environments.Monday, May 21, 12
  88. 88. Posit: Puppet runs frequently, but catalogs change infrequentlyMonday, May 21, 12
  89. 89. Therefore: We’ll often receive the same catalog for a hostMonday, May 21, 12
  90. 90. Feature: Single-instance catalog storageMonday, May 21, 12
  91. 91. Catalog dedupe Compute unique hashes for catalogs We use a Merkle Tree approach (hash tree) for quick comparisons. Puppet Labs sees ~88% catalog duplication Big savings!Monday, May 21, 12
  92. 92. Posit: You have more than one core, though storeconfigs is single-threadedMonday, May 21, 12
  93. 93. Therefore: Throughput is not maximizedMonday, May 21, 12
  94. 94. Feature: Massively parallel operationMonday, May 21, 12
  95. 95. Parallel We can pat our heads and rub our tummies at the same time Database operations don’t block MQ operations don’t block HTTP operations don’t block hash computation operations don’t block metric calculations don’t block... Dozens of threads, zero locksMonday, May 21, 12
  96. 96. Monday, May 21, 12
  97. 97. science & secret alien technology!Monday, May 21, 12
  98. 98. The FutureMonday, May 21, 12
  99. 99. http://github.com/ puppetlabs/ puppetdbMonday, May 21, 12
  100. 100. http:// docs.puppetlabs.com/ puppetdbMonday, May 21, 12
  101. 101. Use it, and tell us about it!Monday, May 21, 12
  102. 102. PuppetDB Thanks for your time! Deepak Giridharagopal Lead Engineer @ Puppet Labs deepak@puppetlabs.com grim_radical, #puppetMonday, May 21, 12
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×