Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
PuppetDB
(ノ°ヮ°)ノ*:・゚✧2014
✧
RUN
PDB
deepak giridharagopal
deepak @puppetlabs.com
@grim_radical
Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB
Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB
Puppet
agent
Puppet
master PuppetDBfacts
PuppetDB
Puppet
agent
Puppet
master
facts
PuppetDB
PuppetDB
Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB
Yum!
Puppet
agent
Puppet
master PuppetDBPuppetDB
Puppet
agent
Puppet
master PuppetDBPuppetDB
catalogcatalog
catalog
Puppet
agent
Puppet
master PuppetDBPuppetDB
catalog
catalog
Puppet
agent
Puppet
master
PuppetDB
PuppetDB
catalog
Puppet
agent
catalog
Puppet
master
facts
PuppetDBPuppetDB
Yum!
Puppet
agent
Puppet
master PuppetDBPuppetDB
catalog
Puppet
agent
Puppet
master PuppetDBPuppetDB
report
Puppet
agent
Puppet
master PuppetDBPuppetDB
report
report
Puppet
agent
Puppet
master PuppetDBPuppetDB
report
Puppet
agent
Puppet
master
PuppetDB
PuppetDB
Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB
Yum!
Puppet
agent
Puppet
master PuppetDBPuppetDB
Puppet
master
catalog
PuppetDB
catalog
Puppet
master
catalog
PuppetDB
catalog
Puppet
master PuppetDB
catalog
catalog
Puppet
master PuppetDB
catalog
Puppet
master
PuppetDcatalog
catalog
Puppet
master
PuppetDcatalog
Puppet
master
Pupcatalog
catalog
Puppet
master
Pupcatalog
Software should be
self-regulating
goo.comfoo.com bar.com
STORAGE
bar.com
Self-regulating catalog & fact storage!
goo.comfoo.com bar.com
STORAGE
bar.combar.combar.combar.com
Self-regulating report storage!
goo.comfoo.com bar.com
STORAGE
Self-regulating node storage!
goo.comfoo.com bar.com
STORAGE
bar.com
Deduplication of catalogs!
foo.com goo.com
/commands MQ Parse
Delayed
Dead Letter
Office
Process
UUID
Querying
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]...
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]...
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]...
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]...
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]...
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]...
• Can query facts, nodes,
resources, reports, events,
metrics
• Advanced queries via AST-
based language
• Aggregates, ord...
A long time ago in a galaxy
far, far away…
!
!
uh, like here a year ago…
Just released 1.4.0,
Around 11k deployments,
Basic streaming support,
Other stuff…
We’ve been busy!
Soft write failures
Paging support
Resource containment paths
in events
Event aggregates
Differential fact storage
Differential edge storage
Pervasive streaming of query
results
3dfx Voodoo2 and
Soundblaster AWE32 support
Improved de-dup...
Resource parameter caching
PostgreSQL Hot Standby
support for faster reads
Debugging of de-duplication
algorithm
Full dire...
Compressed responses
A pony
Certificate chain support
Support for puppet
environments
Event subqueries
Prepared statement caching
Direct POST of json data in
terminus
Brings Aeris from Final
Fantasy VII back to life
Profiling ...
Faster message parsing
Structured fact storage and
querying
Unified query subsystem in v4
API
Riker’s beard
docs.puppetlabs.com/
puppetdb/latest/
release_notes.html
We can’t get through
that entire list, but I’ll try
to highlight a few shiny
bits
1.
Differential storage
goo.comfoo.com bar.com
STORAGE
bar.comfoo.com
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
Fi...
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
Fi...
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
Fi...
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
Fi...
• Edges and facts, too
• Trades reads for writes,
which is a good tradeoff.
• PostgreSQL’s heap-only-
tuples help a lot.
~90%fewer writes
Thanks to the folks at
Spotify, the community,
etc. that helped us with
this!
2.
More effective
de-duplication
Order matters!
{"foo" => "goo",
"bar" => "baz"}
{"bar" => "baz",
"foo" => "goo"}
904d4d…
11c05d…
• Restructuring data prior to
hashing results in much
fewer false negatives
• The fastest way to persist
data is to alread...
~60-70%boost for
users with ordinarily
low dedupe rates
Thanks to the folks at
CERN that helped us
debug this!
3.
Hot standby
PuppetDB PostgreSQL
WRITE
READ
PuppetDB
PostgreSQL
WRITE
READ
PostgreSQL
standby
REPLICATION
Less I/O contention for
reads and writes yields
better throughput
3.
Environment support
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TE...
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TE...
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TE...
STORAGE
File[/foo]
File[/bar] Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TE...
STORAGE
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
...
STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TE...
• Any reception or
transmission of data now
includes the environment
where possible
• Queries can be isolated to a
single ...
3.
Unified query engine
PuppetDB
Parse
Map valid
fields
Map valid
operators
Compile to
SQL
PostgreSQL
Parse
Map valid
fields
Map valid
operators
Com...
PuppetDB
Parse to AST
Term
rewriting
Apply
operators
Compile to
SQL
PostgreSQL
QUERY
• Common query engine
underpinning v4 API
• Operators are available
uniformly across all v4
endpoints
• We can add new end...
4.
Structured/Trusted
fact support
{
"cpus" : {
"cpu1" : {
"bogomips": 6000,
}
},
"networking" : {
"eth0" : {
"ipaddresses" : [ "1.1.1.5" ],
"macaddresses" :...
["=", "path",
["networking",
"eth0",
"macaddresses",
0]]
["~>", "path",
["networking",
"eth.*",
"macaddresses",
".*"]]
!
• PostgreSQL’s pg_trgm index
• Trusted facts
So where are we now?
More features, but also
more speed
Language bindings:
Ruby, Python,
JavaScript, C/C++, Go,
Clojure, Java…
Trapperkeeper
https://github.com/puppetlabs/trapperkeeper
Puppetboard
https://github.com/nedap/puppetboard
Puppet Explorer
https://github.com/spotify/puppetexplorer
A new deployment
every 10 minutes
Coming soon:
more efficient GC
Coming soon:
simplified query syntax
Coming soon:
historical data
Thanks!
PuppetDB: One Year Faster - PuppetConf 2014
PuppetDB: One Year Faster - PuppetConf 2014
PuppetDB: One Year Faster - PuppetConf 2014
Upcoming SlideShare
Loading in …5
×

PuppetDB: One Year Faster - PuppetConf 2014

1,943 views

Published on

PuppetDB: One Year Faster - Deepak Giridharagopal, Puppet Labs

Published in: Technology
  • Be the first to comment

PuppetDB: One Year Faster - PuppetConf 2014

  1. 1. PuppetDB (ノ°ヮ°)ノ*:・゚✧2014 ✧
  2. 2. RUN PDB
  3. 3. deepak giridharagopal deepak @puppetlabs.com @grim_radical
  4. 4. Puppet agent Puppet master facts PuppetDBPuppetDB
  5. 5. Puppet agent Puppet master facts PuppetDBPuppetDB
  6. 6. Puppet agent Puppet master PuppetDBfacts PuppetDB
  7. 7. Puppet agent Puppet master facts PuppetDB PuppetDB
  8. 8. Puppet agent Puppet master facts PuppetDBPuppetDB Yum!
  9. 9. Puppet agent Puppet master PuppetDBPuppetDB
  10. 10. Puppet agent Puppet master PuppetDBPuppetDB catalogcatalog
  11. 11. catalog Puppet agent Puppet master PuppetDBPuppetDB catalog
  12. 12. catalog Puppet agent Puppet master PuppetDB PuppetDB catalog
  13. 13. Puppet agent catalog Puppet master facts PuppetDBPuppetDB Yum!
  14. 14. Puppet agent Puppet master PuppetDBPuppetDB catalog
  15. 15. Puppet agent Puppet master PuppetDBPuppetDB report
  16. 16. Puppet agent Puppet master PuppetDBPuppetDB report
  17. 17. report Puppet agent Puppet master PuppetDBPuppetDB
  18. 18. report Puppet agent Puppet master PuppetDB PuppetDB
  19. 19. Puppet agent Puppet master facts PuppetDBPuppetDB Yum!
  20. 20. Puppet agent Puppet master PuppetDBPuppetDB
  21. 21. Puppet master catalog PuppetDB catalog
  22. 22. Puppet master catalog PuppetDB catalog
  23. 23. Puppet master PuppetDB catalog
  24. 24. catalog Puppet master PuppetDB catalog
  25. 25. Puppet master PuppetDcatalog
  26. 26. catalog Puppet master PuppetDcatalog
  27. 27. Puppet master Pupcatalog
  28. 28. catalog Puppet master Pupcatalog
  29. 29. Software should be self-regulating
  30. 30. goo.comfoo.com bar.com STORAGE bar.com Self-regulating catalog & fact storage!
  31. 31. goo.comfoo.com bar.com STORAGE bar.combar.combar.combar.com Self-regulating report storage!
  32. 32. goo.comfoo.com bar.com STORAGE Self-regulating node storage!
  33. 33. goo.comfoo.com bar.com STORAGE bar.com Deduplication of catalogs! foo.com goo.com
  34. 34. /commands MQ Parse Delayed Dead Letter Office Process UUID
  35. 35. Querying
  36. 36. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/resources
  37. 37. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/resources/Service
  38. 38. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/resources/Service/foo
  39. 39. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/nodes/foo.com/resources
  40. 40. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/nodes/foo.com/resources /Package
  41. 41. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/nodes/foo.com/resources /Package/foo
  42. 42. • Can query facts, nodes, resources, reports, events, metrics • Advanced queries via AST- based language • Aggregates, ordering, paging, streaming, subqueries
  43. 43. A long time ago in a galaxy far, far away… ! ! uh, like here a year ago…
  44. 44. Just released 1.4.0, Around 11k deployments, Basic streaming support, Other stuff…
  45. 45. We’ve been busy!
  46. 46. Soft write failures Paging support Resource containment paths in events Event aggregates Differential fact storage
  47. 47. Differential edge storage Pervasive streaming of query results 3dfx Voodoo2 and Soundblaster AWE32 support Improved de-duplication
  48. 48. Resource parameter caching PostgreSQL Hot Standby support for faster reads Debugging of de-duplication algorithm Full director’s commentary
  49. 49. Compressed responses A pony Certificate chain support Support for puppet environments Event subqueries
  50. 50. Prepared statement caching Direct POST of json data in terminus Brings Aeris from Final Fantasy VII back to life Profiling support for terminus
  51. 51. Faster message parsing Structured fact storage and querying Unified query subsystem in v4 API Riker’s beard
  52. 52. docs.puppetlabs.com/ puppetdb/latest/ release_notes.html
  53. 53. We can’t get through that entire list, but I’ll try to highlight a few shiny bits
  54. 54. 1. Differential storage
  55. 55. goo.comfoo.com bar.com STORAGE bar.comfoo.com
  56. 56. bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[zzz] Exec[echo foo] Exec[echo bar]
  57. 57. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[zzz] Exec[echo foo] Exec[echo bar]
  58. 58. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[zzz] Exec[echo foo] Exec[echo bar]
  59. 59. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[zzz] Exec[echo foo] Exec[echo bar]
  60. 60. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[zzz] Exec[echo foo] Exec[echo bar]
  61. 61. • Edges and facts, too • Trades reads for writes, which is a good tradeoff. • PostgreSQL’s heap-only- tuples help a lot.
  62. 62. ~90%fewer writes
  63. 63. Thanks to the folks at Spotify, the community, etc. that helped us with this!
  64. 64. 2. More effective de-duplication
  65. 65. Order matters!
  66. 66. {"foo" => "goo", "bar" => "baz"} {"bar" => "baz", "foo" => "goo"} 904d4d… 11c05d…
  67. 67. • Restructuring data prior to hashing results in much fewer false negatives • The fastest way to persist data is to already have it persisted!
  68. 68. ~60-70%boost for users with ordinarily low dedupe rates
  69. 69. Thanks to the folks at CERN that helped us debug this!
  70. 70. 3. Hot standby
  71. 71. PuppetDB PostgreSQL WRITE READ
  72. 72. PuppetDB PostgreSQL WRITE READ PostgreSQL standby REPLICATION
  73. 73. Less I/O contention for reads and writes yields better throughput
  74. 74. 3. Environment support
  75. 75. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD
  76. 76. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD
  77. 77. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD All Files, please! Anything for you! !
  78. 78. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD WTF?! I’m trying my best! !
  79. 79. STORAGE File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD All Files for env TEST, plz! Anything for you! ! File[/foo]
  80. 80. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD Thanks! We’re friends again! ❤️
  81. 81. • Any reception or transmission of data now includes the environment where possible • Queries can be isolated to a single environment
  82. 82. 3. Unified query engine
  83. 83. PuppetDB Parse Map valid fields Map valid operators Compile to SQL PostgreSQL Parse Map valid fields Map valid operators Compile to SQL Parse Map valid fields Map valid operators Compile to SQL Parse Map valid fields Map valid operators Compile to SQL Facts Resources Nodes Reports Parse Map valid fields Map valid operators Compile to SQL Events QUERY
  84. 84. PuppetDB Parse to AST Term rewriting Apply operators Compile to SQL PostgreSQL QUERY
  85. 85. • Common query engine underpinning v4 API • Operators are available uniformly across all v4 endpoints • We can add new endpoints and fields much faster
  86. 86. 4. Structured/Trusted fact support
  87. 87. { "cpus" : { "cpu1" : { "bogomips": 6000, } }, "networking" : { "eth0" : { "ipaddresses" : [ "1.1.1.5" ], "macaddresses" : [ "aa:bb:cc:dd:ee:00" ] } } }
  88. 88. ["=", "path", ["networking", "eth0", "macaddresses", 0]]
  89. 89. ["~>", "path", ["networking", "eth.*", "macaddresses", ".*"]]
  90. 90. ! • PostgreSQL’s pg_trgm index • Trusted facts
  91. 91. So where are we now?
  92. 92. More features, but also more speed
  93. 93. Language bindings: Ruby, Python, JavaScript, C/C++, Go, Clojure, Java…
  94. 94. Trapperkeeper https://github.com/puppetlabs/trapperkeeper
  95. 95. Puppetboard https://github.com/nedap/puppetboard
  96. 96. Puppet Explorer https://github.com/spotify/puppetexplorer
  97. 97. A new deployment every 10 minutes
  98. 98. Coming soon: more efficient GC
  99. 99. Coming soon: simplified query syntax
  100. 100. Coming soon: historical data
  101. 101. Thanks!

×