Downloaded 35 times
![File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/resources](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-37-2048.jpg)
![File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/resources/Service](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-38-2048.jpg)
![File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/resources/Service/foo](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-39-2048.jpg)
![File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/nodes/foo.com/resources](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-40-2048.jpg)
![File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/nodes/foo.com/resources
/Package](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-41-2048.jpg)
![File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/nodes/foo.com/resources
/Package/foo](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-42-2048.jpg)
![bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-58-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-59-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-60-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-61-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-62-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-78-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-79-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
All Files,
please!
Anything for
you! !](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-80-2048.jpg)
![STORAGE
File[/foo]
File[/bar] Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
WTF?!
I’m trying my
best! !](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-81-2048.jpg)
![STORAGE
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
All Files for
env TEST, plz!
Anything for
you! !
File[/foo]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-82-2048.jpg)
![STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
Thanks!
We’re friends
again! ❤️](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-83-2048.jpg)
![{
"cpus" : {
"cpu1" : {
"bogomips": 6000,
}
},
"networking" : {
"eth0" : {
"ipaddresses" : [ "1.1.1.5" ],
"macaddresses" : [ "aa:bb:cc:dd:ee:00" ]
}
}
}](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-90-2048.jpg)
![["=", "path",
["networking",
"eth0",
"macaddresses",
0]]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-91-2048.jpg)
![["~>", "path",
["networking",
"eth.*",
"macaddresses",
".*"]]](https://image.slidesharecdn.com/puppetconf2014-140926131724-phpapp01/75/PuppetDB-One-Year-Faster-PuppetConf-2014-92-2048.jpg)
Uploaded byPuppet
PuppetDB: One Year Faster - PuppetConf 2014
1. PuppetDB is a centralized catalog and fact storage system for Puppet. It stores catalogs, facts, reports and other resources from Puppet agents. 2. Recent releases of PuppetDB have added features like differential storage to reduce writes, improved de-duplication, PostgreSQL hot standby for high availability, and support for environments and structured facts. 3. A unified query engine allows querying across facts, resources, nodes and reports using an AST-based language.
More Related Content
Similar to PuppetDB: One Year Faster - PuppetConf 2014
PuppetDB: One Year Faster - PuppetConf 2014
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34. goo.comfoo.com bar.com STORAGE bar.com Deduplication ofcatalogs! foo.com goo.com
- 35. /commands MQ Parse Delayed DeadLetter Office Process UUID
- 36.
- 37. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echofoo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/resources
- 38. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echofoo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/resources/Service
- 39. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echofoo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/resources/Service/foo
- 40. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echofoo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/nodes/foo.com/resources
- 41. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echofoo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/nodes/foo.com/resources /Package
- 42. File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echofoo] Exec[echo bar] bar.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] baz.com File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] goo.com /v4/nodes/foo.com/resources /Package/foo
- 43. • Can queryfacts, nodes, resources, reports, events, metrics • Advanced queries via AST- based language • Aggregates, ordering, paging, streaming, subqueries
- 45. A long timeago in a galaxy far, far away… ! ! uh, like here a year ago…
- 46. Just released 1.4.0, Around11k deployments, Basic streaming support, Other stuff…
- 47.
- 48. Soft write failures Pagingsupport Resource containment paths in events Event aggregates Differential fact storage
- 49. Differential edge storage Pervasivestreaming of query results 3dfx Voodoo2 and Soundblaster AWE32 support Improved de-duplication
- 50. Resource parameter caching PostgreSQLHot Standby support for faster reads Debugging of de-duplication algorithm Full director’s commentary
- 51. Compressed responses A pony Certificatechain support Support for puppet environments Event subqueries
- 52. Prepared statement caching DirectPOST of json data in terminus Brings Aeris from Final Fantasy VII back to life Profiling support for terminus
- 53. Faster message parsing Structuredfact storage and querying Unified query subsystem in v4 API Riker’s beard
- 54.
- 55. We can’t getthrough that entire list, but I’ll try to highlight a few shiny bits
- 56.
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63. • Edges andfacts, too • Trades reads for writes, which is a good tradeoff. • PostgreSQL’s heap-only- tuples help a lot.
- 64.
- 66. Thanks to thefolks at Spotify, the community, etc. that helped us with this!
- 67.
- 68.
- 69. {"foo" => "goo", "bar"=> "baz"} {"bar" => "baz", "foo" => "goo"} 904d4d… 11c05d…
- 70. • Restructuring dataprior to hashing results in much fewer false negatives • The fastest way to persist data is to already have it persisted!
- 71. ~60-70%boost for users withordinarily low dedupe rates
- 72. Thanks to thefolks at CERN that helped us debug this!
- 73.
- 74.
- 75.
- 76. Less I/O contentionfor reads and writes yields better throughput
- 77.
- 78.
- 79.
- 80.
- 81. STORAGE File[/foo] File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echobar] foo.com TEST bar.com PROD WTF?! I’m trying my best! !
- 82. STORAGE File[/bar] Package[foo] Package[bar] Service[foo] Service[bar] Exec[echo foo] Exec[echo bar] foo.com TEST bar.com PROD AllFiles for env TEST, plz! Anything for you! ! File[/foo]
- 83.
- 84. • Any receptionor transmission of data now includes the environment where possible • Queries can be isolated to a single environment
- 85.
- 86. PuppetDB Parse Map valid fields Map valid operators Compileto SQL PostgreSQL Parse Map valid fields Map valid operators Compile to SQL Parse Map valid fields Map valid operators Compile to SQL Parse Map valid fields Map valid operators Compile to SQL Facts Resources Nodes Reports Parse Map valid fields Map valid operators Compile to SQL Events QUERY
- 87. PuppetDB Parse to AST Term rewriting Apply operators Compileto SQL PostgreSQL QUERY
- 88. • Common queryengine underpinning v4 API • Operators are available uniformly across all v4 endpoints • We can add new endpoints and fields much faster
- 89.
- 90. { "cpus" : { "cpu1": { "bogomips": 6000, } }, "networking" : { "eth0" : { "ipaddresses" : [ "1.1.1.5" ], "macaddresses" : [ "aa:bb:cc:dd:ee:00" ] } } }
- 91.
- 92.
- 93. ! • PostgreSQL’s pg_trgmindex • Trusted facts
- 94. So where arewe now?
- 95. More features, butalso more speed
- 96. Language bindings: Ruby, Python, JavaScript,C/C++, Go, Clojure, Java…
- 97.
- 98.
- 99.
- 100. A new deployment every10 minutes
- 101.
- 102.
- 103.
- 104.