• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
All About Storeconfigs
 

All About Storeconfigs

on

  • 13,506 views

Storeconfigs is not a popular feature among Puppet admins, because most don’t know how to use it or fear performance issues. Attend this talk to know how to enhance your Puppet deployments with easy ...

Storeconfigs is not a popular feature among Puppet admins, because most don’t know how to use it or fear performance issues. Attend this talk to know how to enhance your Puppet deployments with easy cross-nodes interactions and collaborations, while conserving system efficiency.

Statistics

Views

Total Views
13,506
Views on SlideShare
7,368
Embed Views
6,138

Actions

Likes
3
Downloads
36
Comments
0

20 Embeds 6,138

http://puppetlabs.com 4246
http://www.masterzen.fr 933
http://www.puppetlabs.com 862
http://feeds.feedburner.com 34
http://puppetlabs.iron-point.com 17
http://coderwall.com 9
https://webcache.googleusercontent.com 8
http://webcache.googleusercontent.com 6
http://www.slideshare.net 6
http://192.168.1.100 3
http://www.netvibes.com 3
http://a0.twimg.com 2
http://localhost 2
https://si0.twimg.com 1
http://ia.puppetlabs.com 1
http://translate.googleusercontent.com 1
http://feeds.masterzen.fr 1
http://puppetdev.iron-point.com 1
http://static.slidesharecdn.com 1
http://planetpuppet.org 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    All About Storeconfigs All About Storeconfigs Presentation Transcript

    • All About Storeconfigs Brice Figureau masterzen (#puppet)
    • Who am I?
    • Puppet Community Developer planetpuppet.org maintainer
    • Whatʼs Storeconfigs?
    • Storeconfigs stores to an RDBMS uses Railsʼ Active Record
    • 2 Possible Usages Inventory Export/Collect
    • Export/Collect Node collaborations
    • Syntax: export @@resource { ... }
    • Syntax: collect Resource <<| query |>>
    • Collect: Query •Resource Parameters •Tags •and, or
    • Tip Always tag exported resources
    • The BIG Picture master collector DB 2 collector 1 Exporter
    • Simple example Web server replication, deployment server
    • Simple example Web server replication, web server part
    • Intermediate example Building /etc/resolv.conf from exported snippets
    • Intermediate example Building /etc/resolv.conf, aggregating snippets
    • Advanced Example Tinydns master
    • Advanced Example Tinydns master
    • Advanced Example Tinydns slave
    • Limitations
    • Inner Workings
    • Inner Workings 1. Read all resources, tags for a node (slow)
    • Inner Workings 2. Compare with in-memory catalog (fast)
    • Inner Workings 3. Store/remove missing/extraneous resources/tags (slow)
    • Inner Workings DB schema
    • Performance Impact
    • Performance Impact Collecting is fast
    • Performance Impact Storeconfigs write can be slow
    • Performance Impact 1st run (1950 resources) Off On 25 19 13 6 0 Time (seconds)
    • Performance Impact Subsequent runs (1950 resources) Off On 8 6 4 2 0 Time (seconds)
    • First Run Impact
    • First Run Impact Failure err: Could not retrieve catalog: execution expired
    • Scaling: some rough math 1 HDD = 250 IOP/s
    • Scaling: some rough math All nodes have 500 resources 4 params/ resources 4 tags/resources => 4500 writes => 1000 IOP
    • Scaling: some rough math Master concurrency: 5
    • Scaling: some rough math IOP/s left per node: 250/5 = 50 IOP/s
    • Scaling: some rough math Time per node 1000 IOP = 20s
    • Scaling: some rough math And for a 500 nodes setup... 500/5 = 100 “runs”
    • Scaling: some rough math With a sleep time of 30min 30 min = 1800s 1800s/20s = 90 90 nodes per 30 min
    • Scaling: some rough math Houston we have a problem 90 nodes when we needed 100
    • Scaling the back-end Dedicated RDBMS server(s)
    • Scaling the back-end Powerful I/O subsystem (RAID,BBU...)
    • Scaling the back-end DB setup tuning
    • Scaling the back-end MySQL, scaling for inserts innodb_buffer_pool_size = 70% of physical RAM
    • Scaling the back-end MySQL, scaling for inserts innodb_log_file_size = up to 5% of physical RAM
    • Scaling the back-end MySQL, scaling for inserts innodb_flush_method = O_DIRECT
    • Scaling the back-end MySQL, scaling for inserts innodb_flush_log_at_trx_commit = 2
    • 0.25 to the help!
    • 0.25 to the help! Async Storeconfigs - no more thundering herd
    • 0.25: Async Storeconfigs puppetqd DB Message Queue master node
    • 0.25: Thin Storeconfigs Only exported resources, tags and facts
    • 0.25: Thin Storeconfigs 1st run (1950 resources) Off On 6 4 3 1 0 Time (seconds)
    • Storeconfigs future?
    • More AR adapters
    • More AR tuning (pooling)
    • Non RDBMS back-ends
    • Non RDBMS back-ends •Tokyo Cabinet/Tyrant •Mongo DB
    • Questions ? Please speak loud and slowly and excuse me by advance if I donʼt understand a word of your question. Being French has some advantages (like health care access or a cute if not stupid accent) it also has drawbacks, especially our educational system for learning foreign languages simply sucks :-D