Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Node collaboration - sharing information between your systems

4,926 views

Published on

Techniques for allowing your systems to share information with each other.
Presented at the October Sydney DevOps Meetup

Published in: Technology
  • Be the first to comment

Node collaboration - sharing information between your systems

  1. 1. Sharing information between your systems Michael Richardson @m_richo
  2. 2. Agenda 1.Chef Search 2.Puppet Exported Resources 3.PuppetDB query 4.Confd + etcd
  3. 3. Agenda 1.Chef Search 2.Puppet Exported Resources 3.PuppetDB query 4.Confd + etcd
  4. 4. Chef Server Stores many useful things including • Policies applied to nodes • Metadata that describes each node (ohai) • And more…
  5. 5. Chef Server This data is searchable!
  6. 6. Chef Server This data is searchable!
  7. 7. Chef Server This data is searchable! But how?
  8. 8. Chef Server 1. ‘search’ method within recipe. 2. Search subcommand of ‘knife’ 3. /search or /search/INDEX endpoints of Chef Server API
  9. 9. Chef Server 1. ‘search’ method within recipe. 2. Search subcommand of ‘knife’ 3. /search or /search/INDEX endpoints of Chef Server API
  10. 10. Chef Server Search within recipe – simple.
  11. 11. Chef Server Search within recipe – simple.
  12. 12. Chef Server Search within recipe – simple.
  13. 13. Chef Server Search within recipe – simple. It’s that simple
  14. 14. Chef Server Search within recipe – more complex.
  15. 15. Chef Server Chef Search is fantastic for • Searching for nodes with particular attributes • Searching for nodes with particular profiles • Using search results to configure services on other nodes.
  16. 16. Chef Server Common use cases • My application talks to a database. What’s its FQDN/IP? • How many memcache nodes are there for my application? What are there FQDN/IPs? • Where should I send my log files? What’s the FQDN of the syslog server? • I want to provide a white-list of IP’s that can request information from a node. • What are the IP addresses of the webservers behind my load balancer? • My Jenkins slaves need to connect to a Jenkins master. What’s it’s IP? • What are the SSH Host keys for particular hosts.
  17. 17. Agenda 1.Chef Search 2.Puppet Exported Resources 3.PuppetDB query 4.Confd + etcd
  18. 18. Exported Resources Definition: An exported resource declaration specifies a desired state for a resource, does not manage the resource on the target system, and publishes the resource for use by other nodes. Any node (including the node that exported it) can then collect the exported resource and manage its own copy of it. * * https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html
  19. 19. Exported Resources Think of it as • Node(s) “publish” resources to the Puppet Master • Other Node(s) consume those resources * https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html
  20. 20. Exported Resources Common Examples • Backend servers configured in a load balancer • Monitoring Servers updated with monitoring clients (nagios_host / nagios_service) • Distribute public keys (ssh and openssl)
  21. 21. Agenda 1.Chef Search 2.Puppet Exported Resources 3.PuppetDB query 4.Confd + etcd
  22. 22. PuppetDB Definition: collects data generated by Puppet. It enables advanced Puppet features like the inventory service and exported resources, and can be the foundation for other applications that use Puppet’s data. *
  23. 23. PuppetDB PuppetDB stores: • The most recent facts from every node • The most recent catalog for every node • Optionally, 14 days of event reports for every node • Exported Resources
  24. 24. PuppetDB PuppetDB stores: • The most recent facts from every node • The most recent catalog for every node • Optionally, 14 days of event reports for every node • Exported Resources
  25. 25. PuppetDB PuppetDB REST API V3 API includes the following endpoints: • Facts • Resources • Nodes • Fact-names, • Metrics • Reports • Events • And more
  26. 26. PuppetDB
  27. 27. PuppetDB
  28. 28. PuppetDB But this talk is about Node Collaboration. How can I make use of this data?
  29. 29. PuppetDB checkout Puppetdbquery module by Erik Dalén https://forge.puppetlabs.com/dalen/puppetdbquery https://github.com/dalen/puppet-puppetdbquery
  30. 30. PuppetDB Puppetdbquery module • Command line tools • Puppet functions to query PuppetDB • Hiera backend to return query results from PuppetDB
  31. 31. PuppetDB Puppetdbquery module • Command line tools • Puppet functions to query PuppetDB • Hiera backend to return query results from PuppetDB
  32. 32. PuppetDB Puppetdbquery functions • query_nodes • query_facts
  33. 33. PuppetDB
  34. 34. PuppetDB note: Output is abbreviated
  35. 35. PuppetDB PuppetDBquery • Works great and very similar to Chef Search (searching for specific hosts, and use their facts for configuring other hosts). • If you use Roles+Profiles pattern, this is a fantastic fit for profiles.
  36. 36. Chef Search and PuppetDB are great!
  37. 37. Chef Search and PuppetDB are great! but what if…
  38. 38. Chef Search and PuppetDB are great! but what if… • You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously).
  39. 39. Chef Search and PuppetDB are great! but what if… • You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously). • You appreciate “desired state” AND “running state” aren’t always the same thing.
  40. 40. Chef Search and PuppetDB are great! but what if… • You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously). • You appreciate “desired state” AND “running state” aren’t always the same thing. • You don’t want to run a Puppet Master or Chef Server (masterless Puppet / Chef-solo).
  41. 41. Chef Search and PuppetDB are great! but what if… • You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously). • You appreciate “desired state” AND “running state” aren’t always the same thing. • You don’t want to run a Puppet Master or Chef Server (masterless Puppet / Chef-solo). • You don’t use Puppet or Chef. 
  42. 42. Agenda 1.Chef Search 2.Puppet Exported Resources 3.PuppetDB query 4.Confd + etcd
  43. 43. Confd • “Lightweight configuration management tool” • https://github.com/kelseyhightower/confd • Written in go • Manage local application configuration files using templates and data from etcd or consul.
  44. 44. etcd • Highly-available key/value store for shared configuration and service discovery • https://github.com/coreos/etcd • Written in go • Typical cluster size or 3-9 peers. • CAP theorem
  45. 45. Confd + etcd Distributed etcd cluster etcd1 etcd2 etcd3 etcd4 etcd5 Hosts running applications and services configured with confd
  46. 46. Confd + etcd • Hosts/Services that wish to share information post data to etcd. • Confd is configured on other hosts to use this data to manage local configuration files and services.
  47. 47. Confd + etcd Distributed etcd cluster etcd1 etcd2 etcd3 etcd4 etcd5 1 2 App server DB server Example 1. App server runs Confd which is polling the following keys. 1. /myapp/db_ip 2. /myapp/db_port 3. /myapp/db_adapter 4. /myapp/db_name 2. Once up and running DB Server, writes details to etcd cluster. 3. Confd reads key/values. Uses template file to write new application configuration file and reloads application.
  48. 48. Confd + etcd Demo A “working” demo is worth a thousand powerpoint slides…
  49. 49. Summary • Share information between your servers. • Let them discover each other. • Let your nodes collaborate with each other.
  50. 50. Thank you

×