Puppet Data Mining

4,690 views

Published on

Talk given at the 6th PuppetCamp in Edinburgh.

Covers some of the useful information puppet stores and how to get at it.

0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,690
On SlideShare
0
From Embeds
0
Number of Embeds
125
Actions
Shares
0
Downloads
76
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Puppet Data Mining

  1. 1. Puppet Data MiningReport processors, stored configs and morePuppetCamp 22nd March 2012gareth rushgrove | morethanseven.net http://www.flickr.com/photos/map408/2412123378
  2. 2. Gareth Rushgrove @garethrgareth rushgrove | morethanseven.net
  3. 3. Blog at morethanseven.netgareth rushgrove | morethanseven.net
  4. 4. Curate devopsweekly.comgareth rushgrove | morethanseven.net
  5. 5. TextWork at UK Government Digital Servicegareth rushgrove | morethanseven.net
  6. 6. Serious Government Businessgareth rushgrove | morethanseven.net
  7. 7. - Why do we want it?- How do we get our hands on it- What can we build with itPuppet datagareth rushgrove | morethanseven.net http://www.flickr.com/photos/iancarroll/5027441664
  8. 8. - Why do we want it?- How do we get our hands on it- What can we build with itPuppet datagareth rushgrove | morethanseven.net http://www.flickr.com/photos/iancarroll/5027441664
  9. 9. - Why do we want it?- How do we get our hands on it- What can we build with itPuppet datagareth rushgrove | morethanseven.net http://www.flickr.com/photos/iancarroll/5027441664
  10. 10. Why
  11. 11. Single source of truthgareth rushgrove | morethanseven.net http://www.flickr.com/photos/joeshlabotnik/4132307070/
  12. 12. - Spreadsheets- Wiki- Deployment scripts- Application code- SSH configs- More deployment scripts- Monitoring configurationExample: find a list of hostnamesgareth rushgrove | morethanseven.net
  13. 13. Operational reportsgareth rushgrove | morethanseven.net http://www.flickr.com/photos/nettsu/5276327339
  14. 14. Company hardwareExample: hardware spreadsheetgareth rushgrove | morethanseven.net
  15. 15. Hooks for toolinggareth rushgrove | morethanseven.net http://www.flickr.com/photos/nickstone333/3135318558
  16. 16. Example: Nagios integrationgareth rushgrove | morethanseven.net
  17. 17. Example: Rundeck integrationgareth rushgrove | morethanseven.net
  18. 18. How
  19. 19. - Report processors- Stored configurations- Puppet internal APIsWhere to find the datagareth rushgrove | morethanseven.net
  20. 20. Report processorsgareth rushgrove | morethanseven.net http://www.flickr.com/photos/nathaninsandiego/3757033518
  21. 21. [agent] report = true pluginsync = true [master] reports = store pluginsync = trueConfigurationgareth rushgrove | morethanseven.net
  22. 22. [agent] report = true pluginsync = true [master] reports = store,http,log,sample pluginsync = trueAdding reportsgareth rushgrove | morethanseven.net
  23. 23. Built ingareth rushgrove | morethanseven.net
  24. 24. Store the YAML report on disk. Each host sends its report as a YAML dump and this just stores the file on disk, in the reportdir directory.Storegareth rushgrove | morethanseven.net
  25. 25. Send all received logs to the local log destinations. Usually the log destination is syslog.Loggareth rushgrove | morethanseven.net
  26. 26. Send report information via HTTP to the reporturl. Each host sends its report as a YAML dump and this sends this YAML to a client via HTTP POST.HTTPgareth rushgrove | morethanseven.net
  27. 27. This report sends specific log messages to specific email addresses based on the tags in the log messages.Tagmailgareth rushgrove | morethanseven.net
  28. 28. Graph all available data about hosts using the RRD library.RRDgraphgareth rushgrove | morethanseven.net
  29. 29. . └── sample-report ├── lib │   └── puppet │   └── reports │   └── sample.rb └── manifests └── init.ppSimple report structuregareth rushgrove | morethanseven.net
  30. 30. require puppet require yaml require logger LOG = Logger.new(/tmp/puppet.log) Puppet::Reports.register_report(:sample) do def process message = "Puppet run for #{self.host} #{self.status}" Puppet.debug "Hello from sample report processor" LOG.info message end endSimple report processorgareth rushgrove | morethanseven.net
  31. 31. require puppet require yaml require logger LOG = Logger.new(/tmp/puppet.log) Puppet::Reports.register_report(:sample) do def process message = "Puppet run for #{self.host} #{self.status}" Puppet.debug "Hello from sample report processor" LOG.info message end endRegister reportgareth rushgrove | morethanseven.net
  32. 32. require puppet require yaml require logger LOG = Logger.new(/tmp/puppet.log) Puppet::Reports.register_report(:sample) do def process message = "Puppet run for #{self.host} #{self.status}" Puppet.debug "Hello from sample report processor" LOG.info message end endDo something with the data in selfgareth rushgrove | morethanseven.net
  33. 33. Open source examplesgareth rushgrove | morethanseven.net http://www.flickr.com/photos/nettsu/5771348892
  34. 34. IRC/Campfiregareth rushgrove | morethanseven.net
  35. 35. IRC/Campfiregareth rushgrove | morethanseven.net
  36. 36. Gangliagareth rushgrove | morethanseven.net
  37. 37. Graphs!gareth rushgrove | morethanseven.net
  38. 38. Zendeskgareth rushgrove | morethanseven.net
  39. 39. James Turnbull will have got there firstgareth rushgrove | morethanseven.net
  40. 40. Stored configuration databasegareth rushgrove | morethanseven.net
  41. 41. [master] storeconfigs = true dbadapter = mysql dbuser = puppet dbpassword = password dbserver = localhost dbsocket = /var/run/mysql.sockConfigurationgareth rushgrove | morethanseven.net
  42. 42. SQLgareth rushgrove | morethanseven.net http://www.flickr.com/photos/nettsu/4997777753
  43. 43. mysql> SHOW TABLES; +------------------+ | Tables_in_puppet | +------------------+ | fact_names | | fact_values | | hosts | | inventory_facts | | inventory_nodes | | param_names | | param_values | | puppet_tags | | resource_tags | | resources | | source_files | +------------------+ 11 rows in set (0.00 sec)Lots of tables to exploregareth rushgrove | morethanseven.net
  44. 44. mysql> SELECT name,ip,environment,last_compile FROM hosts; +--------------------+-----------------+-------------+---------------------+ | name | ip | environment | last_compile | +--------------------+-----------------+-------------+---------------------+ | ubuntu.localdomain | 192.168.157.129 | production | 2012-03-04 21:19:41 | +--------------------+-----------------+-------------+---------------------+ 1 row in set (0.00 sec)Easily get a list of hostsgareth rushgrove | morethanseven.net
  45. 45. mysql> SELECT filename,updated_at FROM source_files; +-----------------------------------------------------+---------------------+ | filename | updated_at | +-----------------------------------------------------+---------------------+ | NULL | 2011-10-11 10:34:36 | | /etc/puppet/modules/users/manifests/init.pp | 2011-10-12 14:13:49 | | /etc/puppet/modules/nagios/manifests/init.pp | 2011-10-12 14:13:49 | | /etc/puppet/modules/ntp/manifests/init.pp | 2011-10-12 14:13:50 | | /etc/puppet/modules/base_packages/manifests/init.pp | 2011-10-12 14:13:50 | | /etc/puppet/modules/sudo/manifests/init.pp | 2011-10-12 14:13:50 | | /etc/puppet/modules/apt/manifests/init.pp | 2011-10-12 14:13:50 | | /etc/puppet/modules/logrotate/manifests/init.pp | 2011-10-12 14:13:51 | | /etc/puppet/modules/ganglia/manifests/init.pp | 2011-10-12 14:13:52 | | /etc/puppet/modules/motd/manifests/init.pp | 2011-10-12 14:13:57 | | /etc/puppet/manifests/classes.pp | 2011-10-12 14:17:25 | | /etc/puppet/modules/mysql/manifests/init.pp | 2011-10-12 14:17:25 | | /etc/puppet/modules/apache2/manifests/init.pp | 2011-10-12 14:17:25 | | /etc/puppet/modules/passenger/manifests/init.pp | 2011-10-12 14:17:26 | +-----------------------------------------------------+---------------------+Other interesting data toogareth rushgrove | morethanseven.net
  46. 46. Active Recordgareth rushgrove | morethanseven.net http://www.flickr.com/photos/simplebitsdan/480696708
  47. 47. #!/usr/bin/env ruby require puppet/rails Puppet[:config] = /etc/puppet/puppet.conf Puppet.parse_config config = Puppet.settings.instance_variable_get(:@values) master_config = config[:master]Load Puppet master configurationgareth rushgrove | morethanseven.net
  48. 48. adapter = master_conf[:dbadapter] args = {:adapter => adapter, :log_level => master_conf[:rails_loglevel]} args[:host] = master_conf[:dbserver] args[:username] = master_conf[:dbuser] args[:password] = master_conf[:dbpassword] args[:database] = master_conf[:dbname] args[:database] = "puppet" unless not args[:database].to_s.empty? args[:port] = master_conf[:dbport] socket = master_conf[:dbsocket] args[:socket] = socket unless socket.to_s.empty?Map database connection settingsgareth rushgrove | morethanseven.net
  49. 49. ActiveRecord::Base.establish_connection(args) Puppet::Rails::Host.all.each { |h| puts "#{h.name}" }Make an active record querygareth rushgrove | morethanseven.net
  50. 50. ⚡ ./list.rb ubuntu.localdomainExample rungareth rushgrove | morethanseven.net
  51. 51. Code spelunkinggareth rushgrove | morethanseven.net
  52. 52. #!/usr/bin/env ruby require puppet Puppet[:config] = "/etc/puppet/puppet.conf" Puppet.parse_config Puppet[:clientyamldir] = Puppet[:yamldir] Puppet::Node.indirection.terminus_class = :yamlLoad Puppet master configurationgareth rushgrove | morethanseven.net
  53. 53. nodes = Puppet::Node.indirection.search("*") nodes.each do |n| facts = Puppet::Node::Facts.indirection.find(n.name) tags = Puppet::Resource::Catalog.indirection.find(n.name).tags puts "#{n.name} - #{tags.join(, )}" endMake a query using the internal APIgareth rushgrove | morethanseven.net
  54. 54. ⚡ sudo ./puppettags.rb ubuntu.localdomain - settings, default, nodeNeeds root permissionsgareth rushgrove | morethanseven.net
  55. 55. What
  56. 56. ⚡ ./puppetlast.rb +-----------+-----------------+ | hostname | last puppet run | +-----------+-----------------+ | hostname4 | 100 minutes | | hostname1 | 25 minutes | | hostname5 | 10 minutes | | hostname2 | 15 minutes | | hostname3 | 5 minutes | +-----------+-----------------+Command line toolsgareth rushgrove | morethanseven.net
  57. 57. Dashboardsgareth rushgrove | morethanseven.net
  58. 58. web-puppetgareth rushgrove | morethanseven.net
  59. 59. JSON over HTTPgareth rushgrove | morethanseven.net
  60. 60. capistrano-puppetgareth rushgrove | morethanseven.net
  61. 61. require capistrano-puppet web_puppet = CapistranoPuppet::Server.new( http://username:password@localhost:9295) role :web do web_puppet.get_servers(webserver) endGet hosts from puppetgareth rushgrove | morethanseven.net
  62. 62. Visualisationgareth rushgrove | morethanseven.net
  63. 63. Your imagination ?gareth rushgrove | morethanseven.net
  64. 64. - web-puppet - https://github.com/garethr/web-puppet- capistrano-puppet - https://github.com/garethr/capistrano-puppet- puppet-ganglia - https://github.com/jamtur01/puppet-ganglia- puppet-zendesk - https://github.com/jamtur01/puppet-zendesk- puppet-irc - https://github.com/jamtur01/puppet-irc- puppet-campfire - https://github.com/jamtur01/puppet-campfireLinksgareth rushgrove | morethanseven.net
  65. 65. One more thinggareth rushgrove | morethanseven.net http://www.flickr.com/photos/benterrett/6852348725/
  66. 66. Questions?gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/

×