Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PuppetConf 2016: Watching the Puppet Show – Sean Porter, Heavy Water Operations

286 views

Published on

Here are the slides from Sean Porter PuppetConf 2016 presentation called Watching the Puppet Show. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Published in: Technology
  • Be the first to comment

PuppetConf 2016: Watching the Puppet Show – Sean Porter, Heavy Water Operations

  1. 1. Watching the Puppet Show Puppet & Sensu Delivering reliable services.
  2. 2. Sean Porter @PorterTech
  3. 3. +
  4. 4. FOCUS ● The challenges ● DevOps & Infrastructure as Code ● Common pitfalls & failure cases ● Sensu ● Puppet & Sensu in practice
  5. 5. Let’s talk about software It is eating the world.
  6. 6. “Software is eating the world” - Marc Andreessen (2011)
  7. 7. SOFTWARE IS EATING THE WORLD! ● Society has an insatiable hunger for software ○ It is becoming part of every facet of our lives ● Companies deliver value with software ● We need to deliver more software, better software, faster, & reliably - Easy right?
  8. 8. Let’s talk about infrastructure A plethora of technologies.
  9. 9. NETFLIX 2013
  10. 10. “Set it all on fire, child” - Overlord Manatee
  11. 11. Let’s talk about DevOps What is DevOps?
  12. 12. WHAT IS DEVOPS? “DevOps is continuously looking for new ways to break down silos, eliminate inefficiencies, and remove the risks that prevent the rapid and reliable delivery of software based services” - Damon Edwards, DevOps Cafe
  13. 13. WHAT IS DEVOPS? ● Continuous improvement - there is no end ● Requires a culture that supports learning ○ Measurement - move the needles ○ Retrospectives (or blameless post-mortems) ● All about delivering better software, faster
  14. 14. Let’s talk about IaC What is Infrastructure as Code?
  15. 15. WHAT IS INFRASTRUCTURE AS CODE? “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources” - Adam Jacob, Web Operations
  16. 16. WHAT IS INFRASTRUCTURE AS CODE? ● It’s not just about reconstruction & repeatability ● IaC techniques scale effectively to manage large numbers of hosts and services ● Apply & revert* changes quickly - move faster! ● All about delivering software, faster, & reliably
  17. 17. BASIC IaC WORKFLOW It’s all software.
  18. 18. BASIC IaC WORKFLOW
  19. 19. No safeties.
  20. 20. INFRASTRUCTURE AS CODE ● Break things at scale! ● Some changes cannot easily be undone ● System state & service health ● Coordinating with application deployments ● “Erosion” - Entropy
  21. 21. BASIC IaC WORKFLOW
  22. 22. BASIC IaC WORKFLOW
  23. 23. BASIC IaC WORKFLOW Improve the feedback loop. Provides continuous testing.
  24. 24. Let’s talk about Sensu What is Sensu?
  25. 25. WHAT IS SENSU? ● It’s a monitoring tool ○ Modern architecture ○ Uses service checks with a simple plugin spec ○ Defined inputs/outputs & very composable ○ Designed for IaC workflows
  26. 26. WHAT IS SENSU? ● A global community ○ 300+ contributors ● Scalable, monitor tens of thousands of systems ● Commercially backed ○ Enterprise version (RBAC etc.) ○ Support, training, & professional services
  27. 27. WHAT PLATFORMS CAN SENSU MONITOR? ● Fantastic multi-platform support! ● Linux (Debian, RHEL) ● Windows ● OS X ● FreeBSD ● Solaris (10, 11) ● AIX
  28. 28. July 11th, 2011
  29. 29. MODERN ARCHITECTURE ● Designed for: ○ Dynamic infrastructure (EC2, Docker, etc.) ○ Public networks ○ Complex network topologies (hybrid cloud) Automatic (de)registration of monitoring clients!
  30. 30. SERVICE CHECKS ● Simple to write & understand ○ STDOUT & exit status code ● Provide context in multiple forms ○ Human readable messages ○ Formatted metrics (PerfData, Graphite, etc.) ● Placed top to bottom - service dependency chain
  31. 31. SENSU CLIENT SOCKET INPUT echo '{ "name": "mysql_backup", "output": "could not connect to mysql", "status": 2, "ttl": 90000 }' | nc localhost 3030
  32. 32. THE SENSU PIPELINE
  33. 33. PLUGINS & EXTENSIONS ● github.com/sensu-plugins (checks, handlers, etc.) ● monitoring-plugins.org ● Many extensions to add protocols etc. ○ StatsD ○ InfluxDB ○ System Profile (metric collection)
  34. 34. JSON CONFIGURATION { "checks": { "mysql_replication": { "command": "check-mysql-replication.rb", "subscribers": ["mysql"], "interval": 30, "playbook": "http://wiki.example.com/mysql-replication-playbook" } } }
  35. 35. Puppet & Sensu In practice.
  36. 36. SENSU PUPPET MODULE forge.puppetlabs.com/sensu/sensu ● A module to install and configure Sensu ● Well documented & tested (score ~ 5.0) ● Types e.g. sensu_check_config ● Awesome contributors! (101+) ○ jlambert121, jamtur01, rodjek, and more!
  37. 37. Let’s configure a Sensu server Sensu servers publish check requests and process check results and events.
  38. 38. SENSU SERVER node 'sensu-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', redis_host => 'redis.foo.com', redis_password => 'p4s5w0rd', server => true, api => true }
  39. 39. Let’s configure a Sensu client On an HTTP API host.
  40. 40. SENSU CLIENT node 'api-01.foo.com' { class { 'sensu': rabbitmq_host => 'rabbit.foo.com', rabbitmq_password => 's3cr3t', subscriptions => [ 'production', 'api' ] } }
  41. 41. Let’s configure a Sensu handler On the Sensu server.
  42. 42. SENSU HANDLER CONFIG sensu::handler { 'slack': command => 'handler-slack.rb', timeout => 30, config => { 'webhook_url' => 'https://...', 'channel' => 'alerts', 'username' => 'sensu' } } sensu::plugin { 'sensu-plugins-slack': type => 'package', pkg_provider => sensu_gem }
  43. 43. Let’s configure a check Run an HTTP endpoint check on ALL API machines. This check is configured on the Sensu server.
  44. 44. SENSU CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, subscribers => ['api'], aggregate => 'api_health', timeout => 60, handlers => ['slack'] }
  45. 45. SENSU CHECK DEPENDENCIES Install the check plugin on hosts expected to run it: sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
  46. 46. Let’s configure a standalone check Run an HTTP endpoint check on the local API machine. This check is configured on the API machine.
  47. 47. SENSU STANDALONE CHECK CONFIG sensu::check { 'api_http_response': command => 'check-http.rb -u https://127.0.0.1/health', interval => 20, standalone => true, aggregate => 'api_health', timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-http': type => 'package', pkg_provider => sensu_gem }
  48. 48. SENSU IN OTHER PUPPET MODULES Create a new class to be included: e.g. apache/manifests/monitoring/sensu.pp class apache::monitoring::sensu { sensu::check { 'apache-running': command => 'check-procs.rb -p /usr/sbin/httpd -w 100 -c 200 -C 1', handlers => ['slack'] } }
  49. 49. SENSU IN OTHER PUPPET MODULES Add client subscriptions and custom attributes: class apache::monitoring::sensu { sensu::subscription { 'apache': 'custom' => { 'ntp_server' => $ntp::servers[0], 'health_endpoint' => '/healthz' } } }
  50. 50. THE SENSU PIPELINE
  51. 51. PUPPET & SENSU
  52. 52. Let’s take it to the next level Puppet module testing & Sensu.
  53. 53. SERVERSPEC RSpec tests for your servers: describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end
  54. 54. RUNNING TESTS ● Test Kitchen ○ github.com/neillturner/kitchen-puppet ● Vagrant plugins ○ github.com/jvoorhis/vagrant-serverspec ● Serverspec SSH ● … choose your own adventure!
  55. 55. TEST ≈ MONITOR TEST ≈ MONITOR
  56. 56. PUPPET MODULE TESTS AS SENSU CHECKS ● Use the Sensu Serverspec check plugin ○ sensu-install -p serverspec check-serverspec.rb -d /etc/sensu/serverspec -t '*_spec.rb'
  57. 57. SENSU SERVERSPEC CHECK CONFIG sensu::check { 'serverspec': command => 'check-serverspec.rb -d /etc/sensu/serverspec', interval => 30, standalone => true, timeout => 60, handlers => ['slack'] } sensu::plugin { 'sensu-plugins-serverspec': type => 'package', pkg_provider => sensu_gem }
  58. 58. SUMMARY ● More software & infrastructure ● DevOps & IaC help us deliver software - faster! ○ No safeties! ● Monitoring MUST be part of the workflow ● Puppet & Sensu have a mutualistic relationship
  59. 59. sensuapp.org Sean Porter - @PorterTech Questions?

×