Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!

0 views

Published on

"Build a Modern Infrastructure in 45 min!" presented by Matthew Barr, HERE at Puppet Camp NYC 2014

  • Be the first to comment

Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!

  1. 1. Prepared by Build a modern infrastructure in 45 min! Matthew Barr Sr. Systems Engineer
  2. 2. Is your infrastructure a mess? Let’s fix it :)
  3. 3. What we’re going to do: • Define a modern infrastructure • Glance at their architectures • Demonstrate how to do this yourselves • … And then the details..
  4. 4. What is a modern infrastructure?
  5. 5. It includes: • Centralized logging • Monitoring • Orchestration • CI (continuous integration) • Metrics*
  6. 6. What we’ll do today: Setup • Mcollective • Sensu (ideal for cloud infra) • Logstash + ElasticSearch + Kibana • Jenkins
  7. 7. MCollective (mco) • Orchestration • Uses ActiveMQ or RabbitMQ • Maintained by Puppet Labs • http://puppetlabs.com/mcollective
  8. 8. • Distributed monitoring system • Uses RabbitMQ • has a easy API • Adding/remove servers without restarting or changing config files on server • http://sensuapp.org Sensu!
  9. 9. Logstash http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash
  10. 10. Elastic Search & Kibana • Elasticsearch (http://www.elasticsearch.com) is a “distributed restful search and analytics tool” • It’s used as a datastore for Logstash. (it’s not the only one, but one of the most used.) • Kibana is a dashboard for use with Elasticsearch & Logstash.
  11. 11. What we’re actually doing: • Show how to use a set of forge modules to build an infrastructure out. • using the mbarr/moderninfra as an opinionated profile module • download the necessary modules using librarian-puppet
  12. 12. We’ll: • Build a RabbitMQ server + sensu server • the admin host (has the mco client) • Build a logstash server • Build a Jenkins host
  13. 13. Each server will also: • be sending logs via logstash-forwarder • run Sensu client checks • run a mco server
  14. 14. Moderninfra module
  15. 15. A forge module just for you! • Sets up the basics of each service • Sets up the requirements correctly to all work together • Has… opinions.
  16. 16. Install from the forge: puppet module install mbarr-moderninfra
  17. 17. The code! ---! moderninfra::rmqserver: 'rabbitmq.aws.mbarr.net'! moderninfra::mco_password: 'shhhh..its.a.secret.'! moderninfra::sensu_password: 'whatsupdoc'! moderninfra::logstash_server: 'logstash.aws.mbarr.net' Hiera data, to make life easier: class moderninfra (! $rmqserver,! $logstash_server,! $rmq=false,! $mco_client=false,! $mco_server=false,! $sensu_client=false,! $sensu_server=false,! $logstash=false,! $logstash_forwarder=true,! $mco_password=undef,! $sensu_password=undef,! ) {...}
  18. 18. node default {! if $role == "mco" {! class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }! ! if $role == "puppet" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! } if $role == "logstash" {! class {'moderninfra':! logstash => true,! mco_server => true,! sensu_client => true,! }! include profiles::logstash! }! ! if $role == "jenkins" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! include jenkins! }! } Site.pp
  19. 19. RabbitMQ, Sensu & Mcollective
  20. 20. RabbitMQ • This is the middle ware that is used by both mco & sensu. • Our module uses the Puppet SSL certs for connections • Adds a second cert for the host, via the puppet-certificate module.
  21. 21. Code class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }
  22. 22. RMQ Note • To be fair: Sensu isn’t running w/ SSL certs • I’ve used other self signed certs before without issue • Looks like there’s a bug that hopefully is actually fixed in Erlang OTP 17.1
  23. 23. Mcollective • Using SSL to secure PSK connections between mco & RabbitMQ • Installs the package, service & puppet agents.
  24. 24. root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet ! * [ ============================================================> ] 4 / 4 ! puppet-us-east-1b-i-346b2a1f.ec2.mbarr.net: puppet-purged. rmq-us-east-1b-i-6a9bda41.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. logstash-us-east-1b-i-979adbbc.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. jenkins-us-east-1b-i-969adbbd.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. ! Summary of Arch: ! No aggregate summary could be computed ! Summary of Ensure: ! 3.6.2-1puppetlabs1 = 3 purged = 1 ! ! Finished processing 4 / 4 hosts in 1172.09 ms
  25. 25. Sensu • Client on all 4 hosts • Server on RMQ box • Distributed checks • Dashboard on 8080 • profiles::sensuchecks installs various checks. (not in module)
  26. 26. Actually making sensu GO: (on server) class profiles::sensuchecks {! sensu::check { 'check_ntp':! command => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H pool.ntp.org -w 20 -c 40',! handlers => 'default',! subscribers => 'general',! standalone => false,! custom => { occurrences => 2 },! }! sensu::check { 'check_cron':! command => '/etc/sensu/plugins/check-procs.rb -p cron -C 1 -c 10 -w 10 ',! handlers => 'default',! subscribers => 'general',! interval => 60,! standalone => false,! custom => { occurrences => 2 },! }! }!
  27. 27. Logstash
  28. 28. • Centralized logging system • Inputs, Outputs, Filters • Inputs: syslog, files, redis.. • Outputs:elasticsearch, etc • Filters: Grok, many others
  29. 29. Logstash profile class profiles::logstash {! ! logstash::configfile { 'basic_config':! source => 'puppet:///modules/profiles/logstash/basic_config',! order => 10! }! ! include kibana3! ! }!
  30. 30. Logstash config input { lumberjack { port => 12345 ssl_certificate => "/etc/logstash/ssl/cert.pem" ssl_key => "/etc/logstash/ssl/key.pem" type => "lumberjack" } } ! input { tcp { port => 5000 type => syslog } udp { port => 5000 type => syslog } } ! output { elasticsearch { host => localhost } stdout { codec => rubydebug } }
  31. 31. Logstash-forwarder • Data is sent from logs on client to Logstash server via SSL • Keeps track of log positions and what’s been sent • Server listens on 12345, for now.
  32. 32. Elasticsearch & Kibana • This is what Kibana looks like with data from logstash fed into elasticsearch • (It’s zoomed a bit, so you can see the good parts.)
  33. 33. Jenkins
  34. 34. Jenkins • Continuous integration tool • There is code to set up slaves in the Jenkins module. • https://forge.puppetlabs.com/rtyler/jenkins
  35. 35. include jenkins
  36. 36. Things this module doesn’t do: • Build your puppet master • DNS names for Puppet master, RMQ, Logstash, etc • Although the cloud formation templates do!
  37. 37. But it might let you sleep at night…
  38. 38. Appendix:! Puppet Master
  39. 39. • Built w/ CloudFormations template • Sorry, not vagrant. Might be added soon. • uses cloud-init to provision puppet & code base • Uses puppet 3.6.2 • Librarian-puppet
  40. 40. Puppet Master • Set host name & domain • Install puppet • rm -rf /etc/puppet • git clone REPO /etc/puppet
  41. 41. Appendix: ! Librarian-puppet
  42. 42. Librarian Puppet • Lets you take a Puppetfile, and manage modules & dependencies • can use forge or git repos • Takes over your modules directory, though. • adds to .gitignore & regenerates the directory from the Puppetfile • I’ve used a pattern of a second directory (modules-local) to allow a slow migration & local files to stay in your existing repo
  43. 43. Modules-local pattern Old: modulepath = $confdir/modules:$confdir/modules-local ! 3.6+ directory environments: environment.conf modulepath = modules:modules-local
  44. 44. Puppetfile forge "https://forgeapi.puppetlabs.com" ! mod "reidmv/puppet_certificate" mod "elasticsearch/logstash" mod "elasticsearch/elasticsearch" mod "sensu/sensu" ! mod "rtyler/jenkins" ! mod "puppetlabs/mcollective" ! mod "thejandroman/kibana3", "0.0.3" ! # mod "mbarr/moderninfra", # :git => "git://github.com/matthewbarr/moderninfra.git" ! #mod "garethr/graphite"
  45. 45. modules ├── activemq ├── apache ├── apt ├── concat ├── datacat ├── elasticsearch ├── epel ├── erlang ├── file_concat ├── git ├── java ├── java_ks ├── jenkins ├── kibana3 ├── logstash ├── mcollective ├── puppet_certificate ├── rabbitmq ├── sensu ├── staging ├── stdlib ├── vcsrepo └── zypprepo modules-local ├── moderninfra └── profiles
  46. 46. We’re hiring! (in Boston)! ! ! Matthew Barr! @matthewbarr (github & twitter)! matthew.barr@here.com! mbarr@mbarr.net! http://github.com/matthewbarr/build-modern-infra

×