Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!

1,606 views
1,289 views

Published on

"Build a Modern Infrastructure in 45 min!" presented by Matthew Barr, HERE at Puppet Camp NYC 2014

0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,606
On SlideShare
0
From Embeds
0
Number of Embeds
251
Actions
Shares
0
Downloads
48
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!

  1. 1. Prepared by Build a modern infrastructure in 45 min! Matthew Barr Sr. Systems Engineer
  2. 2. Is your infrastructure a mess? Let’s fix it :)
  3. 3. What we’re going to do: • Define a modern infrastructure • Glance at their architectures • Demonstrate how to do this yourselves • … And then the details..
  4. 4. What is a modern infrastructure?
  5. 5. It includes: • Centralized logging • Monitoring • Orchestration • CI (continuous integration) • Metrics*
  6. 6. What we’ll do today: Setup • Mcollective • Sensu (ideal for cloud infra) • Logstash + ElasticSearch + Kibana • Jenkins
  7. 7. MCollective (mco) • Orchestration • Uses ActiveMQ or RabbitMQ • Maintained by Puppet Labs • http://puppetlabs.com/mcollective
  8. 8. • Distributed monitoring system • Uses RabbitMQ • has a easy API • Adding/remove servers without restarting or changing config files on server • http://sensuapp.org Sensu!
  9. 9. Logstash http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash
  10. 10. Elastic Search & Kibana • Elasticsearch (http://www.elasticsearch.com) is a “distributed restful search and analytics tool” • It’s used as a datastore for Logstash. (it’s not the only one, but one of the most used.) • Kibana is a dashboard for use with Elasticsearch & Logstash.
  11. 11. What we’re actually doing: • Show how to use a set of forge modules to build an infrastructure out. • using the mbarr/moderninfra as an opinionated profile module • download the necessary modules using librarian-puppet
  12. 12. We’ll: • Build a RabbitMQ server + sensu server • the admin host (has the mco client) • Build a logstash server • Build a Jenkins host
  13. 13. Each server will also: • be sending logs via logstash-forwarder • run Sensu client checks • run a mco server
  14. 14. Moderninfra module
  15. 15. A forge module just for you! • Sets up the basics of each service • Sets up the requirements correctly to all work together • Has… opinions.
  16. 16. Install from the forge: puppet module install mbarr-moderninfra
  17. 17. The code! ---! moderninfra::rmqserver: 'rabbitmq.aws.mbarr.net'! moderninfra::mco_password: 'shhhh..its.a.secret.'! moderninfra::sensu_password: 'whatsupdoc'! moderninfra::logstash_server: 'logstash.aws.mbarr.net' Hiera data, to make life easier: class moderninfra (! $rmqserver,! $logstash_server,! $rmq=false,! $mco_client=false,! $mco_server=false,! $sensu_client=false,! $sensu_server=false,! $logstash=false,! $logstash_forwarder=true,! $mco_password=undef,! $sensu_password=undef,! ) {...}
  18. 18. node default {! if $role == "mco" {! class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }! ! if $role == "puppet" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! } if $role == "logstash" {! class {'moderninfra':! logstash => true,! mco_server => true,! sensu_client => true,! }! include profiles::logstash! }! ! if $role == "jenkins" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! include jenkins! }! } Site.pp
  19. 19. RabbitMQ, Sensu & Mcollective
  20. 20. RabbitMQ • This is the middle ware that is used by both mco & sensu. • Our module uses the Puppet SSL certs for connections • Adds a second cert for the host, via the puppet-certificate module.
  21. 21. Code class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }
  22. 22. RMQ Note • To be fair: Sensu isn’t running w/ SSL certs • I’ve used other self signed certs before without issue • Looks like there’s a bug that hopefully is actually fixed in Erlang OTP 17.1
  23. 23. Mcollective • Using SSL to secure PSK connections between mco & RabbitMQ • Installs the package, service & puppet agents.
  24. 24. root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet ! * [ ============================================================> ] 4 / 4 ! puppet-us-east-1b-i-346b2a1f.ec2.mbarr.net: puppet-purged. rmq-us-east-1b-i-6a9bda41.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. logstash-us-east-1b-i-979adbbc.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. jenkins-us-east-1b-i-969adbbd.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. ! Summary of Arch: ! No aggregate summary could be computed ! Summary of Ensure: ! 3.6.2-1puppetlabs1 = 3 purged = 1 ! ! Finished processing 4 / 4 hosts in 1172.09 ms
  25. 25. Sensu • Client on all 4 hosts • Server on RMQ box • Distributed checks • Dashboard on 8080 • profiles::sensuchecks installs various checks. (not in module)
  26. 26. Actually making sensu GO: (on server) class profiles::sensuchecks {! sensu::check { 'check_ntp':! command => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H pool.ntp.org -w 20 -c 40',! handlers => 'default',! subscribers => 'general',! standalone => false,! custom => { occurrences => 2 },! }! sensu::check { 'check_cron':! command => '/etc/sensu/plugins/check-procs.rb -p cron -C 1 -c 10 -w 10 ',! handlers => 'default',! subscribers => 'general',! interval => 60,! standalone => false,! custom => { occurrences => 2 },! }! }!
  27. 27. Logstash
  28. 28. • Centralized logging system • Inputs, Outputs, Filters • Inputs: syslog, files, redis.. • Outputs:elasticsearch, etc • Filters: Grok, many others
  29. 29. Logstash profile class profiles::logstash {! ! logstash::configfile { 'basic_config':! source => 'puppet:///modules/profiles/logstash/basic_config',! order => 10! }! ! include kibana3! ! }!
  30. 30. Logstash config input { lumberjack { port => 12345 ssl_certificate => "/etc/logstash/ssl/cert.pem" ssl_key => "/etc/logstash/ssl/key.pem" type => "lumberjack" } } ! input { tcp { port => 5000 type => syslog } udp { port => 5000 type => syslog } } ! output { elasticsearch { host => localhost } stdout { codec => rubydebug } }
  31. 31. Logstash-forwarder • Data is sent from logs on client to Logstash server via SSL • Keeps track of log positions and what’s been sent • Server listens on 12345, for now.
  32. 32. Elasticsearch & Kibana • This is what Kibana looks like with data from logstash fed into elasticsearch • (It’s zoomed a bit, so you can see the good parts.)
  33. 33. Jenkins
  34. 34. Jenkins • Continuous integration tool • There is code to set up slaves in the Jenkins module. • https://forge.puppetlabs.com/rtyler/jenkins
  35. 35. include jenkins
  36. 36. Things this module doesn’t do: • Build your puppet master • DNS names for Puppet master, RMQ, Logstash, etc • Although the cloud formation templates do!
  37. 37. But it might let you sleep at night…
  38. 38. Appendix:! Puppet Master
  39. 39. • Built w/ CloudFormations template • Sorry, not vagrant. Might be added soon. • uses cloud-init to provision puppet & code base • Uses puppet 3.6.2 • Librarian-puppet
  40. 40. Puppet Master • Set host name & domain • Install puppet • rm -rf /etc/puppet • git clone REPO /etc/puppet
  41. 41. Appendix: ! Librarian-puppet
  42. 42. Librarian Puppet • Lets you take a Puppetfile, and manage modules & dependencies • can use forge or git repos • Takes over your modules directory, though. • adds to .gitignore & regenerates the directory from the Puppetfile • I’ve used a pattern of a second directory (modules-local) to allow a slow migration & local files to stay in your existing repo
  43. 43. Modules-local pattern Old: modulepath = $confdir/modules:$confdir/modules-local ! 3.6+ directory environments: environment.conf modulepath = modules:modules-local
  44. 44. Puppetfile forge "https://forgeapi.puppetlabs.com" ! mod "reidmv/puppet_certificate" mod "elasticsearch/logstash" mod "elasticsearch/elasticsearch" mod "sensu/sensu" ! mod "rtyler/jenkins" ! mod "puppetlabs/mcollective" ! mod "thejandroman/kibana3", "0.0.3" ! # mod "mbarr/moderninfra", # :git => "git://github.com/matthewbarr/moderninfra.git" ! #mod "garethr/graphite"
  45. 45. modules ├── activemq ├── apache ├── apt ├── concat ├── datacat ├── elasticsearch ├── epel ├── erlang ├── file_concat ├── git ├── java ├── java_ks ├── jenkins ├── kibana3 ├── logstash ├── mcollective ├── puppet_certificate ├── rabbitmq ├── sensu ├── staging ├── stdlib ├── vcsrepo └── zypprepo modules-local ├── moderninfra └── profiles
  46. 46. We’re hiring! (in Boston)! ! ! Matthew Barr! @matthewbarr (github & twitter)! matthew.barr@here.com! mbarr@mbarr.net! http://github.com/matthewbarr/build-modern-infra

×