0
Prepared by
Build a modern infrastructure in 45 min!
Matthew Barr
Sr. Systems Engineer
Is your
infrastructure a
mess?
Let’s fix it :)
What we’re going to do:
• Define a modern infrastructure
• Glance at their architectures
• Demonstrate how to do this yours...
What is a modern infrastructure?
It includes:
• Centralized logging
• Monitoring
• Orchestration
• CI (continuous integration)
• Metrics*
What we’ll do today: Setup
• Mcollective
• Sensu (ideal for cloud infra)
• Logstash + ElasticSearch + Kibana
• Jenkins
MCollective (mco)
• Orchestration
• Uses ActiveMQ or RabbitMQ
• Maintained by Puppet Labs
• http://puppetlabs.com/mcollect...
• Distributed monitoring system
• Uses RabbitMQ
• has a easy API
• Adding/remove servers without restarting or changing co...
Logstash
http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash
Elastic Search & Kibana
• Elasticsearch (http://www.elasticsearch.com) is a “distributed
restful search and analytics tool...
What we’re actually doing:
• Show how to use a set of forge modules to build an infrastructure
out.
• using the mbarr/mode...
We’ll:
• Build a RabbitMQ server + sensu server
• the admin host (has the mco client)
• Build a logstash server
• Build a ...
Each server will also:
• be sending logs via logstash-forwarder
• run Sensu client checks
• run a mco server
Moderninfra module
A forge module just for you!
• Sets up the basics of each service
• Sets up the requirements correctly to all work togethe...
Install from the forge:
puppet module install mbarr-moderninfra
The code!
---!
moderninfra::rmqserver: 'rabbitmq.aws.mbarr.net'!
moderninfra::mco_password: 'shhhh..its.a.secret.'!
modern...
node default {!
if $role == "mco" {!
class {'moderninfra':!
rmq => true,!
mco_client => true,!
sensu_server => true,!
}!
i...
RabbitMQ, Sensu & Mcollective
RabbitMQ
• This is the middle ware that is used by both mco & sensu.
• Our module uses the Puppet SSL certs for connection...
Code
class {'moderninfra':!
rmq => true,!
mco_client => true,!
sensu_server => true,!
}!
include profiles::sensuchecks !
}
RMQ Note
• To be fair: Sensu isn’t running w/ SSL certs
• I’ve used other self signed certs before without issue
• Looks l...
Mcollective
• Using SSL to secure PSK connections between mco & RabbitMQ
• Installs the package, service & puppet agents.
root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet
!
* [ ========================================================...
Sensu
• Client on all 4 hosts
• Server on RMQ box
• Distributed checks
• Dashboard on 8080
• profiles::sensuchecks installs...
Actually making sensu GO: (on server)
class profiles::sensuchecks {!
sensu::check { 'check_ntp':!
command => 'PATH=$PATH:/...
Logstash
• Centralized logging system
• Inputs, Outputs, Filters
• Inputs: syslog, files, redis..
• Outputs:elasticsearch, etc
• Fil...
Logstash profile
class profiles::logstash {!
!
logstash::configfile { 'basic_config':!
source => 'puppet:///modules/profile...
Logstash config
input {
lumberjack {
port => 12345
ssl_certificate => "/etc/logstash/ssl/cert.pem"
ssl_key => "/etc/logstash...
Logstash-forwarder
• Data is sent from logs on client to Logstash server via SSL
• Keeps track of log positions and what’s...
Elasticsearch & Kibana
• This is what Kibana looks like with data from logstash fed into
elasticsearch
• (It’s zoomed a bi...
Jenkins
Jenkins
• Continuous integration tool
• There is code to set up slaves in the Jenkins module.
• https://forge.puppetlabs.c...
include jenkins
Things this module doesn’t do:
• Build your puppet master
• DNS names for Puppet master, RMQ, Logstash, etc
• Although the...
But it might let
you sleep at
night…
Appendix:!
Puppet Master
• Built w/ CloudFormations template
• Sorry, not vagrant. Might be added soon.
• uses cloud-init to provision puppet & cod...
Puppet Master
• Set host name & domain
• Install puppet
• rm -rf /etc/puppet
• git clone REPO /etc/puppet
Appendix: !
Librarian-puppet
Librarian Puppet
• Lets you take a Puppetfile, and manage modules & dependencies
• can use forge or git repos
• Takes over ...
Modules-local pattern
Old:
modulepath = $confdir/modules:$confdir/modules-local
!
3.6+ directory environments: environment...
Puppetfile
forge "https://forgeapi.puppetlabs.com"
!
mod "reidmv/puppet_certificate"
mod "elasticsearch/logstash"
mod "elast...
modules
├── activemq
├── apache
├── apt
├── concat
├── datacat
├── elasticsearch
├── epel
├── erlang
├── file_concat
├── gi...
We’re hiring! (in Boston)!
!
!
Matthew Barr!
@matthewbarr (github & twitter)!
matthew.barr@here.com!
mbarr@mbarr.net!
http...
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Upcoming SlideShare
Loading in...5
×

Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!

1,022

Published on

"Build a Modern Infrastructure in 45 min!" presented by Matthew Barr, HERE at Puppet Camp NYC 2014

0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,022
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
44
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Transcript of "Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!"

  1. 1. Prepared by Build a modern infrastructure in 45 min! Matthew Barr Sr. Systems Engineer
  2. 2. Is your infrastructure a mess? Let’s fix it :)
  3. 3. What we’re going to do: • Define a modern infrastructure • Glance at their architectures • Demonstrate how to do this yourselves • … And then the details..
  4. 4. What is a modern infrastructure?
  5. 5. It includes: • Centralized logging • Monitoring • Orchestration • CI (continuous integration) • Metrics*
  6. 6. What we’ll do today: Setup • Mcollective • Sensu (ideal for cloud infra) • Logstash + ElasticSearch + Kibana • Jenkins
  7. 7. MCollective (mco) • Orchestration • Uses ActiveMQ or RabbitMQ • Maintained by Puppet Labs • http://puppetlabs.com/mcollective
  8. 8. • Distributed monitoring system • Uses RabbitMQ • has a easy API • Adding/remove servers without restarting or changing config files on server • http://sensuapp.org Sensu!
  9. 9. Logstash http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash
  10. 10. Elastic Search & Kibana • Elasticsearch (http://www.elasticsearch.com) is a “distributed restful search and analytics tool” • It’s used as a datastore for Logstash. (it’s not the only one, but one of the most used.) • Kibana is a dashboard for use with Elasticsearch & Logstash.
  11. 11. What we’re actually doing: • Show how to use a set of forge modules to build an infrastructure out. • using the mbarr/moderninfra as an opinionated profile module • download the necessary modules using librarian-puppet
  12. 12. We’ll: • Build a RabbitMQ server + sensu server • the admin host (has the mco client) • Build a logstash server • Build a Jenkins host
  13. 13. Each server will also: • be sending logs via logstash-forwarder • run Sensu client checks • run a mco server
  14. 14. Moderninfra module
  15. 15. A forge module just for you! • Sets up the basics of each service • Sets up the requirements correctly to all work together • Has… opinions.
  16. 16. Install from the forge: puppet module install mbarr-moderninfra
  17. 17. The code! ---! moderninfra::rmqserver: 'rabbitmq.aws.mbarr.net'! moderninfra::mco_password: 'shhhh..its.a.secret.'! moderninfra::sensu_password: 'whatsupdoc'! moderninfra::logstash_server: 'logstash.aws.mbarr.net' Hiera data, to make life easier: class moderninfra (! $rmqserver,! $logstash_server,! $rmq=false,! $mco_client=false,! $mco_server=false,! $sensu_client=false,! $sensu_server=false,! $logstash=false,! $logstash_forwarder=true,! $mco_password=undef,! $sensu_password=undef,! ) {...}
  18. 18. node default {! if $role == "mco" {! class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }! ! if $role == "puppet" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! } if $role == "logstash" {! class {'moderninfra':! logstash => true,! mco_server => true,! sensu_client => true,! }! include profiles::logstash! }! ! if $role == "jenkins" {! class {'moderninfra':! mco_server => true,! sensu_client => true,! }! include jenkins! }! } Site.pp
  19. 19. RabbitMQ, Sensu & Mcollective
  20. 20. RabbitMQ • This is the middle ware that is used by both mco & sensu. • Our module uses the Puppet SSL certs for connections • Adds a second cert for the host, via the puppet-certificate module.
  21. 21. Code class {'moderninfra':! rmq => true,! mco_client => true,! sensu_server => true,! }! include profiles::sensuchecks ! }
  22. 22. RMQ Note • To be fair: Sensu isn’t running w/ SSL certs • I’ve used other self signed certs before without issue • Looks like there’s a bug that hopefully is actually fixed in Erlang OTP 17.1
  23. 23. Mcollective • Using SSL to secure PSK connections between mco & RabbitMQ • Installs the package, service & puppet agents.
  24. 24. root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet ! * [ ============================================================> ] 4 / 4 ! puppet-us-east-1b-i-346b2a1f.ec2.mbarr.net: puppet-purged. rmq-us-east-1b-i-6a9bda41.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. logstash-us-east-1b-i-979adbbc.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. jenkins-us-east-1b-i-969adbbd.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1. ! Summary of Arch: ! No aggregate summary could be computed ! Summary of Ensure: ! 3.6.2-1puppetlabs1 = 3 purged = 1 ! ! Finished processing 4 / 4 hosts in 1172.09 ms
  25. 25. Sensu • Client on all 4 hosts • Server on RMQ box • Distributed checks • Dashboard on 8080 • profiles::sensuchecks installs various checks. (not in module)
  26. 26. Actually making sensu GO: (on server) class profiles::sensuchecks {! sensu::check { 'check_ntp':! command => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H pool.ntp.org -w 20 -c 40',! handlers => 'default',! subscribers => 'general',! standalone => false,! custom => { occurrences => 2 },! }! sensu::check { 'check_cron':! command => '/etc/sensu/plugins/check-procs.rb -p cron -C 1 -c 10 -w 10 ',! handlers => 'default',! subscribers => 'general',! interval => 60,! standalone => false,! custom => { occurrences => 2 },! }! }!
  27. 27. Logstash
  28. 28. • Centralized logging system • Inputs, Outputs, Filters • Inputs: syslog, files, redis.. • Outputs:elasticsearch, etc • Filters: Grok, many others
  29. 29. Logstash profile class profiles::logstash {! ! logstash::configfile { 'basic_config':! source => 'puppet:///modules/profiles/logstash/basic_config',! order => 10! }! ! include kibana3! ! }!
  30. 30. Logstash config input { lumberjack { port => 12345 ssl_certificate => "/etc/logstash/ssl/cert.pem" ssl_key => "/etc/logstash/ssl/key.pem" type => "lumberjack" } } ! input { tcp { port => 5000 type => syslog } udp { port => 5000 type => syslog } } ! output { elasticsearch { host => localhost } stdout { codec => rubydebug } }
  31. 31. Logstash-forwarder • Data is sent from logs on client to Logstash server via SSL • Keeps track of log positions and what’s been sent • Server listens on 12345, for now.
  32. 32. Elasticsearch & Kibana • This is what Kibana looks like with data from logstash fed into elasticsearch • (It’s zoomed a bit, so you can see the good parts.)
  33. 33. Jenkins
  34. 34. Jenkins • Continuous integration tool • There is code to set up slaves in the Jenkins module. • https://forge.puppetlabs.com/rtyler/jenkins
  35. 35. include jenkins
  36. 36. Things this module doesn’t do: • Build your puppet master • DNS names for Puppet master, RMQ, Logstash, etc • Although the cloud formation templates do!
  37. 37. But it might let you sleep at night…
  38. 38. Appendix:! Puppet Master
  39. 39. • Built w/ CloudFormations template • Sorry, not vagrant. Might be added soon. • uses cloud-init to provision puppet & code base • Uses puppet 3.6.2 • Librarian-puppet
  40. 40. Puppet Master • Set host name & domain • Install puppet • rm -rf /etc/puppet • git clone REPO /etc/puppet
  41. 41. Appendix: ! Librarian-puppet
  42. 42. Librarian Puppet • Lets you take a Puppetfile, and manage modules & dependencies • can use forge or git repos • Takes over your modules directory, though. • adds to .gitignore & regenerates the directory from the Puppetfile • I’ve used a pattern of a second directory (modules-local) to allow a slow migration & local files to stay in your existing repo
  43. 43. Modules-local pattern Old: modulepath = $confdir/modules:$confdir/modules-local ! 3.6+ directory environments: environment.conf modulepath = modules:modules-local
  44. 44. Puppetfile forge "https://forgeapi.puppetlabs.com" ! mod "reidmv/puppet_certificate" mod "elasticsearch/logstash" mod "elasticsearch/elasticsearch" mod "sensu/sensu" ! mod "rtyler/jenkins" ! mod "puppetlabs/mcollective" ! mod "thejandroman/kibana3", "0.0.3" ! # mod "mbarr/moderninfra", # :git => "git://github.com/matthewbarr/moderninfra.git" ! #mod "garethr/graphite"
  45. 45. modules ├── activemq ├── apache ├── apt ├── concat ├── datacat ├── elasticsearch ├── epel ├── erlang ├── file_concat ├── git ├── java ├── java_ks ├── jenkins ├── kibana3 ├── logstash ├── mcollective ├── puppet_certificate ├── rabbitmq ├── sensu ├── staging ├── stdlib ├── vcsrepo └── zypprepo modules-local ├── moderninfra └── profiles
  46. 46. We’re hiring! (in Boston)! ! ! Matthew Barr! @matthewbarr (github & twitter)! matthew.barr@here.com! mbarr@mbarr.net! http://github.com/matthewbarr/build-modern-infra
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×