;
The devops approach to monitoring,The devops approach to monitoring,
Open Source and IAC StyleOpen Source and IAC Style
...
;
whoamiwhoami
• sysadmin @ inuitssysadmin @ inuits
• open-source defender for 7+ yearsopen-source defender for 7+ years
•...
;
DevOpsDevOps
• CultureCulture
• AutomationAutomation
• MeasurementMeasurement
• SharingSharing
Damon Edwards and John Wi...
;
Monitoring is usuallyMonitoring is usually
an afterthoughtan afterthought
ENOTIME, ENOBUDGETENOTIME, ENOBUDGET
Julien Pi...
;
#monitoringsucks#monitoringsucks
• A movement started in 2011A movement started in 2011
• http://github.com/monitoringsu...
;
GoalsGoals
• Find when a service is unavailableFind when a service is unavailable
• Understand failure post-mortemUnders...
;
GoalsGoals
• Find when a service is unavailableFind when a service is unavailable
• Understand failure post-mortemUnders...
;
Monitor everythingMonitor everything
• ServersServers
• ServicesServices
• UsageUsage
• HardwareHardware
• SoftwareSoftw...
;
Monitor every environmentMonitor every environment
• See performance changes in devSee performance changes in dev
• Fix ...
;
MetricMetric
• Time + name + value = metricTime + name + value = metric
• Can be anythingCan be anything
Julien Pivotto ...
;
EventEvent
• Time + fields = metricTime + fields = metric
• Logs become usable dataLogs become usable data
• Can be transf...
;
Metrics + eventsMetrics + events
• Overview of your infrastructureOverview of your infrastructure
• Usage and state of t...
;
AutomationAutomation
• Infrastructure as CodeInfrastructure as Code
• Automate everythingAutomate everything
⇒ One sourc...
;
DeploymentDeployment
• Definitions of a service includes monitoringDefinitions of a service includes monitoring
• Deployed...
;
ToolsTools
• No all-in-one toolNo all-in-one tool
• No autodiscovery toolNo autodiscovery tool
• Text-based configuration...
;
IcingaIcinga
• Fork of nagiosFork of nagios
• Large and vibrant communityLarge and vibrant community
• Configuration comp...
;
IcingaIcinga
https://icinga.org
Julien Pivotto DevOps and monitoring
;
SensuSensu
• FlexibilityFlexibility
• Compatible with nagios pluginsCompatible with nagios plugins
• Connects to your so...
;
CollectdCollectd
• Statistics collection daemonStatistics collection daemon
• A lot of plugins available. . .A lot of pl...
;
Collectd pluginsCollectd plugins
http://www.flickr.com/photos/juhansonin/3141561416/
Julien Pivotto DevOps and monitoring
;
Collectd pluginsCollectd plugins
AMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_S...
;
LogstashLogstash
• Ship logs from any sourceShip logs from any source
• Filter themFilter them
• Index themIndex them
• ...
;
LogstashLogstash
Julien Pivotto DevOps and monitoring
;
GraphiteGraphite
• GraphingGraphing
• Accept any metricAccept any metric
• Store data in files (whisper)Store data in file...
;
Send data to graphiteSend data to graphite
echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "...
;
Graphite APIGraphite API
Julien Pivotto DevOps and monitoring
;
StatsdStatsd
• Graphite friendGraphite friend
• Stats aggregationStats aggregation
• Simple countersSimple counters
• Fl...
;
Feeding statsdFeeding statsd
echo "stats.sshd.login:1|c" |
nc -u statsd.example.com 8125
Julien Pivotto DevOps and monit...
;
gdashgdash
https://github.com/ripienaar/gdash
Julien Pivotto DevOps and monitoring
;
giraffegiraffe
Alternative to gdash
Julien Pivotto DevOps and monitoring
;
KibanaKibana
• Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES
• Kibana 1 was written...
;
Kibana 3Kibana 3
• Everything happens in the browserEverything happens in the browser
• The browser is connected to Elas...
;
Kibana queriesKibana queries
Example of a kibana query
@fields.syslog_program:"httpd" AND
@fields.http_host:"test.exampl...
;Julien Pivotto DevOps and monitoring
;Julien Pivotto DevOps and monitoring
;Julien Pivotto DevOps and monitoring
;
Toolchain exampleToolchain example
• Apache ships logs to rsyslogApache ships logs to rsyslog
• Rsyslog ships logs to lo...
;
Reusing Icinga/Nagios perfdataReusing Icinga/Nagios perfdata
• Icinga performs various checksIcinga performs various che...
;
SharingSharing
• Build dashboard: dashing, teamdashBuild dashboard: dashing, teamdash
• Share with developersShare with ...
;
Try them yourselfTry them yourself
https://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagra...
;
Thank youThank you
Any question?Any question?
Julien Pivotto DevOps and monitoring
;
ContactContact
Julien PivottoJulien Pivotto
julien@inuits.eujulien@inuits.eu
@roidelapluie@roidelapluie
INUITS bvbaINUIT...
Upcoming SlideShare
Loading in …5
×

The devops approach to monitoring, Open Source and Infrastructure as Code Style

12,865 views

Published on

Monitoring is critical for every decent application that runs on production. Many of the monitoring tools widely used show their limits at the age of Infrastructure as Code and Cloud computing. Let's investigate how monitoring can face the new challenges: scalability, reproducability and automation

Published in: Technology, Design

The devops approach to monitoring, Open Source and Infrastructure as Code Style

  1. 1. ; The devops approach to monitoring,The devops approach to monitoring, Open Source and IAC StyleOpen Source and IAC Style Julien PivottoJulien Pivotto Open World ForumOpen World Forum October 4, 2013October 4, 2013
  2. 2. ; whoamiwhoami • sysadmin @ inuitssysadmin @ inuits • open-source defender for 7+ yearsopen-source defender for 7+ years • devops believerdevops believer • @roidelapluie on twitter/github@roidelapluie on twitter/github Julien Pivotto DevOps and monitoring
  3. 3. ; DevOpsDevOps • CultureCulture • AutomationAutomation • MeasurementMeasurement • SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Julien Pivotto DevOps and monitoring
  4. 4. ; Monitoring is usuallyMonitoring is usually an afterthoughtan afterthought ENOTIME, ENOBUDGETENOTIME, ENOBUDGET Julien Pivotto DevOps and monitoring
  5. 5. ; #monitoringsucks#monitoringsucks • A movement started in 2011A movement started in 2011 • http://github.com/monitoringsuckshttp://github.com/monitoringsucks • A lot of tools and informationA lot of tools and information Julien Pivotto DevOps and monitoring
  6. 6. ; GoalsGoals • Find when a service is unavailableFind when a service is unavailable • Understand failure post-mortemUnderstand failure post-mortem Julien Pivotto DevOps and monitoring
  7. 7. ; GoalsGoals • Find when a service is unavailableFind when a service is unavailable • Understand failure post-mortemUnderstand failure post-mortem • Learn from your infrastructureLearn from your infrastructure • AnticipateAnticipate Julien Pivotto DevOps and monitoring
  8. 8. ; Monitor everythingMonitor everything • ServersServers • ServicesServices • UsageUsage • HardwareHardware • SoftwareSoftware • PeoplePeople Julien Pivotto DevOps and monitoring
  9. 9. ; Monitor every environmentMonitor every environment • See performance changes in devSee performance changes in dev • Fix them before it hits productionFix them before it hits production Julien Pivotto DevOps and monitoring
  10. 10. ; MetricMetric • Time + name + value = metricTime + name + value = metric • Can be anythingCan be anything Julien Pivotto DevOps and monitoring
  11. 11. ; EventEvent • Time + fields = metricTime + fields = metric • Logs become usable dataLogs become usable data • Can be transformed into metricsCan be transformed into metrics Julien Pivotto DevOps and monitoring
  12. 12. ; Metrics + eventsMetrics + events • Overview of your infrastructureOverview of your infrastructure • Usage and state of the servicesUsage and state of the services • Combine several metricsCombine several metrics • Extract business valuesExtract business values Julien Pivotto DevOps and monitoring
  13. 13. ; AutomationAutomation • Infrastructure as CodeInfrastructure as Code • Automate everythingAutomate everything ⇒ One source of truthOne source of truth Julien Pivotto DevOps and monitoring
  14. 14. ; DeploymentDeployment • Definitions of a service includes monitoringDefinitions of a service includes monitoring • Deployed ⇔ monitoredDeployed ⇔ monitored Julien Pivotto DevOps and monitoring
  15. 15. ; ToolsTools • No all-in-one toolNo all-in-one tool • No autodiscovery toolNo autodiscovery tool • Text-based configurationText-based configuration • ScalableScalable ⇒ The Unix philosphyThe Unix philosphy Julien Pivotto DevOps and monitoring
  16. 16. ; IcingaIcinga • Fork of nagiosFork of nagios • Large and vibrant communityLarge and vibrant community • Configuration compatible with nagiosConfiguration compatible with nagios • User-friendly interfaceUser-friendly interface • Use Icinga Classic!Use Icinga Classic! Julien Pivotto DevOps and monitoring
  17. 17. ; IcingaIcinga https://icinga.org Julien Pivotto DevOps and monitoring
  18. 18. ; SensuSensu • FlexibilityFlexibility • Compatible with nagios pluginsCompatible with nagios plugins • Connects to your source of trustConnects to your source of trust • Relies on RabbitMQRelies on RabbitMQ Julien Pivotto DevOps and monitoring
  19. 19. ; CollectdCollectd • Statistics collection daemonStatistics collection daemon • A lot of plugins available. . .A lot of plugins available. . . • Can send data to graphiteCan send data to graphite • Simple configurationSimple configuration Julien Pivotto DevOps and monitoring
  20. 20. ; Collectd pluginsCollectd plugins http://www.flickr.com/photos/juhansonin/3141561416/ Julien Pivotto DevOps and monitoring
  21. 21. ; Collectd pluginsCollectd plugins AMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND CarbonAMQP Apache APC_UPS Apple_Sensors Ascent Battery BIND Carbon ConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XMLConnTrack ContextSwitch CPU CPUFreq CSV cURL cURL-JSON cURL-XML DBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMXDBI DF Disk DNS E-Mail Entropy Exec FileCount FSCache GenericJMX gmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Loadgmond HDDTemp Interface IPMI IPTables IPVS IRQ Java libvirt Load LogFile LPAR MadWifi MBMon memcachec memcached Memory ModbusLogFile LPAR MadWifi MBMon memcachec memcached Memory Modbus Monitorus Multimeter MySQL NetApp Netlink Network NFS nginxMonitorus Multimeter MySQL NetApp Netlink Network NFS nginx Notify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZNotify_Desktop Notify_Email NTPd NUT olsrd OneWire OpenVPN OpenVZ Oracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols PythonOracle Perl Pinba Ping PostgreSQL PowerDNS Processes Protocols Python Redis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLogRedis RouterOS RRDCacheD RRDtool Sensors Serial SNMP Swap SysLog Table Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSockTable Tail Tape TCPConns TeamSpeak2 TED thermal TokyoTyrant UnixSock Uptime Users UUID Varnish vmem VServer Wireless XMMSUptime Users UUID Varnish vmem VServer Wireless XMMS Write_Graphite Write_HTTP Write_MongoDBWrite_Graphite Write_HTTP Write_MongoDB Write_Redis Write_Riemann ZFS_ARCWrite_Redis Write_Riemann ZFS_ARC Julien Pivotto DevOps and monitoring
  22. 22. ; LogstashLogstash • Ship logs from any sourceShip logs from any source • Filter themFilter them • Index themIndex them • Search themSearch them • Backed with elasticsearchBacked with elasticsearch Julien Pivotto DevOps and monitoring
  23. 23. ; LogstashLogstash Julien Pivotto DevOps and monitoring
  24. 24. ; GraphiteGraphite • GraphingGraphing • Accept any metricAccept any metric • Store data in files (whisper)Store data in files (whisper) • A lot of helpers functionsA lot of helpers functions • Listen on UDP and TCPListen on UDP and TCP Julien Pivotto DevOps and monitoring
  25. 25. ; Send data to graphiteSend data to graphite echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003echo "stats.sshd.login 1 $(date +%s)" | nc -u graphite.example.com 2003 Julien Pivotto DevOps and monitoring
  26. 26. ; Graphite APIGraphite API Julien Pivotto DevOps and monitoring
  27. 27. ; StatsdStatsd • Graphite friendGraphite friend • Stats aggregationStats aggregation • Simple countersSimple counters • Flushes every XX seconds to graphiteFlushes every XX seconds to graphite • UDPUDP Julien Pivotto DevOps and monitoring
  28. 28. ; Feeding statsdFeeding statsd echo "stats.sshd.login:1|c" | nc -u statsd.example.com 8125 Julien Pivotto DevOps and monitoring
  29. 29. ; gdashgdash https://github.com/ripienaar/gdash Julien Pivotto DevOps and monitoring
  30. 30. ; giraffegiraffe Alternative to gdash Julien Pivotto DevOps and monitoring
  31. 31. ; KibanaKibana • Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES • Kibana 1 was written in PHPKibana 1 was written in PHP • Kibana 2 was written in RubyKibana 2 was written in Ruby • Kibana 3 is written in AngularJSKibana 3 is written in AngularJS Julien Pivotto DevOps and monitoring
  32. 32. ; Kibana 3Kibana 3 • Everything happens in the browserEverything happens in the browser • The browser is connected to ElasticsearchThe browser is connected to Elasticsearch • You can save dashboards into ESYou can save dashboards into ES • You can write/template dashboards to filesYou can write/template dashboards to files Julien Pivotto DevOps and monitoring
  33. 33. ; Kibana queriesKibana queries Example of a kibana query @fields.syslog_program:"httpd" AND @fields.http_host:"test.example.com" AND @fields.response:"404" • LLucene query syntax • SSimple and effective • PPoint & click web interface Julien Pivotto DevOps and monitoring
  34. 34. ;Julien Pivotto DevOps and monitoring
  35. 35. ;Julien Pivotto DevOps and monitoring
  36. 36. ;Julien Pivotto DevOps and monitoring
  37. 37. ; Toolchain exampleToolchain example • Apache ships logs to rsyslogApache ships logs to rsyslog • Rsyslog ships logs to logstashRsyslog ships logs to logstash • Logstash ships metrics to statsdLogstash ships metrics to statsd • Statsd ships metrics to GraphiteStatsd ships metrics to Graphite • Icinga query metric from graphiteIcinga query metric from graphite • https://github.com/etsy/nagios_toolshttps://github.com/etsy/nagios_tools Julien Pivotto DevOps and monitoring
  38. 38. ; Reusing Icinga/Nagios perfdataReusing Icinga/Nagios perfdata • Icinga performs various checksIcinga performs various checks • Icinga sends perfdata to graphiteIcinga sends perfdata to graphite • Graphite stores the dataGraphite stores the data • Gdash serves them inside dashboardsGdash serves them inside dashboards • https://github.com/roidelapluie/icinga-to-graphitehttps://github.com/roidelapluie/icinga-to-graphite Julien Pivotto DevOps and monitoring
  39. 39. ; SharingSharing • Build dashboard: dashing, teamdashBuild dashboard: dashing, teamdash • Share with developersShare with developers • Share with managersShare with managers Julien Pivotto DevOps and monitoring
  40. 40. ; Try them yourselfTry them yourself https://github.com/KrisBuytaert/vagrant-graphitehttps://github.com/KrisBuytaert/vagrant-graphite Julien Pivotto DevOps and monitoring
  41. 41. ; Thank youThank you Any question?Any question? Julien Pivotto DevOps and monitoring
  42. 42. ; ContactContact Julien PivottoJulien Pivotto julien@inuits.eujulien@inuits.eu @roidelapluie@roidelapluie INUITS bvbaINUITS bvba Duboisstraat 50Duboisstraat 50 2060 Antwerp2060 Antwerp BelgiumBelgium +32 473 441 636+32 473 441 636 https://inuits.euhttps://inuits.eu Julien Pivotto DevOps and monitoring

×