Monitoring with Elasticsearch, short talk for the ElasticSearch Belux Meetup Pre Fosdem

1. Monitoring with the ELKMonitoring with the ELK
StackStack
Kris Buytaert
@krisbuytaert

2. KrisKris BuytaertBuytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays,
#loadays, #cfgmgmtcamp ...#loadays, #cfgmgmtcamp ...
● Evangelizing devopsEvangelizing devops

3. #devops=~C(L)AMS#devops=~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● Monitoring and MeasurementMonitoring and Measurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

4. #monitoringsucks#monitoringsucks
● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011
● A sub #devops movementA sub #devops movement
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/

5. #monitoringlove#monitoringlove
•
•
Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011
•
A new era of toolingA new era of tooling
•
#monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits
•
#monitorama#monitorama

6. What we wantWhat we want
● Small , well suited componentsSmall , well suited components
•
CollectCollect
•
Transport / MangleTransport / Mangle
•
StoreStore
•
AnalyseAnalyse
•
Act / AlertAct / Alert
•
VisualizeVisualize
•

8. What do YOU do withWhat do YOU do with
your logfiles ?your logfiles ?

9. Logs & CollectionLogs & Collection
● Syslog,Syslog,
● RsyslogRsyslog
● Syslog-ngSyslog-ng
● Log4jLog4j
● Graylog2Graylog2
● ELSA (Enterprise LogELSA (Enterprise Log
Search and Archive)Search and Archive)
● ELK StackELK Stack

10. LogstashLogstash
● Not your average centralized logging toolNot your average centralized logging tool
● Elasticsearch backedElasticsearch backed
● ShipperShipper
● IndexerIndexer
● WebWeb

11. ● Collect fromCollect from
anywhereanywhere
● FilterFilter
● Send anywhereSend anywhere
● QueingQueing

12. Collect logsCollect logs
● rsyslog -> rsyslog inputrsyslog -> rsyslog input
● dedicated shippers (logstash-shipper,dedicated shippers (logstash-shipper,
lumberjack)lumberjack)
•
Direct or via queueDirect or via queue
● Write logs -> json format -> redis -> json inputWrite logs -> json format -> redis -> json input
● * -> your favourite logstash input* -> your favourite logstash input
●

13. FiltersFilters
● GrokGrok
● MutateMutate mutate {mutate {
# Lowercase some values that are always in uppercase# Lowercase some values that are always in uppercase
lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
}}
● dropdrop
● TagsTags

14. GrokGrok
grok {grok {
match => [match => [
"message", "%{IPORHOST:host} - %{USER:remote_user} [%"message", "%{IPORHOST:host} - %{USER:remote_user} [%
{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] %
{QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} %{QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} %
{QS:http_user_agent}"{QS:http_user_agent}"
]]
add_field => ["grok_type", "nginx-access"]add_field => ["grok_type", "nginx-access"]
add_tag => ["grokked"]add_tag => ["grokked"]
}}
}}

15. OutputsOutputs
● Mostly ElasticSearchMostly ElasticSearch
● Plenty of outputsPlenty of outputs
•
EmailEmail
•
NagiosNagios
•
RiemannRiemann
•
StatsdStatsd
•
GraphiteGraphite

19. Same tool used by devs to debugSame tool used by devs to debug
as by ops to debugas by ops to debug

20. Long Term MetricsLong Term Metrics
● Disk space is cheapDisk space is cheap
● But some people don't care about those logsBut some people don't care about those logs
after X weeks / months / yearsafter X weeks / months / years
● Send statistics via statsd to graphite ,Send statistics via statsd to graphite ,
● Keep graphite data for long term storage,Keep graphite data for long term storage,
● Purge elasticsearch contentPurge elasticsearch content
(curator is incomplete, working on patches)(curator is incomplete, working on patches)

21. GraphiteGraphite
● Graphing at ScaleGraphing at Scale
● Graphing at EaseGraphing at Ease
● Any metric is a graphAny metric is a graph
● echo "somestring $somevalue $timestamp" |echo "somestring $somevalue $timestamp" |
nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003

23. Friends of GraphiteFriends of Graphite
● CCollection :ollection :
●
Statsd,Statsd,
●
Collectd + CarbonpluginCollectd + Carbonplugin
●
JmxtransJmxtrans
●
LogsterLogster
● DashboardsDashboards
●
TattleTattle
●
GdashGdash
●
GrafanaGrafana

24. Alerting on Events/LogsAlerting on Events/Logs
● Logstash -> icingaLogstash -> icinga
output {output {
if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc-if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc-
app/ and [program] =~ /^edc-/ {app/ and [program] =~ /^edc-/ {
nagios_nsca {nagios_nsca {
host => "10.0.64.28"host => "10.0.64.28"
nagios_status => "1"nagios_status => "1"
nagios_host => "%{logsource}"nagios_host => "%{logsource}"
nagios_service => "Log check - %{program}"nagios_service => "Log check - %{program}"
}}
}}

25. Buy this guy
a beer

26. Remember, it's notRemember, it's not
about the toolsabout the tools

27. ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Duboistraat 50Duboistraat 50
2060 Antwerpen2060 Antwerpen
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221