Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Bastian Widmer / @dasrecht
Logging with Elasticsearch,
Logstash & Kibana
But why?
„Can you check the errors from yesterday between 15.02 and 15.07“
Visualization > Plaintext
Who are you?
Bastian Widmer
@dasrecht / bastianwidmer.ch
Switzerland
Development and Operations Engineer
Agenda 1 Introduction
2
3
4
5
Architecture
ELK Stack
Tools!
Demo
ELK Stack!ELK Stack!
ELK Stack!ELK Stack!
Elasticsearch
Logstash
Kibana
Elasticsearch
Elasticsearch
• Java
• Search and Index
• Distributed — Copies & Shards
• Clustering
• API — JSON / RESTful
• Apache Lucene
Elasticsearch
• Index

like a Database
• Replica

Copies for Fault Tolerance
• Shard

Lucene Instance which indexes
the Da...
Elasticsearch
Logstash
Logstash
• Multiple Input / Multiple Output
• Centralize Logs
• Collect
• Parse
• Store / Forward
Logstash
The life of an event
• Input
• Filters
• Output
• Codecs
Logstash
• JRuby*
• >1.4.0 - FlatJAR Release is gone
• Instead of running „java -jar logstash.jar“ — „bin/logstash“
• Cont...
Input
• File
• Syslog
• Redis
• logstash-forwarder (former Lumberjack)
Filters
• Grok
• Mutate
• Drop
• Clone
• GeoIP (!!!)
Outputs
• Elasticsearch
• File
• Graphite
• StatsD
Logstash
1 input {!
2 stdin { }!
3 }!
4 !
5 output {!
6 stdout {!
7 codec => rubydebug!
8 }!
9 }!
!
Logstash
1 vagrant@precise64$ ./logstash agent -f 1_simpleconfig.cfg!
2 very important log message!!
3 {!
4 "message" => "...
Logstash
1 input {!
2 stdin { }!
3 }!
4 output {!
5 elasticsearch{!
6 host => "127.0.0.1"!
7 }!
8 stdout {!
9 codec => rub...
Logstash
1 input {!
2 file {!
3 path => "/var/log/syslog"!
4 start_position => beginning!
5 }!
6 }!
7 !
8 output {!
9 stdo...
Logstash
Errno::EBADF: Bad file descriptor - Bad file descriptor
Kibana
Architecture
Architecture
Shipper
Shipper
Shipper Broker Indexer
Search
and
Storage
Architecture
Shipper
Shipper
Shipper Broker Indexer
Search
and
Storage
Syslog
Architecture
Shipper
Shipper
Shipper Broker Indexer
Search
and
Storage
Syslog Logstash
Architecture
Shipper
Shipper
Shipper Broker Indexer
Search
and
Storage
Syslog Logstash Elasticsearch
Architecture
the real deal!
Architecture
Shipper
Shipper
Shipper BrokerBroker Indexer
Search
and
Storage
Logstash Redis Logstash Elasticsearch
Tools!
(because anyone needs a bit help)
Elasticsearch Head
http://mobz.github.io/elasticsearch-head/
• Close	

• Remove	

• Backup	

• Restore
elasticsearch-index-mgmt
http://s.nrdy.ch/eee
But then…
Curator
• Time Series Indices? THIS IS THE TOOL!
• Close Indexes
• Delete (by space or time)
• Disable Bloom Filter
• Opti...
Curator
• Time Series Indexes? THIS IS THE TOOL!
• Close Indexes
• Remove Indexes
• Remove by Space Usage
• Disable Bloom ...
Curator
• Close indices older than 14 days, delete indices older than 30 days 



curator --host my-elasticsearch -d 30 -c...
Curator
1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10!
2 2014-04-21T17:57:19.419 INFO main:333 Job starting...!
...
BigDesk
bigdesk.org
Grok Debugger
grokdebug.herokuapp.com
Logstash Cookbook
cookbook.logstash.net
The Logstash Book
logstashbook.com
Logfiles
Logstash
Elasticsearch
Kibana
DEMO!DEMO!
Take Home
• Centralized Logging saves time
• Is fun with the ELK Stack
• Gives you Graphs to Interpret
• „can you check th...
Thank you for having me
here!
Slides : http://s.nrdy.ch/campus-logging
Images Used
• Elk : https://www.flickr.com/photos/ucumari/353839518/
• Paper Stash : https://www.flickr.com/photos/shehan3...
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Attack monitoring using ElasticSearch Logstash and Kibana
Next
Upcoming SlideShare
Attack monitoring using ElasticSearch Logstash and Kibana
Next
Download to read offline and view in fullscreen.

Share

Logging with Elasticsearch, Logstash & Kibana

Download to read offline

Related Books

Free with a 30 day trial from Scribd

See all

Logging with Elasticsearch, Logstash & Kibana

  1. 1. Bastian Widmer / @dasrecht Logging with Elasticsearch, Logstash & Kibana
  2. 2. But why?
  3. 3. „Can you check the errors from yesterday between 15.02 and 15.07“
  4. 4. Visualization > Plaintext
  5. 5. Who are you? Bastian Widmer @dasrecht / bastianwidmer.ch Switzerland Development and Operations Engineer
  6. 6. Agenda 1 Introduction 2 3 4 5 Architecture ELK Stack Tools! Demo
  7. 7. ELK Stack!ELK Stack!
  8. 8. ELK Stack!ELK Stack! Elasticsearch Logstash Kibana
  9. 9. Elasticsearch
  10. 10. Elasticsearch • Java • Search and Index • Distributed — Copies & Shards • Clustering • API — JSON / RESTful • Apache Lucene
  11. 11. Elasticsearch • Index
 like a Database • Replica
 Copies for Fault Tolerance • Shard
 Lucene Instance which indexes the Data
 see : http://blog.liip.ch/archive/2013/07/19/on-elasticsearch-performance.html
  12. 12. Elasticsearch
  13. 13. Logstash
  14. 14. Logstash • Multiple Input / Multiple Output • Centralize Logs • Collect • Parse • Store / Forward
  15. 15. Logstash
  16. 16. The life of an event • Input • Filters • Output • Codecs
  17. 17. Logstash • JRuby* • >1.4.0 - FlatJAR Release is gone • Instead of running „java -jar logstash.jar“ — „bin/logstash“ • Contrib Plugins • Daily Indices ! * see https://gist.github.com/jordansissel/978956
  18. 18. Input • File • Syslog • Redis • logstash-forwarder (former Lumberjack)
  19. 19. Filters • Grok • Mutate • Drop • Clone • GeoIP (!!!)
  20. 20. Outputs • Elasticsearch • File • Graphite • StatsD
  21. 21. Logstash 1 input {! 2 stdin { }! 3 }! 4 ! 5 output {! 6 stdout {! 7 codec => rubydebug! 8 }! 9 }! !
  22. 22. Logstash 1 vagrant@precise64$ ./logstash agent -f 1_simpleconfig.cfg! 2 very important log message!! 3 {! 4 "message" => "very important log message!",! 5 "@version" => "1",! 6 "@timestamp" => "2014-04-21T16:18:02.952Z",! 7 "host" => "precise64"! 8 }
  23. 23. Logstash 1 input {! 2 stdin { }! 3 }! 4 output {! 5 elasticsearch{! 6 host => "127.0.0.1"! 7 }! 8 stdout {! 9 codec => rubydebug! 10 }! 11 }
  24. 24. Logstash 1 input {! 2 file {! 3 path => "/var/log/syslog"! 4 start_position => beginning! 5 }! 6 }! 7 ! 8 output {! 9 stdout {! 10 codec => rubydebug! 11 }! 12 elasticsearch{! 13 host => "127.0.0.1"! 14 }! 15 }
  25. 25. Logstash Errno::EBADF: Bad file descriptor - Bad file descriptor
  26. 26. Kibana
  27. 27. Architecture
  28. 28. Architecture Shipper Shipper Shipper Broker Indexer Search and Storage
  29. 29. Architecture Shipper Shipper Shipper Broker Indexer Search and Storage Syslog
  30. 30. Architecture Shipper Shipper Shipper Broker Indexer Search and Storage Syslog Logstash
  31. 31. Architecture Shipper Shipper Shipper Broker Indexer Search and Storage Syslog Logstash Elasticsearch
  32. 32. Architecture the real deal!
  33. 33. Architecture Shipper Shipper Shipper BrokerBroker Indexer Search and Storage Logstash Redis Logstash Elasticsearch
  34. 34. Tools! (because anyone needs a bit help)
  35. 35. Elasticsearch Head http://mobz.github.io/elasticsearch-head/
  36. 36. • Close • Remove • Backup • Restore elasticsearch-index-mgmt http://s.nrdy.ch/eee
  37. 37. But then…
  38. 38. Curator • Time Series Indices? THIS IS THE TOOL! • Close Indexes • Delete (by space or time) • Disable Bloom Filter • Optimize / ForceMerge • https://github.com/elasticsearch/curator
  39. 39. Curator • Time Series Indexes? THIS IS THE TOOL! • Close Indexes • Remove Indexes • Remove by Space Usage • Disable Bloom Filter • https://github.com/elasticsearch/curator Curator Perfect for Time Series Indexes
  40. 40. Curator • Close indices older than 14 days, delete indices older than 30 days 
 
 curator --host my-elasticsearch -d 30 -c 14 • Disable bloom filter for indices older than 2 days, close indices older than 14 days, delete indices older than 30 days:
 
 curator --host my-elasticsearch -b 2 -c 14 -d 30
  41. 41. Curator 1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10! 2 2014-04-21T17:57:19.419 INFO main:333 Job starting...! 3 2014-04-21T17:57:19.420 INFO _new_conn:180 Starting new HTTP connection (1): localhost! 4 2014-04-21T17:57:19.422 INFO log_request_success:49 GET http://localhost:9200/ [status:200 request:0.002s]! 5 2014-04-21T17:57:19.423 INFO main:359 Deleting indices older than 10 days...! 6 2014-04-21T17:57:19.430 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings? expand_wildcards=closed [status:200 request:0.007s]! 7 2014-04-21T17:57:19.433 INFO find_expired_indices:209 logstash-2014.04.21 is 10 days, 0:00:00 above the cutoff.! 8 2014-04-21T17:57:19.433 INFO index_loop:309 DELETE index operations completed.! 9 2014-04-21T17:57:19.433 INFO main:364 Closing indices older than 7 days...! 10 2014-04-21T17:57:19.434 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings? expand_wildcards=closed [status:200 request:0.001s]! 11 2014-04-21T17:57:19.435 INFO find_expired_indices:209 logstash-2014.04.21 is 7 days, 0:00:00 above the cutoff.! 12 2014-04-21T17:57:19.435 INFO index_loop:309 CLOSE index operations completed.! 13 2014-04-21T17:57:19.435 INFO main:369 Disabling bloom filter on indices older than 2 days...! 14 2014-04-21T17:57:19.437 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings? expand_wildcards=closed [status:200 request:0.002s]! 15 2014-04-21T17:57:19.438 INFO find_expired_indices:209 logstash-2014.04.21 is 2 days, 0:00:00 above the cutoff.! 16 2014-04-21T17:57:19.438 INFO index_loop:309 DISABLE BLOOM FILTER FOR index operations completed.! 17 2014-04-21T17:57:19.438 INFO main:379 Done in 0:00:00.020348.!
  42. 42. BigDesk bigdesk.org
  43. 43. Grok Debugger grokdebug.herokuapp.com
  44. 44. Logstash Cookbook cookbook.logstash.net
  45. 45. The Logstash Book logstashbook.com
  46. 46. Logfiles Logstash Elasticsearch Kibana DEMO!DEMO!
  47. 47. Take Home • Centralized Logging saves time • Is fun with the ELK Stack • Gives you Graphs to Interpret • „can you check the errors from yesterday between 15.02 and 15.07“ get’s A LOT easier • Start here tomorrow: http://logstash.net/docs/ 1.4.0/tutorials/getting-started-with-logstash
  48. 48. Thank you for having me here! Slides : http://s.nrdy.ch/campus-logging
  49. 49. Images Used • Elk : https://www.flickr.com/photos/ucumari/353839518/ • Paper Stash : https://www.flickr.com/photos/shehan365/8394630603/ • Architecture : https://www.flickr.com/photos/dasrecht/6743411525/
  • ekingg

    Dec. 18, 2020
  • JupiterTsou

    Oct. 3, 2018
  • RicoChen4

    Sep. 17, 2018
  • eayoub

    Aug. 28, 2018
  • PeiLingHung1

    Aug. 26, 2018
  • ramiljoaquin1

    Jul. 22, 2018
  • ZhengZeng8

    Jul. 1, 2018
  • HajimeYoshida2

    Mar. 6, 2018
  • lequocduan

    Feb. 23, 2018
  • MadeleineLee

    Feb. 14, 2018
  • lantince

    Jan. 19, 2018
  • ivywjhua

    Nov. 8, 2017
  • ZaleChiou

    Jun. 23, 2017
  • lljokell

    Jun. 21, 2017
  • NaveenKumar719

    May. 11, 2017
  • hawking8987

    May. 9, 2017
  • ShaminderSingh5

    Apr. 27, 2017
  • liupingyi

    Apr. 17, 2017
  • mknezic

    Mar. 10, 2017
  • garyalley12

    Feb. 25, 2017

Views

Total views

51,742

On Slideshare

0

From embeds

0

Number of embeds

9,441

Actions

Downloads

1,840

Shares

0

Comments

0

Likes

91

×