Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Anaysing your logs with docker and elk

119 views

Published on

Anaysing your logs with docker and elk

Published in: Internet
  • Be the first to comment

Anaysing your logs with docker and elk

  1. 1. Analysing your logs with ELK stack & Docker
  2. 2. Intro 2
  3. 3. Do it yourself
  4. 4. Dockerhub elk Docker hub
  5. 5. https://github.com/deviantony/docker-elk dockercompose Docker ELK repo
  6. 6. Importing data is as simple as Getting started $ nc localhost 5000 < /path/to/logfile.log
  7. 7. Wrong date However ..
  8. 8. filter { date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } } Filters
  9. 9. Rerun
  10. 10. Enter grok grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  11. 11. Grok patterns https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
  12. 12. Own Grok patterns Directory: Patterns filter { grok { patterns_dir => ["./patterns"] match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}" } } } contents of ./patterns/postfix: POSTFIX_QUEUEID [0-9A-F]{10,11}
  13. 13. Duplicates fingerprint { source => ["message"] concatenate_sources => true method => "SHA1" target => "fingerprint" key => "17272737" } output { elasticsearch { hosts => "elasticsearch:9200" document_id => "%{fingerprint}" } }
  14. 14. Agents if [agent] != "-" and [agent] != "" { useragent { add_tag => [ "UA" ] source => "agent" } } if "UA" in [tags] { if [device] == "Other" { mutate { remove_field => "device" } } if [name] == "Other" { mutate { remove_field => "name" } } if [os] == "Other" { mutate { remove_field => "os" } } }
  15. 15. Geo ip geoip { source => "clientip" target => "geoip" database => "/etc/logstash/GeoLiteCity.mmdb" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] }
  16. 16. Graphs
  17. 17. Questions?

×