This document provides instructions for analyzing logs with the ELK stack using Docker. It discusses importing log data, using filters like date and grok patterns to parse logs, deduplicating entries, enriching data with plugins like useragent and geoip, and generating graphs from the analyzed data. The Docker ELK repository on GitHub contains a docker-compose file to quickly get started with log analysis using Docker.

1. Analysing your logs with ELK
stack & Docker

2. Intro
2

3. Do it yourself

4. Dockerhub elk
Docker hub

5. https://github.com/deviantony/docker-elk
dockercompose
Docker ELK repo

6. Importing data is as simple as
Getting started
$ nc localhost 5000 < /path/to/logfile.log

7. Wrong date
However ..

8. filter {
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
Filters

9. Rerun

10. Enter grok
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}

11. Grok patterns
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

12. Own Grok patterns
Directory: Patterns
filter {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}"
}
}
}
contents of ./patterns/postfix:
POSTFIX_QUEUEID [0-9A-F]{10,11}

13. Duplicates
fingerprint {
source => ["message"]
concatenate_sources => true
method => "SHA1"
target => "fingerprint"
key => "17272737"
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
document_id => "%{fingerprint}"
}
}

14. Agents
if [agent] != "-" and [agent] != "" {
useragent {
add_tag => [ "UA" ]
source => "agent"
}
}
if "UA" in [tags] {
if [device] == "Other" { mutate { remove_field => "device" } }
if [name] == "Other" { mutate { remove_field => "name" } }
if [os] == "Other" { mutate { remove_field => "os" } }
}

15. Geo ip
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}

16. Graphs

17. Questions?