How to use Logstash-Elasticsearch-Kibana (ELK) for analyse your server logs

1. Logstash-Elasticsearch-Kibana
How to manage logs
E. Witthauer
November 28, 2015

2. Why logging
• Debugging

3. Why logging
• Debugging
• Metrics

4. Why logging
• Debugging
• Metrics
• Monitoring

5. Old style
• Tail: ssh example.org > tail -f /var/log/some.log

6. Old style
• Tail: ssh example.org > tail -f /var/log/some.log
• Tools for multiple les: like multitail

7. Old style
• Tail: ssh example.org tail -f /var/log/some.log
• Tools for multiple les: like multitail
• Run command synchron in multiple ssh sessions

8. Old style
• Tail: ssh example.org tail -f /var/log/some.log
• Tools for multiple les: like multitail
• Run command synchron in multiple ssh sessions
But for more than one le/server or autmatic statistics:

9. Old style
• Tail: ssh example.org tail -f /var/log/some.log
• Tools for multiple les: like multitail
• Run command synchron in multiple ssh sessions
But for more than one le/server or autmatic statistics:

10. Better style
Better all in one place with option to later analysis

11. The ELK-Stack
E lasticsearch - Searchserver for indexing
the data (NoSQL-DB)

12. The ELK-Stack
E lasticsearch - Searchserver for indexing
the data (NoSQL-DB)
L ogstash - Log data processor for
transform and lter the data

13. The ELK-Stack
E lasticsearch - Searchserver for indexing
the data (NoSQL-DB)
L ogstash - Log data processor for
transform and lter the data
K ibana - WebUI for data visualisation and
analysis (node.js based)

14. The infrastructure
1. Read the logs and put them into a Redis-DB
2. Read from Redis-DB, lter and put into Elasticsearch

15. The infrastructure
Why 2 steps?
• Logs will be read even if Elasticsearch is not active

16. The infrastructure
Why 2 steps?
• Logs will be read even if Elasticsearch is not active
• Monitor Redis to see how many events are there (e.g. per
second)

17. The infrastructure
Why 2 steps?
• Logs will be read even if Elasticsearch is not active
• Monitor Redis to see how many events are there (e.g. per
second)
• Check the event format if we have some index problems (e.g.
wrong eld value or tag)

18. Setup
Logstash
• Install Java (1.9)

19. Setup
Logstash
• Install Java (1.9)
• Download Logstash from
https://www.elastic.co/downloads/logstash

20. Setup
Logstash
• Install Java (1.9)
• Download Logstash from
https://www.elastic.co/downloads/logstash
• Extract the zip le

21. Setup
Logstash
• Install Java (1.9)
• Download Logstash from
https://www.elastic.co/downloads/logstash
• Extract the zip le
• Run it: bin/logstash -f logstash.conf (see cong le below)

22. Setup
Logstash
• Install Java (1.9)
• Download Logstash from
https://www.elastic.co/downloads/logstash
• Extract the zip le
• Run it: bin/logstash -f logstash.conf (see cong le below)
• Or install the deb package and run it

23. Setup
Redis
• Install Redis from your repository system

24. Setup
Elasticsearch
• Install Java (1.9) if not done yet

25. Setup
Elasticsearch
• Install Java (1.9) if not done yet
• Download Elasticsearch from
https://www.elastic.co/downloads/elasticsearch

26. Setup
Elasticsearch
• Install Java (1.9) if not done yet
• Download Elasticsearch from
https://www.elastic.co/downloads/elasticsearch
• Extract the zip le

27. Setup
Elasticsearch
• Install Java (1.9) if not done yet
• Download Elasticsearch from
https://www.elastic.co/downloads/elasticsearch
• Extract the zip le
• Run it: bin/elasticsearch

28. Setup
Elasticsearch
• Install Java (1.9) if not done yet
• Download Elasticsearch from
https://www.elastic.co/downloads/elasticsearch
• Extract the zip le
• Run it: bin/elasticsearch
• Or install the deb package and run it

29. Setup
Kibana
• Install Java (1.9) if not done yet

30. Setup
Kibana
• Install Java (1.9) if not done yet
• Download Kibana from
https://www.elastic.co/downloads/kibana

31. Setup
Kibana
• Install Java (1.9) if not done yet
• Download Kibana from
https://www.elastic.co/downloads/kibana
• Extract the zip le

32. Setup
Kibana
• Install Java (1.9) if not done yet
• Download Kibana from
https://www.elastic.co/downloads/kibana
• Extract the zip le
• Open cong/kibana.yml in editor
• Set the elasticsearch.url to point at your Elasticsearch instance
(e.g. loclhost or 1270.0.1)
• Run it: bin/kibana
• Open url http://yourhost.com:5601

33. Cong
Shipper
For the Shipper we create a cong le:
1 input {
2 f i l e {
3 path = / var / log / apache2 /∗ access ∗. log
4 s t a r t _ p o s i t i o n = beginning
5 type = apache
6 sincedb_path = / opt /. sincedb_apache_access
7 }
8 }
9 output {
10 r e d i s {
11 host = 1 2 7 . 0 . 0 . 1
12 data_type = l i s t
13 key = l o g s t a s h
14 }
15 }

34. Cong
Shipper explained
input {...} Conguration for our input
le {...} Species a le input (all apache access log les)
path Path to our log les (regex)
start_position We start reading the le from the beginning
type adds a eld type with value apache to the output
sincedb_path Path to the internal database that sores the last
reading position in this le(s)
output {...} Conguration for our ouput
redis {...} Conguration for redis output
host Redis host address
data_type Specied that we store the events as a list in redis
key Name of our redis list

35. Cong
Indexer
For the Shipper we create a cong le:
1 input {
2 r e d i s {
3 host = 1 2 7 . 0 . 0 . 1
4 type = r e d i s −input
5 data_type = l i s t
6 key = l o g s t a s h
7 }
8 }
9 f i l t e r {
10 i f [ path ] =~ access { ANALYSE APACHE ACCESS }
11 e l s e i f [ path ] =~ e r r o r { ANALYSE APACHE ERROR }
12 e l s e i f [ type ] == s y s l o g { ANALYSE SYSLOG }
13 e l s e i f [ type ] == auth { ANALYSE AUTH LOG }
14 }
15 output {
16 e l a s t i c s e a r c h { }
17 }

36. Cong
Indexer explained
input {...} Conguration for our input
redis {...} Conguration for redis input
host Redis host address
type adds a eld type with value redis-list to the
output
data_type Specied that we store the events as a list in redis
key Name of our redis list)
lter {...} Our lter for the dierent events (syslog, apache
error, apache access, auth)
if path|type Separate lter congurations for our events (see later)
output {...} Conguration for elasticsearch output
elasticsearch{ } Default conguration for elasticsearch (localhost,
no further conguration needed)

37. Cong - Indexer
Apache Access Filter
The Apache Access Filter:
1 mutate {
2 r e p l a c e = { type = apache_access }
3 remove_tag = [ _ g r o k p a r s e f a i l u r e ]
4 remove_field = [ tags , tag , path ]
5 }
6 grok {
7 patterns_dir = / opt / grok_patterns
8 match = { message = %{VHOSTCOMBINEDAPACHELOG} }
9 }
10 date {
11 match = [ timestamp , dd/MMM/ yyyy :HH:mm: ss Z ]
12 }
13 geoip {
14 source = c l i e n t i p
15 }
16 useragent {
17 source = agent
18 }

38. Cong - Indexer
Apache Access Filter
mutate {...} Change eld values
replace Replace value of eld type with apache_access
remove_tag List of tags to be removed
remove_eld List of eld to be removed
grok {...} Parese text and structure it
pattern_dir Path to our pattern les, if we don't use the internal
ones
match Field and pattern for matching
date {...} Analyse the timestamp eld
geoip Analyse the eld clientip with geoip (city, region,
ip, etc.)
useragent Analyse the eld agent as browser user agent (OS,
Major- and Minor-version browsername, etc.)

39. Cong - Indexer
Apache Error Filter
The Apache Access Filter:
1 grok {
2 patterns_dir = / opt / grok_patterns
3 match = { message = %{APACHERERROR} }
4 }
5 m u l t i l i n e {
6 pattern = ^PHP b( Notice | Warning | Error | Fatal )b :
7 s o u r c e = errorMessage
8 what = next
9 }
10 m u l t i l i n e {
11 pattern = ^PHP[ ]{3 ,} d+. .∗
12 s o u r c e = errorMessage
13 what = p r e v i o u s
14 }
15 mutate {
16 r e p l a c e = { type = apache_error }
17 r e p l a c e = { message = %{errorMessage } }
18 . . .
19 }
20 geoip {
21 s o u r c e = c l i e n t I p
22 }
23 i f [ request ] == / feed {
24 drop {}
25 }

40. Cong - Indexer
Apache Error Filter
grok {...} Parese text and structure it
pattern_dir Path to our pattern les
match Field and pattern for matching
multiline{...} Detect if we have a multiline message
pattern The detection pattern
source The eld for detection
what How to handle it (next =combine with next/previous
message)
mutate {...} Change eld values
replace Replace value of eld type with apache_error
and message with value of errorMessage
geoip Analyse the eld clientip with geoip
request if the eld request has the value /feed drop it,
we don't need it anymore

41. Cong - Indexer
Syslog/Auth Filter
The Apache Access Filter:
1 grok {
2 match = { message = %{SYSLOGT} }
3 add_field = [ received_at , %{@timestamp} ]
4 }
5 s y s l o g _ p r i { }}

42. Cong - Indexer
Syslog/Auth Filter
grok {...} Parese text and structure it
pattern_dir Path to our pattern les
match Field and pattern for matching
add_eld add an additional eld
syslog_prio {...} Handle syslog priority levels

43. Conclusion
• With these cong le and two running logstash instances we
have the log in elasticsearch

44. Conclusion
• With these cong le and two running logstash instances we
have the log in elasticsearch
• Kibana can be used for graphs and analyses

45. Kibana
Combined apache error entry

46. Kibana
Access graph

47. Kibana
Access cities, browser and devices

48. End
That's all
For more infos just search for Kibana, Logstash or Elasticsearch