Real-time data analysis using ELK

REALTIME DATA ANALYSIS
USING ELK
@jettroCoenradie

Jettro Coenradie
http://amsterdam.luminis.eu

REALTIME DATA ANALYSIS
USING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know

indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes
%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css
%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2)
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/
5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-
before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/42.0.2311.90 Safari/537.36"
46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"
46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"
183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1)"
183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1)"
183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET
CLR 4.0.30319)"
46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
www.seokicks.de/robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/
robot.html)"
46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs

Chrome/42.0.2311.90 Safari/537.36"
robot.html)"
CLR 4.0.30319)"
robot.html)"
# tail -fn 100 access-log-2014-04-22

Chrome/42.0.2311.90 Safari/537.36"
robot.html)"
CLR 4.0.30319)"
robot.html)"
# tail -fn 100 access-log-2014-04-22
awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ }
END { for (i in ipcount) {
printf "%15s - %dn", i, ipcount[i] } }' access-log-2015-04-21

EVERY NIGHT A BATCH USING WEBALIZER

HOW MUCH DELAY CAN
YOU ACCEPT?

ARCHITECTURE OF DELAY
access
logs shipper Queue Logstash
elasticsearch
Monitor Send Retrieve
Store
forwarder
logstash
beaver
Redis
Kafka

DATA LIFECYCLE
Obtain Transform

DATA LIFECYCLE
Obtain Transform Store

DATA LIFECYCLE
Obtain Transform Store Use

DATA LIFECYCLE
Learn

DATA LIFECYCLE: ELK
Learn

DATA LIFECYCLE: ELK
Learn
Logstash

DATA LIFECYCLE: ELK
Learn
Logstash Logstash

DATA LIFECYCLE: ELK
Learn
Logstash Logstash Elasticsearch

DATA LIFECYCLE: ELK
Learn
Logstash Logstash Elasticsearch Kibana

DATA LIFECYCLE: ELK
Learn
Logstash Logstash Elasticsearch Kibana
YOU

INTRODUCTION OF ELK
COMPONENTS

LOGSTASH: COMPONENTS
ﬁle
syslog
redis log4j
web
socket
twitter grok
mutate
drop
clone
geoipelastic
search
ﬁle
graphite
statsd

ﬁle
syslog
redis log4j
Input
web
socket
twitter
Filter
Output
grok
mutate
drop
clone
geoipelastic
search
ﬁle
graphite
statsd

ﬁle syslog redislog4jInput
web
socket
twitter
Filter
Output
grok mutatedropclone geoip
elastic
search
ﬁlegraphite statsd

ELASTICSEARCH
cluster
Node Node Node

ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index

ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard

ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard
Mapping

ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard
Mapping
Search API

ELASTICSEARCH
cluster
Node Node Node
Index
Index
Index
Index
Index
Index
shardshard
shardshard
shardshard
shardshard
shardshard
shardshard
Mapping
Search API
Aggregations

AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-
verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0
(compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows
NT 5.1; SV1) ; .NET CLR 1.0.3705)"

AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-
verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0
(compatible; MSIE 6.0;Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0;Windows
NT 5.1; SV1) ; .NET CLR 1.0.3705)"
GET

AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538
"http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1;WOW64)
AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"

AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538
"http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1;WOW64)
AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"
POST

AGGREGATIONS
175989 133343 2008 2
POSTGET HEAD PUT

AGGREGATIONS
Date histogram
Feb Mar Apr
311344 395654 157623

AGGREGATIONS
Date histogram
Feb Mar Apr
311344 395654 157623
Cardinality [client ip] 11848 26152 9064

GET /gridshore-logs-*/_search?search_type=count
{
"aggs": {
"byDate": {
"date_histogram": {
"ﬁeld": "@timestamp",
"interval": "month"
},
"aggs": {
"uniqueVisitors": {
"cardinality": {
"ﬁeld": "clientip"
}
}
}
}
}
}

KIBANA
Discover
Visualise
Analyse

OBTAINING LOGS
daily rolling ﬁle

OBTAINING LOGS
daily rolling ﬁle
shell script ftp

OBTAINING LOGS
daily rolling ﬁle
shell script ftp
logstash

OBTAINING LOGS
daily rolling ﬁle
shell script ftp
logstash
elasticsearch

1. input {
2. ﬁle {
3. path => "/access-log-*"
4. type => "apache"
5. start_position => "beginning"
6. }
7. }
OBTAIN

1. input {
2. ﬁle {
4. type => "apache"
6. }
7. }
OBTAIN
ﬁles to import

1. input {
2. ﬁle {
4. type => "apache"
6. }
7. }
OBTAIN
used for ﬁltering

1. input {
2. ﬁle {
4. type => "apache"
6. }
7. }
OBTAIN
start reading from

Chrome/42.0.2311.90 Safari/537.36"
robot.html)"
CLR 4.0.30319)"
robot.html)"

Chrome/42.0.2311.90 Safari/537.36"
robot.html)"
CLR 4.0.30319)"
robot.html)"
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/
gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver
%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons
%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss
%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/
2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/42.0.2311.90 Safari/537.36"

1. filter {
2. grok {
3. match => { "message" => "%{COMBINEDAPACHELOG}" }
4. remove_field => ["message"]
5. }
6. grok {
7. match => { "request" => "%{URIPATH:request_noparam}"}
8. }
9. geoip {
10. source => "clientip"
11. }
12. useragent {
13. source => "agent"
14. target => "useragent"
15. remove_field => ["agent"]
16. }
17. date {
18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
19. }
20.}
TRANSFORM

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
remove parsed
message
TRANSFORM

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
TRANSFORM
extra parse of request

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
TRANSFORM
request => /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes
%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack
%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content
%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3
request_noparam => /wp-content/plugins/scripts-gzip/gzip.php

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
TRANSFORM
add geo information

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
TRANSFORM
parse useragent
ﬁelds

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
TRANSFORM
agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
(KHTML, like Gecko) Safari/537.36
useragent => {"name": "Safari", "os":“Mac OS X 10.10.2”, "os_name":“Mac OS X”,
"device": "Other", "major": "537", "minor": "36"
}

1. ﬁlter {
2. grok {
5. }
6. grok {
8. }
9. geoip {
11. }
12. useragent {
16. }
17. date {
19. }
20.}
TRANSFORM
take timestamp
from log

STORE
1. output {
2. if "_grokparsefailure" not in [tags] {
3. elasticsearch {
4. protocol => "transport"
5. host => "localhost:9300"
6. cluster => "jc-play"
7. index => "gridshore-logs-%{+YYYY.MM}"
8. manage_template => false
9. template_name => "gridshore-logs"
10. }
11. }
12.}

STORE
1. output {
3. elasticsearch {
10. }
11. }
12.}
in case of an error

STORE
1. output {
3. elasticsearch {
10. }
11. }
12.}
use faster binary
protocol

STORE
1. output {
3. elasticsearch {
10. }
11. }
12.}
format of index to create:
gridshore-logs-2015.02

STORE
1. output {
3. elasticsearch {
10. }
11. }
12.}
provide our own
index template

Integrated
Disease
Management
Control
LESSONS LEARNED

SMAP - Soil Moisture Active Passive
http://smap.jpl.nasa.gov/mission/why-it-matters/
Monitor
Drought
Predict
Floods
Assist
Crop
Productivity
Weather
Forecasting

VERIZON
https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting
“We offer technology products and solutions that
transform the way our customers connect, collaborate
and innovate”

VERIZON
https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting
“We offer technology products and solutions that
transform the way our customers connect, collaborate
and innovate”
Store massive logging data
Store in high rate
Query in acceptable rate

VERIZON
128 Nodes
8 cores - 64 Gb RAM - 6 x 1TB disk
10+ Bilion documents a day
Over 500 Billion documents total

WHAT ABOUT SECURITY
Elastic shield

LOGSTASH
• API for pipeline
• Internal / persistent queues

LOGSTASH
• API for pipeline
• Internal / persistent queues
• Clustered logstash

ELASTICSEARCH
• Better error responses

ELASTICSEARCH
• Reindex API

ELASTICSEARCH
• Reindex API
• Changes API

KIBANA
• Formatting output: numbers, currency, urls, video

KIBANA
• Edit and save or pin ﬁlters

KIBANA
• Choose your own colours in charts

KIBANA
• Choose your own colours in charts
• Create API for custom plugins

SUMMARISE
• Real time data analysis

SUMMARISE
• Obtain and transform data using logstash

SUMMARISE
• Index data in elasticsearch

SUMMARISE
• Show data using Kibana

SUMMARISE
• Show data using Kibana
• What Kibana does well and what not

MORE INFORMATION
@jettroCoenradie
jettro.coenradie@luminis.eu
http://amsterdam.luminis.eu/news/
https://www.elastic.co/products

Real-time data analysis using ELK

More Related Content

What's hot

Viewers also liked

Similar to Real-time data analysis using ELK

More from Jettro Coenradie

Recently uploaded

Real-time data analysis using ELK