The document discusses analyzing performance test results with the ELK stack. It describes the ELK stack components of ElasticSearch for log storing, Logstash for log processing, and Kibana for generating graphs and charts from JSON logs. It discusses using the ELK stack for both live log monitoring and static log analysis. It provides examples of using Kibana to monitor system logs, analyze HTTP request resources, and compare performance testing results over time.

1. 2008–2017 © Tuxera Inc. All Rights Reserved.
Analysing performance
test results with ELK
Anoop Vijayan Maniankara
Tuxera Inc.
@maniankara
22008–2017 © Tuxera Inc. All Rights Reserved.

2. 2008–2017 © Tuxera Inc. All Rights Reserved.

3. 2008–2017 © Tuxera Inc. All Rights Reserved.
Elk (Elastic) stack 5.x
• Elastic search – Logs storing (~NoSQL database)
• Log stash – Log processor
• Kibana – Graphs and charts over JSON
Data shippers
• Filebeat
• Metricbeat
• Packet beat
• …

4. 2008–2017 © Tuxera Inc. All Rights Reserved.
Anoop Vijayan
12+ years experience in designing,
developing, testing Continuous
Integration systems in bigger enterprises
to small companies
DevOps lead at Tuxera Inc.
@maniankara
Github, stackoverflow, etc.

5. 2008–2017 © Tuxera Inc. All Rights Reserved.
Agenda
• All about logs
–Purpose of log handling, tools available
–With elkstack
• Live log monitoring or processing
• Static log/data analysis
–Demo

6. 2008–2017 © Tuxera Inc. All Rights Reserved.
Purpose - Log handling
Time
Activities
Longer Time
• Search – Trace events from past
• Analyze – Tracking trends over time
Event
• Monitor – Detect events
• Compare – Two or more instances

7. 2008–2017 © Tuxera Inc. All Rights Reserved.
Tools in the market
• Free and opensource:
– ELK stack/Elastic stack - More manual configurations
• Commercial (popular ones, not all)
– Splunk – 1GB/day $4.5k (Perpetual)/$1.8k (Annual)
– Loggly – Saas not available in-house $1.18k (1GB/day)/$2.38k (4GB/day)
– …

8. 2008–2017 © Tuxera Inc. All Rights Reserved.
Architecture – ELK stack
Souce: http://blog.arungupta.me/getting-started-elk-stack-wildfly/
L E K

9. 2008–2017 © Tuxera Inc. All Rights Reserved.
ELK stack (Kibana) look and feel
Monitoring system logs Analyzing http request resources

10. 2008–2017 © Tuxera Inc. All Rights Reserved.
ELK stack (Kibana) look and feel contd.
Analyzing event fields Comparing tools

11. 2008–2017 © Tuxera Inc. All Rights Reserved.
Scope
• Live log handling
• Static data comparison

12. 2008–2017 © Tuxera Inc. All Rights Reserved.
Live log parsing (nginx log)
…
192.168.75.130 - - [23/Apr/2017:10:03:34 +0300] "GET /elasticsearch/_site/vendor.js HTTP/1.1" 304 0 "http://sandbox.srv.tuxera.com:3005/elasticsearch/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
…
…
{"@timestamp":"2017-04-
23T07:55:37.886Z","beat":{"hostname":"sandbox.srv.tuxera.com","name":"sandbox.srv.tuxera.com","version":"5.2.2"},"input_type":"log","message":"192.168.75.
130 - - [23/Apr/2017:10:03:34 +0300] "GET /elasticsearch/_site/vendor.js HTTP/1.1" 304 0 "http://sandbox.srv.tuxera.com:3005/elasticsearch/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133
Safari/537.36"","offset":1595,"source":"/var/log/nginx/access.log","type":"log"}
…
FilebeatLog collection
LogstashLog processing
elasticsearchLog storing
KibanaLog visualization
access.log
ELK stack
…
{”client”:” 192.168.75.130 ”,”time”:”23/Apr/2017:10:03:34 +0300”, ”request”:”GET”,”path”:” /elasticsearch/_site/vendor.js”,”response”:”304”,”host”:”
sandbox.srv.tuxera.com”,”client”:”"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133
Safari/537.36"”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2”,
"offset":1595,"source":"/var/log/nginx/access.log","type":"log"}}
…
…
{ "_index" : ”logstash-2017-04-23", "_type" : ”log", "_id" : "AVmDwg6kHxnroGLuXlMG", "_version" : 1, "found": true, "_source" : {”client”:” 192.168.75.130
”,”time”:”23/Apr/2017:10:03:34 +0300”, ”request”:”GET”,”path”:” /elasticsearch/_site/vendor.js”,”response”:”304”,”host”:”
sandbox.srv.tuxera.com”,”client”:”"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133
Safari/537.36"”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2”,
"offset":1595,"source":"/var/log/nginx/access.log","type":"log"}}}

13. 2008–2017 © Tuxera Inc. All Rights Reserved.
Static data comparison
FilebeatLog collection
LogstashLog processing
elasticsearchLog storing
KibanaLog visualization
Syslog
ELK stack
…
{”time”:” Apr 23 06:47:01”, ”host”:” sandbox”,
”application”:”CRON”, ”processid”:” 23259”, ”cmd”:”
(test -x /usr/sbin/anacron || ( cd / u0026u0026 run-
parts --report /etc/cron.weekly ))”,
”beat”:{"hostname" : "
sandbox.srv.tuxera.com","name" : "
sandbox.srv.tuxera.com ","version" : "5.2.2"}}
…
…
{ "_index" : ”logstash-2017-04-23", "_type" : ”log", "_id" : "AVmDwg6kHxnroGLuXlMG", "_version" : 1, "found": true, "_source" :
{”time”:” Apr 23 06:47:01”, ”host”:” sandbox”, ”application”:”CRON”, ”processid”:” 23259”, ”cmd”:” (test -x /usr/sbin/anacron || ( cd /
u0026u0026 run-parts --report /etc/cron.weekly ))”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : "
sandbox.srv.tuxera.com ","version" : "5.2.2"}}}
…
{”time”:” Apr 23 06:47:01”, ”host”:” sandbox”,
”application”:”CRON”, ”processid”:” 23259”, ”cmd”:”
(test -x /usr/sbin/anacron || ( cd / u0026u0026 run-
parts --report /etc/cron.weekly ))”,
”beat”:{"hostname" : "
sandbox.srv.tuxera.com","name" : "
sandbox.srv.tuxera.com ","version" : "5.2.2"}}
…

14. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demos
• Github path: https://github.com/maniankara/elk5.x-perf-testing

15. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo – 1 – Live monitoring of nginx access.log
• Demo1
– Live events
– Filtering based on string
FilebeatLog collection
LogstashLog processing
elasticsearchLog storing
KibanaLog visualization
access.log
ELK stack
• Recap

16. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo1 – live events of nginx access logs

17. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo – 2 – Helsinki air quality stats
• Helsinki metropolitan air quality statistics (from HSY)

18. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo 2 – all pollution dashboard

19. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo2 – NO2 pollution dashboard

20. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo – 3 –Tuxera performance tests
• Terms to know
• Effective read/write
• Cache read/write
–Performance improvement/degradation over time
–commit comparison

21. 2008–2017 © Tuxera Inc. All Rights Reserved.
Demo4 - Alerts
• Elasticsearch aggregations

22. 2008–2017 © Tuxera Inc. All Rights Reserved.
Thank you !
Slides: https://goo.gl/ZZi5EL
Demo: https://goo.gl/4JYBE9
Anoop Vijayan
@maniankara