This document provides information about a DevOpsDays 2016 workshop on creating an IT operations data lake using the ELK stack and Graphite. The workshop will be led by Arnold van Wijnbergen, who has 16 years of experience in IT automation. Attendees will learn how to use Logstash to ingest and process logs and metrics from various sources into Elasticsearch for analysis and visualization with Kibana and Grafana. The main components that will be covered are Logstash, Elasticsearch, Kibana, Graphite, and Kafka.

1. DevOpsDays 2016
Amsterdam
First steps to create your IT Operations data lake
with ELK Stack and Graphite

2. 2
Arnold van Wijnbergen
@BSMConsultant / Arnold.van.Wijnbergen@gmail.com
•16 years working experience with
automating IT
•DevOps Evangelist with great
passion about Architecture and
Automation Tooling
•Full time Tooling Geek 
 Automation
 Monitoring, tasted them all
 Predictive Analytics
 Infrastructure-as-Code
#IlovIT ☁

3. 3
INTRODUCE OURSELVES
Who we are and what do we expect

4. 4
CHOOSE YOUR WAY
Dev, Ops or just DevOps
DOD-AMS-WORKSHOP-BANKIT
TOURIST ROUTE
DOD-AMS-WORKSHOP
DANGER ROUTE

5. JUST ANOTHER DATA LAKE CONCEPT
Metrics, logs,
health-states, etc

6. “ELK Stack + Graphite + Kafka makes a great combination”

7. ElasticsearchLogstash Kibana
GrafanaGraphiteKafka/Zookeeper
Java
Consumer
Java
Producer
BankIT
Logfile
Topic:Events
Topic:Metrics
Syslog

8. Main component for managing your element is LogStash
Reference : https://www.elastic.co/guide/en/logstash/current/introduction.html

9. Logstash Structured way of Working – Processing Pipeline
Reference : https://www.elastic.co/guide/en/logstash/current/pipeline.html

10. Pipeline processing of an event is mainly existing of three
important stages
Reference : https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
(Filter) Processing
Parse
fields out
Enrich
fields values
Correlate
On field value matching

11. Parse unstructured data and make
IT readable structured

12. Enrich & correlate our structured data set

13. Success with the workshop
https://goo.gl/97xOX2
13

14. Backup slides
14

15. Meet Mr LogStash
What can he do for
you ?
I Love your data …
:-)

16. Many test options are available to validate your code and expected
functional behaviour
Code
Validation with
CONFIGTEST
Unit Testing
with RSPEC
Reference : https://www.elastic.co/blog/logstash-functionality-through-testing

17. Now combine the forces and create a structured configuration
Configuration folder
Input.conf
Input.conf
Input.conf
Filter.conf
Input.conf
Output.conf
file
stdi
n
mutat
egro
k
stdou
telasticsear
chredi
s
Reference : https://www.elastic.co/guide/en/logstash/current/configuration-file-
structure.html