SlideShare a Scribd company logo
1 of 29
Download to read offline
ELK
BigData for DevOps
Javantura v3 // February 20, 2016
Maarten Mulders // @mthmulders
Agenda
E, L, K
Real-world use case
Q & A
ELK?
elastic (search)
"search, analyze in real-time. sweet"
logstash
"scrub, parse and enrich. like soap for your data"
kibana
"line graphs, pie charts... yeah we got that"
all together now
logstash → collect log files
elastic → storage and analysis
kibana → visualisation
input {
file {
path => "/path/to/file.log"
}
output {
path => "/path/to/copied.log"
}
}
logstash
very modular: various inputs, filters and outputs
 
input: various application log files, but also syslog, stdin, xmpp, log4j
socket, irc, ...
filter: extract semantics (geo info, grok), add information, remove
information, match fields (cidr, dates, numbers, dns, user agent), ...
output: send events to another system such as graphite,
elasticsearch, email, file, stdout, irc, jira, nagios, s3, redis, xmpp, ...
elastic
search and analytics engine
very scalable
stores collected log events in an uniform way
events can be filtered and queried by clients (e.g. kibana)
kibana
analytics and search dashboard for elastic
 
just html and javascript (dashboards can be saved to elastic, too)
filtering determines what data is used to populate the dashboard,
queries categorise data inside the dashboard
Real-world use case
logstash setup
processess technical logging and audit logging
adds information (hostname, environment, application name)
removes information (sensitive details about customers,
transactions)
transforms information to a more usable form
 
ship events to redis
elastic setup
large cluster that contains data
one month of history
also hosts kibana files and stores its dashboards
kibana configuration
filters based on environment and timestamp (last 24h)
automatically refreshed
queries for 'error', 'orange cell', specific error codes
rows and panels for optimal screen usage
logstash input
input {
file {
path => "/path/to/application.log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
type => "application"
}
file {
path => "/path/to/audit.log"
type => "audit"
}
}
logstash filters
regular application log file
filter {
if [type] == "application" {
grok {
match => {
"message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} [%DATA]
%{LOGLEVEL:level} %{JAVACLASS} %{GREEDYDATA:line}"
}
remove_field => "message"
}
}
}
logstash filters (ctd)
audit log file
2015-01-28 01:32:15,098 [thread-1] INFO nl.ing.application.Class
eventId=1401751935098~|~inChannel=MINGZ~|~odBeneficiaryAccount=NL28INGB0000000001
filter {
if [type] == "audit" {
grok {
match => {
"message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} [%DATA]
%{LOGLEVEL} %{JAVACLASS} %{GREEDYDATA:audit_message}"
}
remove_field => "message"
}
mutate { gsub => ["audit_message", "~|~", "`"] }
kv {
source => "audit_message"
field_split => "`"
remove_field => "audit_message"
}
prune { blacklist_names => "^od.+$" }
}
}
{ timestamp: "2015-01-28 01:32:15,098",
eventId: "1401751935098",
inChannel: "MINGZ" }
logstash filters (ctd)
just in case...
filter {
if "_grokparsefailures" in [tags] {
prune {
blacklist_names => [ "message", "audit_message" ]
}
}
}
logstash output
output {
redis {
host => "redis-host"
data_type => "list"
key => "logstash"
}
}
Questions?

More Related Content

What's hot

PSUG #52 Dataflow and simplified reactive programming with Akka-streams
PSUG #52 Dataflow and simplified reactive programming with Akka-streamsPSUG #52 Dataflow and simplified reactive programming with Akka-streams
PSUG #52 Dataflow and simplified reactive programming with Akka-streamsStephane Manciot
 
Real World Serverless
Real World ServerlessReal World Serverless
Real World ServerlessPetr Zapletal
 
Bellevue Big Data meetup: Dive Deep into Spark Streaming
Bellevue Big Data meetup: Dive Deep into Spark StreamingBellevue Big Data meetup: Dive Deep into Spark Streaming
Bellevue Big Data meetup: Dive Deep into Spark StreamingSantosh Sahoo
 
Logs aggregation and analysis
Logs aggregation and analysisLogs aggregation and analysis
Logs aggregation and analysisDivante
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 
Elk devops
Elk devopsElk devops
Elk devopsIdeato
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stackVikrant Chauhan
 
ELK - Stack - Munich .net UG
ELK - Stack - Munich .net UGELK - Stack - Munich .net UG
ELK - Stack - Munich .net UGSteve Behrendt
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELKYuHsuan Chen
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case studyPaolo Tonin
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured StreamingKnoldus Inc.
 
Scalable real-time processing techniques
Scalable real-time processing techniquesScalable real-time processing techniques
Scalable real-time processing techniquesLars Albertsson
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack IntroductionVikram Shinde
 
Cassandra as event sourced journal for big data analytics
Cassandra as event sourced journal for big data analyticsCassandra as event sourced journal for big data analytics
Cassandra as event sourced journal for big data analyticsAnirvan Chakraborty
 

What's hot (20)

PSUG #52 Dataflow and simplified reactive programming with Akka-streams
PSUG #52 Dataflow and simplified reactive programming with Akka-streamsPSUG #52 Dataflow and simplified reactive programming with Akka-streams
PSUG #52 Dataflow and simplified reactive programming with Akka-streams
 
Real World Serverless
Real World ServerlessReal World Serverless
Real World Serverless
 
Bellevue Big Data meetup: Dive Deep into Spark Streaming
Bellevue Big Data meetup: Dive Deep into Spark StreamingBellevue Big Data meetup: Dive Deep into Spark Streaming
Bellevue Big Data meetup: Dive Deep into Spark Streaming
 
Logs aggregation and analysis
Logs aggregation and analysisLogs aggregation and analysis
Logs aggregation and analysis
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
 
Elk devops
Elk devopsElk devops
Elk devops
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stack
 
Elk
Elk Elk
Elk
 
ELK - Stack - Munich .net UG
ELK - Stack - Munich .net UGELK - Stack - Munich .net UG
ELK - Stack - Munich .net UG
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case study
 
Introducing ELK
Introducing ELKIntroducing ELK
Introducing ELK
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured Streaming
 
Elk scilifelab
Elk scilifelabElk scilifelab
Elk scilifelab
 
Scalable real-time processing techniques
Scalable real-time processing techniquesScalable real-time processing techniques
Scalable real-time processing techniques
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack Introduction
 
Apache airflow
Apache airflowApache airflow
Apache airflow
 
Cassandra as event sourced journal for big data analytics
Cassandra as event sourced journal for big data analyticsCassandra as event sourced journal for big data analytics
Cassandra as event sourced journal for big data analytics
 

Viewers also liked

Viewers also liked (20)

Javantura v3 - FIWARE – from ideas to real projects – Krunoslav Hrnjak
Javantura v3 - FIWARE – from ideas to real projects – Krunoslav HrnjakJavantura v3 - FIWARE – from ideas to real projects – Krunoslav Hrnjak
Javantura v3 - FIWARE – from ideas to real projects – Krunoslav Hrnjak
 
Javantura v3 - Conquer the Internet of Things with Java and Docker – Johan Ja...
Javantura v3 - Conquer the Internet of Things with Java and Docker – Johan Ja...Javantura v3 - Conquer the Internet of Things with Java and Docker – Johan Ja...
Javantura v3 - Conquer the Internet of Things with Java and Docker – Johan Ja...
 
Javantura v3 - Just say it – using language to communicate with the computer ...
Javantura v3 - Just say it – using language to communicate with the computer ...Javantura v3 - Just say it – using language to communicate with the computer ...
Javantura v3 - Just say it – using language to communicate with the computer ...
 
Javantura v3 - The Internet of (Lego) Trains – Johan Janssen, Ingmar van der ...
Javantura v3 - The Internet of (Lego) Trains – Johan Janssen, Ingmar van der ...Javantura v3 - The Internet of (Lego) Trains – Johan Janssen, Ingmar van der ...
Javantura v3 - The Internet of (Lego) Trains – Johan Janssen, Ingmar van der ...
 
Javantura v3 - Apache Spark revolution – what’s it all about – Petar Zečević
Javantura v3 - Apache Spark revolution – what’s it all about – Petar ZečevićJavantura v3 - Apache Spark revolution – what’s it all about – Petar Zečević
Javantura v3 - Apache Spark revolution – what’s it all about – Petar Zečević
 
Javantura v3 - Java & JWT Stateless authentication – Karlo Novak
Javantura v3 - Java & JWT Stateless authentication – Karlo NovakJavantura v3 - Java & JWT Stateless authentication – Karlo Novak
Javantura v3 - Java & JWT Stateless authentication – Karlo Novak
 
Javantura v3 - Going Reactive with RxJava – Hrvoje Crnjak
Javantura v3 - Going Reactive with RxJava – Hrvoje CrnjakJavantura v3 - Going Reactive with RxJava – Hrvoje Crnjak
Javantura v3 - Going Reactive with RxJava – Hrvoje Crnjak
 
Javantura v3 - ES6 – Future Is Now – Nenad Pečanac
Javantura v3 - ES6 – Future Is Now – Nenad PečanacJavantura v3 - ES6 – Future Is Now – Nenad Pečanac
Javantura v3 - ES6 – Future Is Now – Nenad Pečanac
 
Javantura v3 - CQRS – another view on application architecture – Aleksandar S...
Javantura v3 - CQRS – another view on application architecture – Aleksandar S...Javantura v3 - CQRS – another view on application architecture – Aleksandar S...
Javantura v3 - CQRS – another view on application architecture – Aleksandar S...
 
Javantura v3 - Husky – (y)our tool for tracking value in data – Mladen Marovi...
Javantura v3 - Husky – (y)our tool for tracking value in data – Mladen Marovi...Javantura v3 - Husky – (y)our tool for tracking value in data – Mladen Marovi...
Javantura v3 - Husky – (y)our tool for tracking value in data – Mladen Marovi...
 
Javantura v3 - Microservice – no fluff the REAL stuff – Nakul Mishra
Javantura v3 - Microservice – no fluff the REAL stuff – Nakul MishraJavantura v3 - Microservice – no fluff the REAL stuff – Nakul Mishra
Javantura v3 - Microservice – no fluff the REAL stuff – Nakul Mishra
 
Javantura v3 - What really motivates developers – Ivan Krnić
Javantura v3 - What really motivates developers – Ivan KrnićJavantura v3 - What really motivates developers – Ivan Krnić
Javantura v3 - What really motivates developers – Ivan Krnić
 
Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...
Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...
Javantura v3 - Real-time BigData ingestion and querying of aggregated data – ...
 
Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...
Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...
Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...
 
Javantura v4 - CroDuke Indy and the Kingdom of Java Skills - Branko Mihaljevi...
Javantura v4 - CroDuke Indy and the Kingdom of Java Skills - Branko Mihaljevi...Javantura v4 - CroDuke Indy and the Kingdom of Java Skills - Branko Mihaljevi...
Javantura v4 - CroDuke Indy and the Kingdom of Java Skills - Branko Mihaljevi...
 
Javantura v4 - DMN – supplement your BPMN - Željko Šmaguc
Javantura v4 - DMN – supplement your BPMN - Željko ŠmagucJavantura v4 - DMN – supplement your BPMN - Željko Šmaguc
Javantura v4 - DMN – supplement your BPMN - Željko Šmaguc
 
Javantura v4 - JVM++ The GraalVM - Martin Toshev
Javantura v4 - JVM++ The GraalVM - Martin ToshevJavantura v4 - JVM++ The GraalVM - Martin Toshev
Javantura v4 - JVM++ The GraalVM - Martin Toshev
 
Javantura v4 - FreeMarker in Spring web - Marin Kalapać
Javantura v4 - FreeMarker in Spring web - Marin KalapaćJavantura v4 - FreeMarker in Spring web - Marin Kalapać
Javantura v4 - FreeMarker in Spring web - Marin Kalapać
 
Javantura v4 - Getting started with Apache Spark - Dinko Srkoč
Javantura v4 - Getting started with Apache Spark - Dinko SrkočJavantura v4 - Getting started with Apache Spark - Dinko Srkoč
Javantura v4 - Getting started with Apache Spark - Dinko Srkoč
 
Javantura v4 - Let me tell you a story why Scrum is not for you - Roko Roić
Javantura v4 - Let me tell you a story why Scrum is not for you - Roko RoićJavantura v4 - Let me tell you a story why Scrum is not for you - Roko Roić
Javantura v4 - Let me tell you a story why Scrum is not for you - Roko Roić
 

Similar to Javantura v3 - ELK – Big Data for DevOps – Maarten Mulders

Real-Time Spark: From Interactive Queries to Streaming
Real-Time Spark: From Interactive Queries to StreamingReal-Time Spark: From Interactive Queries to Streaming
Real-Time Spark: From Interactive Queries to StreamingDatabricks
 
Spark what's new what's coming
Spark what's new what's comingSpark what's new what's coming
Spark what's new what's comingDatabricks
 
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...Lucidworks
 
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...randyguck
 
Elk with Openstack
Elk with OpenstackElk with Openstack
Elk with OpenstackArun prasath
 
Hadoop & Hive Change the Data Warehousing Game Forever
Hadoop & Hive Change the Data Warehousing Game ForeverHadoop & Hive Change the Data Warehousing Game Forever
Hadoop & Hive Change the Data Warehousing Game ForeverDataWorks Summit
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Microsoft Tech Community
 
Meet the squirrel @ #CSHUG
Meet the squirrel @ #CSHUGMeet the squirrel @ #CSHUG
Meet the squirrel @ #CSHUGMárton Balassi
 
Writing Continuous Applications with Structured Streaming in PySpark
Writing Continuous Applications with Structured Streaming in PySparkWriting Continuous Applications with Structured Streaming in PySpark
Writing Continuous Applications with Structured Streaming in PySparkDatabricks
 
Elk presentation 2#3
Elk presentation 2#3Elk presentation 2#3
Elk presentation 2#3uzzal basak
 
A Deep Dive into Structured Streaming in Apache Spark
A Deep Dive into Structured Streaming in Apache Spark A Deep Dive into Structured Streaming in Apache Spark
A Deep Dive into Structured Streaming in Apache Spark Anyscale
 
Cassandra at Finn.io — May 30th 2013
Cassandra at Finn.io — May 30th 2013Cassandra at Finn.io — May 30th 2013
Cassandra at Finn.io — May 30th 2013DataStax Academy
 
Fast NoSQL from HDDs?
Fast NoSQL from HDDs? Fast NoSQL from HDDs?
Fast NoSQL from HDDs? ScyllaDB
 
ELK: a log management framework
ELK: a log management frameworkELK: a log management framework
ELK: a log management frameworkGiovanni Bechis
 
Spline 0.3 and Plans for 0.4
Spline 0.3 and Plans for 0.4 Spline 0.3 and Plans for 0.4
Spline 0.3 and Plans for 0.4 Vaclav Kosar
 
JDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A Logstack
JDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A LogstackJDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A Logstack
JDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A LogstackPROIDEA
 
nuclio Overview October 2017
nuclio Overview October 2017nuclio Overview October 2017
nuclio Overview October 2017iguazio
 
Apache: Big Data - Starting with Apache Spark, Best Practices
Apache: Big Data - Starting with Apache Spark, Best PracticesApache: Big Data - Starting with Apache Spark, Best Practices
Apache: Big Data - Starting with Apache Spark, Best Practicesfelixcss
 
from source to solution - building a system for event-oriented data
from source to solution - building a system for event-oriented datafrom source to solution - building a system for event-oriented data
from source to solution - building a system for event-oriented dataEric Sammer
 

Similar to Javantura v3 - ELK – Big Data for DevOps – Maarten Mulders (20)

Real-Time Spark: From Interactive Queries to Streaming
Real-Time Spark: From Interactive Queries to StreamingReal-Time Spark: From Interactive Queries to Streaming
Real-Time Spark: From Interactive Queries to Streaming
 
Spark what's new what's coming
Spark what's new what's comingSpark what's new what's coming
Spark what's new what's coming
 
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...
 
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
 
Elk with Openstack
Elk with OpenstackElk with Openstack
Elk with Openstack
 
Hadoop & Hive Change the Data Warehousing Game Forever
Hadoop & Hive Change the Data Warehousing Game ForeverHadoop & Hive Change the Data Warehousing Game Forever
Hadoop & Hive Change the Data Warehousing Game Forever
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
 
Meet the squirrel @ #CSHUG
Meet the squirrel @ #CSHUGMeet the squirrel @ #CSHUG
Meet the squirrel @ #CSHUG
 
Streaming ETL for All
Streaming ETL for AllStreaming ETL for All
Streaming ETL for All
 
Writing Continuous Applications with Structured Streaming in PySpark
Writing Continuous Applications with Structured Streaming in PySparkWriting Continuous Applications with Structured Streaming in PySpark
Writing Continuous Applications with Structured Streaming in PySpark
 
Elk presentation 2#3
Elk presentation 2#3Elk presentation 2#3
Elk presentation 2#3
 
A Deep Dive into Structured Streaming in Apache Spark
A Deep Dive into Structured Streaming in Apache Spark A Deep Dive into Structured Streaming in Apache Spark
A Deep Dive into Structured Streaming in Apache Spark
 
Cassandra at Finn.io — May 30th 2013
Cassandra at Finn.io — May 30th 2013Cassandra at Finn.io — May 30th 2013
Cassandra at Finn.io — May 30th 2013
 
Fast NoSQL from HDDs?
Fast NoSQL from HDDs? Fast NoSQL from HDDs?
Fast NoSQL from HDDs?
 
ELK: a log management framework
ELK: a log management frameworkELK: a log management framework
ELK: a log management framework
 
Spline 0.3 and Plans for 0.4
Spline 0.3 and Plans for 0.4 Spline 0.3 and Plans for 0.4
Spline 0.3 and Plans for 0.4
 
JDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A Logstack
JDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A LogstackJDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A Logstack
JDD 2016 - Tomasz Gagor, Pawel Torbus - A Needle In A Logstack
 
nuclio Overview October 2017
nuclio Overview October 2017nuclio Overview October 2017
nuclio Overview October 2017
 
Apache: Big Data - Starting with Apache Spark, Best Practices
Apache: Big Data - Starting with Apache Spark, Best PracticesApache: Big Data - Starting with Apache Spark, Best Practices
Apache: Big Data - Starting with Apache Spark, Best Practices
 
from source to solution - building a system for event-oriented data
from source to solution - building a system for event-oriented datafrom source to solution - building a system for event-oriented data
from source to solution - building a system for event-oriented data
 

More from HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

More from HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association (20)

Java cro'21 the best tools for java developers in 2021 - hujak
Java cro'21   the best tools for java developers in 2021 - hujakJava cro'21   the best tools for java developers in 2021 - hujak
Java cro'21 the best tools for java developers in 2021 - hujak
 
JavaCro'21 - Java is Here To Stay - HUJAK Keynote
JavaCro'21 - Java is Here To Stay - HUJAK KeynoteJavaCro'21 - Java is Here To Stay - HUJAK Keynote
JavaCro'21 - Java is Here To Stay - HUJAK Keynote
 
Javantura v7 - Behaviour Driven Development with Cucumber - Ivan Lozić
Javantura v7 - Behaviour Driven Development with Cucumber - Ivan LozićJavantura v7 - Behaviour Driven Development with Cucumber - Ivan Lozić
Javantura v7 - Behaviour Driven Development with Cucumber - Ivan Lozić
 
Javantura v7 - The State of Java - Today and Tomowwow - HUJAK's Community Key...
Javantura v7 - The State of Java - Today and Tomowwow - HUJAK's Community Key...Javantura v7 - The State of Java - Today and Tomowwow - HUJAK's Community Key...
Javantura v7 - The State of Java - Today and Tomowwow - HUJAK's Community Key...
 
Javantura v7 - Learning to Scale Yourself: The Journey from Coder to Leader -...
Javantura v7 - Learning to Scale Yourself: The Journey from Coder to Leader -...Javantura v7 - Learning to Scale Yourself: The Journey from Coder to Leader -...
Javantura v7 - Learning to Scale Yourself: The Journey from Coder to Leader -...
 
JavaCro'19 - The State of Java and Software Development in Croatia - Communit...
JavaCro'19 - The State of Java and Software Development in Croatia - Communit...JavaCro'19 - The State of Java and Software Development in Croatia - Communit...
JavaCro'19 - The State of Java and Software Development in Croatia - Communit...
 
Javantura v6 - Java in Croatia and HUJAK - Branko Mihaljević, Aleksander Radovan
Javantura v6 - Java in Croatia and HUJAK - Branko Mihaljević, Aleksander RadovanJavantura v6 - Java in Croatia and HUJAK - Branko Mihaljević, Aleksander Radovan
Javantura v6 - Java in Croatia and HUJAK - Branko Mihaljević, Aleksander Radovan
 
Javantura v6 - On the Aspects of Polyglot Programming and Memory Management i...
Javantura v6 - On the Aspects of Polyglot Programming and Memory Management i...Javantura v6 - On the Aspects of Polyglot Programming and Memory Management i...
Javantura v6 - On the Aspects of Polyglot Programming and Memory Management i...
 
Javantura v6 - Case Study: Marketplace App with Java and Hyperledger Fabric -...
Javantura v6 - Case Study: Marketplace App with Java and Hyperledger Fabric -...Javantura v6 - Case Study: Marketplace App with Java and Hyperledger Fabric -...
Javantura v6 - Case Study: Marketplace App with Java and Hyperledger Fabric -...
 
Javantura v6 - How to help customers report bugs accurately - Miroslav Čerkez...
Javantura v6 - How to help customers report bugs accurately - Miroslav Čerkez...Javantura v6 - How to help customers report bugs accurately - Miroslav Čerkez...
Javantura v6 - How to help customers report bugs accurately - Miroslav Čerkez...
 
Javantura v6 - When remote work really works - the secrets behind successful ...
Javantura v6 - When remote work really works - the secrets behind successful ...Javantura v6 - When remote work really works - the secrets behind successful ...
Javantura v6 - When remote work really works - the secrets behind successful ...
 
Javantura v6 - Kotlin-Java Interop - Matej Vidaković
Javantura v6 - Kotlin-Java Interop - Matej VidakovićJavantura v6 - Kotlin-Java Interop - Matej Vidaković
Javantura v6 - Kotlin-Java Interop - Matej Vidaković
 
Javantura v6 - Spring HATEOAS hypermedia-driven web services, and clients tha...
Javantura v6 - Spring HATEOAS hypermedia-driven web services, and clients tha...Javantura v6 - Spring HATEOAS hypermedia-driven web services, and clients tha...
Javantura v6 - Spring HATEOAS hypermedia-driven web services, and clients tha...
 
Javantura v6 - End to End Continuous Delivery of Microservices for Kubernetes...
Javantura v6 - End to End Continuous Delivery of Microservices for Kubernetes...Javantura v6 - End to End Continuous Delivery of Microservices for Kubernetes...
Javantura v6 - End to End Continuous Delivery of Microservices for Kubernetes...
 
Javantura v6 - Istio Service Mesh - The magic between your microservices - Ma...
Javantura v6 - Istio Service Mesh - The magic between your microservices - Ma...Javantura v6 - Istio Service Mesh - The magic between your microservices - Ma...
Javantura v6 - Istio Service Mesh - The magic between your microservices - Ma...
 
Javantura v6 - How can you improve the quality of your application - Ioannis ...
Javantura v6 - How can you improve the quality of your application - Ioannis ...Javantura v6 - How can you improve the quality of your application - Ioannis ...
Javantura v6 - How can you improve the quality of your application - Ioannis ...
 
Javantura v6 - Just say it v2 - Pavao Varela Petrac
Javantura v6 - Just say it v2 - Pavao Varela PetracJavantura v6 - Just say it v2 - Pavao Varela Petrac
Javantura v6 - Just say it v2 - Pavao Varela Petrac
 
Javantura v6 - Automation of web apps testing - Hrvoje Ruhek
Javantura v6 - Automation of web apps testing - Hrvoje RuhekJavantura v6 - Automation of web apps testing - Hrvoje Ruhek
Javantura v6 - Automation of web apps testing - Hrvoje Ruhek
 
Javantura v6 - Master the Concepts Behind the Java 10 Challenges and Eliminat...
Javantura v6 - Master the Concepts Behind the Java 10 Challenges and Eliminat...Javantura v6 - Master the Concepts Behind the Java 10 Challenges and Eliminat...
Javantura v6 - Master the Concepts Behind the Java 10 Challenges and Eliminat...
 
Javantura v6 - Building IoT Middleware with Microservices - Mario Kusek
Javantura v6 - Building IoT Middleware with Microservices - Mario KusekJavantura v6 - Building IoT Middleware with Microservices - Mario Kusek
Javantura v6 - Building IoT Middleware with Microservices - Mario Kusek
 

Recently uploaded

From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 

Recently uploaded (20)

From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 

Javantura v3 - ELK – Big Data for DevOps – Maarten Mulders

  • 1. ELK BigData for DevOps Javantura v3 // February 20, 2016 Maarten Mulders // @mthmulders
  • 2. Agenda E, L, K Real-world use case Q & A
  • 5. logstash "scrub, parse and enrich. like soap for your data"
  • 6. kibana "line graphs, pie charts... yeah we got that"
  • 7. all together now logstash → collect log files elastic → storage and analysis kibana → visualisation
  • 8.
  • 9. input { file { path => "/path/to/file.log" } output { path => "/path/to/copied.log" } }
  • 10. logstash very modular: various inputs, filters and outputs   input: various application log files, but also syslog, stdin, xmpp, log4j socket, irc, ... filter: extract semantics (geo info, grok), add information, remove information, match fields (cidr, dates, numbers, dns, user agent), ... output: send events to another system such as graphite, elasticsearch, email, file, stdout, irc, jira, nagios, s3, redis, xmpp, ...
  • 11. elastic search and analytics engine very scalable stores collected log events in an uniform way events can be filtered and queried by clients (e.g. kibana)
  • 12. kibana analytics and search dashboard for elastic   just html and javascript (dashboards can be saved to elastic, too) filtering determines what data is used to populate the dashboard, queries categorise data inside the dashboard
  • 14.
  • 15.
  • 16. logstash setup processess technical logging and audit logging adds information (hostname, environment, application name) removes information (sensitive details about customers, transactions) transforms information to a more usable form   ship events to redis
  • 17. elastic setup large cluster that contains data one month of history also hosts kibana files and stores its dashboards
  • 18. kibana configuration filters based on environment and timestamp (last 24h) automatically refreshed queries for 'error', 'orange cell', specific error codes rows and panels for optimal screen usage
  • 19. logstash input input { file { path => "/path/to/application.log" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601} " negate => true what => previous } type => "application" } file { path => "/path/to/audit.log" type => "audit" } }
  • 20. logstash filters regular application log file filter { if [type] == "application" { grok { match => { "message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} [%DATA] %{LOGLEVEL:level} %{JAVACLASS} %{GREEDYDATA:line}" } remove_field => "message" } } }
  • 21. logstash filters (ctd) audit log file 2015-01-28 01:32:15,098 [thread-1] INFO nl.ing.application.Class eventId=1401751935098~|~inChannel=MINGZ~|~odBeneficiaryAccount=NL28INGB0000000001 filter { if [type] == "audit" { grok { match => { "message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} [%DATA] %{LOGLEVEL} %{JAVACLASS} %{GREEDYDATA:audit_message}" } remove_field => "message" } mutate { gsub => ["audit_message", "~|~", "`"] } kv { source => "audit_message" field_split => "`" remove_field => "audit_message" } prune { blacklist_names => "^od.+$" } } } { timestamp: "2015-01-28 01:32:15,098", eventId: "1401751935098", inChannel: "MINGZ" }
  • 22. logstash filters (ctd) just in case... filter { if "_grokparsefailures" in [tags] { prune { blacklist_names => [ "message", "audit_message" ] } } }
  • 23. logstash output output { redis { host => "redis-host" data_type => "list" key => "logstash" } }
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.