SlideShare a Scribd company logo
Using Logstash, ElasticSearch and Kibana
Alejandro E Brito Monedero
@ae_bm
2013 / 05 / 23
Business as usual
So many hosts to check
Is there life out there?
http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
Time to play whack a log
http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
http://brotality.com/wp-content/uploads/2012/12/madness.jpg
I need a new toy
http://www.youtube.com/watch?v=8L6Dpq5kY_A
Video time
Logstash ✔ collects logs
✔ parses logs
✔ stores logs
✔ indexes logs
✔ searches logs
✔ and fixes timestamps
You only need
● JVM
● logstash.jar
$ log_producer | grep ... | sed … | awk … | tee output 
| sort | uniq -c | sort -n
Log source Logstash
Logstash
(optional)
ElasticSearch
DB
statsd
Pipes
Inputs Filters Outputs
File
Redis
Syslog
Lumberjack
Rabbitmq
SQS
…
Alter
Date
Grok
Multiline
Grep
...
AMQP
Cloudwatch
Elasticsearch
Mongodb
Redis
File
...
Plugins
Not here yet? JRuby to the rescue
ElasticSearch
Distributed RESTful search server
● Near real-time search
● RESTful API
● Easy to scale horizontally
● HA
● Full text search
● YAML config file / JSON format!!
● Document oriented JSON
Getting started: Logstash JAR includes it / download and set cluster.name
This is where it will be worth to spend some time tuning
Kibana
✔ Nice UI
✔ Better than the old frontend logstash
included
✔ Ruby / framework Sinatra
Web frontend to search / graph and more
Original plan
Apache
(ligthweight shipper)
Tomcat
(ligthweight shipper)
broker logstash
ElasticSearch
Kibana
After a few workarounds
Apache
(logstash shipper)
Tomcat
(logstash shipper)
Logstash ElasticSearch
Kibana
SSH tunnels
Example config 1/3
Logstash-httpd.conf
input {
file {
type => "httpd"
path => ["/var/log/httpd/*-logstash.log"]
exclude => ["*.gz"]
start_position => "beginning"
format => "json_event"
}
}
output {
tcp {
host => "0.0.0.0"
mode => "server"
port => 1666
}
}
Logstash-server.conf
input {
tcp {
type => "httpd"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "1666"
}
tcp {
type => "app"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "2666"
}
}
output {
elasticsearch {
cluster => "logstash"
}
}
Example config 2/3
Logstash-tomcat.conf
filter {
# Tomcat
# Remove blank lines
grep {
type => "tomcat"
match => [ "@message", "(.+)" ]
drop => true
add_tag => [ "no_blank_lines" ]
}
# make the multilines be treated like a single line
multiline {
type => "tomcat"
pattern => "^dddd"
negate => true
what => "previous"
}
Example config 3/3
Logstash-tomcat.conf
# mark the exceptions (multiline)
grep {
type => "tomcat"
tags => [ "multiline" ]
match => [ "@message", ".+Exception: .+" ]
drop => false
add_tag => [ "java_exception" ]
}
# get the log level, operation id, module and timestamp as separated fields
grok {
type => "tomcat"
pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."
add_tag => [ "groked" ]
}
# fix the timestamp
date {
type => "tomcat"
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]
add_tag => [ "timestamp_fix" ]
}
}
I need a new toy
Demo
Some remarks
● Don't forget about security
● The applications should be flexible enough for allowing to
publish their logs using brokers or other methods beyond
files and syslog
● Logging in JSON format is a nice to have
● Share the log visualization
● Use the brokers Luke
● If you develop internalize this
http://www.masterzen.fr/2013/01/13/the-10-
commandments-of-logging/
Extras
● http://logstash.net/
● http://www.logstashbook.com/code/ only $10.09
● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns
● http://grokdebug.herokuapp.com/
● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)
●
http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
● http://kibana.org/
●
https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html
● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/
●
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
Do you want to join the <some fancy words here> team?
I am not hiring, but I can tell you about some places where
it is better to stay away
Have a nice day
All the images, videos and stuff are property of their respective owners, look at the cat
and don't sue me
http://stuffpoint.com/cats/image/41633/cute-cat-picture/

More Related Content

What's hot

Elk stack
Elk stackElk stack
Elk stack
Jilles van Gurp
 
Logstash
LogstashLogstash
Logstash
琛琳 饶
 
Web authentication & authorization
Web authentication & authorizationWeb authentication & authorization
Web authentication & authorization
Alexandru Pasaila
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of Lucene
Rahul Jain
 
Intro To Mongo Db
Intro To Mongo DbIntro To Mongo Db
Intro To Mongo Db
chriskite
 
Elasticsearch
ElasticsearchElasticsearch
Elasticsearch
Hermeto Romano
 
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
Amazon Web Services Korea
 
Elasticsearch
ElasticsearchElasticsearch
Elasticsearch
Shagun Rathore
 
Hacking and Defending APIs - Red and Blue make Purple.pdf
Hacking and Defending APIs - Red and Blue make Purple.pdfHacking and Defending APIs - Red and Blue make Purple.pdf
Hacking and Defending APIs - Red and Blue make Purple.pdf
Matt Tesauro
 
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020 AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWSKRUG - AWS한국사용자모임
 
OpenSearch
OpenSearchOpenSearch
OpenSearch
hchen1
 
Data pipeline and data lake
Data pipeline and data lakeData pipeline and data lake
Data pipeline and data lake
DaeMyung Kang
 
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
Juhong Park
 
Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
hypto
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to Kibana
Vineet .
 
Elastic search Walkthrough
Elastic search WalkthroughElastic search Walkthrough
Elastic search Walkthrough
Suhel Meman
 
서버학개론(백엔드 서버 개발자를 위한)
서버학개론(백엔드 서버 개발자를 위한)서버학개론(백엔드 서버 개발자를 위한)
서버학개론(백엔드 서버 개발자를 위한)
수보 김
 
Search and analyze your data with elasticsearch
Search and analyze your data with elasticsearchSearch and analyze your data with elasticsearch
Search and analyze your data with elasticsearch
Anton Udovychenko
 
第一次Elasticsearch就上手
第一次Elasticsearch就上手第一次Elasticsearch就上手
第一次Elasticsearch就上手
Aaron King
 
How Netflix Is Solving Authorization Across Their Cloud
How Netflix Is Solving Authorization Across Their CloudHow Netflix Is Solving Authorization Across Their Cloud
How Netflix Is Solving Authorization Across Their Cloud
Torin Sandall
 

What's hot (20)

Elk stack
Elk stackElk stack
Elk stack
 
Logstash
LogstashLogstash
Logstash
 
Web authentication & authorization
Web authentication & authorizationWeb authentication & authorization
Web authentication & authorization
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of Lucene
 
Intro To Mongo Db
Intro To Mongo DbIntro To Mongo Db
Intro To Mongo Db
 
Elasticsearch
ElasticsearchElasticsearch
Elasticsearch
 
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
 
Elasticsearch
ElasticsearchElasticsearch
Elasticsearch
 
Hacking and Defending APIs - Red and Blue make Purple.pdf
Hacking and Defending APIs - Red and Blue make Purple.pdfHacking and Defending APIs - Red and Blue make Purple.pdf
Hacking and Defending APIs - Red and Blue make Purple.pdf
 
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020 AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
 
OpenSearch
OpenSearchOpenSearch
OpenSearch
 
Data pipeline and data lake
Data pipeline and data lakeData pipeline and data lake
Data pipeline and data lake
 
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
[KAIST 채용설명회] 데이터 엔지니어는 무슨 일을 하나요?
 
Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to Kibana
 
Elastic search Walkthrough
Elastic search WalkthroughElastic search Walkthrough
Elastic search Walkthrough
 
서버학개론(백엔드 서버 개발자를 위한)
서버학개론(백엔드 서버 개발자를 위한)서버학개론(백엔드 서버 개발자를 위한)
서버학개론(백엔드 서버 개발자를 위한)
 
Search and analyze your data with elasticsearch
Search and analyze your data with elasticsearchSearch and analyze your data with elasticsearch
Search and analyze your data with elasticsearch
 
第一次Elasticsearch就上手
第一次Elasticsearch就上手第一次Elasticsearch就上手
第一次Elasticsearch就上手
 
How Netflix Is Solving Authorization Across Their Cloud
How Netflix Is Solving Authorization Across Their CloudHow Netflix Is Solving Authorization Across Their Cloud
How Netflix Is Solving Authorization Across Their Cloud
 

Similar to Using Logstash, elasticsearch & kibana

(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석
INSIGHT FORENSIC
 
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang MeetupДоклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Badoo Development
 
Logstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtimeLogstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtime
Andrea Cardinale
 
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Yauheni Akhotnikau
 
Machine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens LourensMachine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens Lourens
OpenCredo
 
Elk scilifelab
Elk scilifelabElk scilifelab
Troubleshooting real production problems
Troubleshooting real production problemsTroubleshooting real production problems
Troubleshooting real production problems
Tier1 app
 
Elk devops
Elk devopsElk devops
Elk devops
Ideato
 
Mario
MarioMario
Mario
宗志 陈
 
Why you should be using structured logs
Why you should be using structured logsWhy you should be using structured logs
Why you should be using structured logs
Stefan Krawczyk
 
#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template
Hadziq Fabroyir
 
Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)
Tier1 app
 
Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존
동수 장
 
Undelete (and more) rows from the binary log
Undelete (and more) rows from the binary logUndelete (and more) rows from the binary log
Undelete (and more) rows from the binary log
Frederic Descamps
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
PROIDEA
 
Fixing Web Data in Production
Fixing Web Data in ProductionFixing Web Data in Production
Fixing Web Data in Production
Aaron Knight
 
Openstack 簡介
Openstack 簡介Openstack 簡介
Openstack 簡介
kao kuo-tung
 
Kommons
KommonsKommons
Hartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter SapporoHartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter Sapporo
Jun OHWADA
 
Introduction of RiotJS
Introduction of RiotJSIntroduction of RiotJS
Introduction of RiotJS
Ryo Iinuma
 

Similar to Using Logstash, elasticsearch & kibana (20)

(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석
 
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang MeetupДоклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
 
Logstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtimeLogstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtime
 
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
 
Machine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens LourensMachine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens Lourens
 
Elk scilifelab
Elk scilifelabElk scilifelab
Elk scilifelab
 
Troubleshooting real production problems
Troubleshooting real production problemsTroubleshooting real production problems
Troubleshooting real production problems
 
Elk devops
Elk devopsElk devops
Elk devops
 
Mario
MarioMario
Mario
 
Why you should be using structured logs
Why you should be using structured logsWhy you should be using structured logs
Why you should be using structured logs
 
#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template
 
Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)
 
Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존
 
Undelete (and more) rows from the binary log
Undelete (and more) rows from the binary logUndelete (and more) rows from the binary log
Undelete (and more) rows from the binary log
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
 
Fixing Web Data in Production
Fixing Web Data in ProductionFixing Web Data in Production
Fixing Web Data in Production
 
Openstack 簡介
Openstack 簡介Openstack 簡介
Openstack 簡介
 
Kommons
KommonsKommons
Kommons
 
Hartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter SapporoHartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter Sapporo
 
Introduction of RiotJS
Introduction of RiotJSIntroduction of RiotJS
Introduction of RiotJS
 

More from Alejandro E Brito Monedero

Mad scalability (perfomance debugging)
Mad scalability (perfomance debugging)Mad scalability (perfomance debugging)
Mad scalability (perfomance debugging)
Alejandro E Brito Monedero
 
Tres historias
Tres historiasTres historias
AMQP vs GRAPHITE
AMQP vs GRAPHITEAMQP vs GRAPHITE
AMQP vs GRAPHITE
Alejandro E Brito Monedero
 
Sysdig
SysdigSysdig
Sysdig SRECon 16 Europe
Sysdig SRECon 16 EuropeSysdig SRECon 16 Europe
Sysdig SRECon 16 Europe
Alejandro E Brito Monedero
 
Funcional para trollear
Funcional para trollearFuncional para trollear
Funcional para trollear
Alejandro E Brito Monedero
 
Top Bug
Top BugTop Bug
Fabric más allá de lo básico
Fabric más allá de lo básicoFabric más allá de lo básico
Fabric más allá de lo básico
Alejandro E Brito Monedero
 
Experiencias con PostgreSQL en AWS
Experiencias con PostgreSQL en AWSExperiencias con PostgreSQL en AWS
Experiencias con PostgreSQL en AWS
Alejandro E Brito Monedero
 
Fabric Fast & Furious edition
Fabric Fast & Furious editionFabric Fast & Furious edition
Fabric Fast & Furious edition
Alejandro E Brito Monedero
 
Así que pusiste MongoDB. Dime ¿cómo lo administras?
Así que pusiste MongoDB. Dime ¿cómo lo administras?Así que pusiste MongoDB. Dime ¿cómo lo administras?
Así que pusiste MongoDB. Dime ¿cómo lo administras?
Alejandro E Brito Monedero
 
AWS Baby steps circa 2008
AWS Baby steps circa 2008AWS Baby steps circa 2008
AWS Baby steps circa 2008
Alejandro E Brito Monedero
 
Wireshark tips
Wireshark tipsWireshark tips
Mi experiencia con Amazon AWS EC2 y S3
Mi experiencia con Amazon AWS EC2 y S3Mi experiencia con Amazon AWS EC2 y S3
Mi experiencia con Amazon AWS EC2 y S3
Alejandro E Brito Monedero
 

More from Alejandro E Brito Monedero (14)

Mad scalability (perfomance debugging)
Mad scalability (perfomance debugging)Mad scalability (perfomance debugging)
Mad scalability (perfomance debugging)
 
Tres historias
Tres historiasTres historias
Tres historias
 
AMQP vs GRAPHITE
AMQP vs GRAPHITEAMQP vs GRAPHITE
AMQP vs GRAPHITE
 
Sysdig
SysdigSysdig
Sysdig
 
Sysdig SRECon 16 Europe
Sysdig SRECon 16 EuropeSysdig SRECon 16 Europe
Sysdig SRECon 16 Europe
 
Funcional para trollear
Funcional para trollearFuncional para trollear
Funcional para trollear
 
Top Bug
Top BugTop Bug
Top Bug
 
Fabric más allá de lo básico
Fabric más allá de lo básicoFabric más allá de lo básico
Fabric más allá de lo básico
 
Experiencias con PostgreSQL en AWS
Experiencias con PostgreSQL en AWSExperiencias con PostgreSQL en AWS
Experiencias con PostgreSQL en AWS
 
Fabric Fast & Furious edition
Fabric Fast & Furious editionFabric Fast & Furious edition
Fabric Fast & Furious edition
 
Así que pusiste MongoDB. Dime ¿cómo lo administras?
Así que pusiste MongoDB. Dime ¿cómo lo administras?Así que pusiste MongoDB. Dime ¿cómo lo administras?
Así que pusiste MongoDB. Dime ¿cómo lo administras?
 
AWS Baby steps circa 2008
AWS Baby steps circa 2008AWS Baby steps circa 2008
AWS Baby steps circa 2008
 
Wireshark tips
Wireshark tipsWireshark tips
Wireshark tips
 
Mi experiencia con Amazon AWS EC2 y S3
Mi experiencia con Amazon AWS EC2 y S3Mi experiencia con Amazon AWS EC2 y S3
Mi experiencia con Amazon AWS EC2 y S3
 

Recently uploaded

Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
Edge AI and Vision Alliance
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 

Recently uploaded (20)

Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfareArtificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic Warfare
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 

Using Logstash, elasticsearch & kibana

  • 1. Using Logstash, ElasticSearch and Kibana Alejandro E Brito Monedero @ae_bm 2013 / 05 / 23
  • 3.
  • 4. So many hosts to check Is there life out there? http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
  • 5. Time to play whack a log http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
  • 7. I need a new toy http://www.youtube.com/watch?v=8L6Dpq5kY_A Video time
  • 8. Logstash ✔ collects logs ✔ parses logs ✔ stores logs ✔ indexes logs ✔ searches logs ✔ and fixes timestamps You only need ● JVM ● logstash.jar
  • 9. $ log_producer | grep ... | sed … | awk … | tee output | sort | uniq -c | sort -n Log source Logstash Logstash (optional) ElasticSearch DB statsd Pipes
  • 11. ElasticSearch Distributed RESTful search server ● Near real-time search ● RESTful API ● Easy to scale horizontally ● HA ● Full text search ● YAML config file / JSON format!! ● Document oriented JSON Getting started: Logstash JAR includes it / download and set cluster.name This is where it will be worth to spend some time tuning
  • 12. Kibana ✔ Nice UI ✔ Better than the old frontend logstash included ✔ Ruby / framework Sinatra Web frontend to search / graph and more
  • 13. Original plan Apache (ligthweight shipper) Tomcat (ligthweight shipper) broker logstash ElasticSearch Kibana
  • 14. After a few workarounds Apache (logstash shipper) Tomcat (logstash shipper) Logstash ElasticSearch Kibana SSH tunnels
  • 15. Example config 1/3 Logstash-httpd.conf input { file { type => "httpd" path => ["/var/log/httpd/*-logstash.log"] exclude => ["*.gz"] start_position => "beginning" format => "json_event" } } output { tcp { host => "0.0.0.0" mode => "server" port => 1666 } } Logstash-server.conf input { tcp { type => "httpd" format => "json_event" host => "127.0.0.1" mode => "client" port => "1666" } tcp { type => "app" format => "json_event" host => "127.0.0.1" mode => "client" port => "2666" } } output { elasticsearch { cluster => "logstash" } }
  • 16. Example config 2/3 Logstash-tomcat.conf filter { # Tomcat # Remove blank lines grep { type => "tomcat" match => [ "@message", "(.+)" ] drop => true add_tag => [ "no_blank_lines" ] } # make the multilines be treated like a single line multiline { type => "tomcat" pattern => "^dddd" negate => true what => "previous" }
  • 17. Example config 3/3 Logstash-tomcat.conf # mark the exceptions (multiline) grep { type => "tomcat" tags => [ "multiline" ] match => [ "@message", ".+Exception: .+" ] drop => false add_tag => [ "java_exception" ] } # get the log level, operation id, module and timestamp as separated fields grok { type => "tomcat" pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..." add_tag => [ "groked" ] } # fix the timestamp date { type => "tomcat" match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ] add_tag => [ "timestamp_fix" ] } }
  • 18. I need a new toy Demo
  • 19. Some remarks ● Don't forget about security ● The applications should be flexible enough for allowing to publish their logs using brokers or other methods beyond files and syslog ● Logging in JSON format is a nice to have ● Share the log visualization ● Use the brokers Luke ● If you develop internalize this http://www.masterzen.fr/2013/01/13/the-10- commandments-of-logging/
  • 20. Extras ● http://logstash.net/ ● http://www.logstashbook.com/code/ only $10.09 ● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns ● http://grokdebug.herokuapp.com/ ● http://www.infoq.com/articles/review-the-logstash-book (better diagrams) ● http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/ ● http://kibana.org/ ● https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html ● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ ● http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
  • 21. Do you want to join the <some fancy words here> team? I am not hiring, but I can tell you about some places where it is better to stay away Have a nice day All the images, videos and stuff are property of their respective owners, look at the cat and don't sue me http://stuffpoint.com/cats/image/41633/cute-cat-picture/