Your SlideShare is downloading. ×
0
#servertraining

Log Analysis Open Source
con

Logstash
Elasticsearch &
Kibana
www.servermanaged.it

twitter.com/serverman...
#servertraining

Slides a cura di
Valentino Gagliardi
Technical Manager at ServerManaged.it
Devop & Sysadmin vecchia scuol...
#servertraining

Sommario

Log analysis, a cosa serve?
●In principio era...
●Cos'e' Logstash
●Cos'e' Elasticsearch
●Cos'e'...
#servertraining

Log analysis:

“is an art and science
seeking to make sense
out of computergenerated records”
www.serverm...
#servertraining

Tradotto:

dare un senso ad una
montagna di log
provenienti da server,
routers, ecc
www.servermanaged.it
...
#servertraining

Log:

I registri di un sistema.
Se c'e' un problema sul
server Y c'e' anche una
traccia nei log.
www.serv...
#servertraining

Log analysis per:

- tracciare i problemi
- prevenire incidenti di
sicurezza
- ricostruirli se avvengono
...
#servertraining

In principio
era tailf
...
www.servermanaged.it

twitter.com/servermanagedit
#servertraining

# tailf /var/log/secure

www.servermanaged.it

twitter.com/servermanagedit
#servertraining

# tailf /var/log/secure
# tailf /var/log/messages

www.servermanaged.it

twitter.com/servermanagedit
#servertraining

# tailf /var/log/secure
# tailf /var/log/messages
# multitail /var/log/httpd/error.log
/var/log/httpd/acc...
#servertraining

Oggi e'
data
visualization
www.servermanaged.it

twitter.com/servermanagedit
#servertraining

www.servermanaged.it

twitter.com/servermanagedit
#servertraining

Cos'e' Logstash?

“Logstash helps you take
logs and other event data
from your systems and
store them in ...
#servertraining

Logstash:

trasforma qualsiasi fonte
di eventi e log in qualcosa
di digeribile e
processabile
www.serverm...
#servertraining

Logstash:

36 inputs (and growing)
14 codecs (and growing)
40 filtri (and growing)
50 outputs (and growin...
#servertraining
## Una configurazione minimale di Logstash
input {
file {
type => "linux-syslog"
path => ["/var/log/*.log"...
#servertraining

Cos'e' Redis?

“Redis is an open source,
BSD licensed, advanced
key-value store.“

www.servermanaged.it

...
#servertraining

Redis:

in un sistema di logging
centralizzato puo' essere
usato come buffer per i
log
www.servermanaged....
#servertraining

Cos'e' Elasticsearch?

“flexible and powerful
open source, distributed
real-time
search and analytics eng...
#servertraining

Elasticsearch:

in un sistema di logging
centralizzato puo' essere
usato come output per
indicizzare i lo...
#servertraining

Cos'e' Kibana?

“Make Sense of your
Data“

www.servermanaged.it

twitter.com/servermanagedit
#servertraining

Kibana:

una dashboard per
estrarre i dati da
Elasticsearch
www.servermanaged.it

twitter.com/servermanag...
#servertraining

www.servermanaged.it

twitter.com/servermanagedit
#servertraining

Big picture:
logging
centralizzato
www.servermanaged.it

twitter.com/servermanagedit
#servertraining

Un setup tipico per il
logging centralizzato
con Rsyslog, Logstash,
Redis, Elasticsearch e
Kibana.
www.se...
#servertraining

server

Server di Logging Centralizzato

server
server

rsyslog

server

www.servermanaged.it

twitter.co...
#servertraining

server

Server di Logging Centralizzato
server

server
server

rsyslog

server
server

www.servermanaged....
#servertraining

server

Server di Logging Centralizzato
server

server
server

rsyslog

server
server

www.servermanaged....
#servertraining

server

Server di Logging Centralizzato
server

server
server

rsyslog

server
server

www.servermanaged....
#servertraining

server

Server di Logging Centralizzato
server

server
server

rsyslog

server
server

www.servermanaged....
#servertraining

server

Server di Logging Centralizzato
server

server
server

rsyslog

server
server

www.servermanaged....
#servertraining

Ok tutto bello. “Ma
cosa te ne fai di questi
grafici?” (tratto da una
domanda vera)
www.servermanaged.it
...
#servertraining

Centralizzare i log:
- visibilita' dei trend
- visibilita' dei problemi
- analisi di sicurezza
www.server...
#servertraining

Caso di studio.
Mitigazione di una
serie di potenti
attacchi bruteforce
www.servermanaged.it

twitter.com...
#servertraining
Visualizzare le conseguenze di un attacco, anticipare il trend e mitigare la
minaccia.

In rosso: attacco ...
#servertraining

Non solo Logstash:
Le alternative costose
al logging
centralizzato open
source.
www.servermanaged.it

twi...
#servertraining

Splunk>

Grab a 20GB license for
12187631461319$/month
(gratuito fino a
500MB/giorno)
www.servermanaged.i...
#servertraining

Splunk>

www.servermanaged.it

twitter.com/servermanagedit
#servertraining

Loggly

10GB di log al giorno
con una ritenzione di
90 giorni costano circa
1482 dollari al mese.
www.ser...
#servertraining

Ogni prodotto ha pro e
contro. Splunk e
Loggly: compliance e
immediatezza.
Logstash: per i nerd.
www.serv...
#servertraining

Logstash: vantaggi
- open source
- gratuito
- una folta community
- in continuo sviluppo

www.servermanag...
#servertraining

KEEP CALM
AND
LOGSTASH
www.servermanaged.it

twitter.com/servermanagedit
#servertraining

http://www.logstash.net
http://www.redis.io
http://www.elasticsearch.org

www.servermanaged.it

twitter.c...
#servertraining

Slides a cura di
Valentino Gagliardi
Technical Manager at ServerManaged.it
Devop & Sysadmin vecchia scuol...
Upcoming SlideShare
Loading in...5
×

Log analysis OpenSource con Logstash, Elasticsearch e Kibana

5,723

Published on

Log analysis Open Source con Logstash, Elasticsearch e Kibana.
A cosa serve la log analysis? Panoramica sulle possibilita' offerte da Logstash, Elasticsearch e Kibana per la gestione centralizzata open source di log.

Published in: Technology, Education
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,723
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
62
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "Log analysis OpenSource con Logstash, Elasticsearch e Kibana"

  1. 1. #servertraining Log Analysis Open Source con Logstash Elasticsearch & Kibana www.servermanaged.it twitter.com/servermanagedit
  2. 2. #servertraining Slides a cura di Valentino Gagliardi Technical Manager at ServerManaged.it Devop & Sysadmin vecchia scuola, consulente informatico per small/medium business, cloud, hosting operations. www.servermanaged.it twitter.com/servermanagedit
  3. 3. #servertraining Sommario Log analysis, a cosa serve? ●In principio era... ●Cos'e' Logstash ●Cos'e' Elasticsearch ●Cos'e' Kibana ●The big picture, un setup tipico ●Ma Splunk? E Loggly? ●Logstash, vantaggi ● www.servermanaged.it twitter.com/servermanagedit
  4. 4. #servertraining Log analysis: “is an art and science seeking to make sense out of computergenerated records” www.servermanaged.it twitter.com/servermanagedit
  5. 5. #servertraining Tradotto: dare un senso ad una montagna di log provenienti da server, routers, ecc www.servermanaged.it twitter.com/servermanagedit
  6. 6. #servertraining Log: I registri di un sistema. Se c'e' un problema sul server Y c'e' anche una traccia nei log. www.servermanaged.it twitter.com/servermanagedit
  7. 7. #servertraining Log analysis per: - tracciare i problemi - prevenire incidenti di sicurezza - ricostruirli se avvengono www.servermanaged.it twitter.com/servermanagedit
  8. 8. #servertraining In principio era tailf ... www.servermanaged.it twitter.com/servermanagedit
  9. 9. #servertraining # tailf /var/log/secure www.servermanaged.it twitter.com/servermanagedit
  10. 10. #servertraining # tailf /var/log/secure # tailf /var/log/messages www.servermanaged.it twitter.com/servermanagedit
  11. 11. #servertraining # tailf /var/log/secure # tailf /var/log/messages # multitail /var/log/httpd/error.log /var/log/httpd/access.log www.servermanaged.it twitter.com/servermanagedit
  12. 12. #servertraining Oggi e' data visualization www.servermanaged.it twitter.com/servermanagedit
  13. 13. #servertraining www.servermanaged.it twitter.com/servermanagedit
  14. 14. #servertraining Cos'e' Logstash? “Logstash helps you take logs and other event data from your systems and store them in a central place. “ www.servermanaged.it twitter.com/servermanagedit
  15. 15. #servertraining Logstash: trasforma qualsiasi fonte di eventi e log in qualcosa di digeribile e processabile www.servermanaged.it twitter.com/servermanagedit
  16. 16. #servertraining Logstash: 36 inputs (and growing) 14 codecs (and growing) 40 filtri (and growing) 50 outputs (and growing) www.servermanaged.it twitter.com/servermanagedit
  17. 17. #servertraining ## Una configurazione minimale di Logstash input { file { type => "linux-syslog" path => ["/var/log/*.log"] exclude => [ "*.gz" ] } } output { redis { host => "127.0.0.1" data_type => "list" key => "syslog" } } ## www.servermanaged.it twitter.com/servermanagedit
  18. 18. #servertraining Cos'e' Redis? “Redis is an open source, BSD licensed, advanced key-value store.“ www.servermanaged.it twitter.com/servermanagedit
  19. 19. #servertraining Redis: in un sistema di logging centralizzato puo' essere usato come buffer per i log www.servermanaged.it twitter.com/servermanagedit
  20. 20. #servertraining Cos'e' Elasticsearch? “flexible and powerful open source, distributed real-time search and analytics engine for the cloud“ www.servermanaged.it twitter.com/servermanagedit
  21. 21. #servertraining Elasticsearch: in un sistema di logging centralizzato puo' essere usato come output per indicizzare i log www.servermanaged.it twitter.com/servermanagedit
  22. 22. #servertraining Cos'e' Kibana? “Make Sense of your Data“ www.servermanaged.it twitter.com/servermanagedit
  23. 23. #servertraining Kibana: una dashboard per estrarre i dati da Elasticsearch www.servermanaged.it twitter.com/servermanagedit
  24. 24. #servertraining www.servermanaged.it twitter.com/servermanagedit
  25. 25. #servertraining Big picture: logging centralizzato www.servermanaged.it twitter.com/servermanagedit
  26. 26. #servertraining Un setup tipico per il logging centralizzato con Rsyslog, Logstash, Redis, Elasticsearch e Kibana. www.servermanaged.it twitter.com/servermanagedit
  27. 27. #servertraining server Server di Logging Centralizzato server server rsyslog server www.servermanaged.it twitter.com/servermanagedit
  28. 28. #servertraining server Server di Logging Centralizzato server server server rsyslog server server www.servermanaged.it twitter.com/servermanagedit
  29. 29. #servertraining server Server di Logging Centralizzato server server server rsyslog server server www.servermanaged.it twitter.com/servermanagedit
  30. 30. #servertraining server Server di Logging Centralizzato server server server rsyslog server server www.servermanaged.it twitter.com/servermanagedit
  31. 31. #servertraining server Server di Logging Centralizzato server server server rsyslog server server www.servermanaged.it twitter.com/servermanagedit
  32. 32. #servertraining server Server di Logging Centralizzato server server server rsyslog server server www.servermanaged.it twitter.com/servermanagedit
  33. 33. #servertraining Ok tutto bello. “Ma cosa te ne fai di questi grafici?” (tratto da una domanda vera) www.servermanaged.it twitter.com/servermanagedit
  34. 34. #servertraining Centralizzare i log: - visibilita' dei trend - visibilita' dei problemi - analisi di sicurezza www.servermanaged.it twitter.com/servermanagedit
  35. 35. #servertraining Caso di studio. Mitigazione di una serie di potenti attacchi bruteforce www.servermanaged.it twitter.com/servermanagedit
  36. 36. #servertraining Visualizzare le conseguenze di un attacco, anticipare il trend e mitigare la minaccia. In rosso: attacco bruteforce massivo su siti web Joomla In verde: mitigazione dell'attacco. Le richieste anomale vengono respinte www.servermanaged.it twitter.com/servermanagedit
  37. 37. #servertraining Non solo Logstash: Le alternative costose al logging centralizzato open source. www.servermanaged.it twitter.com/servermanagedit
  38. 38. #servertraining Splunk> Grab a 20GB license for 12187631461319$/month (gratuito fino a 500MB/giorno) www.servermanaged.it twitter.com/servermanagedit
  39. 39. #servertraining Splunk> www.servermanaged.it twitter.com/servermanagedit
  40. 40. #servertraining Loggly 10GB di log al giorno con una ritenzione di 90 giorni costano circa 1482 dollari al mese. www.servermanaged.it twitter.com/servermanagedit
  41. 41. #servertraining Ogni prodotto ha pro e contro. Splunk e Loggly: compliance e immediatezza. Logstash: per i nerd. www.servermanaged.it twitter.com/servermanagedit
  42. 42. #servertraining Logstash: vantaggi - open source - gratuito - una folta community - in continuo sviluppo www.servermanaged.it twitter.com/servermanagedit
  43. 43. #servertraining KEEP CALM AND LOGSTASH www.servermanaged.it twitter.com/servermanagedit
  44. 44. #servertraining http://www.logstash.net http://www.redis.io http://www.elasticsearch.org www.servermanaged.it twitter.com/servermanagedit
  45. 45. #servertraining Slides a cura di Valentino Gagliardi Technical Manager at ServerManaged.it Devop & Sysadmin vecchia scuola, consulente informatico per small/medium business, cloud, hosting operations. (Vieni a trovarmi su Google+, LinkedIn e Twitter) Immagine di sfondo: http://medialoot.com/item/free-dark-noise-backgrounds www.servermanaged.it twitter.com/servermanagedit
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×