The document details a webinar presentation by NETWAYS GmbH on Logstash, covering its architecture, integration into IT environments, and application possibilities. Founded in 1995, NETWAYS specializes in open source systems management and offers various services like workshops, consulting, and training. The session aims to demonstrate how Logstash can effectively manage and analyze large quantities of log data.

1. www.netways.de // blog.netways.de // @netways
Make IT do more with less
06.12.2013 | WEBINAR
LOGSTASH: LOGFILE-ARCHIVIERUNG
LEICHT GEMACHT
BLERIM SHEQA UND CHRISTIAN STEIN | NETWAYS GMBH

2. www.netways.de // blog.netways.de // @netways
Make IT do more with less
VORSTELLUNG MITARBEITER
￭ Christian Stein
• Account Manager
• Bei NETWAYS seit 2012
￭ Blerim Sheqa
• Systems Engineer
• Bei NETWAYS seit 2013

3. www.netways.de // blog.netways.de // @netways
Make IT do more with less
AGENDA
￭ Kurzvorstellung NETWAYS
￭ Logstash
￭ Logstash Architektur
￭ Integration in die IT-Landschaft
￭ Anwendungsmöglichkeiten
￭ Kibana
￭ Live Demo
￭ Fragen & Antworten

4. www.netways.de // blog.netways.de // @netways
Make IT do more with less
KURZVORSTELLUNG NETWAYS

5. www.netways.de // blog.netways.de // @netways
Make IT do more with less
KURZVORSTELLUNG NETWAYS
• Firmengründung 1995
• Open Source seit 1997
• 40 Mitarbeiter
• Spezialisierung in den Bereichen
Open Source Systems Management
und Open Source Datacenter
Infrastructure

6. www.netways.de // blog.netways.de // @netways
Make IT do more with less
NETWAYS KOMPETENZEN
• Monitoring & Reporting
• Configuration Management
• Service Management
• Knowledge Management
• Backup & Recovery
• High Availability & Clustering
• Cloud Computing
• Load Balancing
• Virtualization
• Database Management
OPEN SOURCE
SYSTEMS MANAGEMENT
OPEN SOURCE
DATA CENTER
MANAGED SERVICES MONITORING HARDWARE KONFERENZEN

7. www.netways.de // blog.netways.de // @netways
Make IT do more with less
Open Source Data Center Conference
• 09. – 10. April 2014 Berlin
• 120 Teilnehmer (2013)
• 2 Tracks mit Vorträgen & Workshops
Puppet Camp 2014
• 11. April 2014 Berlin
• 170 Teilnehmer (November 2013)
NETWAYS KONFERENZEN

8. www.netways.de // blog.netways.de // @netways
Make IT do more with less
Puppet Fundamentals Schulung (3 Tage)
• Nächster Termin: 21. - 23. Januar 2014
• Ort: Nürnberg
Logstash Schulung (2 Tage)
• Nächster Termin: 03. - 04. Februar 2014
• Ort: Nürnberg
NETWAYS SCHULUNGEN

9. www.netways.de // blog.netways.de // @netways
Make IT do more with less
www.netways.org
• NETWAYS Addons
• NETWAYS Plugins
www.icinga.org
• Development
• Hosting
www.monitoringexchange.org
• Icinga / Nagios Addons und Plugins
• ~2000 Projekte
COMMUNITYARBEIT

10. www.netways.de // blog.netways.de // @netways
Make IT do more with less
UNSERE KUNDEN (AUSZUG)

11. www.netways.de // blog.netways.de // @netways
Make IT do more with less
UNSERE LEISTUNGEN IM ÜBERBLICK
• Workshops & Consulting zur Implementierung vor Ort
• Betrieb
• Komplette Monitoringsysteme
• Satellitensysteme
• Entwicklungsleistungen
• Plugins
• Systemintegration
• Schulungen
• Standardisierte Schulungsmodule
• Individuell vor Ort
• Support
• Standardverträge
• Individuelle Supportkonzepte
• Konferenzen

12. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH

13. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH
• Ziele
• Logs in großen Mengen behandeln
• Analyse der Daten

14. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH
• Problem
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 6666 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 492955 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:10 +0100] "GET /app/partials/inspector.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 17140 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 498624 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/micropanel.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/histogram/styleEditor.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/editor.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/histogram/queriesEditor.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/histogram/editor.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/pagination.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/partials/querySelect.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:42 +0100] "POST /_all/_search HTTP/1.1" 200 667 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0„
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 6666 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 492955 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:10 +0100] "GET /app/partials/inspector.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 17140 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 498624 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/micropanel.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/histogram/styleEditor.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/editor.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/histogram/queriesEditor.html HTTP/1.1" 304 245
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 17140 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 498624 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/micropanel.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 17140 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 498624 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
• 91.198.2.65 - logstash [04/Dec/2013:08:53:11 +0100] "GET /app/panels/table/micropanel.html HTTP/1.1" 304 245 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101
• 91.198.2.65 - - [04/Dec/2013:08:53:10 +0100] "POST /_all/_search HTTP/1.1" 200 17140 "https://logstash.demo.netways.de/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"

15. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH
• Regex Magic
• ^([d.]+) (S+) (S+) [([w:/]+s[+-
]d{4})] "(.+?)" (d{3}) (d+) "([^"]+)"
"([^"]+)"
• Pro Logformat eine eigene Regex
• Aufwendig und zeitintensiv
• Nur für technisch versierte
• Schlecht für die Teamarbeit

16. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH

17. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH
• Zentrales Speichern von Logs und Events
• Sammeln von verschiedenen Quellen
• Logs verändern / strukturieren
• Transportieren der Logs
• Einfache Installation
• Integration in bestehende Infrastruktur
• Lightweight
• Open Source

18. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR

19. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR
Logstash:
Pipe on Steroids
Input
Filter
Output

20. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR
• Vorher
• 192.168.1.10 – guest [04/Dec/2013:08:54:23 +0100] "POST /icinga-
web/web/api/json HTTP/1.1" 200 788 "https://icinga-
private.demo.netways.de/icinga-web/modules/web/portal" "Mozilla/5.0 (X11;
Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0"
• Nachher
• "http_clientip": "192.168.1.10",
• "http_ident": "-",
• "http_auth": "guest",
• "timestamp": "04/Dec/2013:08:54:23 +0100",
• "http_verb": "POST",
• "http_request": "/icinga-web/web/api/json",
• "http_httpversion": "1.1",
• "http_response": "200",
• "http_bytes": "788",
• "http_referrer": "https://icinga-private.demo.netways.de/icinga-
web/modules/web/portal",
• "http_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:22.0) Gecko/20100101
Firefox/22.0"
• Pattern
• "%{COMBINEDAPACHELOG}"

21. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR
Server
Switch
Webapplikation
• Input
• Output
• Key-Value
Store
• Puffer
• Input
• Filter
• Output
• Index
• Queries

22. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR
• Redis
• Key-Value Store
• Einfache Installation
• Replikation
• Sehr schnell
• Schreibt auf die Festplatte wenn der Speicher voll
ist
• Einträge werden nach Abholug gelöscht

23. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR
• Elasticsearch
• Java
• Basiert auf Apache Lucene
• Effizientes Speichern von Daten (JSON)
• RESTful API
• Clusterfähig (Nodes, Shards, Replika)
• Realtime Analyse der Daten
• Empfohlener Output

24. www.netways.de // blog.netways.de // @netways
Make IT do more with less
INTEGRATION IN DIE IT-LANDSCHAFT

25. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LOGSTASH ARCHITEKTUR
drupal_dblog
elasticsearch
eventlog
exec
file
ganglia
gelf
gemfire
generator
graphite
heroku
imap
irc
log4j
lumberjack
pipe
rabbitmq
redis
relp
s3
snmptrap
sqlite
sqs
stdin
stomp
syslog
tcp
twitter
udp
unix
varnishlog
websocket
wmi
xmpp
zenoss
zeromq
advisor
alter
anonymize
checksum
cidr
cipher
clone
collate
csv
date
dns
drop
environment
extractnumbers
gelfify
geoip
grep
grok
grokdiscovery
json
json_encode
kv
metaevent
metrics
multiline
mutate
noop
prune
railsparallelrequest
range
ruby
sleep
split
syslog_pri
translate
urldecode
useragent
uuid
xml
zeromq
boundary
circonus
cloudwatch
datadog
datadog_metrics
elasticsearch
elasticsearch_http
elasticsearch_river
email
exec
file
ganglia
gelf
gemfire
google_cloud_storage
graphite
graphtastic
hipchat
http
irc
jira
juggernaut
librato
loggly
lumberjack
metriccatcher
mongodb
nagios
nagios_nsca
null
opentsdb
pagerduty
pipe
rabbitmq
redis
riak
riemann
s3
sns
sqs
statsd
stdout
stomp
syslog
tcp
udp
websocket
xmpp
zabbix
zeromq
Input (36) Filter (40) Output (50)

26. www.netways.de // blog.netways.de // @netways
Make IT do more with less
INTEGRATION IN DIE IT-LANDSCHAFT
• Integration in nahezu jede Umgebungen
• Syslog
• Beaver (Python)
• Woodchuck (Ruby)
• Awesant (Perl)
• Logstash-Forwarder (ehem. Lumberjack) (Go)
• Snare (Windows Eventlog) (C)
• Node-logstash (NodeJS)
• Skalierbarkeit
• Auslagerung auf dedizierte Server
• Multiple Redis-Server
• Mehrere Logstash-Indexer möglich
• Elasticsearch ist Clusterfähig (Nodes, Shards, Replika)

27. www.netways.de // blog.netways.de // @netways
Make IT do more with less
ANWENDUNGSMÖGLICHKEITEN

28. www.netways.de // blog.netways.de // @netways
Make IT do more with less
ANWENDUNGSMÖGLICHKEITEN
• Systemlogs (Problemanalyse)
• Webserver Access- und Errorlogs
• Logs der eigenen Applikation (Profiling)
• Alerts via Icinga, E-Mail, XMPP, …
• Statistiken (Graphite)
• …

29. www.netways.de // blog.netways.de // @netways
Make IT do more with less
KIBANA

30. www.netways.de // blog.netways.de // @netways
Make IT do more with less
ANWENDUNGSMÖGLICHKEITEN
• Integriert in Logstash
• Javascript
• Alles wird vom Browser ausgeführt
• Queries direkt an Elasticsearch
• Visuelle Darstellung

31. www.netways.de // blog.netways.de // @netways
Make IT do more with less
LIVE DEMO

32. www.netways.de // blog.netways.de // @netways
Make IT do more with less
FRAGEN & ANTWORTEN

33. www.netways.de // blog.netways.de // @netways
Make IT do more with less
NETWAYS GmbH
Deutschherrnstrasse 15-19
90429 Nürmberg
Tel: +49 911 92885-0
Fax: +49 911 92885-77
Email: info@netways.de
Website: www.netways.de
Twitter: twitter.com/netways
Facebook: facebook.com/netways
Blog: blog.netways.de
FRAGEN & ANTWORTEN
?