Logstash - CeBIT 2014 - Open Source Forum
Upcoming SlideShare
Loading in...5
×
 

Logstash - CeBIT 2014 - Open Source Forum

on

  • 1,821 views

Presentation @ Open Source Forum CeBIT 2014. Including introduction into Redis, Logstash, Elasticsearch

Presentation @ Open Source Forum CeBIT 2014. Including introduction into Redis, Logstash, Elasticsearch

Statistics

Views

Total Views
1,821
Views on SlideShare
1,787
Embed Views
34

Actions

Likes
4
Downloads
30
Comments
0

2 Embeds 34

https://twitter.com 33
http://www.slideee.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Logstash - CeBIT 2014 - Open Source Forum Logstash - CeBIT 2014 - Open Source Forum Presentation Transcript

  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 CEBIT 2014 – 12.03.2014 LOG- UND EVENTMANAGEMENT MIT LOGSTASH BERND ERK | NETWAYS GMBH
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 AGENDA ■ Kurzvorstellung ■ Einführung ■ Architektur ■ Installation ■ Routing und Filterung von Events ■ Interfaces & API ■ Integration in Nagios und Icinga ■ Fragen & Antworten
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KURZVORSTELLUNG
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KURZVORSTELLUNG NETWAYS • Firmengründung 1995 • Open Source seit 1997 • 40 Mitarbeiter • Spezialisierung in den Bereichen Open Source Systems Management und Open Source Datacenter Infrastructure http://jobs.netways.de
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 NETWAYS KOMPETENZEN • Monitoring & Reporting • Configuration Management • Service Management • Knowledge Management • Backup & Recovery • High Availability & Clustering • Cloud Computing • Load Balancing • Virtualization • Database Management OPEN SOURCE SYSTEMS MANAGEMENT OPEN SOURCE DATA CENTER MANAGED SERVICES MONITORING HARDWARE KONFERENZEN
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 Open Source Datacenter Conference • 08. – 10. April 2014 • Datacenter | Automation | DevOps PuppetCamp 2014 • 11. April Berlin Open Source Monitoring Conference • 11. April Berlin NETWAYS KONFERENZEN
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 EINFÜHRUNG
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGS Logs -> Fluss an unstrukturierten Daten Oct 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11: disconnected by user bestehend aus Timestamp und Message
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 EVENTS Event -> Fluss an strukturierten Daten Event { Time: Oct 4 16:57:24 Process: sshd State: Received disconnect from 10.10.0.31 Client: 10.10.0.31 } bestehend aus konkreten Attributen
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOG & EVENTMANAGEMENT Logs > Event > Analyse (Korrelation) > Aktion
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 TOOLS ■ Nagios & Icinga Addons • check_logfiles • NagTrap • EventDB • EDBC ■ Logmanagement-Tools • Graylog • Fluentd • Logstash
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGSTASH Logstash
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ARCHITEKTUR & INSTALLATION
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGSTASH ■ Logmanagement auf Basis von JRuby ■ Konfigurierbare “Pipe” ■ Flexible Plugin-Architektur für • Input • Filter • Output ■ Standardplugins für alle gängige Protokolle ■ Webinterface ■ Single File Deployment
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 Outputs • amqp • boundary • circonus • cloudwatch • datadog • datadog_metrics • elasticsearch • elasticsearch_http • elasticsearch_river • email • exec • file • ganglia • gelf • gemfire • google_cloud_storage • graphite • graphtastic • hipchat LOGSTASH - IO Inputs • amqp • drupal_dblog • elasticsearch • eventlog • exec • file • ganglia • gelf • gemfire • generator • graphite • heroku • imap • irc • log4j • lumberjack • pipe • rabbitmq • redis • relp • s3 • snmptrap • sqlite • sqs • stdin • stomp • syslog • tcp • twitter • udp • unix • varnishlog • websocket • wmi • xmpp • zenoss • zeromq • http • irc • jira • juggernaut • librato • loggly • lumberjack • metriccatcher • mongodb • nagios • nagios_nsca • null • opentsdb • pagerduty • pipe • rabbitmq • redis • riak • riemann • s3 • sns • sqs • statsd • stdout • stomp • syslog • tcp • udp • websocket • xmpp • zabbix • zeromq
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION - LOGSTASH ■ Download - http://logstash.net ■ java -jar logstash-x.x.x-flatjar.jar agent -f <config-file>
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ARCHITEKTUR Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 REDIS ■ NoSQL in memory auf Basis von C ■ Unterstützung verschiedener “Datentypen” • strings • hashes • lists • sets and sorted sets ■ Support für verschiedene Replikationsszenarien ■ SAUSCHNELL $ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q SET: 122556.53 requests per second GET: 123601.76 requests per second LPUSH: 136752.14 requests per second LPOP: 132424.03 requests per second
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION - REDIS ■ Download - http://redis.io/download ■ make ■ make test ■ make install ■ /usr/local/bin/redis-server
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ELASTICSEARCH ■ Schemafreier RESTful Suchserver auf Basis von Java ■ Basierend auf Lucene Core ■ “Vergleichbar” mit Apache Solr ■ Verteilte Architektur durch • Shards • Replicas • Gateways ■ Realtime-Suche als Basis für Kibana
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION - ELASTICSEARCH ■ Download – http://elasticsearch.org/download/ ■ Entpacken des Archives ■ Ausführung von bin/elasticsearch
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ROUTING UND FILTERUNG VON EVENTS
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ÜBERSICHT Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH - SHIPPER ■ Übermittlung von Logs an Logstash • Logstash • Lumberjack • Syslog • Log4J • Gelf • File-Read • u.v.a.m.
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH - SHIPPER ■ Konfiguration input { file { path => "/root/osmc/demodata/access.log.1” type => "apache-access" } } output { stdout { debug => true } redis { host => "127.0.0.1" data_type => "list" key => "logstash.apache" } } ■ java -jar logstash-current.jar agent -f logstash_shipper.conf Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH - INDEXER ■ Konfiguration input { redis { host => "127.0.0.1" type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache” } } output { stdout { debug => true } elasticsearch_http { host => "127.0.0.1" } } Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - APACHE ■ Konfiguration für Apache-Logs input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event" } } filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } } } output { elasticsearch_http { host => "127.0.0.1” } } Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - GEOIP ■ Konfiguration für Geo-Daten input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” } } filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG}" } geoip { source => "clientip" add_tag => ["geotag"] } } output { elasticsearch_http {host => "127.0.0.1”} } Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INTERFACES & API
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KIBANA Kibana
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KIBANA
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ELASTICHQ
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KIBANA - DEMO DEMO
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INTEGRATION NAGIOS UND ICINGA
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 REALTIME LOGANALYSE ■ Analyse verschiedener Quellen in Realtime ■ Prüfung auf Patterns und States • Facilitites • Regex • Programs ■ Übermittlung als Passiver Event
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ÜBERSICHT LOGSTASH UND ICINGA Search & Storage WebinterfaceIndexer Icinga –WebIcinga - Commandpipe
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - ICINGA ■ Konfiguration für Icinga-Alert input { … } filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE}" ] } grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host}", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2”] }}} output { elasticsearch {host => "127.0.0.1”} nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd" }} Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGSTASH – ICINGA - DEMO DEMO
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ZUGABE
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 REALTIME GRAPHING
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 STATSD & GRAPHITE ■ StatsD • Netzwerkdaemon auf Basis von UDP • Bucket -> Value -> Flush • Entkoppelte Zwischenaggretion für Statisik ■ Graphite • Graphing-Framework bestehend aus • Whisper (Datenbank) • Carbon (Engine) • Graphite-Web (Interface)
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – STATSD - NODEJS ■ apt-get install make python g++ checkinstall ■ mkdir nodejs && cd nodejs ■ wget -N http://nodejs.org/dist/node-latest.tar.gz ■ tar xzvf node-latest.tar.gz && cd `ls -rd node-v*` ■ checkinstall
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – STATSD ■ wget https://github.com/etsy/statsd/archive/master.zip ■ unzip master.zip ■ node stats.js config.js
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 MONITORING - STATSD ■ Status Informationen • echo stats | nc 127.0.0.1 8126 • echo health | nc 127.0.0.1 8126 ■ Timer- und Counterinfo • echo counters | nc 127.0.0.1 8126 • echo timers| nc 127.0.0.1 8126
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – GRAPHITE ■ Download der Sources • git clone https://github.com/graphite- project/graphite-web.git • git clone https://github.com/graphite- project/carbon.git • git clone https://github.com/graphite- project/whisper.git
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – GRAPHITE ■ Installation Whisper pushd whisper sudo python setup.py install popd ■ Installation Carbon pushd carbon sudo python setup.py install popd ■ Konfiguration Carbon pushd /opt/graphite/conf cp carbon.conf.example carbon.conf cp storage-schemas.conf.example storage-schemas.conf
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – GRAPHITE - WEBAPP ■ Check Dependencies Graphite webapp pushd graphite-web python check-dependencies.py popd ■ Installation Graphite webapp pushd graphite-web python setup.py install popd ■ Konfiguration Apache example-graphite-vhost.conf
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ÜBERSICHT STATSD UND GRAPHITE Search & Storage WebinterfaceIndexer GraphiteStatsd
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - STATSD ■ Konfiguration für Statsd input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event” add_field=> ["sitename","www.icinga.org"] } } filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG}" ] } }} output { stdout { debug => true } if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename}.response.%{response}” count => ["apache.%{sitename}.bytes", "%{bytes}"] } } elasticsearch_http {host => "127.0.0.1”}} Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer StatsD
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 GRAPHITE - DEMO DEMO
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 FRAGEN & ANTWORTEN
  • www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 NETWAYS GmbH Deutschherrnstrasse 15-19 90429 Nürnberg Tel: +49 911 92885-0 Fax: +49 911 92885-77 Email: info@netways.de Website: www.netways.de Twitter: twitter.com/netways Facebook: facebook.com/netways Blog: blog.netways.de FRAGEN & ANTWORTEN DANKE