Logstash - CeBIT 2014 - Open Source Forum

  • 1,925 views
Uploaded on

Presentation @ Open Source Forum CeBIT 2014. Including introduction into Redis, Logstash, Elasticsearch

Presentation @ Open Source Forum CeBIT 2014. Including introduction into Redis, Logstash, Elasticsearch

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,925
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
33
Comments
0
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 CEBIT 2014 – 12.03.2014 LOG- UND EVENTMANAGEMENT MIT LOGSTASH BERND ERK | NETWAYS GMBH
  • 2. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 AGENDA ■ Kurzvorstellung ■ Einführung ■ Architektur ■ Installation ■ Routing und Filterung von Events ■ Interfaces & API ■ Integration in Nagios und Icinga ■ Fragen & Antworten
  • 3. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KURZVORSTELLUNG
  • 4. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KURZVORSTELLUNG NETWAYS • Firmengründung 1995 • Open Source seit 1997 • 40 Mitarbeiter • Spezialisierung in den Bereichen Open Source Systems Management und Open Source Datacenter Infrastructure http://jobs.netways.de
  • 5. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 NETWAYS KOMPETENZEN • Monitoring & Reporting • Configuration Management • Service Management • Knowledge Management • Backup & Recovery • High Availability & Clustering • Cloud Computing • Load Balancing • Virtualization • Database Management OPEN SOURCE SYSTEMS MANAGEMENT OPEN SOURCE DATA CENTER MANAGED SERVICES MONITORING HARDWARE KONFERENZEN
  • 6. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 Open Source Datacenter Conference • 08. – 10. April 2014 • Datacenter | Automation | DevOps PuppetCamp 2014 • 11. April Berlin Open Source Monitoring Conference • 11. April Berlin NETWAYS KONFERENZEN
  • 7. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 EINFÜHRUNG
  • 8. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGS Logs -> Fluss an unstrukturierten Daten Oct 4 16:57:24 web sshd[25828]: Received disconnect from 10.10.0.31: 11: disconnected by user bestehend aus Timestamp und Message
  • 9. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 EVENTS Event -> Fluss an strukturierten Daten Event { Time: Oct 4 16:57:24 Process: sshd State: Received disconnect from 10.10.0.31 Client: 10.10.0.31 } bestehend aus konkreten Attributen
  • 10. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOG & EVENTMANAGEMENT Logs > Event > Analyse (Korrelation) > Aktion
  • 11. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 TOOLS ■ Nagios & Icinga Addons • check_logfiles • NagTrap • EventDB • EDBC ■ Logmanagement-Tools • Graylog • Fluentd • Logstash
  • 12. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGSTASH Logstash
  • 13. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ARCHITEKTUR & INSTALLATION
  • 14. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGSTASH ■ Logmanagement auf Basis von JRuby ■ Konfigurierbare “Pipe” ■ Flexible Plugin-Architektur für • Input • Filter • Output ■ Standardplugins für alle gängige Protokolle ■ Webinterface ■ Single File Deployment
  • 15. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 Outputs • amqp • boundary • circonus • cloudwatch • datadog • datadog_metrics • elasticsearch • elasticsearch_http • elasticsearch_river • email • exec • file • ganglia • gelf • gemfire • google_cloud_storage • graphite • graphtastic • hipchat LOGSTASH - IO Inputs • amqp • drupal_dblog • elasticsearch • eventlog • exec • file • ganglia • gelf • gemfire • generator • graphite • heroku • imap • irc • log4j • lumberjack • pipe • rabbitmq • redis • relp • s3 • snmptrap • sqlite • sqs • stdin • stomp • syslog • tcp • twitter • udp • unix • varnishlog • websocket • wmi • xmpp • zenoss • zeromq • http • irc • jira • juggernaut • librato • loggly • lumberjack • metriccatcher • mongodb • nagios • nagios_nsca • null • opentsdb • pagerduty • pipe • rabbitmq • redis • riak • riemann • s3 • sns • sqs • statsd • stdout • stomp • syslog • tcp • udp • websocket • xmpp • zabbix • zeromq
  • 16. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION - LOGSTASH ■ Download - http://logstash.net ■ java -jar logstash-x.x.x-flatjar.jar agent -f <config-file>
  • 17. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ARCHITEKTUR Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 18. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 REDIS ■ NoSQL in memory auf Basis von C ■ Unterstützung verschiedener “Datentypen” • strings • hashes • lists • sets and sorted sets ■ Support für verschiedene Replikationsszenarien ■ SAUSCHNELL $ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q SET: 122556.53 requests per second GET: 123601.76 requests per second LPUSH: 136752.14 requests per second LPOP: 132424.03 requests per second
  • 19. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION - REDIS ■ Download - http://redis.io/download ■ make ■ make test ■ make install ■ /usr/local/bin/redis-server
  • 20. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ELASTICSEARCH ■ Schemafreier RESTful Suchserver auf Basis von Java ■ Basierend auf Lucene Core ■ “Vergleichbar” mit Apache Solr ■ Verteilte Architektur durch • Shards • Replicas • Gateways ■ Realtime-Suche als Basis für Kibana
  • 21. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION - ELASTICSEARCH ■ Download – http://elasticsearch.org/download/ ■ Entpacken des Archives ■ Ausführung von bin/elasticsearch
  • 22. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ROUTING UND FILTERUNG VON EVENTS
  • 23. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ÜBERSICHT Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 24. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH - SHIPPER ■ Übermittlung von Logs an Logstash • Logstash • Lumberjack • Syslog • Log4J • Gelf • File-Read • u.v.a.m.
  • 25. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH - SHIPPER ■ Konfiguration input { file { path => "/root/osmc/demodata/access.log.1” type => "apache-access" } } output { stdout { debug => true } redis { host => "127.0.0.1" data_type => "list" key => "logstash.apache" } } ■ java -jar logstash-current.jar agent -f logstash_shipper.conf Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 26. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH - INDEXER ■ Konfiguration input { redis { host => "127.0.0.1" type => "redis-input" # these settings should match the output of the agent data_type => "list" key => "logstash.apache” } } output { stdout { debug => true } elasticsearch_http { host => "127.0.0.1" } } Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 27. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - APACHE ■ Konfiguration für Apache-Logs input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event" } } filter { if [type] == "apache-access" { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } } } output { elasticsearch_http { host => "127.0.0.1” } } Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 28. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - GEOIP ■ Konfiguration für Geo-Daten input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” } } filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG}" } geoip { source => "clientip" add_tag => ["geotag"] } } output { elasticsearch_http {host => "127.0.0.1”} } Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 29. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INTERFACES & API
  • 30. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KIBANA Kibana
  • 31. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KIBANA
  • 32. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ELASTICHQ
  • 33. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KIBANA - DEMO DEMO
  • 34. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INTEGRATION NAGIOS UND ICINGA
  • 35. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 REALTIME LOGANALYSE ■ Analyse verschiedener Quellen in Realtime ■ Prüfung auf Patterns und States • Facilitites • Regex • Programs ■ Übermittlung als Passiver Event
  • 36. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ÜBERSICHT LOGSTASH UND ICINGA Search & Storage WebinterfaceIndexer Icinga –WebIcinga - Commandpipe
  • 37. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - ICINGA ■ Konfiguration für Icinga-Alert input { … } filter { if [type] == "syslog" { grok {match => [ "message", "%{SYSLOGBASE}" ] } grep { match => [ "message", "Error" ] drop => false add_tag => "nagios-update" add_field => [ # "nagios_host", "%{@source_host}", "nagios_host", "localhost", "nagios_service", "Logstash", "nagios_level", "2”] }}} output { elasticsearch {host => "127.0.0.1”} nagios { commandfile => "/var/lib/icinga/rw/icinga.cmd" }} Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer
  • 38. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 LOGSTASH – ICINGA - DEMO DEMO
  • 39. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ZUGABE
  • 40. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 REALTIME GRAPHING
  • 41. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 STATSD & GRAPHITE ■ StatsD • Netzwerkdaemon auf Basis von UDP • Bucket -> Value -> Flush • Entkoppelte Zwischenaggretion für Statisik ■ Graphite • Graphing-Framework bestehend aus • Whisper (Datenbank) • Carbon (Engine) • Graphite-Web (Interface)
  • 42. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – STATSD - NODEJS ■ apt-get install make python g++ checkinstall ■ mkdir nodejs && cd nodejs ■ wget -N http://nodejs.org/dist/node-latest.tar.gz ■ tar xzvf node-latest.tar.gz && cd `ls -rd node-v*` ■ checkinstall
  • 43. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – STATSD ■ wget https://github.com/etsy/statsd/archive/master.zip ■ unzip master.zip ■ node stats.js config.js
  • 44. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 MONITORING - STATSD ■ Status Informationen • echo stats | nc 127.0.0.1 8126 • echo health | nc 127.0.0.1 8126 ■ Timer- und Counterinfo • echo counters | nc 127.0.0.1 8126 • echo timers| nc 127.0.0.1 8126
  • 45. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – GRAPHITE ■ Download der Sources • git clone https://github.com/graphite- project/graphite-web.git • git clone https://github.com/graphite- project/carbon.git • git clone https://github.com/graphite- project/whisper.git
  • 46. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – GRAPHITE ■ Installation Whisper pushd whisper sudo python setup.py install popd ■ Installation Carbon pushd carbon sudo python setup.py install popd ■ Konfiguration Carbon pushd /opt/graphite/conf cp carbon.conf.example carbon.conf cp storage-schemas.conf.example storage-schemas.conf
  • 47. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 INSTALLATION – GRAPHITE - WEBAPP ■ Check Dependencies Graphite webapp pushd graphite-web python check-dependencies.py popd ■ Installation Graphite webapp pushd graphite-web python setup.py install popd ■ Konfiguration Apache example-graphite-vhost.conf
  • 48. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 ÜBERSICHT STATSD UND GRAPHITE Search & Storage WebinterfaceIndexer GraphiteStatsd
  • 49. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 KONFIGURATION - LOGSTASH – INDEXER - STATSD ■ Konfiguration für Statsd input { redis { host => "127.0.0.1" type => "apache-access” data_type => "list" key => "logstash.apache” format => "json_event” add_field=> ["sitename","www.icinga.org"] } } filter { if [type] == "apache-access" { grok {match => [ "message", "%{COMBINEDAPACHELOG}" ] } }} output { stdout { debug => true } if [type] == "apache-access" { statsd { host => "localhost" port => 8125 namespace => "logstash" debug => false increment => "apache.%{sitename}.response.%{response}” count => ["apache.%{sitename}.bytes", "%{bytes}"] } } elasticsearch_http {host => "127.0.0.1”}} Shipper Shipper Shipper Broker Search & Storage WebinterfaceIndexer StatsD
  • 50. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 GRAPHITE - DEMO DEMO
  • 51. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 FRAGEN & ANTWORTEN
  • 52. www.netways.de // blog.netways.de // @netways We love Open Source#CeBIT 2014 - Halle 6 / E16 / 310 NETWAYS GmbH Deutschherrnstrasse 15-19 90429 Nürnberg Tel: +49 911 92885-0 Fax: +49 911 92885-77 Email: info@netways.de Website: www.netways.de Twitter: twitter.com/netways Facebook: facebook.com/netways Blog: blog.netways.de FRAGEN & ANTWORTEN DANKE