Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Centralized + Unified Logging

1,646 views

Published on

Vday 2014 , Fluentd Introduction

Published in: Technology
  • Login to see the comments

Centralized + Unified Logging

  1. 1. Centralized + Unified Logging Gabor Kozma / gabo@ustream.tv / @kozmag82
  2. 2. Everybody wants to write logs! ✓ Application Logs (frontend / backend) ➢ php, java, ruby, python, bash ✓ Access Logs ➢ apache, nginx, tomcat, jetty ✓ System Logs ➢ syslog, hardware error log ✓ Database Logs ➢ history, transaction
  3. 3. Centralized Logging
  4. 4. Central Logging Architecture ✓ Collection ➢ file, syslog, database ✓ Transport ➢ chukwa, heka, syslog, logstash, flume, fluentd, kafka, nsq, nxlog, other custom solution. Typical: syslog-ng, rsyslog ✓ Storage / Store ➢ Amazon S3, Glacier, NAS ...
  5. 5. Central Logging Architecture ✓ Analysis (You need a way to analyze them!) ➢ Apache Hadoop + HDFS + Map-Reduce jobs ■ Hive, Pig, HBase, Impala.... ➢ Elasticsearch + Graylog2 / Kibana ➢ MongoDB + Map-Reduce/Aggregation Framework ➢ Graphite, Statsd + Dashboards ✓ Alerting (Errors almost always indicate a problem!) ➢ Airbreak/Errbit, Sentry, Honeybadger, Nagios, Zabbix, Open/PagerDuty
  6. 6. Unified Logging Layer
  7. 7. Unified Logging Layer ✓ Ubiquity ➢ Various format problem ➢ Various source and destination ➢ You must be optimize most of use case! ✓ Rigidity vs. Flexibility ➢ Apache Thrift , Apache Avro, Protocol Buffer , JSON / BSON, MessagePack
  8. 8. Unified Logging Layer ✓ Reliability and Scalability ➢ Scalable ➢ Support retryable data transfer ➢ Sync / Async data transfer ➢ Push / Pull base system ✓ Extensibility ➢ Support new input / output ■ You don’t have to modify anything else.
  9. 9. Fluentd - Pluggable architecture ✓ Input, Output, Buffer, Parser, Formatter 300+ plugins
  10. 10. Fluentd - Minimum res. require ✓ Combination of C language and Ruby ✓ 1 node ✓ 30/40 Mbyte RAM ✓ 1 CPU core 13.000 event / sec
  11. 11. Fluentd - Built-in Reliability ✓ Buffer ➢ file or memory ✓ Retrying ✓ Error handling ➢ transaction, failover, secondary node support (heartbeat)
  12. 12. Fluentd - Event structure (log) ✓ Time ➢ Second unit ➢ From data source or adding parsed time ✓ Tag ➢ for message routing ✓ Record ➢ JSON format ■ MessagePack internally :) ■ none structured
  13. 13. Fluentd - Useful plugins ✓ Output ➢ stdout, file, forest, graphite, mongo, mysql, elasticsearch, splunk, null, s3, geoip, webhdfs ✓ Input ➢ syslog, tail, http, udp, tcp, scribe ✓ Buffer ➢ memory, file ✓ Formatter and/or Parser ➢ lstv, json, multiline
  14. 14. Examples
  15. 15. Fluentd - Examples <source> type tail format /^(?<host>[^ ]*):(?<port>[^ ]*) (?<ip>[^ ]*) (?<user>[^ ]*) (?<remotelog>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*) (?: "(?<referer>[^"]*)" ""(?<agent>[^"]*)"")?(?: "(?<referer>[^"]*)" "(?<agent> [^"]*)")?$/ path /var/log/apache2/other_vhosts_access.log.* pos_file /var/log/fluent/apache2.other_vhosts_access.log.pos time_format %d/%b/%Y:%H:%M:%S %z tag apache2.access.raw read_from_head true </source>
  16. 16. Fluentd - Examples <match apache2.*.raw> type record_reformer enable_ruby false renew_record false remove_keys remotelog tag ${tag_prefix[-2]}.reformed <record> hostname ${hostname} </record> </match> <match apache2.*.reformed> type geoip geoip_lookup_key ip geoip_database /usr/share/GeoIP/GeoIPCity.dat <record> geo_city ${city['ip']} ... geo_region ${region['ip']} </record> add_tag_suffix .geoip flush_interval 5s </match>
  17. 17. Fluentd - Examples <match apache2.access.reformed.geoip> type forward flush_interval 5s buffer_type file buffer_queue_limit 512 buffer_chunk_limit 100M buffer_path /opt/fluent/buffer/apache2/ <server> name hostname host xxx.xxx.xxx.xxx weight 10 </server> ... <server> name hostname host xxx.xxx.xxx.xxx standby </server> <secondary> type file path /var/log/fluent/forward-failed/ apache2/ </secondary> </match>
  18. 18. Fluentd - Examples <match apache2.access.**> type copy <store> type file path /opt/fluent/apache2/access time_format %Y%m%dT%H%M%S%z flush_interval 60s append true compress gzip utc num_threads 4 ... ... </store> <store> type datacounter ... </store> <store> type graphite ... </store> </match>
  19. 19. Fluentd - Testimonials
  20. 20. Questions? http://www.fluentd.org http://docs.fluentd.org/ http://fluentular.herokuapp.com/ https://rubygems.org/search?query=fluent-plugin-http:// msgpack.org/

×