Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we use Fluentd in Treasure Data

1,548 views

Published on

Talk at Fluentd meetup at Slideshare
http://www.meetup.com/Fluentd-User-Group/events/126799482/

Published in: Technology
  • Be the first to comment

How we use Fluentd in Treasure Data

  1. 1. Sadayuki Furuhashi Treasuare Data, Inc. Founder & Software Architect How we use Fluentd in Treasure Data for system metrics A creator of Fluentd
  2. 2. Self-introduction > Sadayuki Furuhashi > twitter/github: frsyuki > Treasure Data, Inc. Founder & Software Architect > Open source projects MessagePack - efficient serializer (original author) Fluentd - event collector (original author)
  3. 3. Metrics collection from Rails apps fluentd Rails app fluentd fluentd Rails app MetricSense for Ruby aggregate logs using out_forward plugin local fluentd collects logs and buffer them in local disk
  4. 4. class ApplicationController before_filter :setup_metrics after_filter :post_metrics def setup_metrics MetricSense.clear! MetricSense.tag "#{params[:controller]}_#{params[:action]}" if current_user.present? MetricSense.segment( :account=>current_user.account_id, :agent=>request.env[‘HTTP_AGENT’]) end end def post_metrics MetricSense.measure! end # in application code: # MetricSense.value :size=>10
  5. 5. Metrics collection from Hadoop fluentd Hadoop Cluster Hadoop Clusterhadoop metrics plugin (original plugin) ✓ number of failed tasks ✓ number of used slots ✓ blacklisted nodes ✓ memory usage of JobTracker ✓ memory usage of NameNode/DataNode ✓ HDFS put/get latency etc...
  6. 6. Metrics collection from DB fluentd script in_exec plugin runs a script every 1 hour runs SQL statements ✓ users’ activity and stats ✓ job queue length (PerfectQueue stats) etc...
  7. 7. Metrics collection from Hadoop fluentd Rails app fluentd Rails app fluentd Hadoop JobTracker Hadoop JobTracker script
  8. 8. Metrics utilization fluentd Librato Metrics for realtime analysis Treasure Data for historical analysis Pager Duty for alerts out_tdlogout_metricsense
  9. 9. Next version of Fluentd > <filter> > <match> in <source> > <label> for error records > New process model & Live restart
  10. 10. <source> type tail path /var/log/httpd.log format apache tag not_filtered.apache </source> <match not_filetered.**> type rewrite remove_prefix not_filtered <rule> key status pattern ^500$ ignore true </rule> </match> <match **> type forward host log.server </match> Before Mysterious tag tag operations
  11. 11. <source> type tail path /var/log/httpd.log format apache tag apache </source> <filter **> type rewrite <rule> key status pattern ^500$ ignore true </rule> </filter> <match **> type forward host log.server </match> After (v11) Filter plugins!
  12. 12. <source> type tail path /var/log/httpd.log format apache tag apache <filter **> type rewrite <rule> key status pattern ^500$ ignore true </rule> </filter> </source> <match **> type forward host log.server </match> After (v11) <filter>/<match> in <source>
  13. 13. <source> type tail path /var/log/httpd.log tag apache </source> <match **> type forward host log.server </match> Before I want to add flowcounter here...
  14. 14. <source> type tail path /var/log/httpd.log tag apache </source> <match flow.traffic> type forward host traffic.server </match> <match **> type copy <store> type flowcounter tag flow.traffic </store> <store> type forward host log.server </store> </match> Before Nested!
  15. 15. <source> type tail path /var/log/httpd.log tag apache </source> <filter **> type copy <match> type flowcounter tag flow.traffic <match> type forward host traffic.server </match> </match> </filter> <match **> type forward host log.server </match> After (v11) Filtering pipeline
  16. 16. <source> type forward </source> <filter **> type copy <match> type file path /mnt/local_archive </match> </filter> <label alert> <match **> ... </match> </label> <label analysis> ... </label> # copy & label & forward <filter **> type copy <match> type forward label alert host alerting.server </match> </filter> # copy & label & forward <filter **> type copy <match> type forward label analysis host analysis.server </match> </filter> After (v11)

×