• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Fluentd for Small Projects @ Fluentd Casual 20120518
 

Fluentd for Small Projects @ Fluentd Casual 20120518

on

  • 5,207 views

 

Statistics

Views

Total Views
5,207
Views on SlideShare
3,395
Embed Views
1,812

Actions

Likes
17
Downloads
45
Comments
0

10 Embeds 1,812

http://www.socialtoprunners.jp 773
http://okochang.hatenablog.jp 622
http://www.socialtoprunners.com 243
https://twitter.com 122
http://webcache.googleusercontent.com 14
http://feedplanets.com 12
http://us-w1.rockmelt.com 11
http://www35.jimdo.com 8
http://socialtoprunners.jimdo.com 6
http://translate.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Fluentd for Small Projects @ Fluentd Casual 20120518 Fluentd for Small Projects @ Fluentd Casual 20120518 Presentation Transcript

  • Fluentd ForSmall Projects Structured logging Pluggable architecture Reliable forwardingKazuki OhtaTreasure Data, Inc.@kzk_mover
  • Self-Introduction• Kazuki Ohta > twitter: @kzk_mover > github: kzk• Treasure Data, Inc. > Chief Technology Officer; Founder > Original Fluentd Author @frsyuki is another co-founder.• Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > td-agent package manager
  • Fluentd∼
  • What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Storage record
  • What’s Fluentd? Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • What’s Fluentd?syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • Fluentd is Casual...? @tagomoris @hotchpotch 15,000,000 UU @ Cookpad∼80,000 msgs/sec
  • Yes. Fluentd isCASUAL
  • Fluentd Solves Daily Problems• Operation Problems > ssh root@bad-node && tail -f log.txt > cap deploy && pray > How many PV/UUs we have now? > How many users which did XXX and YYY?• Support > User A claimed that he bought 10 items, could you confirm that?
  • But Your Boss Might Say....• 「Log... For What?」• 「What’s Fluentd?」• 「OK, Who uses It?」• 「We have syslogd」
  • Start Fluentd, Silently...• Log Everything As JSON! > TSV/CSV logs are the nightmare.• Install td-agent > rpm/deb/chef/puppet etc.• tail plugin + MongoDB > tailing existing files. > no need to change existing systems.
  • Figure 1: Common Log Operations Analytics Error Notify Recommendation
  • Figure 2: Types of Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others
  • 5 ways to convince your boss to use Fluentd
  • 1) Support System• All logs are stored in MongoDB• No need to write ad-hoc log analysis scripts. before after#!/usr/bin/ruby $ mongo i-hate-mongoopen( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345}) ls = l.strip.split user = ls[4] puts l if user == 12345} $ ruby a.rb > foo.txt
  • 2) Realtime System Stats SCREENSHOT
  • 2) Realtime System Stats• Librato Metrics > Realtime Stats > Realtime Alert > Graphite+Nagios as a Service > GrowthForecast+Nagios http://d.hatena.ne.jp/tagomoris/20120330/1333077746• Important Metrics.... > queue depth, # of concurrent users, # of sql queries, # of slow queries, etc.
  • 3) Error Log Collection# waf.rb App Fluentd MongoDBbegin WAF app.run ERROR!rescue => e Fluent::Logger.post("error", {"e"=>e.to_s})end$ mongo db> db.error.find() # Recent 10 errors
  • 4) Error Notifications • “Semi-Realtime” Error Notifications • IRC, HipChat, Campfire, PagerDuty, AirBrake, Skype, etc. TD s Internal Chat <match app.error> type campfire room DevOpsRoom </match>
  • 5) Log Analysis MongoDB• MongoDB App Fluentd $ mongo db TD > db.logs.distinct( uid ).count()• Treasure Data $ td query SELECT count(distinct uid) FROM logs
  • Conclusion.Fluentd is for... • Support System • Realtime System Stats • Realtime Alert • Error Notifications • Log Analysis • ...
  • #3
  • Github Issues #3
  • Fluentdconveysmessage
  • 伝書鳩