Fluentd for Small Projects @ Fluentd Casual 20120518
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Fluentd for Small Projects @ Fluentd Casual 20120518

on

  • 5,518 views

 

Statistics

Views

Total Views
5,518
Views on SlideShare
3,673
Embed Views
1,845

Actions

Likes
17
Downloads
45
Comments
0

10 Embeds 1,845

http://www.socialtoprunners.jp 773
http://okochang.hatenablog.jp 625
http://www.socialtoprunners.com 273
https://twitter.com 122
http://webcache.googleusercontent.com 14
http://feedplanets.com 12
http://us-w1.rockmelt.com 11
http://www35.jimdo.com 8
http://socialtoprunners.jimdo.com 6
http://translate.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Fluentd for Small Projects @ Fluentd Casual 20120518 Presentation Transcript

  • 1. Fluentd ForSmall Projects Structured logging Pluggable architecture Reliable forwardingKazuki OhtaTreasure Data, Inc.@kzk_mover
  • 2. Self-Introduction• Kazuki Ohta > twitter: @kzk_mover > github: kzk• Treasure Data, Inc. > Chief Technology Officer; Founder > Original Fluentd Author @frsyuki is another co-founder.• Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > td-agent package manager
  • 3. Fluentd∼
  • 4. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Storage record
  • 5. What’s Fluentd? Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • 6. What’s Fluentd?syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • 7. Fluentd is Casual...? @tagomoris @hotchpotch 15,000,000 UU @ Cookpad∼80,000 msgs/sec
  • 8. Yes. Fluentd isCASUAL
  • 9. Fluentd Solves Daily Problems• Operation Problems > ssh root@bad-node && tail -f log.txt > cap deploy && pray > How many PV/UUs we have now? > How many users which did XXX and YYY?• Support > User A claimed that he bought 10 items, could you confirm that?
  • 10. But Your Boss Might Say....• 「Log... For What?」• 「What’s Fluentd?」• 「OK, Who uses It?」• 「We have syslogd」
  • 11. Start Fluentd, Silently...• Log Everything As JSON! > TSV/CSV logs are the nightmare.• Install td-agent > rpm/deb/chef/puppet etc.• tail plugin + MongoDB > tailing existing files. > no need to change existing systems.
  • 12. Figure 1: Common Log Operations Analytics Error Notify Recommendation
  • 13. Figure 2: Types of Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others
  • 14. 5 ways to convince your boss to use Fluentd
  • 15. 1) Support System• All logs are stored in MongoDB• No need to write ad-hoc log analysis scripts. before after#!/usr/bin/ruby $ mongo i-hate-mongoopen( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345}) ls = l.strip.split user = ls[4] puts l if user == 12345} $ ruby a.rb > foo.txt
  • 16. 2) Realtime System Stats SCREENSHOT
  • 17. 2) Realtime System Stats• Librato Metrics > Realtime Stats > Realtime Alert > Graphite+Nagios as a Service > GrowthForecast+Nagios http://d.hatena.ne.jp/tagomoris/20120330/1333077746• Important Metrics.... > queue depth, # of concurrent users, # of sql queries, # of slow queries, etc.
  • 18. 3) Error Log Collection# waf.rb App Fluentd MongoDBbegin WAF app.run ERROR!rescue => e Fluent::Logger.post("error", {"e"=>e.to_s})end$ mongo db> db.error.find() # Recent 10 errors
  • 19. 4) Error Notifications • “Semi-Realtime” Error Notifications • IRC, HipChat, Campfire, PagerDuty, AirBrake, Skype, etc. TD s Internal Chat <match app.error> type campfire room DevOpsRoom </match>
  • 20. 5) Log Analysis MongoDB• MongoDB App Fluentd $ mongo db TD > db.logs.distinct( uid ).count()• Treasure Data $ td query SELECT count(distinct uid) FROM logs
  • 21. Conclusion.Fluentd is for... • Support System • Realtime System Stats • Realtime Alert • Error Notifications • Log Analysis • ...
  • 22. #3
  • 23. Github Issues #3
  • 24. Fluentdconveysmessage
  • 25. 伝書鳩