Your SlideShare is downloading. ×
Fluentd for Small Projects @ Fluentd Casual 20120518
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Fluentd for Small Projects @ Fluentd Casual 20120518

5,269
views

Published on

Published in: Technology, Education

0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,269
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
56
Comments
0
Likes
19
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript

    • 1. Fluentd ForSmall Projects Structured logging Pluggable architecture Reliable forwardingKazuki OhtaTreasure Data, Inc.@kzk_mover
    • 2. Self-Introduction• Kazuki Ohta > twitter: @kzk_mover > github: kzk• Treasure Data, Inc. > Chief Technology Officer; Founder > Original Fluentd Author @frsyuki is another co-founder.• Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > td-agent package manager
    • 3. Fluentd∼
    • 4. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Storage record
    • 5. What’s Fluentd? Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
    • 6. What’s Fluentd?syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
    • 7. Fluentd is Casual...? @tagomoris @hotchpotch 15,000,000 UU @ Cookpad∼80,000 msgs/sec
    • 8. Yes. Fluentd isCASUAL
    • 9. Fluentd Solves Daily Problems• Operation Problems > ssh root@bad-node && tail -f log.txt > cap deploy && pray > How many PV/UUs we have now? > How many users which did XXX and YYY?• Support > User A claimed that he bought 10 items, could you confirm that?
    • 10. But Your Boss Might Say....• 「Log... For What?」• 「What’s Fluentd?」• 「OK, Who uses It?」• 「We have syslogd」
    • 11. Start Fluentd, Silently...• Log Everything As JSON! > TSV/CSV logs are the nightmare.• Install td-agent > rpm/deb/chef/puppet etc.• tail plugin + MongoDB > tailing existing files. > no need to change existing systems.
    • 12. Figure 1: Common Log Operations Analytics Error Notify Recommendation
    • 13. Figure 2: Types of Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others
    • 14. 5 ways to convince your boss to use Fluentd
    • 15. 1) Support System• All logs are stored in MongoDB• No need to write ad-hoc log analysis scripts. before after#!/usr/bin/ruby $ mongo i-hate-mongoopen( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345}) ls = l.strip.split user = ls[4] puts l if user == 12345} $ ruby a.rb > foo.txt
    • 16. 2) Realtime System Stats SCREENSHOT
    • 17. 2) Realtime System Stats• Librato Metrics > Realtime Stats > Realtime Alert > Graphite+Nagios as a Service > GrowthForecast+Nagios http://d.hatena.ne.jp/tagomoris/20120330/1333077746• Important Metrics.... > queue depth, # of concurrent users, # of sql queries, # of slow queries, etc.
    • 18. 3) Error Log Collection# waf.rb App Fluentd MongoDBbegin WAF app.run ERROR!rescue => e Fluent::Logger.post("error", {"e"=>e.to_s})end$ mongo db> db.error.find() # Recent 10 errors
    • 19. 4) Error Notifications • “Semi-Realtime” Error Notifications • IRC, HipChat, Campfire, PagerDuty, AirBrake, Skype, etc. TD s Internal Chat <match app.error> type campfire room DevOpsRoom </match>
    • 20. 5) Log Analysis MongoDB• MongoDB App Fluentd $ mongo db TD > db.logs.distinct( uid ).count()• Treasure Data $ td query SELECT count(distinct uid) FROM logs
    • 21. Conclusion.Fluentd is for... • Support System • Realtime System Stats • Realtime Alert • Error Notifications • Log Analysis • ...
    • 22. #3
    • 23. Github Issues #3
    • 24. Fluentdconveysmessage
    • 25. 伝書鳩