Fluentd for Small Projects @ Fluentd Casual 20120518

6,446 views
6,198 views

Published on

Published in: Technology, Education
0 Comments
21 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,446
On SlideShare
0
From Embeds
0
Number of Embeds
2,002
Actions
Shares
0
Downloads
66
Comments
0
Likes
21
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Fluentd for Small Projects @ Fluentd Casual 20120518

    1. 1. Fluentd ForSmall Projects Structured logging Pluggable architecture Reliable forwardingKazuki OhtaTreasure Data, Inc.@kzk_mover
    2. 2. Self-Introduction• Kazuki Ohta > twitter: @kzk_mover > github: kzk• Treasure Data, Inc. > Chief Technology Officer; Founder > Original Fluentd Author @frsyuki is another co-founder.• Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > td-agent package manager
    3. 3. Fluentd∼
    4. 4. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Storage record
    5. 5. What’s Fluentd? Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
    6. 6. What’s Fluentd?syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
    7. 7. Fluentd is Casual...? @tagomoris @hotchpotch 15,000,000 UU @ Cookpad∼80,000 msgs/sec
    8. 8. Yes. Fluentd isCASUAL
    9. 9. Fluentd Solves Daily Problems• Operation Problems > ssh root@bad-node && tail -f log.txt > cap deploy && pray > How many PV/UUs we have now? > How many users which did XXX and YYY?• Support > User A claimed that he bought 10 items, could you confirm that?
    10. 10. But Your Boss Might Say....• 「Log... For What?」• 「What’s Fluentd?」• 「OK, Who uses It?」• 「We have syslogd」
    11. 11. Start Fluentd, Silently...• Log Everything As JSON! > TSV/CSV logs are the nightmare.• Install td-agent > rpm/deb/chef/puppet etc.• tail plugin + MongoDB > tailing existing files. > no need to change existing systems.
    12. 12. Figure 1: Common Log Operations Analytics Error Notify Recommendation
    13. 13. Figure 2: Types of Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others
    14. 14. 5 ways to convince your boss to use Fluentd
    15. 15. 1) Support System• All logs are stored in MongoDB• No need to write ad-hoc log analysis scripts. before after#!/usr/bin/ruby $ mongo i-hate-mongoopen( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345}) ls = l.strip.split user = ls[4] puts l if user == 12345} $ ruby a.rb > foo.txt
    16. 16. 2) Realtime System Stats SCREENSHOT
    17. 17. 2) Realtime System Stats• Librato Metrics > Realtime Stats > Realtime Alert > Graphite+Nagios as a Service > GrowthForecast+Nagios http://d.hatena.ne.jp/tagomoris/20120330/1333077746• Important Metrics.... > queue depth, # of concurrent users, # of sql queries, # of slow queries, etc.
    18. 18. 3) Error Log Collection# waf.rb App Fluentd MongoDBbegin WAF app.run ERROR!rescue => e Fluent::Logger.post("error", {"e"=>e.to_s})end$ mongo db> db.error.find() # Recent 10 errors
    19. 19. 4) Error Notifications • “Semi-Realtime” Error Notifications • IRC, HipChat, Campfire, PagerDuty, AirBrake, Skype, etc. TD s Internal Chat <match app.error> type campfire room DevOpsRoom </match>
    20. 20. 5) Log Analysis MongoDB• MongoDB App Fluentd $ mongo db TD > db.logs.distinct( uid ).count()• Treasure Data $ td query SELECT count(distinct uid) FROM logs
    21. 21. Conclusion.Fluentd is for... • Support System • Realtime System Stats • Realtime Alert • Error Notifications • Log Analysis • ...
    22. 22. #3
    23. 23. Github Issues #3
    24. 24. Fluentdconveysmessage
    25. 25. 伝書鳩

    ×