Fluentd ForSmall Projects        Structured logging                      Pluggable architecture                      Relia...
Self-Introduction•   Kazuki Ohta    >   twitter: @kzk_mover    >   github: kzk•   Treasure Data, Inc.    >   Chief Technol...
Fluentd∼
What’s Fluentd?                                   time    Application                     tag                  2012-02-04 ...
What’s Fluentd?             Application                           filter / buffer /              Fluentd                   ...
What’s Fluentd?syslogd    Scribe    Application          File Plug-in                                   tailPlug-in    Plu...
Fluentd is Casual...?   @tagomoris           @hotchpotch                   15,000,000 UU @ Cookpad∼80,000 msgs/sec
Yes. Fluentd isCASUAL
Fluentd Solves Daily Problems•   Operation Problems    >   ssh root@bad-node && tail -f log.txt    >   cap deploy && pray ...
But Your Boss      Might Say....•   「Log... For What?」•   「What’s Fluentd?」•   「OK, Who uses It?」•   「We have syslogd」
Start Fluentd, Silently...•   Log Everything As JSON!    >   TSV/CSV logs are the nightmare.•   Install td-agent    >   rp...
Figure 1: Common Log Operations                             Analytics                             Error Notify            ...
Figure 2: Types of Logs                          App Log                          Access Log                          (Apa...
5 ways to convince    your boss  to use Fluentd
1) Support System• All logs are stored in MongoDB• No need to write ad-hoc log analysis scripts.            before        ...
2) Realtime System Stats          SCREENSHOT
2) Realtime System Stats•   Librato Metrics    >   Realtime Stats    >   Realtime Alert    >   Graphite+Nagios        as a...
3) Error Log Collection# waf.rb            App                             Fluentd  MongoDBbegin              WAF   app.ru...
4) Error Notifications •   “Semi-Realtime” Error Notifications •   IRC, HipChat, Campfire, PagerDuty,     AirBrake, Skype, et...
5) Log Analysis                          MongoDB•   MongoDB                     App       Fluentd    $ mongo db           ...
Conclusion.Fluentd is for... •   Support System •   Realtime System Stats •   Realtime Alert •   Error Notifications •   Lo...
#3
Github Issues     #3
Fluentdconveysmessage
伝書鳩
Upcoming SlideShare
Loading in...5
×

Fluentd for Small Projects @ Fluentd Casual 20120518

5,673

Published on

Published in: Technology, Education
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,673
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
60
Comments
0
Likes
20
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Fluentd for Small Projects @ Fluentd Casual 20120518

    1. 1. Fluentd ForSmall Projects Structured logging Pluggable architecture Reliable forwardingKazuki OhtaTreasure Data, Inc.@kzk_mover
    2. 2. Self-Introduction• Kazuki Ohta > twitter: @kzk_mover > github: kzk• Treasure Data, Inc. > Chief Technology Officer; Founder > Original Fluentd Author @frsyuki is another co-founder.• Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > td-agent package manager
    3. 3. Fluentd∼
    4. 4. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Storage record
    5. 5. What’s Fluentd? Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
    6. 6. What’s Fluentd?syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
    7. 7. Fluentd is Casual...? @tagomoris @hotchpotch 15,000,000 UU @ Cookpad∼80,000 msgs/sec
    8. 8. Yes. Fluentd isCASUAL
    9. 9. Fluentd Solves Daily Problems• Operation Problems > ssh root@bad-node && tail -f log.txt > cap deploy && pray > How many PV/UUs we have now? > How many users which did XXX and YYY?• Support > User A claimed that he bought 10 items, could you confirm that?
    10. 10. But Your Boss Might Say....• 「Log... For What?」• 「What’s Fluentd?」• 「OK, Who uses It?」• 「We have syslogd」
    11. 11. Start Fluentd, Silently...• Log Everything As JSON! > TSV/CSV logs are the nightmare.• Install td-agent > rpm/deb/chef/puppet etc.• tail plugin + MongoDB > tailing existing files. > no need to change existing systems.
    12. 12. Figure 1: Common Log Operations Analytics Error Notify Recommendation
    13. 13. Figure 2: Types of Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others
    14. 14. 5 ways to convince your boss to use Fluentd
    15. 15. 1) Support System• All logs are stored in MongoDB• No need to write ad-hoc log analysis scripts. before after#!/usr/bin/ruby $ mongo i-hate-mongoopen( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345}) ls = l.strip.split user = ls[4] puts l if user == 12345} $ ruby a.rb > foo.txt
    16. 16. 2) Realtime System Stats SCREENSHOT
    17. 17. 2) Realtime System Stats• Librato Metrics > Realtime Stats > Realtime Alert > Graphite+Nagios as a Service > GrowthForecast+Nagios http://d.hatena.ne.jp/tagomoris/20120330/1333077746• Important Metrics.... > queue depth, # of concurrent users, # of sql queries, # of slow queries, etc.
    18. 18. 3) Error Log Collection# waf.rb App Fluentd MongoDBbegin WAF app.run ERROR!rescue => e Fluent::Logger.post("error", {"e"=>e.to_s})end$ mongo db> db.error.find() # Recent 10 errors
    19. 19. 4) Error Notifications • “Semi-Realtime” Error Notifications • IRC, HipChat, Campfire, PagerDuty, AirBrake, Skype, etc. TD s Internal Chat <match app.error> type campfire room DevOpsRoom </match>
    20. 20. 5) Log Analysis MongoDB• MongoDB App Fluentd $ mongo db TD > db.logs.distinct( uid ).count()• Treasure Data $ td query SELECT count(distinct uid) FROM logs
    21. 21. Conclusion.Fluentd is for... • Support System • Realtime System Stats • Realtime Alert • Error Notifications • Log Analysis • ...
    22. 22. #3
    23. 23. Github Issues #3
    24. 24. Fluentdconveysmessage
    25. 25. 伝書鳩
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×