1. Fluentd For
Small Projects Structured logging
Pluggable architecture
Reliable
forwarding
Kazuki Ohta
Treasure Data, Inc.
@kzk_mover

2. Self-Introduction
• Kazuki Ohta
> twitter: @kzk_mover
> github: kzk
• Treasure Data, Inc.
> Chief Technology Officer; Founder
> Original Fluentd Author @frsyuki is another co-founder.
• Open-Source Enthusiast
> KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc.
> td-agent package manager

3. Fluentd∼

4. What’s Fluentd?
time
Application tag
2012-02-04 01:33:51
myapp.buylog {
Fluentd “user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
Storage record

5. What’s Fluentd?
Application
filter / buffer /
Fluentd
routing
SaaS Storage Fluentd
Plug-in Plug-in Plug-in

6. What’s Fluentd?
syslogd Scribe Application File Plug-in
tail
Plug-in Plug-in
Fluentd filter / buffer / routing
SaaS Storage Fluentd
Plug-in Plug-in Plug-in

7. Fluentd is Casual...?
@tagomoris @hotchpotch
15,000,000 UU @ Cookpad
∼80,000 msgs/sec

8. Yes.
Fluentd is
CASUAL

9. Fluentd Solves Daily Problems
• Operation Problems
> ssh root@bad-node && tail -f log.txt
> cap deploy && pray
> How many PV/UUs we have now?
> How many users which did XXX and YYY?
• Support
> User A claimed that he bought 10 items, could
you confirm that?

10. But Your Boss
Might Say....
• 「Log... For What?」
• 「What’s Fluentd?」
• 「OK, Who uses It?」
• 「We have syslogd」

11. Start Fluentd, Silently...
• Log Everything As JSON!
> TSV/CSV logs are the nightmare.
• Install td-agent
> rpm/deb/chef/puppet etc.
• tail plugin + MongoDB
> tailing existing files.
> no need to change existing systems.

12. Figure 1: Common Log Operations
Analytics
Error Notify
Recommendation

13. Figure 2: Types of Logs
App Log
Access Log
(Apache, Rails, etc.)
System Log
(syslog etc.)
Others

14. 5 ways to convince
your boss
to use Fluentd

15. 1) Support System
• All logs are stored in MongoDB
• No need to write ad-hoc log analysis scripts.
before after
#!/usr/bin/ruby $ mongo i-hate-mongo
open( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345})
ls = l.strip.split
user = ls[4]
puts l if user == 12345
}
$ ruby a.rb > foo.txt

16. 2) Realtime System Stats
SCREENSHOT

17. 2) Realtime System Stats
• Librato Metrics
> Realtime Stats
> Realtime Alert
> Graphite+Nagios
as a Service
> GrowthForecast+Nagios
http://d.hatena.ne.jp/tagomoris/20120330/1333077746
• Important Metrics....
> queue depth, # of concurrent users, # of sql queries, #
of slow queries, etc.

18. 3) Error Log Collection
# waf.rb App
Fluentd MongoDB
begin WAF
app.run
ERROR!
rescue => e
Fluent::Logger.post("error", {"e"=>e.to_s})
end
$ mongo db
> db.error.find() # Recent 10 errors

19. 4) Error Notifications
• “Semi-Realtime” Error Notifications
• IRC, HipChat, Campfire, PagerDuty,
AirBrake, Skype, etc.
TD s Internal Chat
<match app.error>
type campfire
room DevOpsRoom
</match>

20. 5) Log Analysis MongoDB
• MongoDB App Fluentd
$ mongo db TD
> db.logs.distinct( uid ).count()
• Treasure Data
$ td query
SELECT count(distinct uid) FROM logs

21. Conclusion.
Fluentd is for...
• Support System
• Realtime System Stats
• Realtime Alert
• Error Notifications
• Log Analysis
• ...

22. #3

23. Github Issues
#3

24. Fluentd
conveys
message

25. 伝書鳩