Introducing Fluentd v11 features.

1. Fluentd v11
Jan 23, 2014
Masahiro Nakagawa
Treasure Data, Inc
www.treasuredata.com/
Sunday, January 26, 14
1

2. Who are you?
•
Masahiro Nakagawa
•
•
Treasure Data, Inc.
•
•
•
@repeatedly
Senior Software Engineer
Fluentd, td-agent, etc...
Dlang, MessagePack, ...
2
Sunday, January 26, 14

3. Structured logging
http://fluentd.org/
Reliable forwarding
Pluggable architecture
Sunday, January 26, 14

4. MxN→M+N
Access logs
Alerting
Apache
Nagios
App logs
Analysis
Frontend
MongoDB
Backend
MySQL
Hadoop
System logs
syslogd
buffer / buffer / routing
Databases
Archiving
Amazon S3
4
Sunday, January 26, 14

5. v10
• Mainly for log forwarding
• with good performance
• working in many productions reliably
• Various plugins are released
• There are 200+ plugins!
• Mainly for CRuby
5
Sunday, January 26, 14

6. Why v11?
• We want more useful features, but...
• Changing the internal architecture is hard
• Keeping the external API is even harder
• Originally, v10 was started as a prototype
• Need drastic changes for the future :)
6
Sunday, January 26, 14

7. What’s new?
• ServerEngine based reliable architecture
• Zero-downtime restarting
• Filter / Label for flexible record handling
• Improved configuration
• Improved plugin development
7
Sunday, January 26, 14

8. ServerEngine based
• Robust signal handling
• Put a signal into Queue first
• Built-in supervisor
• Multiprocess support
• No need for in_multiprocess plugin
8
Sunday, January 26, 14

9. Multi Process
Supervisor
Worker
Worker
Worker
<worker>
input tail
output mongo
</worker>
<worker>
input forward
output webhdfs
</worker>
<worker>
input foo
output bar
</worker>
Separate stream pipelines in one instance!
9
Sunday, January 26, 14

10. Zero downtime restart
• SocketManager shares resources with
workers
Supervisor
1. Listen to TCP socket
TCP
10
Sunday, January 26, 14

11. Zero downtime restart
• SocketManager shares resources with
workers
Supervisor
TCP
1. Listen to TCP socket
2. Pass its socket to worker
heartbeat
Worker
TCP
11
Sunday, January 26, 14

12. Zero downtime restart
• SocketManager shares resources with
workers
Supervisor
TCP
heartbeat
Worker
1. Listen to TCP socket
2. Pass its socket to worker
3. Do same action
at worker restarting
with keeping TCP socket
Worker
TCP
12
Sunday, January 26, 14

13. Filter / Label support
• No more tag-related tricks!
• add_tag_xxx, remove_tag_xxx, etc...
• Redirect events to another group
• Much easier to group and share plugins
13
Sunday, January 26, 14

14. Filter
• <match> can have nested <match>
• Configuration format is not fixed!
v10:
<match access.**>
type flowcounter
add_tag_prefix counted
</match>
<match counted.**>
typo growthforecast
</match>
v11:
<match access.** copy>
type flowcounter
<match **>
typo growthforecast
</match>
</match>
14
Sunday, January 26, 14

15. Label
• <label> can contain multiple <match>
• out_redirect can forward events to <label>
<match access.**>
type rewrite_tag_filter
...
<match bang.**>
type redirect
to_label blackhole
</match>
...
</match>
<label blackhole>
<match **>
type null
</match>
</label>
bang’s record go away!
15
Sunday, January 26, 14

16. Improved configuration
• Can use Hash, Array and others
• No need for “,” or similar tricks
• You can write Ruby directly
• Worker pragma
• Separate plugins in each process
• Support the v10 compatible mode
16
Sunday, January 26, 14

17. New parameter types
• Can write complex values without DSL!
• Can use Ruby code for configuration
Hash, Array, etc:
Embedded Ruby code:
<source>
type my_tail
keys ["k1", "k2", "k3"]
</source>
<match **>
typo my_filter
add_keys {"k1" : "v1"}
</match>
Sunday, January 26, 14
<match ** copy>
type my_filter
env "#{ENV['KEY']}"
</match>
•
•
•
Socket.gethostname
`command`
etc...
17

18. Improved plugin
• Actor
• Abstraction layer for several actions
• Error stream
• with @ERROR label
• Log level per plugin
• No more global API, Engine.emit, $log, etc...
18
Sunday, January 26, 14

19. Actor
• Easy to write popular routines
v10:
v11:
class TimerWatcher <
Coolio::TimerWatcher
...
end
def start
@loop = Coolio::Loop.new
@timer = ...
@loop.attach(@timer)
@thread = ...
end
Sunday, January 26, 14
actor.every(@interval) {
event_router.emit(...)
}
19

20. Error stream
• Can handle an error at each record level
Input
{"event":1, ...}
chunk1
{"event":2, ...}
{"event":3, ...}
{"event":4, ...}
Output
OK
ERROR!
OK
OK
…
{"event":5, ...}
{"event":6, ...}
chunk2
ERROR!
OK
Sunday, January 26, 14
Error stream
<label @ERROR>
<match **>
type file
...
</match>
</label>
Built-in @ERROR is used
when error occurred in “emit”
20

21. Lastly...
• Super alpha status!
• Adding / Removing features may occur
• There are several “to be implemented” features
• The release date is not fixed yet
• The initial release should be useful from day one
• Need feedbacks!
21
Sunday, January 26, 14

22. We’re Hiring!
Jan 23, 2014
www.treasuredata.com/
Sunday, January 26, 14
22