Fluentd v11
Jan 23, 2014
Masahiro Nakagawa
Treasure Data, Inc

www.treasuredata.com/
Sunday, January 26, 14

1
Who are you?
•

Masahiro Nakagawa

•
•

Treasure Data, Inc.

•
•
•

@repeatedly

Senior Software Engineer
Fluentd, td-agen...
Structured logging
http://fluentd.org/

Reliable forwarding
Pluggable architecture

Sunday, January 26, 14
MxN→M+N
Access logs

Alerting

Apache

Nagios

App logs

Analysis

Frontend

MongoDB

Backend

MySQL
Hadoop

System logs
s...
v10
• Mainly for log forwarding
• with good performance
• working in many productions reliably
• Various plugins are relea...
Why v11?
• We want more useful features, but...
• Changing the internal architecture is hard
• Keeping the external API is...
What’s new?
• ServerEngine based reliable architecture
• Zero-downtime restarting
• Filter / Label for flexible record hand...
ServerEngine based
• Robust signal handling
• Put a signal into Queue first
• Built-in supervisor
• Multiprocess support
• ...
Multi Process
Supervisor

Worker

Worker

Worker

<worker>
input tail
output mongo
</worker>

<worker>
input forward
outpu...
Zero downtime restart
• SocketManager shares resources with
workers

Supervisor

1. Listen to TCP socket

TCP

10
Sunday, ...
Zero downtime restart
• SocketManager shares resources with
workers

Supervisor
TCP

1. Listen to TCP socket
2. Pass its s...
Zero downtime restart
• SocketManager shares resources with
workers

Supervisor
TCP
heartbeat

Worker

1. Listen to TCP so...
Filter / Label support
• No more tag-related tricks!
• add_tag_xxx, remove_tag_xxx, etc...
• Redirect events to another gr...
Filter
• <match> can have nested <match>
• Configuration format is not fixed!
v10:
<match access.**>
type flowcounter
add_tag...
Label
• <label> can contain multiple <match>
• out_redirect can forward events to <label>
<match access.**>
type rewrite_t...
Improved configuration
• Can use Hash, Array and others
• No need for “,” or similar tricks
• You can write Ruby directly
•...
New parameter types
• Can write complex values without DSL!
• Can use Ruby code for configuration
Hash, Array, etc:

Embedd...
Improved plugin
• Actor
• Abstraction layer for several actions
• Error stream
• with @ERROR label
• Log level per plugin
...
Actor
• Easy to write popular routines
v10:

v11:

class TimerWatcher <
Coolio::TimerWatcher
...
end
def start
@loop = Coo...
Error stream
• Can handle an error at each record level
Input
{"event":1, ...}

chunk1

{"event":2, ...}
{"event":3, ...}
...
Lastly...
• Super alpha status!
• Adding / Removing features may occur
• There are several “to be implemented” features
• ...
We’re Hiring!
Jan 23, 2014

www.treasuredata.com/
Sunday, January 26, 14

22
Upcoming SlideShare
Loading in …5
×

Fluentd v11 at tokuben

1,442 views

Published on

Introducing Fluentd v11 features.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,442
On SlideShare
0
From Embeds
0
Number of Embeds
345
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Fluentd v11 at tokuben

  1. 1. Fluentd v11 Jan 23, 2014 Masahiro Nakagawa Treasure Data, Inc www.treasuredata.com/ Sunday, January 26, 14 1
  2. 2. Who are you? • Masahiro Nakagawa • • Treasure Data, Inc. • • • @repeatedly Senior Software Engineer Fluentd, td-agent, etc... Dlang, MessagePack, ... 2 Sunday, January 26, 14
  3. 3. Structured logging http://fluentd.org/ Reliable forwarding Pluggable architecture Sunday, January 26, 14
  4. 4. MxN→M+N Access logs Alerting Apache Nagios App logs Analysis Frontend MongoDB Backend MySQL Hadoop System logs syslogd buffer / buffer / routing Databases Archiving Amazon S3 4 Sunday, January 26, 14
  5. 5. v10 • Mainly for log forwarding • with good performance • working in many productions reliably • Various plugins are released • There are 200+ plugins! • Mainly for CRuby 5 Sunday, January 26, 14
  6. 6. Why v11? • We want more useful features, but... • Changing the internal architecture is hard • Keeping the external API is even harder • Originally, v10 was started as a prototype • Need drastic changes for the future :) 6 Sunday, January 26, 14
  7. 7. What’s new? • ServerEngine based reliable architecture • Zero-downtime restarting • Filter / Label for flexible record handling • Improved configuration • Improved plugin development 7 Sunday, January 26, 14
  8. 8. ServerEngine based • Robust signal handling • Put a signal into Queue first • Built-in supervisor • Multiprocess support • No need for in_multiprocess plugin 8 Sunday, January 26, 14
  9. 9. Multi Process Supervisor Worker Worker Worker <worker> input tail output mongo </worker> <worker> input forward output webhdfs </worker> <worker> input foo output bar </worker> Separate stream pipelines in one instance! 9 Sunday, January 26, 14
  10. 10. Zero downtime restart • SocketManager shares resources with workers Supervisor 1. Listen to TCP socket TCP 10 Sunday, January 26, 14
  11. 11. Zero downtime restart • SocketManager shares resources with workers Supervisor TCP 1. Listen to TCP socket 2. Pass its socket to worker heartbeat Worker TCP 11 Sunday, January 26, 14
  12. 12. Zero downtime restart • SocketManager shares resources with workers Supervisor TCP heartbeat Worker 1. Listen to TCP socket 2. Pass its socket to worker 3. Do same action at worker restarting with keeping TCP socket Worker TCP 12 Sunday, January 26, 14
  13. 13. Filter / Label support • No more tag-related tricks! • add_tag_xxx, remove_tag_xxx, etc... • Redirect events to another group • Much easier to group and share plugins 13 Sunday, January 26, 14
  14. 14. Filter • <match> can have nested <match> • Configuration format is not fixed! v10: <match access.**> type flowcounter add_tag_prefix counted </match> <match counted.**> typo growthforecast </match> v11: <match access.** copy> type flowcounter <match **> typo growthforecast </match> </match> 14 Sunday, January 26, 14
  15. 15. Label • <label> can contain multiple <match> • out_redirect can forward events to <label> <match access.**> type rewrite_tag_filter ... <match bang.**> type redirect to_label blackhole </match> ... </match> <label blackhole> <match **> type null </match> </label> bang’s record go away! 15 Sunday, January 26, 14
  16. 16. Improved configuration • Can use Hash, Array and others • No need for “,” or similar tricks • You can write Ruby directly • Worker pragma • Separate plugins in each process • Support the v10 compatible mode 16 Sunday, January 26, 14
  17. 17. New parameter types • Can write complex values without DSL! • Can use Ruby code for configuration Hash, Array, etc: Embedded Ruby code: <source> type my_tail keys ["k1", "k2", "k3"] </source> <match **> typo my_filter add_keys {"k1" : "v1"} </match> Sunday, January 26, 14 <match ** copy> type my_filter env "#{ENV['KEY']}" </match> • • • Socket.gethostname `command` etc... 17
  18. 18. Improved plugin • Actor • Abstraction layer for several actions • Error stream • with @ERROR label • Log level per plugin • No more global API, Engine.emit, $log, etc... 18 Sunday, January 26, 14
  19. 19. Actor • Easy to write popular routines v10: v11: class TimerWatcher < Coolio::TimerWatcher ... end def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ... end Sunday, January 26, 14 actor.every(@interval) { event_router.emit(...) } 19
  20. 20. Error stream • Can handle an error at each record level Input {"event":1, ...} chunk1 {"event":2, ...} {"event":3, ...} {"event":4, ...} Output OK ERROR! OK OK … {"event":5, ...} {"event":6, ...} chunk2 ERROR! OK Sunday, January 26, 14 Error stream <label @ERROR> <match **> type file ... </match> </label> Built-in @ERROR is used when error occurred in “emit” 20
  21. 21. Lastly... • Super alpha status! • Adding / Removing features may occur • There are several “to be implemented” features • The release date is not fixed yet • The initial release should be useful from day one • Need feedbacks! 21 Sunday, January 26, 14
  22. 22. We’re Hiring! Jan 23, 2014 www.treasuredata.com/ Sunday, January 26, 14 22

×