Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Masahiro Nakagawa
Jan 27, 2015
Data Transfer Middleware Meetup #1
Fluentd- road to v1 -
#dtm_meetup
Who are you?
> Masahiro Nakagawa
> github/twitter: @repeatedly
> Treasure Data, Inc.
> Senior Software Engineer
> Fluentd ...
Structured logging	

!
Reliable forwarding	

!
Pluggable architecture
http://fluentd.org/
What’s Fluentd?
> Data collector for unified logging layer
> Streaming data transfer based on JSON
> Written in Ruby
> Gem ...
Divide & Conquer & Retry
error retry
error retry retry
retry
Batch
Stream
Other stream
Core Plugins
> Divide & Conquer

> Buffering & Retrying

> Error handling

> Message routing

> Parallelism
> read / recei...
Architecture (v0.12 or later)
EngineInput
Filter Output
Buffer
> grep
> record_transfomer	

> …
> Forward	

> File tail	

...
Before
After
or Embulk
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs...
v0.10 (old stable)
> Mainly for log forwarding
> with good performance
> working in production
> http://www.fluentd.org/tes...
v0.11 (dead version)
http://repeatedly.github.io/ja/2014/03/about-fluentd-v11/
v0.12 (current stable)
> Event handling improvement
> Filter
> Label
> At-least-once semantics in forwarding
> require_ack...
> Apply filtering routine to event stream
> No more tag tricks!











Filter
<match access.**>	

type record_reformer	...
> Internal event routing
> Redirect events to another group
> much easier to group and share plugins











Label
<sou...
Error stream with Label
> Can handle an error at each record level
> It is still prototype












 ERROR!
{"event":1,...
v0.14 (next stable)
> New plugin APIs
> Actor
> New base classes (#309)
> ServerEngine based engine
> Robust supervisor 
>...
Actor
> Easy to write popular routines
> Hide implementation details















class TimerWatcher <	

Coolio::TimerWa...
> Socket manager shared resources with
workers











19
Supervisor
TCP
1. Listen to TCP socket
Zero downtime restart
20
Worker
Supervisor
heartbeat
TCP
TCP
1. Listen to TCP socket	

2. Pass its socket to worker
Zero downtime restart
> Sock...
21
Worker
Supervisor
Worker
TCP
TCP
1. Listen to TCP socket	

2. Pass its socket to worker	

3. Do same action

at worker ...
v1 (future stable)
> Fix new features / APIs
> Plugin APIs
> Default configurations
> Clear versioning and stability
> No b...
Roadmap summary
> v0.10 (old stable)
> v0.12 (current stable)
> Filter / Label / At-least-once
> v0.14 (spring, 2015)
> Ne...
Other TODO
> Windows support
> Need feedback!
> https://github.com/fluent/fluentd/tree/windows
> Also check: http://qiita.co...
Ecosystem
Treasure Agent (td-agent)
> Treasure Data distribution of Fluentd
> Treasure Agent 2 is current stable
> Update core compo...
fluentd-forwarder
> Forwarding agent written in Go
> Focusing log forwarding to Fluentd
> Work on Windows
> Bundle TCP inpu...
fluentd-ui
> Manage Fluentd instance via Web UI
> https://github.com/fluent/fluentd-ui











Check: treasuredata.com
Cloud service for the entire data pipeline
Upcoming SlideShare
Loading in …5
×

Fluentd - road to v1 -

15,750 views

Published on

"Fluentd - road to v1 -" at Data Transfer Middleware Meetup.

Published in: Technology
  • Be the first to comment

Fluentd - road to v1 -

  1. 1. Masahiro Nakagawa Jan 27, 2015 Data Transfer Middleware Meetup #1 Fluentd- road to v1 - #dtm_meetup
  2. 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of Presto Source Code Reading / meetup > etc…
  3. 3. Structured logging ! Reliable forwarding ! Pluggable architecture http://fluentd.org/
  4. 4. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
  5. 5. Divide & Conquer & Retry error retry error retry retry retry Batch Stream Other stream
  6. 6. Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > read / receive data > from API, database,
 command, etc… > write / send data > to API, database, alert, graph, etc…
  7. 7. Architecture (v0.12 or later) EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
  8. 8. Before
  9. 9. After or Embulk
  10. 10. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffering / processing / routing M x N → M + N
  11. 11. v0.10 (old stable) > Mainly for log forwarding > with good performance > working in production > http://www.fluentd.org/testimonials > Various plugins > http://www.fluentd.org/plugins
  12. 12. v0.11 (dead version) http://repeatedly.github.io/ja/2014/03/about-fluentd-v11/
  13. 13. v0.12 (current stable) > Event handling improvement > Filter > Label > At-least-once semantics in forwarding > require_ack_response parameter > http://ogibayashi.github.io/blog/2014/12/16/try- fluentd-v0-dot-12-at-least-once/
  14. 14. > Apply filtering routine to event stream > No more tag tricks!
 
 
 
 
 
 Filter <match access.**> type record_reformer tag reformed.${tag} </match> ! <match reformed.**> type growthforecast </match> <filter access.**> type record_transformer … </filter> v0.10: v0.12: <match access.**> type growthforecast </match>
  15. 15. > Internal event routing > Redirect events to another group > much easier to group and share plugins
 
 
 
 
 
 Label <source> type forward </source> ! <match app1.**> type record_reformer </match> ! … <source> type forward @label @APP1 </source> <label @APP1> <match access.**> type s3 </match> </label> v0.10: v0.12:
  16. 16. Error stream with Label > Can handle an error at each record level > It is still prototype
 
 
 
 
 
 
 ERROR! {"event":1, ...} {"event":2, ...} {"event":3, ...} chunk1 {"event":4, ...} {"event":5, ...} {"event":6, ...} chunk2 … Input OK ERROR! OK OK OK Output <label @ERROR> <match **> type file ... </match> </label> Error stream Built-in @ERROR is used when error occurred in “emit”
  17. 17. v0.14 (next stable) > New plugin APIs > Actor > New base classes (#309) > ServerEngine based engine > Robust supervisor > Sub-second time support (#461) > Zero downtime restart
  18. 18. Actor > Easy to write popular routines > Hide implementation details
 
 
 
 
 
 
 
 class TimerWatcher < Coolio::TimerWatcher ... end ! def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ... end def configure(conf) actor.every(@interval) { router.emit(...) } end ! def start actor.start end v10: v0.14:
  19. 19. > Socket manager shared resources with workers
 
 
 
 
 
 19 Supervisor TCP 1. Listen to TCP socket Zero downtime restart
  20. 20. 20 Worker Supervisor heartbeat TCP TCP 1. Listen to TCP socket 2. Pass its socket to worker Zero downtime restart > Socket manager shared resources with workers
 
 
 
 
 

  21. 21. 21 Worker Supervisor Worker TCP TCP 1. Listen to TCP socket 2. Pass its socket to worker 3. Do same action
 at worker restarting
 with keeping TCP socket heartbeat Zero downtime restart > Socket manager shared resources with workers
 
 
 
 
 
 TODO: How to implement on JRuby?
  22. 22. v1 (future stable) > Fix new features / APIs > Plugin APIs > Default configurations > Clear versioning and stability > No breaking API compatibility! > Breaking compatibility by Fluentd v2 ?
  23. 23. Roadmap summary > v0.10 (old stable) > v0.12 (current stable) > Filter / Label / At-least-once > v0.14 (spring, 2015) > New plugin APIs, ServerEngine, Time… > v1 (early summer, 2015) > Fix new features / APIs https://github.com/fluent/fluentd/wiki/V1-Roadmap
  24. 24. Other TODO > Windows support > Need feedback! > https://github.com/fluent/fluentd/tree/windows > Also check: http://qiita.com/okahashi117 > JRuby support > msgpack / cool.io now work on JRuby > https://github.com/fluent/fluentd/issues/317
  25. 25. Ecosystem
  26. 26. Treasure Agent (td-agent) > Treasure Data distribution of Fluentd > Treasure Agent 2 is current stable > Update core components > We recommend to use v2, not v1 > Next version, 2.1.4, uses fluentd v0.12 > In this week or next week
  27. 27. fluentd-forwarder > Forwarding agent written in Go > Focusing log forwarding to Fluentd > Work on Windows > Bundle TCP input/output and TD output > No flexible plugin mechanizm > We have a plan to add some input/output > Need feedback!
  28. 28. fluentd-ui > Manage Fluentd instance via Web UI > https://github.com/fluent/fluentd-ui
 
 
 
 
 

  29. 29. Check: treasuredata.com Cloud service for the entire data pipeline

×