SlideShare a Scribd company logo
1 of 23
Download to read offline
Masahiro Nakagawa 
Nov 6, 2014 
Cloudera World Tokyo 
Fluentd 
Unified Logging Layer
Who are you? 
> Masahiro Nakagawa 
> github/twitter: @repeatedly 
> Treasure Data, Inc. 
> Senior Software Engineer 
> Fluentd / td-agent developer 
> I love OSS :) 
> D language - Phobos committer 
> Fluentd - Main maintainer 
> MessagePack / RPC- D and Python (only RPC) 
> The organizer of Presto Source Code Reading 
> etc…
Structured logging 
! 
Reliable forwarding 
! 
Pluggable architecture 
http://fluentd.org/
Data Processing 
Data source 
Collect Store Process Visualize 
Reporting 
Monitoring
Related Products 
Collect Store Process 
Visualize 
Cloudera 
Horton Works 
Treasure Data 
Excel 
Tableau 
R 
easier & shorter time 
???
What’s Fluentd? 
• Open sourced log collector written in Ruby 
• Customization is essential 
small core + many plugins 
6 
It’s like syslogd, but 
uses JSON for log messages
Divide & Conquer & Retry 
7 
error retry 
retry 
error retry retry
Core Plugins 
! 
• Divide & Conquer 
• Buffring & Retrying 
• Error handling 
• Message routing 
• Parallelism 
! 
! 
• read / receive data 
• write / send data 
8
Pluggable Architecture 
9 
Pluggable Pluggable 
Buffer Output 
Input 
> Forward 
> HTTP 
> File tail 
> dstat 
> ... 
> Forward 
> File 
> MongoDB 
> ... 
> File 
> Memory 
Engine 
Output 
> rewrite 
> ...
Before 
10
After 
11
M x N → M + N 
12 
Alerting 
Nagios 
Analysis 
MongoDB 
MySQL 
Hadoop 
Archiving 
Amazon S3 
Access logs 
Apache 
App logs 
Frontend 
Backend 
System logs 
syslogd 
Databases 
buffer / buffer / routing
Ecosystem
Treasure Agent 
• Treasure Data distribution of Fluentd 
• including Ruby, core libraries and 
third party plugins 
• Current stable is 2.1.1 
• rpm / deb / dmg 
• 2.1.2 will be released soon with fluentd-ui 
14
fluentd-forwarder 
• Forwarding agent written in Go 
• mainly for Windows support 
• immature than Fluentd 
• Bundle TCP input/output and TD output 
• No plugin mechanizm 
• Similar product: ik, fluent-agent-hydra, etc 
15
fluentd-ui 
• Manage Fluentd instance using Web GUI 
16
Use-case
Users 
• Companies 
• Web service, Ad Tech, Game, Media, 
EC, IoT, PoS, etc.. 
• Products 
• Kubernetes, Cloudn, Harvester, 
Bloomberg Clustered Private Cloud, etc.. 
18
Simple forwarding 
19 
HDFS 
HBase 
Mongo 
S3
Separate log stream 
20
CEP for stream processing 
21
Roadmap 
• v0.10 (current) 
• v0.12 (Nov, 2014) 
• Filter / Label / at-least-once 
• v0.14 (TBD) 
• New Plugin APIs, ServerEngine 
• v1 (TBD. In the first half of 2015) 
22
Cloud service for the entire data pipeline 
Check: treasuredata.com

More Related Content

What's hot

MongoDB backup service overview Boston MUG
MongoDB backup service overview Boston MUGMongoDB backup service overview Boston MUG
MongoDB backup service overview Boston MUG
MongoDB
 

What's hot (20)

Fluent Bit: Log Forwarding at Scale
Fluent Bit: Log Forwarding at ScaleFluent Bit: Log Forwarding at Scale
Fluent Bit: Log Forwarding at Scale
 
Logging Application Behavior to MongoDB
Logging Application Behavior to MongoDBLogging Application Behavior to MongoDB
Logging Application Behavior to MongoDB
 
Containers and Logging
Containers and LoggingContainers and Logging
Containers and Logging
 
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkFluentd v1 and future at techtalk
Fluentd v1 and future at techtalk
 
GeoDistributed datacenter: the DNS way
GeoDistributed datacenter: the DNS wayGeoDistributed datacenter: the DNS way
GeoDistributed datacenter: the DNS way
 
Fluent Bit
Fluent BitFluent Bit
Fluent Bit
 
Log forwarding at Scale
Log forwarding at ScaleLog forwarding at Scale
Log forwarding at Scale
 
Logging for Containers
Logging for ContainersLogging for Containers
Logging for Containers
 
Cloud Native Logging / Fluentd Summit Tokyo
Cloud Native Logging / Fluentd Summit TokyoCloud Native Logging / Fluentd Summit Tokyo
Cloud Native Logging / Fluentd Summit Tokyo
 
The basics of fluentd
The basics of fluentdThe basics of fluentd
The basics of fluentd
 
Fluentd Intro for OpenShift Commons Briefing
Fluentd Intro for OpenShift Commons BriefingFluentd Intro for OpenShift Commons Briefing
Fluentd Intro for OpenShift Commons Briefing
 
P99CONF — What We Need to Unlearn About Persistent Storage
P99CONF — What We Need to Unlearn About Persistent StorageP99CONF — What We Need to Unlearn About Persistent Storage
P99CONF — What We Need to Unlearn About Persistent Storage
 
Infrastructure as Code with Terraform: Koombea TechTalks
Infrastructure as Code with Terraform: Koombea TechTalksInfrastructure as Code with Terraform: Koombea TechTalks
Infrastructure as Code with Terraform: Koombea TechTalks
 
Redis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech TalkRedis vs Infinispan | DevNation Tech Talk
Redis vs Infinispan | DevNation Tech Talk
 
Breaking the RpiDocker challenge
Breaking the RpiDocker challenge Breaking the RpiDocker challenge
Breaking the RpiDocker challenge
 
Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...
Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...
Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...
 
Rust programming-language
Rust programming-languageRust programming-language
Rust programming-language
 
IPv4aaS tutorial and hands-on
IPv4aaS tutorial and hands-onIPv4aaS tutorial and hands-on
IPv4aaS tutorial and hands-on
 
MongoDB backup service overview Boston MUG
MongoDB backup service overview Boston MUGMongoDB backup service overview Boston MUG
MongoDB backup service overview Boston MUG
 
Application Caching: The Hidden Microservice (SAConf)
Application Caching: The Hidden Microservice (SAConf)Application Caching: The Hidden Microservice (SAConf)
Application Caching: The Hidden Microservice (SAConf)
 

Viewers also liked (6)

Fluentd at HKOScon
Fluentd at HKOSconFluentd at HKOScon
Fluentd at HKOScon
 
LogStash - skuteczne narzędzie dla entuzjastów logowania
LogStash  - skuteczne narzędzie dla entuzjastów logowaniaLogStash  - skuteczne narzędzie dla entuzjastów logowania
LogStash - skuteczne narzędzie dla entuzjastów logowania
 
Presto changes
Presto changesPresto changes
Presto changes
 
Application Logging With Logstash
Application Logging With LogstashApplication Logging With Logstash
Application Logging With Logstash
 
Fluentd and Kafka
Fluentd and KafkaFluentd and Kafka
Fluentd and Kafka
 
E-Commerce Case Studies
E-Commerce Case StudiesE-Commerce Case Studies
E-Commerce Case Studies
 

Similar to Fluentd: Unified Logging Layer at CWT2014

Similar to Fluentd: Unified Logging Layer at CWT2014 (20)

Fluentd and AWS at classmethod
Fluentd and AWS at classmethodFluentd and AWS at classmethod
Fluentd and AWS at classmethod
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
 
Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65
 
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and Then
 
Fluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At FossasiaFluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At Fossasia
 
Serverless Boston @ Oracle Meetup
Serverless Boston @ Oracle MeetupServerless Boston @ Oracle Meetup
Serverless Boston @ Oracle Meetup
 
The Fn Project by Jesse Butler
 The Fn Project by Jesse Butler The Fn Project by Jesse Butler
The Fn Project by Jesse Butler
 
Fluentd meetup
Fluentd meetupFluentd meetup
Fluentd meetup
 
Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the Cloud
 
Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the Cloud
 
Treasure Data and OSS
Treasure Data and OSSTreasure Data and OSS
Treasure Data and OSS
 
DBCC 2021 - FLiP Stack for Cloud Data Lakes
DBCC 2021 - FLiP Stack for Cloud Data LakesDBCC 2021 - FLiP Stack for Cloud Data Lakes
DBCC 2021 - FLiP Stack for Cloud Data Lakes
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
The Fn Project: A Quick Introduction (December 2017)
The Fn Project: A Quick Introduction (December 2017)The Fn Project: A Quick Introduction (December 2017)
The Fn Project: A Quick Introduction (December 2017)
 
Docker and Fluentd (revised)
Docker and Fluentd (revised)Docker and Fluentd (revised)
Docker and Fluentd (revised)
 
Monkey Server
Monkey ServerMonkey Server
Monkey Server
 
Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoring
 
Lua as a business logic language in high load application
Lua as a business logic language in high load applicationLua as a business logic language in high load application
Lua as a business logic language in high load application
 
Introduction to Apache Flink
Introduction to Apache FlinkIntroduction to Apache Flink
Introduction to Apache Flink
 

More from N Masahiro

More from N Masahiro (17)

Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellFluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshell
 
Fluentd v0.14 Overview
Fluentd v0.14 OverviewFluentd v0.14 Overview
Fluentd v0.14 Overview
 
Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12
 
Technologies for Data Analytics Platform
Technologies for Data Analytics PlatformTechnologies for Data Analytics Platform
Technologies for Data Analytics Platform
 
How to create Treasure Data #dotsbigdata
How to create Treasure Data #dotsbigdataHow to create Treasure Data #dotsbigdata
How to create Treasure Data #dotsbigdata
 
Fluentd v0.12 master guide
Fluentd v0.12 master guideFluentd v0.12 master guide
Fluentd v0.12 master guide
 
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015
 
Fluentd - road to v1 -
Fluentd - road to v1 -Fluentd - road to v1 -
Fluentd - road to v1 -
 
SQL for Everything at CWT2014
SQL for Everything at CWT2014SQL for Everything at CWT2014
SQL for Everything at CWT2014
 
Can you say the same words even in oss
Can you say the same words even in ossCan you say the same words even in oss
Can you say the same words even in oss
 
I am learing the programming
I am learing the programmingI am learing the programming
I am learing the programming
 
Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)
 
D vs OWKN Language at LLnagoya
D vs OWKN Language at LLnagoyaD vs OWKN Language at LLnagoya
D vs OWKN Language at LLnagoya
 
Goodbye Doost
Goodbye DoostGoodbye Doost
Goodbye Doost
 
Final presentation at pfintern
Final presentation at pfinternFinal presentation at pfintern
Final presentation at pfintern
 
Kernel VM 5 LT
Kernel VM 5 LTKernel VM 5 LT
Kernel VM 5 LT
 
D言語のコミッタになる一つの方法
D言語のコミッタになる一つの方法D言語のコミッタになる一つの方法
D言語のコミッタになる一つの方法
 

Recently uploaded

“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 

Recently uploaded (20)

WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 

Fluentd: Unified Logging Layer at CWT2014

  • 1. Masahiro Nakagawa Nov 6, 2014 Cloudera World Tokyo Fluentd Unified Logging Layer
  • 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC- D and Python (only RPC) > The organizer of Presto Source Code Reading > etc…
  • 3. Structured logging ! Reliable forwarding ! Pluggable architecture http://fluentd.org/
  • 4. Data Processing Data source Collect Store Process Visualize Reporting Monitoring
  • 5. Related Products Collect Store Process Visualize Cloudera Horton Works Treasure Data Excel Tableau R easier & shorter time ???
  • 6. What’s Fluentd? • Open sourced log collector written in Ruby • Customization is essential small core + many plugins 6 It’s like syslogd, but uses JSON for log messages
  • 7. Divide & Conquer & Retry 7 error retry retry error retry retry
  • 8. Core Plugins ! • Divide & Conquer • Buffring & Retrying • Error handling • Message routing • Parallelism ! ! • read / receive data • write / send data 8
  • 9. Pluggable Architecture 9 Pluggable Pluggable Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ...
  • 12. M x N → M + N 12 Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3 Access logs Apache App logs Frontend Backend System logs syslogd Databases buffer / buffer / routing
  • 14. Treasure Agent • Treasure Data distribution of Fluentd • including Ruby, core libraries and third party plugins • Current stable is 2.1.1 • rpm / deb / dmg • 2.1.2 will be released soon with fluentd-ui 14
  • 15. fluentd-forwarder • Forwarding agent written in Go • mainly for Windows support • immature than Fluentd • Bundle TCP input/output and TD output • No plugin mechanizm • Similar product: ik, fluent-agent-hydra, etc 15
  • 16. fluentd-ui • Manage Fluentd instance using Web GUI 16
  • 18. Users • Companies • Web service, Ad Tech, Game, Media, EC, IoT, PoS, etc.. • Products • Kubernetes, Cloudn, Harvester, Bloomberg Clustered Private Cloud, etc.. 18
  • 19. Simple forwarding 19 HDFS HBase Mongo S3
  • 21. CEP for stream processing 21
  • 22. Roadmap • v0.10 (current) • v0.12 (Nov, 2014) • Filter / Label / at-least-once • v0.14 (TBD) • New Plugin APIs, ServerEngine • v1 (TBD. In the first half of 2015) 22
  • 23. Cloud service for the entire data pipeline Check: treasuredata.com