Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fluent Bit: Log Forwarding at Scale

2,648 views

Published on

Fluent Bit is a fast and scalable Log Forwarder for Cloud Native environments.

Published in: Technology
  • Be the first to comment

Fluent Bit: Log Forwarding at Scale

  1. 1. Log Forwarding at Scale CloudNativeCon, Berlin 2017 Eduardo Silva (@edsiper) eduardo@treasure-data.com
  2. 2. /about ● Open Source Engineer at Treasure Data ● Repositories / Projects ○ github.com/edsiper ○ fluentbit.io ○ duda.io ○ monkey-project.com
  3. 3. “Logging is Simple”
  4. 4. “Logging is Simple”
  5. 5. “Logging is Simple” LogsApp Analysis Logging exists because of Analysis needs
  6. 6. Before Analysis Logs Database Someone have to do some work ?
  7. 7. In a galaxy not so far away... Logs Database
  8. 8. Analysis
  9. 9. Internally, Logging is not Simple
  10. 10. Scale Logging Requires Understanding
  11. 11. Logging Pipeline
  12. 12. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others
  13. 13. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others ● Log files ● Journald ● TCP
  14. 14. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others Format / Structure: ● JSON ● Regex / Named Capture
  15. 15. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others Alter content: Grep, Exclude, Metadata
  16. 16. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others Memory / Filesystem
  17. 17. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others Deliver buffers to N destinations Logging Pipeline
  18. 18. Logging Pipeline Log Messages Parse Filter Buffer Routing Elasticsearch InfluxDB Others Logging Pipeline
  19. 19. Logging Pipeline ● How to deal with the Logging Pipeline ? ● Is there any solution around ?
  20. 20. About Fluentd ● Created by ● Now hosted at
  21. 21. About Fluentd ● More than 600 plugins available ● Pluggable Architecture ● Built-in Reliability ● Full integration with Docker and Kubernetes ● Written in Ruby + C
  22. 22. Fluentd Modes ● Log Forwarder ● Log Aggregator
  23. 23. Log Aggregator = (Forwarder + Buffering Capabilities)
  24. 24. Edge Nodes / Forward to Aggregators App Node 1 DB App DB
  25. 25. Edge Nodes & Costs
  26. 26. Edge Nodes & Costs ● Fluentd requires ~40MB as minimum ● Deploying a few hundred could be expensive ● Can we make Forward cheaper ?
  27. 27. Forwarder & Aggregator Log Forwarder Log Aggregator
  28. 28. About Fluent Bit ● Written in C ● Pluggable Architecture ● Built-in Reliability ● Event Driven - Async I/O
  29. 29. Why Fluent Bit as a Forwarder ● Features ○ Input, Filter and Output Plugins ○ Built-in parsing support ○ Minimum memory required 450KB
  30. 30. Edge Nodes / Forward to Aggregators
  31. 31. Cheap Forwarding
  32. 32. Cloud Native Features ● Docker & Kubernetes Support ● Buffering fully controled ○ pause() / resume() for input plugins ● Easy to containerize ○ Small memory footprint ○ No dependencies (all are built-in)
  33. 33. Cloud Native Features ● Backpressure
  34. 34. Cloud Native Features ● Backpressure
  35. 35. Cloud Native Features ● Backpressure
  36. 36. Cloud Native Features ● Backpressure
  37. 37. Cloud Native Features ● Backpressure Solved
  38. 38. Hands on! DEMO #1 Unstructured vs Structured data
  39. 39. Unstructured v/s Structured ● Why ○ Structured data have a schema ○ Easy to convert to different representations ○ It can be filtered
  40. 40. Kubernetes use case ● Applications runs in Containers ● Containers runs in a POD ● Multiple PODs can exists in a Node ● How to solve logging ?
  41. 41. Hands on! DEMO #2 Kubernetes: parse logs and append Metadata API Server Nodes
  42. 42. Kubernetes Metadata Support Status The new kubernetes filter takes care of the following metadata handling: ○ Local data: POD Name, Namespace, Container Name and Container ID. ○ Remote (API Server): Labels and Annotations
  43. 43. Fluent Bit, what else ? Networking and Co-routines Easier implementation of output plugins that interact with networking operations like socket(), connect(), read(), write(), etc. Fluent Bit provides non-blocking networking API that uses the event-loop with co-routines to implement: ● Network I/O ● TLS/SSL usage ● HTTP Client
  44. 44. Kubernetes DaemonSet Github Repository ● https://github.com/fluent/fluent-bit-kubernetes-daemonset Docker Hub Image (ubuntu-slim) ● fluent/fluent-bit-kubernetes-daemonset
  45. 45. Roadmap Next Release v0.11 (March 2017) <- Today! ● Kubernetes support (filter_kubernetes) ● Parsers & Filters ● Memory optimizations ● Kubernetes DaemonSet Release v0.12 (May 2017) ● in_tail + Multiline support ● Monitoring - re-enable HTTP service end-point: memory, records flow, others.
  46. 46. Thanks! Project information ● Web site Fluentbit.io ● Documentation http://fluentbit.io/documentation/ ● Github http://github.com/fluent/fluent-bit Contact ● Slack http://slack.fluentd.org (fluent-bit channel) ● Twitter @fluentbit

×