Scaling the logging pipeline requires better understanding of each phase behind the scenes. Everything about Fluentd as an aggregator and Fluent Bit as it Log Forwarder

1. Log Forwarding at Scale
Scale 15X, Pasadena, CA
Eduardo Silva (@edsiper)
eduardo@treasure-data.com

2. /about
● Open Source Engineer at Treasure Data
● Repositories / Projects
○ github.com/edsiper
○ fluentbit.io
○ duda.io
○ monkey-project.com

3. “Logging is Simple”

4. “Logging is Simple”

5. “Logging is Simple”
LogsApp Analysis
Logging exists because of Analysis needs

6. Before Analysis
Logs Database
Someone have to do some work
?

7. In a galaxy not so far away...
Logs Database

8. Analysis

9. Internally, Logging
is not Simple

10. Scale Logging Requires
Understanding

11. Logging Pipeline

12. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others

13. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
● Log files
● Journald
● TCP

14. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Format / Structure:
● JSON
● Regex / Named Capture

15. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Alter content: Grep, Exclude, Metadata

16. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Memory / Filesystem

17. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Deliver buffers to N destinations
Logging Pipeline

18. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Logging Pipeline

19. Logging Pipeline
● How to deal with the Logging Pipeline ?
● Is there any solution around ?

21. About Fluentd
● Created by
● Now hosted at

22. About Fluentd
● More than 600 plugins available
● Pluggable Architecture
● Built-in Reliability
● Full integration with Docker and Kubernetes
● Written in Ruby + C

23. Fluentd Modes
● Log Forwarder
● Log Aggregator

24. Log Aggregator =
(Forwarder + Buffering Capabilities)

25. Edge Nodes / Forward to Aggregators
App
Node 1
DB
App DB

26. Edge Nodes & Costs

27. Edge Nodes & Costs
● Fluentd requires ~40MB as minimum
● Deploying a few hundred could be expensive
● Can we make Forward cheaper ?

28. Forwarder & Aggregator
Log Forwarder Log Aggregator

30. About Fluentd
● Written in C
● Pluggable Architecture
● Built-in Reliability
● Event Driven - Async I/O

31. Why Fluent Bit as a Forwarder
● Features
○ Input, Filter and Output Plugins
○ Built-in parsing support
○ Minimum memory required 450KB

32. Edge Nodes / Forward to Aggregators

33. Cheap Forwarding

34. Cloud Native Features
● Docker & Kubernetes Support
● Buffering fully controled
○ pause() / resume() for input plugins
● Easy to containerize
○ Small memory footprint
○ No dependencies (all are built-in)

35. Hands on!
DEMO #1
Unstructured vs Structured data

36. Unstructured v/s Structured
● Why
○ Structured data have a schema
○ Easy to convert to different representations
○ It can be filtered

37. Hands on!
DEMO #2
Process Docker Logs

38. Kubernetes use case
● Applications runs in Containers
● Containers runs in a POD
● Multiple PODs can exists in a Node
● How to solve logging ?

39. Hands on!
DEMO #3
Kubernetes: parse logs and append Metadata
API Server Nodes

40. Kubernetes
Metadata Support Status
The new kubernetes filter takes care of the following metadata
handling:
○ Local data: POD Name, Namespace, Container Name
and Container ID.
○ Remote (API Server): Labels and Annotations

41. Fluent Bit, what else ?
Networking and Co-routines
Easier implementation of output plugins that interact with networking
operations like socket(), connect(), read(), write(), etc.
Fluent Bit provides non-blocking networking API that uses the
event-loop with co-routines to implement:
● Network I/O
● TLS/SSL usage
● HTTP Client

42. Kubernetes DaemonSet
Github Repository
● https://github.com/fluent/fluent-bit-kubernetes-daemonset
Docker Image (ubuntu-slim)
● quay.io/fluent/fluent-bit-kubernetes-daemonset

43. Roadmap
Next Release v0.11 (March 2017)
● Kubernetes support (filter_kubernetes)
● Parsers & Filters
● Memory optimizations
Release v0.12 (May 2017)
● in_tail + Multiline support
● Monitoring - re-enable HTTP service end-point: memory, records
flow, others.

44. Thanks!
Project information
● Web site Fluentbit.io
● Documentation http://fluentbit.io/documentation/
● Github http://github.com/fluent/fluent-bit
Contact
● Slack http://slack.fluentd.org (fluent-bit channel)
● Twitter @fluentbit