This document summarizes a presentation about log forwarding at scale. It discusses how logging works internally and requires understanding the logging pipeline of parsing, filtering, buffering and routing logs. It then introduces Fluent Bit as a lightweight log forwarder that can be used to cheaply forward logs from edge nodes to log aggregators in a scalable way, especially in cloud native environments like Kubernetes. Hands-on demos show how Fluent Bit can parse and add metadata to Kubernetes logs.

1. Log Forwarding at Scale
CloudNativeCon, Berlin 2017
Eduardo Silva (@edsiper)
eduardo@treasure-data.com

2. /about
● Open Source Engineer at Treasure Data
● Repositories / Projects
○ github.com/edsiper
○ fluentbit.io
○ duda.io
○ monkey-project.com

3. “Logging is Simple”

4. “Logging is Simple”

5. “Logging is Simple”
LogsApp Analysis
Logging exists because of Analysis needs

6. Before Analysis
Logs Database
Someone have to do some work
?

7. In a galaxy not so far away...
Logs Database

8. Analysis

9. Internally, Logging
is not Simple

10. Scale Logging Requires
Understanding

11. Logging Pipeline

12. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others

13. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
● Log files
● Journald
● TCP

14. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Format / Structure:
● JSON
● Regex / Named Capture

15. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Alter content: Grep, Exclude, Metadata

16. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Memory / Filesystem

17. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Deliver buffers to N destinations
Logging Pipeline

18. Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Logging Pipeline

19. Logging Pipeline
● How to deal with the Logging Pipeline ?
● Is there any solution around ?

21. About Fluentd
● Created by
● Now hosted at

22. About Fluentd
● More than 600 plugins available
● Pluggable Architecture
● Built-in Reliability
● Full integration with Docker and Kubernetes
● Written in Ruby + C

23. Fluentd Modes
● Log Forwarder
● Log Aggregator

24. Log Aggregator =
(Forwarder + Buffering Capabilities)

25. Edge Nodes / Forward to Aggregators
App
Node 1
DB
App DB

26. Edge Nodes & Costs

27. Edge Nodes & Costs
● Fluentd requires ~40MB as minimum
● Deploying a few hundred could be expensive
● Can we make Forward cheaper ?

28. Forwarder & Aggregator
Log Forwarder Log Aggregator

30. About Fluent Bit
● Written in C
● Pluggable Architecture
● Built-in Reliability
● Event Driven - Async I/O

31. Why Fluent Bit as a Forwarder
● Features
○ Input, Filter and Output Plugins
○ Built-in parsing support
○ Minimum memory required 450KB

32. Edge Nodes / Forward to Aggregators

33. Cheap Forwarding

34. Cloud Native Features
● Docker & Kubernetes Support
● Buffering fully controled
○ pause() / resume() for input plugins
● Easy to containerize
○ Small memory footprint
○ No dependencies (all are built-in)

35. Cloud Native Features
● Backpressure

36. Cloud Native Features
● Backpressure

37. Cloud Native Features
● Backpressure

38. Cloud Native Features
● Backpressure

39. Cloud Native Features
● Backpressure Solved

40. Hands on!
DEMO #1
Unstructured vs Structured data

41. Unstructured v/s Structured
● Why
○ Structured data have a schema
○ Easy to convert to different representations
○ It can be filtered

42. Kubernetes use case
● Applications runs in Containers
● Containers runs in a POD
● Multiple PODs can exists in a Node
● How to solve logging ?

43. Hands on!
DEMO #2
Kubernetes: parse logs and append Metadata
API Server Nodes

44. Kubernetes
Metadata Support Status
The new kubernetes filter takes care of the following metadata
handling:
○ Local data: POD Name, Namespace, Container Name
and Container ID.
○ Remote (API Server): Labels and Annotations

45. Fluent Bit, what else ?
Networking and Co-routines
Easier implementation of output plugins that interact with networking
operations like socket(), connect(), read(), write(), etc.
Fluent Bit provides non-blocking networking API that uses the
event-loop with co-routines to implement:
● Network I/O
● TLS/SSL usage
● HTTP Client

46. Kubernetes DaemonSet
Github Repository
● https://github.com/fluent/fluent-bit-kubernetes-daemonset
Docker Hub Image (ubuntu-slim)
● fluent/fluent-bit-kubernetes-daemonset

47. Roadmap
Next Release v0.11 (March 2017) <- Today!
● Kubernetes support (filter_kubernetes)
● Parsers & Filters
● Memory optimizations
● Kubernetes DaemonSet
Release v0.12 (May 2017)
● in_tail + Multiline support
● Monitoring - re-enable HTTP service end-point: memory, records
flow, others.

48. Thanks!
Project information
● Web site Fluentbit.io
● Documentation http://fluentbit.io/documentation/
● Github http://github.com/fluent/fluent-bit
Contact
● Slack http://slack.fluentd.org (fluent-bit channel)
● Twitter @fluentbit