This document summarizes a presentation about cloud native logging using Fluentd and Fluent Bit. It discusses why logging is important, describes the logging pipeline and challenges with microservices. It then introduces Fluentd and Fluent Bit, how they simplify the logging pipeline including features like plugins, reliability, and integration with Docker and Kubernetes. Specific techniques for log collection and storage with Kubernetes using Fluentd and Fluent Bit are also covered.

1. Cloud Native Logging
Fluentd Summit, June 01th, 2017 - Japan
Eduardo Silva @edsiper
eduardo@treasure-data.com

2. “Logging... Why?”

3. Logging: Why ?
Analyze Applications Behavior

4. Logging: Why ?
Analyze Applications Behavior

5. Logging: Why ?

6. Logging: Why ?

7. Logging Pipeline

8. Logging Pipeline

9. Logging Pipeline

10. It’s a mess

11. “Logging... Why?”
(Micro Services)

12. OK, Where’s My Log in Container Era?

13. Logging has
Never Been Easier

15. About Fluentd
● Invented by
● Now hosted at

16. Logging Pipeline

17. How Fluentd Simplifies Logging Pipeline

18. How Fluentd Simplifies Unified Logging Layer

19. About Fluentd
● More than 600 plugins available
● Pluggable Architecture
● Built-in Reliability
● Full integration with Docker and Kubernetes
● Written in Ruby + C

20. Fluentd Plugin Ecosystem

21. Fluentd Modular Architecture

22. Fluentd Structured Logging

23. Fluentd at Scale

24. Kubernetes & Fluentd

25. Kubernetes 101

26. Kubernetes 101

27. Kubernetes 101

28. Kubernetes 101

29. Kubernetes 101

30. Kubernetes 101

31. Fluentd Deployment: DaemonSet

32. Kubernetes: Fluentd Log Collection

33. Fluentd: Enrich Logs with Kubernetes Metadata

34. Storing the Logs

35. Example: Logs from the Container

36. Logs Stored by Fluentd

37. Fluentd: Error Handling

38. Fluentd: Retrying

39. Fluentd: Kubernetes DaemonSet
Pre-built Yaml file
https://github.com/fluent/fluentd-kubernetes-daemonset/

40. Extending the Fluentd
Ecosystem

41. Fluentd Modes
● Log Forwarder
● Log Aggregator

42. Log Aggregator =
(Forwarder + Buffering Capabilities)

43. Edge Nodes / Forward to Aggregators
App
Node 1
DB
App DB

44. Edge Nodes & Costs

45. Edge Nodes & Costs
● Fluentd requires ~40MB as minimum
● Deploying a few hundred could be expensive
● Can we make Forward cheaper ?

46. Forwarder & Aggregator
Log Forwarder Log Aggregator

48. About Fluent Bit
● Written in C
● Pluggable Architecture
● Built-in Reliability
● Event Driven - Async I/O

49. Why Fluent Bit as a Forwarder
● Features
○ Input, Filter and Output Plugins
○ Built-in parsing support
○ Minimum memory required 450KB

50. Edge Nodes / Forward to Aggregators

51. Cheap Forwarding

52. Cloud Native Features
● Backpressure Handling
● Security
● Kubernetes Metadata Filter
● Timestamps w/Nanoseconds (upcoming v0.12)

53. Cloud Native Features
● Backpressure

54. Cloud Native Features
● Backpressure

55. Cloud Native Features
● Backpressure

56. Cloud Native Features
● Backpressure

57. Cloud Native Features
● Backpressure Solved

58. Security
● Native integration for TLS/SSL
■ Optional for plugins with network I/O
■ Certificates support

59. Features, plugins and more
● Plugins (~30)
○ Input
■ cpu, disk, forward, mem, syslog, tcp, kmsg...
○ Filter
■ Grep, Kubernetes & Record Modifier
○ Output
■ Elasticsearch, HTTP, InfluxDB, Forward, Treasure Data

60. Fluent Bit, what else ?
Networking and Co-routines
Easier implementation of output plugins that interact with networking
operations like socket(), connect(), read(), write(), etc.
Fluent Bit provides non-blocking networking API that uses the
event-loop with co-routines to implement:
● Network I/O
● TLS/SSL usage
● HTTP Client

61. Let’s “try” a demo
● Minikube!
Disclaimer: I had the great idea to upgrade
minutes ago!!!! :O

62. Fluent Bit Kubernetes DaemonSet
Github Repository
● https://github.com/fluent/fluent-bit-kubernetes-daemonset
Docker Hub Image
● fluent/fluent-bit-kubernetes-daemonset

63. Fluent Bit
Project information
● Web site fluentbit.io
● Github fluent/fluent-bit
Contact / Community
● Slack http://slack.fluentd.org