- The document discusses logging for containers using Fluentd, an open source data collector. It describes how Fluentd can provide a unified logging layer, reliably forwarding and aggregating logs from multiple containers and applications in a pluggable way. - Key points covered include using Fluentd with the new Docker logging drivers to directly collect logs from containers, avoiding performance penalties from other approaches. A demo of Fluentd is also mentioned.

1. and
Docker meetup Tokyo #5
Aug 19, 2015

2. Who are you?
• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data Inc.
• Fluentd / td-agent developer
• https://jobs.lever.co/treasure-data
• I love OSS :)
• D Language, MessagePack, The organizer of several meetups, etc…

3. Logging for containers
• Storage:
• should be outside of containers / hosts
• Transferring:
• should be over network
• Aggregation:
• should be done per container / per service

4. Real world…

5. Container-based world
needs unified logging layer

6. Structured logging
Reliable forwarding
Pluggable architecturehttp://fluentd.org/

7. What’s Fluentd?
• Data collector for unified logging layer
• Streaming data transfer based on JSON
• Written in Ruby
• Gem based various plugins
• http://www.fluentd.org/plugins
• Working on lots of productions
• http://www.fluentd.org/testimonials

8. Before…

9. After!

10. M x N → M + N
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
buffering / processing /

11. Core Plugins
• Divide & Conquer
• Buffering & Retrying
• Error handling
• Message routing
• Parallelism
• Read data
• Parse data
• Filter data
• Format data
• Write data

12. Architecture
EngineInput
Filter Output
Buffer
> grep
> record_transfomer
> …
> Forward
> File tail
> ...
> Forward
> File
> ...
Output
> File
> Memory
not pluggable
FormatterParser

13. Aggregation patterns
1) 1-level aggregation 2) 2-level aggregation
Pros:
• simple configuration
Cons:
• fixed aggr, address
• many connections
• high load for aggr
Pros:
• less connections
• lower load for aggr
Cons:
• more containers

14. Aggregation patterns
1) 1-level aggregation 2) 2-level aggregation
Apps sends logs
over fluent-logger
or …
There are some patterns
- via udp / tcp
- tail file
- etc…

15. 2-level aggr w/ Fluentd
1. Transfer logs over network
2. Container logs & tail generated log files

16. 1) Network transferring
• Apps sends logs:
• using fluent-logger-*
• over inter-container network
• Pros:
• no performance penalty
• Cons:
• docker logs is out of scope
• a bit complex for logging
over TCP
app

17. 2) Tail container logs
• Apps write logs to
STDOUT
• docker writes it to log file
• fluentd container reads its file
• Pros:
• simple conf for apps & docker
• logs include container logs
• Cons:
• in_tail performance penalty &
a bit complex configuration
log file
app
http://www.fluentd.org/guides/recipes/docker-logging

18. Seems good?
But we now have smarter way!

19. Logging drivers
• New from docker v1.6
• We can get docker logs directly via drivers
• “fluentd" driver is coming from v1.8
• enabled by “—log-driver=fluentd”
• contributed by @tagomoris
https://github.com/docker/docker/pull/12876
• https://docs.docker.com/reference/logging/fluentd/

20. 3) Logging driver
• Apps write logs to STDOUT
• Docker sends its logs to
fluentd directly
• Pros:
• simple conf for apps & docker
• Logs includes container logs
• Cons:
• Do you see any concerns?
over TCP
app

22. Demo?

23. Fluentd docker image
• Official image by fluent organization
• Main maintainer is @tagomoris
• https://registry.hub.docker.com/u/fluent/fluentd/
• Or build your container!
• https://github.com/fluent/fluentd-docker-image

24. With Kubernetes
• Kubernetes provides fluentd + Elasticsearch +
Kibana image officially
• https://hub.docker.com/r/kubernetes/fluentd-
elasticsearch/
• http://www.dasblinkenlichten.com/logging-in-
kubernetes-with-fluentd-and-elasticsearch/

25. Cloud service for
entire data pipeline!