2. Who are you?
• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data Inc.
• Fluentd / td-agent developer
• https://jobs.lever.co/treasure-data
• I love OSS :)
• D Language, MessagePack, The organizer of several meetups, etc…
3. Logging for containers
• Storage:
• should be outside of containers / hosts
• Transferring:
• should be over network
• Aggregation:
• should be done per container / per service
7. What’s Fluentd?
• Data collector for unified logging layer
• Streaming data transfer based on JSON
• Written in Ruby
• Gem based various plugins
• http://www.fluentd.org/plugins
• Working on lots of productions
• http://www.fluentd.org/testimonials
10. M x N → M + N
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
buffering / processing /
11. Core Plugins
• Divide & Conquer
• Buffering & Retrying
• Error handling
• Message routing
• Parallelism
• Read data
• Parse data
• Filter data
• Format data
• Write data
13. Aggregation patterns
1) 1-level aggregation 2) 2-level aggregation
Pros:
• simple configuration
Cons:
• fixed aggr, address
• many connections
• high load for aggr
Pros:
• less connections
• lower load for aggr
Cons:
• more containers
14. Aggregation patterns
1) 1-level aggregation 2) 2-level aggregation
Apps sends logs
over fluent-logger
or …
There are some patterns
- via udp / tcp
- tail file
- etc…
15. 2-level aggr w/ Fluentd
1. Transfer logs over network
2. Container logs & tail generated log files
16. 1) Network transferring
• Apps sends logs:
• using fluent-logger-*
• over inter-container network
• Pros:
• no performance penalty
• Cons:
• docker logs is out of scope
• a bit complex for logging
over TCP
app
17. 2) Tail container logs
• Apps write logs to
STDOUT
• docker writes it to log file
• fluentd container reads its file
• Pros:
• simple conf for apps & docker
• logs include container logs
• Cons:
• in_tail performance penalty &
a bit complex configuration
log file
app
http://www.fluentd.org/guides/recipes/docker-logging
19. Logging drivers
• New from docker v1.6
• We can get docker logs directly via drivers
• “fluentd" driver is coming from v1.8
• enabled by “—log-driver=fluentd”
• contributed by @tagomoris
https://github.com/docker/docker/pull/12876
• https://docs.docker.com/reference/logging/fluentd/
20. 3) Logging driver
• Apps write logs to STDOUT
• Docker sends its logs to
fluentd directly
• Pros:
• simple conf for apps & docker
• Logs includes container logs
• Cons:
• Do you see any concerns?
over TCP
app
23. Fluentd docker image
• Official image by fluent organization
• Main maintainer is @tagomoris
• https://registry.hub.docker.com/u/fluent/fluentd/
• Or build your container!
• https://github.com/fluent/fluentd-docker-image