Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Docker and Fluentd
2015/06/04
Golang & Docker users meetup
Satoshi Tagomori (@tagomoris)
@tagomoris
Treasure Data, Inc.
Fluentd, Norikra, Hadoop, ...
Which more cute than another?
Logging of container architecture
Storage:
> should be outside of container / hosts
Transferring:
> should be over network...
Container-based world
needs unified logging layer!
Structured logging
Reliable forwarding
Pluggable architecture
http://fluentd.org/
What’s Fluentd?
> Data collector for unified logging layer
> Streaming data transfer based on JSON
> Written in Ruby
> Gem ...
Architecture
EngineInput
Filter Output
Buffer
> grep
> record_transfomer
> …
> Forward
> File tail
> ...
> Forward
> File
...
Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
pros:
simple configuration
cons:
fixed aggr. address
ma...
Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
Apps sends logs
over fluent-logger
there're some patte...
Pattern of 2-level aggr. w/ Fluentd
1. Network transferring
2. Container log & tail
(1) Network transferring
Apps sends logs:
> using fluent-logger-*
> over inter-container network
Pros:
> no performance pen...
(2) Container logger & tail
Apps write logs to STDOUT:
> docker writes it to log file
> fluentd container reads file
Pros:
> ...
Logging drivers
New from docker v1.6!
'Add new Logging driver "fluentd"'
> --log-driver=fluentd
> https://github.com/docker...
over TCP
(3) Container logging driver "fluentd"
Apps write logs to STDOUT:
> docker send it to fluentd
directly!
Pros:
> sim...
Fluentd docker image
Official image by fluent organization
https://registry.hub.docker.com/u/fluent/fluentd/
Use it as it is, ...
❤
Docker and Fluentd (revised)
Docker and Fluentd (revised)
Docker and Fluentd (revised)
Upcoming SlideShare
Loading in …5
×

Docker and Fluentd (revised)

6,046 views

Published on

at Golang and Docker users meetup Tokyo

Published in: Technology

Docker and Fluentd (revised)

  1. 1. Docker and Fluentd 2015/06/04 Golang & Docker users meetup Satoshi Tagomori (@tagomoris)
  2. 2. @tagomoris Treasure Data, Inc. Fluentd, Norikra, Hadoop, ...
  3. 3. Which more cute than another?
  4. 4. Logging of container architecture Storage: > should be outside of container / hosts Transferring: > should be over network Aggregation: > should be done per container / per service
  5. 5. Container-based world needs unified logging layer!
  6. 6. Structured logging Reliable forwarding Pluggable architecture http://fluentd.org/
  7. 7. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
  8. 8. Architecture EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
  9. 9. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation
  10. 10. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation pros: simple configuration cons: fixed aggr. address many connections high load for aggregator pros: less connections lower load for aggr. cons: a bit more container
  11. 11. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation Apps sends logs over fluent-logger there're some patterns... or ...
  12. 12. Pattern of 2-level aggr. w/ Fluentd 1. Network transferring 2. Container log & tail
  13. 13. (1) Network transferring Apps sends logs: > using fluent-logger-* > over inter-container network Pros: > no performance penalty Cons: > logs of docker is out of scope > a bit complex apps for logging over TCP app
  14. 14. (2) Container logger & tail Apps write logs to STDOUT: > docker writes it to log file > fluentd container reads file Pros: > simple conf for apps & docker > logs include container logs Cons: > in_tail performance penalty log file app http://www.fluentd.org/guides/recipes/docker-logging
  15. 15. Logging drivers New from docker v1.6! 'Add new Logging driver "fluentd"' > --log-driver=fluentd > https://github.com/docker/docker/pull/12876 > New for docker v1.7.0?
  16. 16. over TCP (3) Container logging driver "fluentd" Apps write logs to STDOUT: > docker send it to fluentd directly! Pros: > simple conf for apps & docker > logs include container logs Cons: > none? app Stay tune!
  17. 17. Fluentd docker image Official image by fluent organization https://registry.hub.docker.com/u/fluent/fluentd/ Use it as it is, or build your own container! https://github.com/fluent/fluentd-docker-image
  18. 18.

×