Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Distributed Logging Architecture in Container Era

5,278 views

Published on

Tech talk in LinuxCon/ContainerCon Japan 2016

Published in: Technology
  • Be the first to comment

Distributed Logging Architecture in Container Era

  1. 1. Distributed Logging Architecture in Container Era LinuxCon Japan 2016 at Jun 13 2016 Satoshi "Moris" Tagomori (@tagomoris)
  2. 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
  3. 3. http://www.linuxfoundation.org/news-media/announcements/2016/06/chaosuan-crunchy-data-qbox-storageos-and-treasure-data-join-cloud
  4. 4. Topics • Microservices and logging in various industries • Difficulties of logging with containers • Distributed logging architecture • Patterns of distributed logging architecture • Case Study: Docker and Fluentd
  5. 5. Logging
  6. 6. Logging in Various Industries • Web access logs • Views/visitors on media • Views/clicks on Ads • Commercial transactions (EC, Game, ...) • Data from devices • Operation logs on Apps of phones • Various sensor data
  7. 7. Microservices and Logging • Monolithic service • a service produces all data about an user's behavior • Microservices • many services produce data about an user's access • it's needed to collect logs from many services to know what is happening Users Service (Application) Logs Users Logs
  8. 8. Logging and Containers
  9. 9. Containers: "a must" for microservices • Dividing a service into services • a service requires less computing resources
 (VM -> containers) • Making services independent from each other • but it is very difficult :( • some dependency must be solved even in development environment
 (containers on desktop)
  10. 10. Redesign Logging: Why? • No permanent storages • No fixed physical/network address • No fixed mapping between servers and roles • We should parse/label logs at the source, ship these logs by pushing to destination ASAP
  11. 11. Containers: immutable & disposable • No permanent storages • Where to write logs? • files in the container
 → gone w/ container instance 😞 • directories shared from hosts
 → hosts are shared by many containers/services ☹ • TODO: ship logs from container to anywhere ASAP
  12. 12. Containers: unfixed addresses • No fixed physical / network address • Where should we go to fetch logs? • Service discovery (e.g., consul)
 → one more component 😞 • rsync? ssh+tail? or ..? Is it installed in containers?
 → one more tool to depend on ☹ • TODO: push logs to anywhere from containers
  13. 13. Containers: instances per roles • No fixed mapping between servers and roles • How can we parse / store these logs? • Central repository about log syntax
 → very hard to maintain 😞 • Label logs by source address
 → many containers/roles in a host ☹ • TODO: label & parse logs at source of logs
  14. 14. Distributed Logging Architecture
  15. 15. Core Architecture • Collector nodes • Aggregator nodes • Destinations Collector nodes (Docker containers + agent) Destinations
 (Storage, Database, ...) Aggregator nodes
  16. 16. • Parse/Label (collector) • Raw logs are not good for processing • Convert logs to structured data (key-value pairs) • Split/Sort (aggregator) • Mixed logs are not good for searching • Split whole data stream into streams per services • Store (destination) • Format logs(records) as destination expects Collecting and Storing Data
  17. 17. Scaling Logging • Network traffic • CPU load to parse / format • Parse logs on each collector (distributed) • Format logs on aggregator (to be distributed) • Capability • Make aggregators redundant • Controlling delay • to make sure when we can know what's happening in our systems
  18. 18. Patterns
  19. 19. source aggregation NO source aggregation YES destination aggregation NO destination aggregation YES Aggregation Patterns
  20. 20. Source Side Aggregation Patterns w/o source aggregation w/ source aggregation collector aggregator / destination aggregate container
  21. 21. Without Source Aggregation • Pros: • Simple configuration • Cons: • fixed aggregator (endpoint) address • many network connections • high load in aggregator collector aggregator
  22. 22. With Source Aggregation • Pros: • less connections • lower load in aggregator • less configuration in containers
 (by specifying localhost) • highly flexible configuration
 (by deployment only of aggregate containers) • Cons: • a bit much resource (+1 container per host) aggregate container aggregator
  23. 23. Destination Side Aggregation Patterns w/o destination aggregation w/ destination aggregation aggregator collector destination
  24. 24. Without Destination Aggregation • Pros: • Less nodes • Simpler configuration • Cons: • Storage side change affects collector side • Worse performance: many small write requests on storage
  25. 25. With Destination Aggregation • Pros: • Collector side configuration is
 free from storage side changes • Better performance with fine tune
 on destination side aggregator • Cons: • More nodes • A bit complex configuration aggregator
  26. 26. Scaling Patterns Scaling Up Endpoints HTTP/TCP load balancer Huge queue + workers Scaling Out Endpoints Round-robin clients Load balancer Backend nodes Collector nodes Aggregator nodes
  27. 27. Scaling Up Endpoints • Pros: • Simple configuration
 in collector nodes • Cons: • Limits about scaling up Load balancer Backend nodes
  28. 28. Scaling Out Endpoints • Pros: • Unlimited scaling
 by adding aggregator nodes • Cons: • Complex configuration • Client features for round-robin
  29. 29. Without
 Destination Aggregation With
 Destination Aggregation Scaling Up Endpoints Systems in early stages Collecting logs over Internet or Using queues Scaling Out Endpoints Impossible :( Collector nodes must know all endpoints ↓ Uncontrollable Collecting logs in datacenter
  30. 30. Case Studies
  31. 31. Case Study: Docker+Fluentd • Destination aggregation + scaling up • Fluent logger + Fluentd • Source aggregation + scaling up • Docker json logger + Fluentd + Elasticsearch • Docker fluentd logger + Fluentd + Kafka • Source/Destination aggregation + scaling out • Docker fluentd logger + Fluentd
  32. 32. Why Fluentd? • Docker Fluentd logging driver • Docker containers can send logs to Fluentd directly - less overhead • Pluggable architecture • Various destination systems • Small memory footprint • Source aggregation requires +1 container per host • Less additional resource usage ( < 100MB )
  33. 33. Destination aggregation + scaling up • Sending logs directly over TCP by Fluentd logger library in application code • Same with patterns of New Relic • Easy to implement
 - good for startups Application code
  34. 34. Source aggregation + scaling up • Kubernetes: Json logger + Fluentd + Elasticsearch • Applications write logs to STDOUT • Docker writes logs as JSON in files • Fluentd
 reads logs from file
 parse JSON objects
 writes logs to Elasticsearch • EFK stack (like ELK stack) http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/ Elasticsearch Application code Files (JSON)
  35. 35. Source aggregation + scaling up/out • Docker fluentd logging driver + Fluentd + Kafka • Applications write logs to STDOUT • Docker sends logs
 to localhost Fluentd • Fluentd
 gets logs over TCP
 pushes logs into Kafka • Highly scalable & less overhead
 - very good for huge deployment Kafka Application code
  36. 36. Application code Source/Destination aggregation + scaling out • Docker fluentd logging driver + Fluentd • Applications write logs to STDOUT • Docker sends logs
 to localhost Fluentd • Fluentd
 gets logs over TCP
 sends logs into Aggregator Fluentd
 w/ round-robin load balance • Highly flexible
 - good for complex data processing
 requirements Any other storages
  37. 37. What's the Best? • Writing logs from containers: Some way to do it • Docker logging driver • Write logs on files + read/parse it • Send logs from apps directly • Make the platform scalable! • Source aggregation: Fluentd on localhost • Scalable storage: (Kafka, external services, ...) • No destination aggregation + Scaling up • Non-scalable storage: (Filesystems, RDBMSs, ...) • Destination aggregation + Scaling out
  38. 38. Why OSS Are Important For Logging?
  39. 39. Why OSS? • Logging layer is interface • transparency • interoperability • Keep the platform scalable • number of nodes • number of types of source/destination
  40. 40. Use OSS, Make Logging Scalable Thank you!

×