Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Distributed Logging with Kubernetes

23 views

Published on

Gregory Eric Sanderson, software developer at Jive, spoke about the architecture solution for distributed logging with Kubernetes leveraged by Jive/LogMeIn at the Spring 2019 Kubernetes and Cloud Native meetup in Quebec City.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Distributed Logging with Kubernetes

  1. 1. K8S LOG DISTRIBUTION
  2. 2. 2 HELLO! I am Gregory Fodé Sanderson Jive DevOps team You can email me at gregory.sanderson@logmein.com
  3. 3. 3
  4. 4. PROBLEM DEFINITION 1
  5. 5. 8 DATACENTERS Distributed across US, Brazil & Europe 5
  6. 6. 4000 PODS Distributed across US, Brazil & Europe 6
  7. 7. 10600 CONTAINERS Distributed across US, Brazil & Europe 7
  8. 8. 2TB LOGS PER DAY Distributed across US, Brazil & Europe 8
  9. 9. A LOT OF LOGS ! How do we store them ?!?!?!?!?! 9
  10. 10. 10
  11. 11. SOLUTION & DESIGN 2
  12. 12. SOLUTION & DESIGN
  13. 13. SOLUTION: LOG PRODUCERS  Docker reconfigured to add log metadata  DOCKER_OPTS= --log-driver=json-file --log-opt env=K8S_CLUSTER --log-opt labels= io.kubernetes.pod.name, io.kubernetes.container.name, io.kubernetes.container.restartCount, io.kubernetes.pod.namespace --log-opt max-size=20m --log-opt max-file=2
  14. 14. SOLUTION: LOG PRODUCERS  Filebeat  Daemonset  /var/lib/docker/containers/*/*.log  Push logs to kafka  Journalbeat  Daemonset  Reads logs from journald (system logs)  Push logs to kafka
  15. 15. 15
  16. 16. SOLUTION: LOG TRANSPORT  Kafka  Distributed data streaming pipelines  Producer & Consumer model  Fault tolerance for cluster & consumers  Logstash ▻ Consumes logs and sends to ES cluster ▻ All instances part of the same consumer group ▻ Adds metadata to logs
  17. 17. SOLUTION: LOG TRANSPORT grok { match => { '[attrs][io.kubernetes.pod.name]' => '(?<service>.*).[0-9]+-(?<pod_id>[A-Za-z0-9]{5})' } } mutate { add_field => { 'container' => '%{[attrs][io.kubernetes.container.name]}' 'provider' => ‘K8S' } replace => { "hostname" => "%{[beat][name]}.%{site}"} }
  18. 18. SOLUTION: LOG STORAGE  Elasticsearch  Search engine + distributed document index  Fault tolerant: shard replication  Vast community of open source tooling  Retention of 5 days  Kibana ▻ Web UI for filtering Elasticsearch ▻ Used for searching and consulting logs ▻ Tag-based filtering
  19. 19. FUTURE PLANS: MOVE ES TO AWS  Current cluster ▻ 3 bare metal servers ▻ 56 CPUs ▻ 256 RAM ▻ 2 Disks (2TB + 4TB)  AWS cluster ▻ AWS ES managed ▻ 10 i3.2xlarge instances ▻ 64GB RAM ▻ NVMe 1.9 TB disks ▻ 2 AZ
  20. 20. 20
  21. 21. 21 THANKS! Any questions? You can find me at gregory.sanderson@logmein.com

×