Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Moving towards unified logging


Published on

Moving towards unified logging covers thoughts on moving from proprietary log consolidation tools to open source options such as Elasticsearch, Logstash, Kibana (the ELK stack) along with other ideas like using FluentD in place of Logstash.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Moving towards unified logging

  1. 1. Phil Cryer, September 2014 Moving towards unified logging
  2. 2. goal => decouple data sources from backend systems by providing a unified logging layer to route logs as needed
  3. 3. currently => A host runs a Splunk app and forwards all of its logs to Splunk
  4. 4. Host a host
  5. 5. Splunk Host sends logs to splunk Splunk
  6. 6. => this works, but we want a more flexible, open source solution that doesn’t restrict us with specific tools or size quotas why is this a problem
  7. 7. => create a unified logging layer to handle logs with FluentD, an open source, flexible and lightweight alternative to route logs idea
  8. 8. Host the ELK stack in development…
  9. 9. Elasticsearch Host Logstash logstash writes to elasticsearch
  10. 10. Elasticsearch Host Host Logstashrsyslogd but this can be done just with rsyslogd
  11. 11. Elasticsearch Host Host Logstash FluentD and can also be done with FluentD Host rsyslogd
  12. 12. Elasticsearch Host Host Host Host FluentD Host Logstash FluentD Dockerrsyslogd FluentD but FluentD can be used for more, like routing
  13. 13. Elasticsearch Host Host Host FluentD Host Logstash FluentD Docker FluentD Host rsyslogd and handle input from various data sources
  14. 14. Elasticsearch Host Host Host FluentD Host Splunk Logstash FluentD rsyslogd FluentD FluentD Host rsyslogd and output to various backends, even Splunk
  15. 15. Elasticsearch Host Host Host FluentD Host Splunk Host Logstash FluentD rsyslogd Splunk FluentD Host rsyslogd it could do this independently of Splunk
  16. 16. Elasticsearch Host Host Host FluentD Host Splunk Host Logstash FluentD rsyslogd Splunk FluentD FluentD Host rsyslogd or in parallel
  17. 17. Elasticsearch DB Memcache DNS FluentD Host Splunk IDS Logstash FluentD rsyslogd FluentD App rsyslogd and these could be from a variety applications rsyslogd FluentD
  18. 18. Docker TCP Socket FluentD Host MySQL Scala App Data sources AWS using various data sources AWS, Docker Containers, Flume, Java Apps, MySQL Slow Query Logs, Scala Apps, TCP Socket, and more DB Memcache DNS IDSApp
  19. 19. FluentD Host Kafka FluentD HDFS FluentD Data outputs sent to various data outputs AWS, Kafka, CouchDB, Elasticsearch, Hbase, HDFS, Mongo DB, MySQL, Redis, Riak, Splunk, and more
  20. 20. Moving towards unified logging Thanks. so let’s start…