Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fluentd and docker monitoring

4,831 views

Published on

Fluentd and docker monitoring @ dockerbangalore meetup

Published in: Technology
  • Be the first to comment

Fluentd and docker monitoring

  1. 1. FluentD and Docker Vinay Krishna SolutionsIQ
  2. 2. Monitoring
  3. 3. Is it simple? • Monitoring • Applications • Servers
  4. 4. Logging
  5. 5. What? • Status of application • Keeps information about errors/failure • Status of Network
  6. 6. Why? • Developers • Get help in Debugging • IT admin / support • Get help in Trouble-shooting • Apps running smoothly • Security • Business • Input data – analytics • User interaction / behaviors • Improvements
  7. 7. Assumptions • I have enough disk space • I/O operations will not block • Log messages are human readable • My logging mechanism scale • Basically, yeah.. it should work.
  8. 8. Concerns • Logs increase = data increase • Message format get more complex • Did the Kernel flush the buffers ? (sync(2)) • Multi-thread application ?, locking ? • Multiple Applications = Multiple Logs • If Multiple Applications = Multiple logs • Multiple Hosts x Multiple Applications = ???
  9. 9. How to parse/store multiple data sources ?
  10. 10. Fluentd is an open source data collector for unified logging layer. It allows you to unify data collection and consumption for a better use and understanding of data. • Structured logging • Reliable forwarding • Pluggable architecture
  11. 11. Fluentd • Data collection for unified logging layer • Streaming data transfer based on JSON • Written in Ruby • Gem based various plugins • http://www.fluentd.org/plugins • Working on lots of productions • http://www/fluentd.org/testimonials
  12. 12. Before
  13. 13. After
  14. 14. Highlights • Unified Logging Layer • Fluentd tries to structure data as JSON as much as possible • Simple and yet flexible • 300+ plugins • Open Source • Proven Reliability and Performance • 2000+ data-driven companies rely on FluentD • Minimum resources required - vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core • Data loss should never happen. • Fluentd supports memory- and file-based buffering to prevent inter-node data loss. • Fluentd also supports robust failover and can be set up for high availability • Community
  15. 15. Docker Monitoring
  16. 16. Monitor • Resource utilization • How much RAM and CPU is each container using? • Health of docker environments • As the Docker ecosystem continues to evolve, we have to ask ourselves the following questions: • How can we log and monitor Docker effectively? • This includes logging the Docker runtime infrastructure, the container itself and what goes on inside of it, and how to ensure to collect log data from ephemeral containers. • How can we use feedback from containers to manage and improve the quality of our services? • Can we build off of decades of experience logging monolithic applications, or do we have to start from scratch? • If we have to start from scratch, how can we build a solution that helps us make better decisions?
  17. 17. FluentD + Docker
  18. 18. Logging of container architecture • Storage: • should be outside of container / hosts • Transferring: • should be over network • Aggregation: • should be done per container / per service
  19. 19. FluentD Architecture
  20. 20. Logging Driver • Docker v1.6 released the concept of logging drivers • Route container output • Add new logging driver – fluentd • --log-driver=fluentd • https://github.com/docker/docker/pull/12876 • New for docker v1.7.0?
  21. 21. Container logging driver “fluentd” • Apps write logs to STDOUT: • docker sends it to fluentd directly! • Pros: • simple conf for apps and docker • logs include container logs • Cons: • ?
  22. 22. Fluentd docker image • Official image by fluentd organization https://registry.hub.docker.com/u/fluent/fluentd/ • Use it as it is, or build your own container! https://github.com/fluent/fluentd-docker-image
  23. 23. Demo
  24. 24. Install fluentd • Install fluentd via td-agent curl -L http://toolbelt.treasuredata.com/sh/install- ubuntu-trusty-td-agent2.sh | sh • Start td-agent sudo /etc/init.d/td-agent start
  25. 25. Verify installation • Check the logs to make sure it was installed successfully • tail /var/log/td-agent/td-agent.log
  26. 26. Build fluentd image • Create a new directory for your Fluentd Docker resources, and move into it • mkdir ~/fluentd-docker && cd ~/fluentd-docker • Create the following Dockerfile • sudo nano Dockerfile • Add the following content: FROM ruby:2.2.0 MAINTAINER kiyoto@treausuredata.com RUN apt-get update RUN gem install fluentd -v "~>0.12.3" RUN mkdir /etc/fluent RUN apt-get install -y libcurl4-gnutls-dev make RUN /usr/local/bin/gem install fluent-plugin-elasticsearch ADD fluent.conf /etc/fluent/ ENTRYPOINT ["/usr/local/bundle/bin/fluentd", "-c", "/etc/fluent/fluent.conf"]
  27. 27. Build fluentd image • Create a fluent.conf file in the same directory • sudo nano fluent.conf
  28. 28. <source> type tail read_from_head true path /var/lib/docker/containers/*/*-json.log pos_file /var/log/fluentd-docker.pos time_format %Y-%m-%dT%H:%M:%S tag docker.* format json </source> # Using filter to add container IDs to each event <filter docker.var.lib.docker.containers.*.*.log> type record_transformer <record> container_id ${tag_parts[5]} </record> </filter> <match docker.var.lib.docker.containers.*.*.log> type elasticsearch logstash_format true host "#{ENV['ES_PORT_9200_TCP_ADDR']}" # dynamically configured to use Docker's link feature port 9200 flush_interval 5s </match>
  29. 29. • Build docker image • docker build -t fluentd-es . • Check successfully built the images • docker images
  30. 30. ElasticSearch Container • Move to home directory • Cd ~ • Download and start the Elasticsearch container • docker run -d -p 9200:9200 -p 9300:9300 --name es elasticsearch • Check elasticsearch container is running • docker ps
  31. 31. Start the Fluentd-to-Elasticsearch Container • Start the container that runs Fluentd, collects the logs, and sends them to Elastcisearch • docker run -d --link es:es -v /var/lib/docker/containers:/var/lib/docker/containers fluentd-es • Check that container is running • docker ps
  32. 32. Confirm ElasticSearch receives events • curl -XGET 'http://localhost:9200/_all/_search?q=*‘
  33. 33. What’s next • Setup Dashboard
  34. 34. References • http://www.fluentd.org/guides/recipes/docker- logging • http://www.slideshare.net/repeatedly/docker-and- fluentd-51821582 • http://www.slideshare.net/tagomoris/docker-and- fluentd-revised • https://www.socallinuxexpo.org/sites/default/files/ presentations/fluentd.pdf
  35. 35. Questions?

×