Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker Monitoring Webinar

5,301 views

Published on

For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.

Published in: Data & Analytics

Docker Monitoring Webinar

  1. 1. Docker Monitoring Webinar 20% OFF 201509WNR20S 201509WNR20L sematext.com/spm sematext.com/logsene
  2. 2. Housekeeping / Questions
  3. 3. Intro SPM: Performance monitoring, Anomaly Detection and Alerting Logsene: Centralized Log Management Search and Big Data Consulting Support for Solr and Elasticsearch
  4. 4. Agenda ● Docker Monitoring ○ Docker Monitoring Basics ○ Deployment Options for Docker Agents ● Application Monitoring on Docker ○ Monitoring Containerized Applications ○ Deployment Options for Application Monitoring Agents
  5. 5. Why we like Blue Whales ● Portable packaging ● Optimized resource usage ● Configurable resource limits ● Isolated environment for apps
  6. 6. What is Docker made of?
  7. 7. Docker Platforms
  8. 8. Docker Challenges ● New workflows for deployments ○ from developer laptop straight to the Cloud ● Orchestration of applications ● Logging ○ Webinars 29/30 September, @sematext ● Monitoring
  9. 9. Docker Monitoring Basics Let’s understand what we monitor!
  10. 10. Why Docker Monitoring You can’t fix problems you don’t see! ● Tuning ○ Resources / Container Limits ○ Application Performance ● Quality assurance ● Capacity planning ● Availability / SLA’s
  11. 11. What to monitor? Cluster Server 1 Container 1 Container 2 Container N Container 1 Elasticsearch Container 2 Node.js - web service Container N NGINX - reverse proxy Server N JVM Runtime Env JS Runtime Env
  12. 12. The Metrics Stack Changed! Server Metrics Runtime Metrics App Metrics Server Metrics Runtime Metrics App Metrics Container Metrics Runtime Metrics = JVM Metrics or Node.js Metrics, … Without Docker With Docker
  13. 13. Poll What is the result of ‘vmstat’ when started in a container? A) OS Metrics limited to the container B) OS Metrics from the docker host
  14. 14. Docker Stats Command > docker stats $(docker ps -q) CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O 19a9f4c9ecb7 0.00% 1.421 MB/3.944 GB 0.04% 0 B/0 B 7f043a32307c 0.00% 1.413 MB/3.944 GB 0.04% 0 B/0 B 889646bd003f 0.11% 4.235 MB/3.944 GB 0.11% 0 B/0 B 9a29f494bf7d 0.27% 156.8 MB/3.944 GB 3.98% 0 B/0 B Too simple!
  15. 15. Docker API for Metrics > echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn" | nc -U /var/run/docker.sock HTTP/1.0 200 OK Date: Thu, 30 Jul 2015 08:37:31 GMT Content-Type: text/plain; charset=utf-8 {"read":"2015-07-30T04:37:31.13789208-04:00","network":{"rx_bytes":636450,"rx_packets":1944," rx_errors":0,"rx_dropped":0,"tx_bytes":495897,"tx_packets":2415,"tx_errors":0,"tx_dropped":0}," precpu_stats":{"cpu_usage":{"total_usage":0,"percpu_usage":null,"usage_in_kernelmode":0," usage_in_usermode":0},"system_cpu_usage":0,"throttling_data":{"periods":0,"throttled_periods": 0,"throttled_time":0}},"cpu_stats ... Too verbose! More than 70 Metrics are exposed
  16. 16. Watching a Metric > echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn" | nc -U /var/run/docker.sock | logagent -y | grep rx_bytes rx_bytes: 667326 rx_bytes: 667354 rx_bytes: 667413 rx_bytes: 667538 github.com/sematext/logagent-js Which Metrics are most relevant?
  17. 17. Docker Key Metrics
  18. 18. Server Metrics ● CPU ● Memory Usage ● Disk Space ○ Docker Images consume a lot of disk space
  19. 19. Container Metrics ● Containers CPU + Throttled CPU time / CPU limits! ● Memory Limits + Memory Allocation Fail Counters ● Disk I/O + Wait Times ● Network I/O + Network Errors
  20. 20. Docker Key Metrics in SPM
  21. 21. Beyond Metrics: Events ● Docker Events ○ container auditing ○ create, destroy, die, export, kill, pause, restart, start, stop, unpause, oom
  22. 22. Beyond Metrics: Logs ● Logs ○ Docker collects console output of each container ○ API or Log Drivers for forwarding ● Correlation with Metrics ○ Faster troubleshooting blog.sematext.com/2015/08/11/docker-logging-webinar/ Docker Logging Webinar
  23. 23. Deploying Docker Monitoring Agents
  24. 24. Directly on each Docker Host ● The traditional way! ● Full access to the operating system ● Limitations in ’Container only’ systems - CoreOS - RancherOS - Many PaaS Solutions, e.g., DEIS
  25. 25. One Container on each Docker Host ● Yes. The Docker Way! ● Portable to any ‘Docker platform’ ● Requirements: - Access to the Host Metrics + Docker API - Extended privileges (e.g., for SELinux, GCE,...)
  26. 26. Containerized Monitoring Agents
  27. 27. SPM for Docker Monitoring apps.sematext.com/spm-reports/registerApplication.do hub.docker.com/r/sematext/spm-agent-docker/ github.com/sematext/spm-agent-docker
  28. 28. Example 1: Run SPM for Docker docker run -d --name spm-agent -v /var/run/docker.sock:/var/run/docker.sock -e SPM_TOKEN=fe31fc3a-xxxx-47c6-b83c-be376bfxxx -e HOSTNAME=$HOSTNAME -e SPM_MONITORING_TAGS="project:dev-test,role:demo" -e LOGSENE_TOKEN=53a6c7e7-xxxx-4725-962e-ea47cebxxx --privileged sematext/spm-agent-docker
  29. 29. Example 2: CoreOS + DEIS ● CoreOS has no package manager ● systemd - init system ● fleet - as distributed init system ● etcd - for central configurations How to distribute a Monitoring Agent to all CoreOS servers in a cluster?
  30. 30. Example 2
  31. 31. Example 2: Distribute the Agent # Configure SPM app token etcdctl set /sematext.com/myapp/spm/token YOUR_SPM_TOKEN # Download the fleet unit file for SPM wget https://raw.githubusercontent.com/sematext/spm-agent-docker/master/coreos/spm-agent.service -O spm-agent.service # Start SPM Agent in the whole cluster fleetctl load spm-agent.service fleetctl start spm-agent.service
  32. 32. Example 3: RANCHER OS ● Minimalistic (~25 MB !) ● No package manager ● All system services are containers ● No cluster manager (by default) ● Use restart policy ‘always’ for services docker run --restart=always … sematext/spm-agent-docker
  33. 33. Application Monitoring layers.next()
  34. 34. Application Metrics
  35. 35. How are Applications monitored? ● Standalone Agents ○ Remote interface: JMX, MySQL, NGINX, Apache ○ No restart required, when agent gets updated ● In-Process Agents ○ Resource friendly ○ Node.js ○ JVM Apps (Elasticsearch, Solr, Kafka, ...) - Function Transaction/Call Tracing possible
  36. 36. Transaction Tracing + AppMap ● Only possible with in-process monitoring
  37. 37. Deployment Options for Application Monitoring … on Docker
  38. 38. Everything in Each Container
  39. 39. Standalone in Separate Container
  40. 40. In-Process in Separate Container
  41. 41. Setup for Elasticsearch Image spm-client: image: sematext/spm-client # Exports Volume /opt/spm container_name: spm-client environment: - SPM_CFG="${SPM_TOKEN} es javaagent jvmname:ES1" ELASTICSEARCH: image: elasticsearch ports: - "9200:9200" volumes_from: - spm-client environment: # inject in-process monitor from SPM client volume /opt/spm - ES_JAVA_OPTS="-Dcom.sun.management.jmxremote -javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::ES1"
  42. 42. Setup for node.js Apps Dockerfile: FROM nodejs RUN git clone my/app RUN npm i RUN npm i spm-agent-nodejs CMD node -r spm-agent-nodejs app.js --- docker build -t myapp . docker run -e SPM_TOKEN=43A... myapp
  43. 43. Application + Docker Metrics
  44. 44. SPM Logsene METRICS + LOGS ⇒ BETTER TOGETHER
  45. 45. Summary Stefan Thies Twitter: @seti321 stefan.thies@sematext.com info@sematext.com blog.sematext.com sematext.com/spm hub.docker.com/r/sematext/spm-agent-docker/ github.com/sematext/spm-agent-docker
  46. 46. Docker Monitoring Webinar 20% OFF 201509WNR20S 201509WNR20L sematext.com/spm sematext.com/logsene
  47. 47. Thank you for your attention

×