For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.

1. Docker Monitoring
Webinar
20% OFF 201509WNR20S 201509WNR20L
sematext.com/spm sematext.com/logsene

2. Housekeeping / Questions

3. Intro
SPM: Performance monitoring,
Anomaly Detection and Alerting
Logsene: Centralized Log Management
Search and Big Data Consulting
Support for Solr and Elasticsearch

4. Agenda
● Docker Monitoring
○ Docker Monitoring Basics
○ Deployment Options for Docker Agents
● Application Monitoring on Docker
○ Monitoring Containerized Applications
○ Deployment Options for
Application Monitoring Agents

5. Why we like Blue Whales
● Portable packaging
● Optimized resource usage
● Configurable resource limits
● Isolated environment for apps

6. What is Docker made of?

7. Docker Platforms

8. Docker Challenges
● New workflows for deployments
○ from developer laptop straight to the Cloud
● Orchestration of applications
● Logging
○ Webinars 29/30 September, @sematext
● Monitoring

9. Docker Monitoring Basics
Let’s understand what we monitor!

10. Why Docker Monitoring
You can’t fix problems you don’t see!
● Tuning
○ Resources / Container Limits
○ Application Performance
● Quality assurance
● Capacity planning
● Availability / SLA’s

11. What to monitor?
Cluster
Server 1
Container 1
Container 2
Container N
Container 1
Elasticsearch
Container 2
Node.js - web service
Container N
NGINX - reverse proxy
Server N
JVM Runtime Env
JS Runtime Env

12. The Metrics Stack Changed!
Server Metrics
Runtime Metrics
App Metrics
Server Metrics
Runtime Metrics
App Metrics
Container Metrics
Runtime Metrics = JVM Metrics or Node.js Metrics, …
Without Docker With Docker

13. Poll
What is the result of ‘vmstat’ when started in a
container?
A) OS Metrics limited to the container
B) OS Metrics from the docker host

14. Docker Stats Command
> docker stats $(docker ps -q)
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
19a9f4c9ecb7 0.00% 1.421 MB/3.944 GB 0.04% 0 B/0 B
7f043a32307c 0.00% 1.413 MB/3.944 GB 0.04% 0 B/0 B
889646bd003f 0.11% 4.235 MB/3.944 GB 0.11% 0 B/0 B
9a29f494bf7d 0.27% 156.8 MB/3.944 GB 3.98% 0 B/0 B
Too simple!

15. Docker API for Metrics
> echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn"
| nc -U /var/run/docker.sock
HTTP/1.0 200 OK
Date: Thu, 30 Jul 2015 08:37:31 GMT
Content-Type: text/plain; charset=utf-8
{"read":"2015-07-30T04:37:31.13789208-04:00","network":{"rx_bytes":636450,"rx_packets":1944,"
rx_errors":0,"rx_dropped":0,"tx_bytes":495897,"tx_packets":2415,"tx_errors":0,"tx_dropped":0},"
precpu_stats":{"cpu_usage":{"total_usage":0,"percpu_usage":null,"usage_in_kernelmode":0,"
usage_in_usermode":0},"system_cpu_usage":0,"throttling_data":{"periods":0,"throttled_periods":
0,"throttled_time":0}},"cpu_stats ...
Too verbose!
More than 70 Metrics
are exposed

16. Watching a Metric
> echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn"
| nc -U /var/run/docker.sock
| logagent -y
| grep rx_bytes
rx_bytes: 667326
rx_bytes: 667354
rx_bytes: 667413
rx_bytes: 667538
github.com/sematext/logagent-js
Which Metrics are
most relevant?

17. Docker Key Metrics

18. Server Metrics
● CPU
● Memory Usage
● Disk Space
○ Docker Images consume a lot of disk space

19. Container Metrics
● Containers CPU
+ Throttled CPU time / CPU limits!
● Memory Limits
+ Memory Allocation Fail Counters
● Disk I/O + Wait Times
● Network I/O + Network Errors

20. Docker Key Metrics in SPM

21. Beyond Metrics: Events
● Docker Events
○ container auditing
○ create, destroy, die, export, kill,
pause, restart, start, stop,
unpause, oom

22. Beyond Metrics: Logs
● Logs
○ Docker collects console output
of each container
○ API or Log Drivers for forwarding
● Correlation with Metrics
○ Faster troubleshooting
blog.sematext.com/2015/08/11/docker-logging-webinar/
Docker
Logging
Webinar

23. Deploying Docker
Monitoring Agents

24. Directly on each Docker Host
● The traditional way!
● Full access to the operating system
● Limitations in ’Container only’ systems
- CoreOS
- RancherOS
- Many PaaS Solutions, e.g., DEIS

25. One Container on each Docker Host
● Yes. The Docker Way!
● Portable to any ‘Docker platform’
● Requirements:
- Access to the Host Metrics + Docker API
- Extended privileges (e.g., for SELinux, GCE,...)

26. Containerized Monitoring Agents

27. SPM for Docker Monitoring
apps.sematext.com/spm-reports/registerApplication.do
hub.docker.com/r/sematext/spm-agent-docker/
github.com/sematext/spm-agent-docker

28. Example 1: Run SPM for Docker
docker run -d --name spm-agent
-v /var/run/docker.sock:/var/run/docker.sock
-e SPM_TOKEN=fe31fc3a-xxxx-47c6-b83c-be376bfxxx
-e HOSTNAME=$HOSTNAME
-e SPM_MONITORING_TAGS="project:dev-test,role:demo"
-e LOGSENE_TOKEN=53a6c7e7-xxxx-4725-962e-ea47cebxxx
--privileged
sematext/spm-agent-docker

29. Example 2: CoreOS + DEIS
● CoreOS has no package manager
● systemd - init system
● fleet - as distributed init system
● etcd - for central configurations
How to distribute a Monitoring
Agent to all CoreOS servers in a cluster?

30. Example 2

31. Example 2: Distribute the Agent
# Configure SPM app token
etcdctl set /sematext.com/myapp/spm/token YOUR_SPM_TOKEN
# Download the fleet unit file for SPM
wget
https://raw.githubusercontent.com/sematext/spm-agent-docker/master/coreos/spm-agent.service
-O spm-agent.service
# Start SPM Agent in the whole cluster
fleetctl load spm-agent.service
fleetctl start spm-agent.service

32. Example 3: RANCHER OS
● Minimalistic (~25 MB !)
● No package manager
● All system services are containers
● No cluster manager (by default)
● Use restart policy ‘always’ for services
docker run --restart=always … sematext/spm-agent-docker

33. Application Monitoring
layers.next()

34. Application Metrics

35. How are Applications monitored?
● Standalone Agents
○ Remote interface: JMX, MySQL, NGINX, Apache
○ No restart required, when agent gets updated
● In-Process Agents
○ Resource friendly
○ Node.js
○ JVM Apps (Elasticsearch, Solr, Kafka, ...)
- Function Transaction/Call Tracing possible

36. Transaction Tracing + AppMap
● Only possible with in-process monitoring

37. Deployment Options for
Application Monitoring
… on Docker

38. Everything in Each Container

39. Standalone in Separate Container

40. In-Process in Separate Container

41. Setup for Elasticsearch Image
spm-client:
image: sematext/spm-client # Exports Volume /opt/spm
container_name: spm-client
environment:
- SPM_CFG="${SPM_TOKEN} es javaagent jvmname:ES1"
ELASTICSEARCH:
image: elasticsearch
ports:
- "9200:9200"
volumes_from:
- spm-client
environment: # inject in-process monitor from SPM client volume /opt/spm
- ES_JAVA_OPTS="-Dcom.sun.management.jmxremote
-javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::ES1"

42. Setup for node.js Apps
Dockerfile:
FROM nodejs
RUN git clone my/app
RUN npm i
RUN npm i spm-agent-nodejs
CMD node -r spm-agent-nodejs app.js
---
docker build -t myapp .
docker run -e SPM_TOKEN=43A... myapp

43. Application + Docker Metrics

44. SPM Logsene
METRICS + LOGS ⇒ BETTER TOGETHER

45. Summary
Stefan Thies
Twitter: @seti321
stefan.thies@sematext.com
info@sematext.com
blog.sematext.com
sematext.com/spm
hub.docker.com/r/sematext/spm-agent-docker/
github.com/sematext/spm-agent-docker

46. Docker Monitoring
Webinar
20% OFF 201509WNR20S 201509WNR20L
sematext.com/spm sematext.com/logsene

47. Thank you for your attention