Alex will be talking about how docker container monitoring was built at Outlyer. He'll be diving into the details behind how you actually monitor everything in such an environment and the challenges that come with it. Namely, how the Docker API, Cgroups, and the Netlink Linux kernel interface can be leveraged to get specific metrics for each container.

1. Monitoring for Cloud Scale Microservices
Alexandre Dias | www.outlyer.com | @outlyerapp
How to build a Docker Monitoring
Solution
23-May-2017

2. Some Fun Facts About Docker
The Average
Host Runs 8
Containers
=
8 times more
metrics per
host
The Average
Container
Runs for 2
days
=
More Metric
Churn
12% of Hosts
Run
Containers
& % Growing
Fast
Source: Outlyer Customers

3. Monitoring Docker – The Basics
Physical Server
Hypervisor
OS OS
MonitoringAgent
MySQL
Java
Tomcat
MonitoringAgent
MySQL
PHP
Apache
VM Monitoring
Physical Server
Hypervisor
OS OS
Monitoring
Agent
MySQL
Java
Tomcat
Container Monitoring
Docker
Monitoring
Agent
MySQL
PHP
Apache
Docker

4. Monitoring Docker – The Basics
VM Monitoring
• All processes are accessible from ‘localhost’
• Agent runs in each VM
• Simple plugins can monitor each process
Container Monitoring
• All processes are siloed into containers
• Agent on each host inside its own container or
on the host VM
• Can’t monitor inside containers so you have to
monitor from the outside like a remote mini-
host

5. Monitoring Docker – Monitoring Processes
Shell Command Monitoring
Need to run “docker exec” with container ID
Endpoint Monitoring
Need to inject container IP address on internal
Docker network

6. Monitoring Docker – Organized Chaos via Orchestration

7. What Container Monitoring Looks Like in the Real World

8. Configuration Management is replaced with Auto-discovery

9. Summary: Everything’s dynamic & needs smart automation.

10. Docker - how do we start?

11. Docker Stats

12. Docker Remote API

13. What about /proc ?

14. What about /proc ?

15. Cgroups - Linux Control Groups
•Works on groups of processes
•Limit resource usage
•Check resource usage
•Docker
•CoreOS
•LXC
•systemd

16. Cgroups - Linux Control Groups
•Organised via hierarchies
•Each hierarchy (group to monitor or control) connected to subsystems
•blkio
•cpu
•cpuacct
•memory
•…

17. Cgroups - Linux Control Groups
$ cat /proc/mounts

18. Cgroups - Linux Control Groups

19. Cgroups - Linux Control Groups

20. Cgroups - Linux Control Groups

21. Cgroups - Linux Control Groups
• CPU
• Memory
• Disk
• Load average?
• Network via /proc/<container_pid>/net

22. Netlink - userspace-kernel communication
• Socket family (AF_NETLINK)
• Designed for network information
• Duplex (kernel can initiate sessions)
• Extensible

23. Netlink - Taskstats
• Netlink protocol
• Per-process statistics
https://github.com/facebook/gnlpy

24. Netlink - Cgroupstats
https://en.wikipedia.org/wiki/Load_(computing)#Reckoning_CPU_load
• Extension to Taskstats
• Per-cgroup statistics

25. Making Nagios Plugins Work Against Containers
Shell Command Monitoring
Need to run “docker exec” with container ID
Endpoint Monitoring
Need to inject container IP address on internal
Docker network

26. Making Nagios Plugins Work Against Containers:
Making it (Dark) Magic

27. Making Nagios Plugins Work Against Containers:
Making it (Dark) Magic

28. Making it work with Orchestrators

29. Making it work with Orchestrators
Service
Image from http://blog.arungupta.me/kubernetes-design-patterns/

30. Making it work with Orchestrators: Dimensional Labels
Every container and their metrics gets applied the following
dimensional labels via Kubernetes:
• Node
• Pod
• Service
• Custom Labels

31. Making it work with Orchestrators: Host View

32. What’s Next?

33. Monitoring, done differently.
blog.outlyer.com

34. Monitoring, done differently.
Signup for free at:
www.outlyer.com