Recommended
More Related Content
What's hot
What's hot (20)
Similar to ContainerCon sysdig Slides
Similar to ContainerCon sysdig Slides (20)
Recently uploaded
Recently uploaded (20)
ContainerCon sysdig Slides
- 1. The Dark Art of Container Monitoring Loris Degioanni
- 2. Me Loris Degioanni Creator and CEO of sysdig Past: WinPcap, Wireshark @lorisdegio
- 3. Containers Are Great… • less overhead • faster deployments • reproducibility of environments • cost optimizations • Isolation • flexibility
- 4. …But Inspecting Containers Is Not Easy Containers are: • Isolated • Self-Contained • Simple • Lightweight
- 5. …But Inspecting Containers Is Not Easy Containers are: • Isolated • Self-Contained • Simple • Lightweight Can monitoring and troubleshooting respect these properties?
- 6. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 7. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 8. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 10. Hypervisor Monitoring VMs, Option 1 VM1 VM3VM2 Hypervisor-level instrumentation, Amazon CloudWatch
- 11. Hypervisor Monitoring VMs, Option 2 VM1 VM3VM2 Monitoring Agent
- 13. OS Monitoring Containers, Option 1 Container1 Container3Container2 Monitoring Agent
- 14. OS Monitoring Containers, Option 1 Container1 Container3Container2 Monitoring Agent • Not scalable • Not composable • Adds dependencies/size • Kills the concept of one process per container
- 15. OS Monitoring Containers, Option 2 Container1 Container3Container2 Container runtime – level monitoring Kernel-level instrumentation
- 16. OS Monitoring Containers, Option 3 Container1 Monitoring Container Container2
- 17. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 18. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 19. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
- 20. In The Rest of this Talk • Overview of Open Source monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos
- 21. In The Rest of this Talk • Overview of Open Source monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos
- 22. Some Things We Want to Monitor • Resource usage (CPU/Memory/Disk) • Network activity • File I/O activity • Errors/faults • Application activity/Logs • Topology
- 23. Resource Usage (CPU/Memory/Disk) • ps/top/htop from the host • Pro: always there • Cons: Very limited container context (full cgroup string only), work only from the main host • docker ps / docker top / docker stats • Pro: always there, APIs • Cons: Docker only, limited interactivity • cAdvisor • sysdig
- 24. Installing cAdvisor sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest
- 25. cAdvisor Pros and Cons Pros • easy to install • works from a container • user friendly (web UI) • API • Integrations (e.g. InfluxDB, Prometheus, Heapster) Cons: • (very) Limited set of metrics • Containers are black boxes
- 26. Docker Stats API The most sophisticated Docker stats poller ever: echo -e "GET /containers/mysql/stats HTTP/1.1rn" | nc -U /var/run/docker.sock Info: http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-stats- based-on-resource-usage
- 27. Docker Stats API Pros and Cons Pros • Part of Docker • Richer than cAdvisor • REST API • Integrations • docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin • commercial vendors Cons: • Docker only • Containers are black boxes
- 28. sysdig • Capture system events, filter them, run useful scripts • strace + tcpdump + lsof + htop + Lua • Open Source • Nice curses UI
- 29. Design Goals • Production-ready • Simple • lightweight • Rich data • Instant value • Natural workflow • Native support for containers
- 30. Design Goals • Production-ready • Simple • lightweight • Rich data • Instant value • Natural workflow • Native support for containers
- 31. Installing sysdig docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig
- 35. sysdig Architecture Kernel Container1 Docker Container2 Docker Container3 LXCAppApp sysdig Docker (optionally) Save to a trace file foo.scap
- 36. Viewing Network Activity •cAdvisor •docker stats <container> •iftop/tcpdump/tshark in the container •sysdig
- 37. Viewing Disk Activity •iotop/lsof in the container •sysdig
- 38. Viewing Application Activity •Code/container instrumentation •Logs •sysdig
Editor's Notes
- Top: f->cgroups
- Resource usage with sysdig: sysdig –n1000 ./sysdig -pc fd.port=80 and evt.buffer contains GET ./sysdig –c topcontainers_net Top processes Show cpu, network and disk I/O Show internal pid Show etcd and flannel Go inside the Ubuntu container Top containers
- Select haproxy Ports F5 Connections F5 tcpdump docker exec -i -t frontend1 /bin/bash apt-get install tcpdump Top ports in the machine Top users for these ports All connections in the machine Traffic by container Connections in the container
- Select wordpress4 Top directories /etc F5 Errors ENOENT F6 Top files for the whole host Top directories for the whole host Top files and top directories for cadvisor Container errors Drill down into the Ubuntu container Show error with sysdig
- Go to frontend container Top ports view F5 on 80