Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Container Monitoring with Sysdig

6,397 views

Published on

The slidedeck covers monitoring Docker and Kubernetes with Sysdig, csysdig and Sysdig cloud. Demo video links are embedded inside.

Published in: Technology
  • Be the first to comment

Container Monitoring with Sysdig

  1. 1. CONTAINER MONITORING WITH SYSDIG Presenter Name: Sreenivas Makam Presented at: Docker Meetup Bangalore Presentation Date: Feb 27, 2016
  2. 2. About me • Senior Engineering Manager at Cisco Systems Data Center group • Personal blog can be found at https://sreeninet.wordpress.com/ and my hacky code at https://github.com/smakam • Author of “Mastering CoreOS” book, published on Feb 2016. (https://sreeninet.wordpress.com/ 2016/02/27/mastering-coreos- book-got-published/) • You can reach me on LinkedIn at https://in.linkedin.com/in/sreeniva smakam
  3. 3. Linux Debug tools and Container monitoring • Strace – trace system calls • Tcpdump, netstat, iftop – monitor network activity • Top, htop – track cpu, memory usage • Lsof – List open files • Iotop – track process io Since Containers run in their own namespace, it is not straightforward to monitor Containers using these tools.
  4. 4. Container monitoring options 1. Install monitoring tools inside Container. – This defeats purpose of Container and it’s not scalable. 2. Install monitoring tool inside the host machine where Container runs. – Difficult to do this in Container optimized OS like CoreOS, RancherOS, Atomic 3. Install monitoring tool as a Container with system level privileges. – Preferred option Sysdig follows a combination of 2 and 3.
  5. 5. Native Container monitoring using Docker tools • Docker stats – cpu, memory, io • Docker top – processes in container • Docker logs – Container logs • Docker events – Container events What cannot be done using above approach? • Top network connections • Which Containers are talking to each other and which Containers are talking externally? • Top files being used • System calls made
  6. 6. Sysdig Overview • Sysdig is a monitoring software for bare metal, VM as well as Containers. • Sysdig documentation calls sysdig as “strace + tcpdump + htop + iftop + lsof + ...awesome sauce” • Sysdig monitors kernel system calls to get monitoring visibility • Sysdig integrates with Docker, LXC and Rkt for Container monitoring • Sysdig integrates with Kubernetes and Mesos for visibility into Container orchestration • Post-monitoring can be done using “.scap” files similar to “.pcap” files with Wireshark. • Sysdig works mainly in Linux systems. Sysdig for windows can analyze trace files but not do monitoring.
  7. 7. Sysdig Architecture • Sysdig-probe is installed as kernel module. • Sysdig does monitoring with minimal kernel and CPU overhead. Reference: https://sysdig.com/interpreting-sysdig-output/
  8. 8. Sysdig Container Architecture • Sysdig can be installed as a Container or as a binary in the host Linux system Reference: https://sysdig.com/let-light-sysdig-adds-container-visibility/
  9. 9. Sysdig software • Sysdig CLI – Open source CLI tool. • csysdig - Open source Text based ncurses interface on top of Sysdig. • Sysdig cloud – Commercial product – Available for 14 day free trial. – Combines Sysdig output from multiple hosts to a central Sysdig cloud server – Can be installed on-premise
  10. 10. Sysdig format • Incremental event number • Event timestamp – customize this with the -t command line flag (more info) • CPU ID • Command name • Thread ID • Event direction – ‘>’ means ‘process input’, while ‘<’ means ‘process output’ • Event type • Event arguments Eg: 90772 21:19:18.249796600 0 nginx (3212) < accept fd=3(<4t>172.19.0.4:35831- >172.19.0.2:http) tuple=172.19.0.4:35831->172.19.0.2:http queuepct=0 queuelen=0 queuemax=128 90780 21:19:18.249846551 0 nginx (3212) < open fd=11(<f>/usr/share/nginx/html/index.html) name=/usr/share/nginx/html/index.html flags=65(O_NONBLOCK|O_RDONLY) mode=0
  11. 11. Sysdig examples • sysdig -pc -c topprocs_cpu – List top processes by CPU usage • sysdig -pc -c topprocs_net - List top processes by network usage • sysdig -pc -c topprocs_file - List top processes by io usage • sysdig -pc -c spy_users – List all commands executed by user • sysdig -qw dumpfile.scap – Dump all system transactions into dumpfile.scap tracefile • sysdig -r dumpfile.scap -c echo_fds container.name=haproxy – Read trace file and filter output by file io and Container name • sysdig -pc -A -c echo_fds container.name=haproxy – List all file activity by Container “haproxy” in ascii format • sysdig -l -> list filters • sysdig -cl -> list chisels • Csysdig –pc -> Start csysdig with Container visibility
  12. 12. Sysdig Kubernetes Integration • By integrating with Kubernetes, Sysdig becomes aware of Kubernetes constructs like Namespaces, Replication controllers, Pods and Services. • Sysdig becomes aware of Kubernetes constructs by getting details from Kubernetes API server. • By grouping monitoring data at Kubernetes construct, user gets better visibility into the resource usage as a collection. • Sysdig cloud has better integration with Kubernetes than Sysdig since monitoring data at cluster level is possible only with Sysdig cloud.
  13. 13. Demo-1(Video - https://www.youtube.com/watch?v=otiHinxObE4) Network FE Network BE Ubuntu haproxy Nginx1 nginx2 nginx3 docker network create be docker network create fe docker run --name nginx1 --net be -v ~/haproxy/nginx1.html:/usr/share/nginx/html/index.html -d nginx docker run --name nginx2 --net be -v ~/haproxy/nginx2.html:/usr/share/nginx/html/index.html -d nginx docker run --name nginx3 --net be -v ~/haproxy/nginx3.html:/usr/share/nginx/html/index.html -d nginx docker run -d --name haproxy --net be -v ~/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg haproxy docker network connect fe haproxy docker run -it --rm --net fe --name ubuntu myubuntu bash Configurations present at: https://github.com/smakam/docker/tree/master/haproxy
  14. 14. Demo – 2(Guestbook Video - https://www.youtube.com/watch?v=oQw_2ZNpMd0) Front end RC Redis master RC Redis slave RC P P P P P P Frontend service RedisM service RedisS service Service Replication Controller Pods ./cluster/kubectl.sh create -f examples/guestbook/redis-master-controller.yaml ./cluster/kubectl.sh create -f examples/guestbook/redis-master-service.yaml ./cluster/kubectl.sh create -f examples/guestbook/redis-slave-controller.yaml ./cluster/kubectl.sh create -f examples/guestbook/redis-slave-service.yaml ./cluster/kubectl.sh create -f examples/guestbook/frontend-controller.yaml ./cluster/kubectl.sh create -f examples/guestbook/frontend-service.yaml
  15. 15. References • Sysdig install (http://www.sysdig.org/install/) • Interpreting sysdig (https://sysdig.com/interpreting-sysdig-output/) • Sysdig Internals (https://sysdig.com/sysdig-vs-dtrace-vs-strace-a- technical-discussion/) • Sysdig for Containers (https://sysdig.com/let-light-sysdig-adds-container- visibility/) • csysdig manpage (http://man7.org/linux/man- pages/man8/csysdig.8.html) • Sysdig with Kubernetes (https://sysdig.com/digging-into-kubernetes- with-sysdig/) • Sysdig with Mesos, Marathon (http://support.sysdigcloud.com/hc/en- us/articles/207886103-Sysdig-Cloud-Agent-Mesos-Marathon ) • Sysdig with Rkt (https://sysdig.com/monitoring-rkt-sysdig/ ) • Sysdig with CoreOS (https://sysdig.com/coreos-sysdig-part-1-digging- into-coreos-environments/)
  16. 16. QUESTIONS?
  17. 17. Setting up Sysdig cloud • To try it out, I got a 14 day free trial account from Sysdig website. • Install Sysdig cloud agent on each node by using the command specified in Sysdig cloud settings tab. • For Kubernetes integration, Sysdig cloud needs to be installed in both Kubernetes master and slave nodes.
  18. 18. Setting up Kubernetes cluster For installing Kubernetes cluster on AWS, I followed these steps after downloading Kubernetes. export KUBERNETES_PROVIDER=aws export NUM_MINIONS=2 export MASTER_SIZE=t2.micro export MINION_SIZE=t2.micro export KUBE_OS_DISTRIBUTION=trusty ./cluster/kube-up.sh Note: • I hit this issue with Kubernetes 1.1.7(http://stackoverflow.com/questions/34993716/failed- to-run-install-fedora-deps-when-starting-up-local-kubernetes-cluster). I solved it by using the workaround mentioned in the link. • To access Guestbook application externally, I used “Nodeport” based load balancer and opened up the specified port in AWS Security group on the slave nodes. • To login to Kubernetes AWS nodes, ssh as “ubuntu” user with public key under .ssh/kube_aws_rsa • Setting up Kubernetes cluster on Vagrant has a problem with Sysdig cloud since Sysdig cloud seems to get confused with multiple nodes residing behind a firewall and it shows up as a single node.

×