SlideShare a Scribd company logo
1 of 18
CONTAINER MONITORING
WITH SYSDIG
Presenter Name: Sreenivas Makam
Presented at: Docker Meetup Bangalore
Presentation Date: Feb 27, 2016
About me
• Senior Engineering Manager at
Cisco Systems Data Center group
• Personal blog can be found at
https://sreeninet.wordpress.com/
and my hacky code at
https://github.com/smakam
• Author of “Mastering CoreOS”
book, published on Feb 2016.
(https://sreeninet.wordpress.com/
2016/02/27/mastering-coreos-
book-got-published/)
• You can reach me on LinkedIn at
https://in.linkedin.com/in/sreeniva
smakam
Linux Debug tools and Container
monitoring
• Strace – trace system calls
• Tcpdump, netstat, iftop – monitor network
activity
• Top, htop – track cpu, memory usage
• Lsof – List open files
• Iotop – track process io
Since Containers run in their own namespace, it is
not straightforward to monitor Containers using
these tools.
Container monitoring options
1. Install monitoring tools inside Container.
– This defeats purpose of Container and it’s not scalable.
2. Install monitoring tool inside the host machine
where Container runs.
– Difficult to do this in Container optimized OS like CoreOS,
RancherOS, Atomic
3. Install monitoring tool as a Container with system
level privileges.
– Preferred option
Sysdig follows a combination of 2 and 3.
Native Container monitoring using
Docker tools
• Docker stats – cpu, memory, io
• Docker top – processes in container
• Docker logs – Container logs
• Docker events – Container events
What cannot be done using above approach?
• Top network connections
• Which Containers are talking to each other and which
Containers are talking externally?
• Top files being used
• System calls made
Sysdig Overview
• Sysdig is a monitoring software for bare metal, VM as well as
Containers.
• Sysdig documentation calls sysdig as “strace + tcpdump + htop +
iftop + lsof + ...awesome sauce”
• Sysdig monitors kernel system calls to get monitoring visibility
• Sysdig integrates with Docker, LXC and Rkt for Container
monitoring
• Sysdig integrates with Kubernetes and Mesos for visibility into
Container orchestration
• Post-monitoring can be done using “.scap” files similar to “.pcap”
files with Wireshark.
• Sysdig works mainly in Linux systems. Sysdig for windows can
analyze trace files but not do monitoring.
Sysdig Architecture
• Sysdig-probe is installed as kernel module.
• Sysdig does monitoring with minimal kernel and CPU overhead.
Reference:
https://sysdig.com/interpreting-sysdig-output/
Sysdig Container Architecture
• Sysdig can be installed as a Container or as a binary in the host Linux
system
Reference:
https://sysdig.com/let-light-sysdig-adds-container-visibility/
Sysdig software
• Sysdig CLI – Open source CLI tool.
• csysdig - Open source Text based ncurses
interface on top of Sysdig.
• Sysdig cloud – Commercial product
– Available for 14 day free trial.
– Combines Sysdig output from multiple hosts to a
central Sysdig cloud server
– Can be installed on-premise
Sysdig format
• Incremental event number
• Event timestamp – customize this with the -t command line flag (more info)
• CPU ID
• Command name
• Thread ID
• Event direction – ‘>’ means ‘process input’, while ‘<’ means ‘process output’
• Event type
• Event arguments
Eg:
90772 21:19:18.249796600 0 nginx (3212) < accept fd=3(<4t>172.19.0.4:35831-
>172.19.0.2:http) tuple=172.19.0.4:35831->172.19.0.2:http queuepct=0 queuelen=0
queuemax=128
90780 21:19:18.249846551 0 nginx (3212) < open
fd=11(<f>/usr/share/nginx/html/index.html)
name=/usr/share/nginx/html/index.html flags=65(O_NONBLOCK|O_RDONLY)
mode=0
Sysdig examples
• sysdig -pc -c topprocs_cpu – List top processes by CPU usage
• sysdig -pc -c topprocs_net - List top processes by network usage
• sysdig -pc -c topprocs_file - List top processes by io usage
• sysdig -pc -c spy_users – List all commands executed by user
• sysdig -qw dumpfile.scap – Dump all system transactions into
dumpfile.scap tracefile
• sysdig -r dumpfile.scap -c echo_fds container.name=haproxy –
Read trace file and filter output by file io and Container name
• sysdig -pc -A -c echo_fds container.name=haproxy – List all file
activity by Container “haproxy” in ascii format
• sysdig -l -> list filters
• sysdig -cl -> list chisels
• Csysdig –pc -> Start csysdig with Container visibility
Sysdig Kubernetes Integration
• By integrating with Kubernetes, Sysdig becomes
aware of Kubernetes constructs like Namespaces,
Replication controllers, Pods and Services.
• Sysdig becomes aware of Kubernetes constructs by
getting details from Kubernetes API server.
• By grouping monitoring data at Kubernetes construct,
user gets better visibility into the resource usage as a
collection.
• Sysdig cloud has better integration with Kubernetes
than Sysdig since monitoring data at cluster level is
possible only with Sysdig cloud.
Demo-1(Video -
https://www.youtube.com/watch?v=otiHinxObE4)
Network
FE
Network
BE
Ubuntu haproxy Nginx1 nginx2 nginx3
docker network create be
docker network create fe
docker run --name nginx1 --net be -v ~/haproxy/nginx1.html:/usr/share/nginx/html/index.html -d nginx
docker run --name nginx2 --net be -v ~/haproxy/nginx2.html:/usr/share/nginx/html/index.html -d nginx
docker run --name nginx3 --net be -v ~/haproxy/nginx3.html:/usr/share/nginx/html/index.html -d nginx
docker run -d --name haproxy --net be -v ~/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg haproxy
docker network connect fe haproxy
docker run -it --rm --net fe --name ubuntu myubuntu bash
Configurations present at: https://github.com/smakam/docker/tree/master/haproxy
Demo – 2(Guestbook Video -
https://www.youtube.com/watch?v=oQw_2ZNpMd0)
Front end RC Redis master RC Redis slave RC
P P P P P P
Frontend
service
RedisM
service
RedisS
service
Service
Replication
Controller
Pods
./cluster/kubectl.sh create -f examples/guestbook/redis-master-controller.yaml
./cluster/kubectl.sh create -f examples/guestbook/redis-master-service.yaml
./cluster/kubectl.sh create -f examples/guestbook/redis-slave-controller.yaml
./cluster/kubectl.sh create -f examples/guestbook/redis-slave-service.yaml
./cluster/kubectl.sh create -f examples/guestbook/frontend-controller.yaml
./cluster/kubectl.sh create -f examples/guestbook/frontend-service.yaml
References
• Sysdig install (http://www.sysdig.org/install/)
• Interpreting sysdig (https://sysdig.com/interpreting-sysdig-output/)
• Sysdig Internals (https://sysdig.com/sysdig-vs-dtrace-vs-strace-a-
technical-discussion/)
• Sysdig for Containers (https://sysdig.com/let-light-sysdig-adds-container-
visibility/)
• csysdig manpage (http://man7.org/linux/man-
pages/man8/csysdig.8.html)
• Sysdig with Kubernetes (https://sysdig.com/digging-into-kubernetes-
with-sysdig/)
• Sysdig with Mesos, Marathon (http://support.sysdigcloud.com/hc/en-
us/articles/207886103-Sysdig-Cloud-Agent-Mesos-Marathon )
• Sysdig with Rkt (https://sysdig.com/monitoring-rkt-sysdig/ )
• Sysdig with CoreOS (https://sysdig.com/coreos-sysdig-part-1-digging-
into-coreos-environments/)
QUESTIONS?
Setting up Sysdig cloud
• To try it out, I got a 14 day free trial account
from Sysdig website.
• Install Sysdig cloud agent on each node by
using the command specified in Sysdig cloud
settings tab.
• For Kubernetes integration, Sysdig cloud
needs to be installed in both Kubernetes
master and slave nodes.
Setting up Kubernetes cluster
For installing Kubernetes cluster on AWS, I followed these steps after downloading Kubernetes.
export KUBERNETES_PROVIDER=aws
export NUM_MINIONS=2
export MASTER_SIZE=t2.micro
export MINION_SIZE=t2.micro
export KUBE_OS_DISTRIBUTION=trusty
./cluster/kube-up.sh
Note:
• I hit this issue with Kubernetes 1.1.7(http://stackoverflow.com/questions/34993716/failed-
to-run-install-fedora-deps-when-starting-up-local-kubernetes-cluster). I solved it by using
the workaround mentioned in the link.
• To access Guestbook application externally, I used “Nodeport” based load balancer and
opened up the specified port in AWS Security group on the slave nodes.
• To login to Kubernetes AWS nodes, ssh as “ubuntu” user with public key under
.ssh/kube_aws_rsa
• Setting up Kubernetes cluster on Vagrant has a problem with Sysdig cloud since Sysdig cloud
seems to get confused with multiple nodes residing behind a firewall and it shows up as a
single node.

More Related Content

What's hot

What's hot (20)

왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
 
대용량 분산 아키텍쳐 설계 #5. rest
대용량 분산 아키텍쳐 설계 #5. rest대용량 분산 아키텍쳐 설계 #5. rest
대용량 분산 아키텍쳐 설계 #5. rest
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton
 
Google Cloud IAM 계정, 권한 및 조직 관리
Google Cloud IAM 계정, 권한 및 조직 관리Google Cloud IAM 계정, 권한 및 조직 관리
Google Cloud IAM 계정, 권한 및 조직 관리
 
Terraform modules restructured
Terraform modules restructuredTerraform modules restructured
Terraform modules restructured
 
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
 
Terraform on Azure
Terraform on AzureTerraform on Azure
Terraform on Azure
 
Welcome to the Jungle: Pentesting AWS
Welcome to the Jungle: Pentesting AWSWelcome to the Jungle: Pentesting AWS
Welcome to the Jungle: Pentesting AWS
 
[NDC17] Kubernetes로 개발서버 간단히 찍어내기
[NDC17] Kubernetes로 개발서버 간단히 찍어내기[NDC17] Kubernetes로 개발서버 간단히 찍어내기
[NDC17] Kubernetes로 개발서버 간단히 찍어내기
 
OpenShift-Technical-Overview.pdf
OpenShift-Technical-Overview.pdfOpenShift-Technical-Overview.pdf
OpenShift-Technical-Overview.pdf
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
Understanding container security
Understanding container securityUnderstanding container security
Understanding container security
 
Terraform modules and (some of) best practices
Terraform modules and (some of) best practicesTerraform modules and (some of) best practices
Terraform modules and (some of) best practices
 
대용량 분산 아키텍쳐 설계 #1 아키텍쳐 설계 방법론
대용량 분산 아키텍쳐 설계 #1 아키텍쳐 설계 방법론대용량 분산 아키텍쳐 설계 #1 아키텍쳐 설계 방법론
대용량 분산 아키텍쳐 설계 #1 아키텍쳐 설계 방법론
 
[웨비나] 클라우드 마이그레이션 수행 시 가장 많이 하는 질문 Top 10!
[웨비나] 클라우드 마이그레이션 수행 시 가장 많이 하는 질문 Top 10![웨비나] 클라우드 마이그레이션 수행 시 가장 많이 하는 질문 Top 10!
[웨비나] 클라우드 마이그레이션 수행 시 가장 많이 하는 질문 Top 10!
 
Working with Terraform on Azure
Working with Terraform on AzureWorking with Terraform on Azure
Working with Terraform on Azure
 
Final terraform
Final terraformFinal terraform
Final terraform
 
Terraform: An Overview & Introduction
Terraform: An Overview & IntroductionTerraform: An Overview & Introduction
Terraform: An Overview & Introduction
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb Sharding
 
AWS Security Best Practices in a Zero Trust Security Model - DEM06 - Atlanta ...
AWS Security Best Practices in a Zero Trust Security Model - DEM06 - Atlanta ...AWS Security Best Practices in a Zero Trust Security Model - DEM06 - Atlanta ...
AWS Security Best Practices in a Zero Trust Security Model - DEM06 - Atlanta ...
 

Viewers also liked

Viewers also liked (12)

CI, CD with Docker, Jenkins and Tutum
CI, CD with Docker, Jenkins and TutumCI, CD with Docker, Jenkins and Tutum
CI, CD with Docker, Jenkins and Tutum
 
Docker 1.11 Presentation
Docker 1.11 PresentationDocker 1.11 Presentation
Docker 1.11 Presentation
 
Devops in Networking
Devops in NetworkingDevops in Networking
Devops in Networking
 
CoreOS Overview and Current Status
CoreOS Overview and Current StatusCoreOS Overview and Current Status
CoreOS Overview and Current Status
 
Docker Networking - Common Issues and Troubleshooting Techniques
Docker Networking - Common Issues and Troubleshooting TechniquesDocker Networking - Common Issues and Troubleshooting Techniques
Docker Networking - Common Issues and Troubleshooting Techniques
 
Docker Networking Overview
Docker Networking OverviewDocker Networking Overview
Docker Networking Overview
 
Docker Networking Tip - Macvlan driver
Docker Networking Tip - Macvlan driverDocker Networking Tip - Macvlan driver
Docker Networking Tip - Macvlan driver
 
Compare Docker deployment options in the public cloud
Compare Docker deployment options in the public cloudCompare Docker deployment options in the public cloud
Compare Docker deployment options in the public cloud
 
Docker Networking Tip - Load balancing options
Docker Networking Tip - Load balancing optionsDocker Networking Tip - Load balancing options
Docker Networking Tip - Load balancing options
 
Docker Security Overview
Docker Security OverviewDocker Security Overview
Docker Security Overview
 
What is Artificial Intelligence | Artificial Intelligence Tutorial For Beginn...
What is Artificial Intelligence | Artificial Intelligence Tutorial For Beginn...What is Artificial Intelligence | Artificial Intelligence Tutorial For Beginn...
What is Artificial Intelligence | Artificial Intelligence Tutorial For Beginn...
 
10 facts about jobs in the future
10 facts about jobs in the future10 facts about jobs in the future
10 facts about jobs in the future
 

Similar to Container Monitoring with Sysdig

Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...
DataWorks Summit
 

Similar to Container Monitoring with Sysdig (20)

Docker Runtime Security
Docker Runtime SecurityDocker Runtime Security
Docker Runtime Security
 
containerD
containerDcontainerD
containerD
 
Alex Dias: how to build a docker monitoring solution
Alex Dias: how to build a docker monitoring solution Alex Dias: how to build a docker monitoring solution
Alex Dias: how to build a docker monitoring solution
 
WSO2Con USA 2015: Revolutionizing WSO2 PaaS with Kubernetes & App Factory
WSO2Con USA 2015: Revolutionizing WSO2 PaaS with Kubernetes & App FactoryWSO2Con USA 2015: Revolutionizing WSO2 PaaS with Kubernetes & App Factory
WSO2Con USA 2015: Revolutionizing WSO2 PaaS with Kubernetes & App Factory
 
Docker Mentorweek beginner workshop notes
Docker Mentorweek beginner workshop notesDocker Mentorweek beginner workshop notes
Docker Mentorweek beginner workshop notes
 
Oscon 2017: Build your own container-based system with the Moby project
Oscon 2017: Build your own container-based system with the Moby projectOscon 2017: Build your own container-based system with the Moby project
Oscon 2017: Build your own container-based system with the Moby project
 
Container & kubernetes
Container & kubernetesContainer & kubernetes
Container & kubernetes
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
 
PostgreSQL and Linux Containers
PostgreSQL and Linux ContainersPostgreSQL and Linux Containers
PostgreSQL and Linux Containers
 
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
 
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOpsMake stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOps
 
Best Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker ContainersBest Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker Containers
 
Kubernetes Story - Day 1: Build and Manage Containers with Podman
Kubernetes Story - Day 1: Build and Manage Containers with PodmanKubernetes Story - Day 1: Build and Manage Containers with Podman
Kubernetes Story - Day 1: Build and Manage Containers with Podman
 
TIAD 2016 : Migrating 100% of your production services to containers
TIAD 2016 : Migrating 100% of your production services to containersTIAD 2016 : Migrating 100% of your production services to containers
TIAD 2016 : Migrating 100% of your production services to containers
 
Clocker - How to Train your Docker Cloud
Clocker - How to Train your Docker CloudClocker - How to Train your Docker Cloud
Clocker - How to Train your Docker Cloud
 
Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...
 
Postgre sql linuxcontainers by Jignesh Shah
Postgre sql linuxcontainers by Jignesh ShahPostgre sql linuxcontainers by Jignesh Shah
Postgre sql linuxcontainers by Jignesh Shah
 
Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Lions, Tigers and Deers: What building zoos can teach us about securing micro...Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Lions, Tigers and Deers: What building zoos can teach us about securing micro...
 
DevOPS training - Day 2/2
DevOPS training - Day 2/2DevOPS training - Day 2/2
DevOPS training - Day 2/2
 
Docker Advanced registry usage
Docker Advanced registry usageDocker Advanced registry usage
Docker Advanced registry usage
 

More from Sreenivas Makam

More from Sreenivas Makam (10)

GKE Tip Series - Usage Metering
GKE Tip Series -  Usage MeteringGKE Tip Series -  Usage Metering
GKE Tip Series - Usage Metering
 
GKE Tip Series how do i choose between gke standard, autopilot and cloud run
GKE Tip Series   how do i choose between gke standard, autopilot and cloud run GKE Tip Series   how do i choose between gke standard, autopilot and cloud run
GKE Tip Series how do i choose between gke standard, autopilot and cloud run
 
Kubernetes design principles, patterns and ecosystem
Kubernetes design principles, patterns and ecosystemKubernetes design principles, patterns and ecosystem
Kubernetes design principles, patterns and ecosystem
 
My kubernetes toolkit
My kubernetes toolkitMy kubernetes toolkit
My kubernetes toolkit
 
Top 3 reasons why you should run your Enterprise workloads on GKE
Top 3 reasons why you should run your Enterprise workloads on GKETop 3 reasons why you should run your Enterprise workloads on GKE
Top 3 reasons why you should run your Enterprise workloads on GKE
 
How Kubernetes helps Devops
How Kubernetes helps DevopsHow Kubernetes helps Devops
How Kubernetes helps Devops
 
Deep dive into Kubernetes Networking
Deep dive into Kubernetes NetworkingDeep dive into Kubernetes Networking
Deep dive into Kubernetes Networking
 
Service Discovery using etcd, Consul and Kubernetes
Service Discovery using etcd, Consul and KubernetesService Discovery using etcd, Consul and Kubernetes
Service Discovery using etcd, Consul and Kubernetes
 
Docker 1.9 Feature Overview
Docker 1.9 Feature OverviewDocker 1.9 Feature Overview
Docker 1.9 Feature Overview
 
Docker Networking - Current Status and goals of Experimental Networking
Docker Networking - Current Status and goals of Experimental NetworkingDocker Networking - Current Status and goals of Experimental Networking
Docker Networking - Current Status and goals of Experimental Networking
 

Recently uploaded

“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 

Recently uploaded (20)

Navigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiNavigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi Daparthi
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 

Container Monitoring with Sysdig

  • 1. CONTAINER MONITORING WITH SYSDIG Presenter Name: Sreenivas Makam Presented at: Docker Meetup Bangalore Presentation Date: Feb 27, 2016
  • 2. About me • Senior Engineering Manager at Cisco Systems Data Center group • Personal blog can be found at https://sreeninet.wordpress.com/ and my hacky code at https://github.com/smakam • Author of “Mastering CoreOS” book, published on Feb 2016. (https://sreeninet.wordpress.com/ 2016/02/27/mastering-coreos- book-got-published/) • You can reach me on LinkedIn at https://in.linkedin.com/in/sreeniva smakam
  • 3. Linux Debug tools and Container monitoring • Strace – trace system calls • Tcpdump, netstat, iftop – monitor network activity • Top, htop – track cpu, memory usage • Lsof – List open files • Iotop – track process io Since Containers run in their own namespace, it is not straightforward to monitor Containers using these tools.
  • 4. Container monitoring options 1. Install monitoring tools inside Container. – This defeats purpose of Container and it’s not scalable. 2. Install monitoring tool inside the host machine where Container runs. – Difficult to do this in Container optimized OS like CoreOS, RancherOS, Atomic 3. Install monitoring tool as a Container with system level privileges. – Preferred option Sysdig follows a combination of 2 and 3.
  • 5. Native Container monitoring using Docker tools • Docker stats – cpu, memory, io • Docker top – processes in container • Docker logs – Container logs • Docker events – Container events What cannot be done using above approach? • Top network connections • Which Containers are talking to each other and which Containers are talking externally? • Top files being used • System calls made
  • 6. Sysdig Overview • Sysdig is a monitoring software for bare metal, VM as well as Containers. • Sysdig documentation calls sysdig as “strace + tcpdump + htop + iftop + lsof + ...awesome sauce” • Sysdig monitors kernel system calls to get monitoring visibility • Sysdig integrates with Docker, LXC and Rkt for Container monitoring • Sysdig integrates with Kubernetes and Mesos for visibility into Container orchestration • Post-monitoring can be done using “.scap” files similar to “.pcap” files with Wireshark. • Sysdig works mainly in Linux systems. Sysdig for windows can analyze trace files but not do monitoring.
  • 7. Sysdig Architecture • Sysdig-probe is installed as kernel module. • Sysdig does monitoring with minimal kernel and CPU overhead. Reference: https://sysdig.com/interpreting-sysdig-output/
  • 8. Sysdig Container Architecture • Sysdig can be installed as a Container or as a binary in the host Linux system Reference: https://sysdig.com/let-light-sysdig-adds-container-visibility/
  • 9. Sysdig software • Sysdig CLI – Open source CLI tool. • csysdig - Open source Text based ncurses interface on top of Sysdig. • Sysdig cloud – Commercial product – Available for 14 day free trial. – Combines Sysdig output from multiple hosts to a central Sysdig cloud server – Can be installed on-premise
  • 10. Sysdig format • Incremental event number • Event timestamp – customize this with the -t command line flag (more info) • CPU ID • Command name • Thread ID • Event direction – ‘>’ means ‘process input’, while ‘<’ means ‘process output’ • Event type • Event arguments Eg: 90772 21:19:18.249796600 0 nginx (3212) < accept fd=3(<4t>172.19.0.4:35831- >172.19.0.2:http) tuple=172.19.0.4:35831->172.19.0.2:http queuepct=0 queuelen=0 queuemax=128 90780 21:19:18.249846551 0 nginx (3212) < open fd=11(<f>/usr/share/nginx/html/index.html) name=/usr/share/nginx/html/index.html flags=65(O_NONBLOCK|O_RDONLY) mode=0
  • 11. Sysdig examples • sysdig -pc -c topprocs_cpu – List top processes by CPU usage • sysdig -pc -c topprocs_net - List top processes by network usage • sysdig -pc -c topprocs_file - List top processes by io usage • sysdig -pc -c spy_users – List all commands executed by user • sysdig -qw dumpfile.scap – Dump all system transactions into dumpfile.scap tracefile • sysdig -r dumpfile.scap -c echo_fds container.name=haproxy – Read trace file and filter output by file io and Container name • sysdig -pc -A -c echo_fds container.name=haproxy – List all file activity by Container “haproxy” in ascii format • sysdig -l -> list filters • sysdig -cl -> list chisels • Csysdig –pc -> Start csysdig with Container visibility
  • 12. Sysdig Kubernetes Integration • By integrating with Kubernetes, Sysdig becomes aware of Kubernetes constructs like Namespaces, Replication controllers, Pods and Services. • Sysdig becomes aware of Kubernetes constructs by getting details from Kubernetes API server. • By grouping monitoring data at Kubernetes construct, user gets better visibility into the resource usage as a collection. • Sysdig cloud has better integration with Kubernetes than Sysdig since monitoring data at cluster level is possible only with Sysdig cloud.
  • 13. Demo-1(Video - https://www.youtube.com/watch?v=otiHinxObE4) Network FE Network BE Ubuntu haproxy Nginx1 nginx2 nginx3 docker network create be docker network create fe docker run --name nginx1 --net be -v ~/haproxy/nginx1.html:/usr/share/nginx/html/index.html -d nginx docker run --name nginx2 --net be -v ~/haproxy/nginx2.html:/usr/share/nginx/html/index.html -d nginx docker run --name nginx3 --net be -v ~/haproxy/nginx3.html:/usr/share/nginx/html/index.html -d nginx docker run -d --name haproxy --net be -v ~/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg haproxy docker network connect fe haproxy docker run -it --rm --net fe --name ubuntu myubuntu bash Configurations present at: https://github.com/smakam/docker/tree/master/haproxy
  • 14. Demo – 2(Guestbook Video - https://www.youtube.com/watch?v=oQw_2ZNpMd0) Front end RC Redis master RC Redis slave RC P P P P P P Frontend service RedisM service RedisS service Service Replication Controller Pods ./cluster/kubectl.sh create -f examples/guestbook/redis-master-controller.yaml ./cluster/kubectl.sh create -f examples/guestbook/redis-master-service.yaml ./cluster/kubectl.sh create -f examples/guestbook/redis-slave-controller.yaml ./cluster/kubectl.sh create -f examples/guestbook/redis-slave-service.yaml ./cluster/kubectl.sh create -f examples/guestbook/frontend-controller.yaml ./cluster/kubectl.sh create -f examples/guestbook/frontend-service.yaml
  • 15. References • Sysdig install (http://www.sysdig.org/install/) • Interpreting sysdig (https://sysdig.com/interpreting-sysdig-output/) • Sysdig Internals (https://sysdig.com/sysdig-vs-dtrace-vs-strace-a- technical-discussion/) • Sysdig for Containers (https://sysdig.com/let-light-sysdig-adds-container- visibility/) • csysdig manpage (http://man7.org/linux/man- pages/man8/csysdig.8.html) • Sysdig with Kubernetes (https://sysdig.com/digging-into-kubernetes- with-sysdig/) • Sysdig with Mesos, Marathon (http://support.sysdigcloud.com/hc/en- us/articles/207886103-Sysdig-Cloud-Agent-Mesos-Marathon ) • Sysdig with Rkt (https://sysdig.com/monitoring-rkt-sysdig/ ) • Sysdig with CoreOS (https://sysdig.com/coreos-sysdig-part-1-digging- into-coreos-environments/)
  • 17. Setting up Sysdig cloud • To try it out, I got a 14 day free trial account from Sysdig website. • Install Sysdig cloud agent on each node by using the command specified in Sysdig cloud settings tab. • For Kubernetes integration, Sysdig cloud needs to be installed in both Kubernetes master and slave nodes.
  • 18. Setting up Kubernetes cluster For installing Kubernetes cluster on AWS, I followed these steps after downloading Kubernetes. export KUBERNETES_PROVIDER=aws export NUM_MINIONS=2 export MASTER_SIZE=t2.micro export MINION_SIZE=t2.micro export KUBE_OS_DISTRIBUTION=trusty ./cluster/kube-up.sh Note: • I hit this issue with Kubernetes 1.1.7(http://stackoverflow.com/questions/34993716/failed- to-run-install-fedora-deps-when-starting-up-local-kubernetes-cluster). I solved it by using the workaround mentioned in the link. • To access Guestbook application externally, I used “Nodeport” based load balancer and opened up the specified port in AWS Security group on the slave nodes. • To login to Kubernetes AWS nodes, ssh as “ubuntu” user with public key under .ssh/kube_aws_rsa • Setting up Kubernetes cluster on Vagrant has a problem with Sysdig cloud since Sysdig cloud seems to get confused with multiple nodes residing behind a firewall and it shows up as a single node.

Editor's Notes

  1. Microsoft Confidential
  2. Microsoft Confidential