Successfully reported this slideshow.
Your SlideShare is downloading. ×

Make Your Containers Faster: Linux Container Performance Tools

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 23 Ad

Make Your Containers Faster: Linux Container Performance Tools

Download to read offline

If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.

Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.


You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/

If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.

Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.


You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Make Your Containers Faster: Linux Container Performance Tools (20)

Advertisement

More from Kernel TLV (20)

Recently uploaded (20)

Advertisement

Make Your Containers Faster: Linux Container Performance Tools

  1. 1. Linux Container Performance Analysis Sasha Goldshtein CTO, Sela Group goldshtn goldshtn
  2. 2. The Plan • This talk explains how to analyze Linux container performance and which challenges you’ll encounter along the way • You’ll learn: To retrieve basic resource utilization per-container Which deployment strategies to use with container performance tools To dig into high-CPU issues using perf To make symbol resolution work across namespaces To use BPF-based tools with containers 2
  3. 3. Container Resource Utilization 3
  4. 4. Containers Under The Hood in 30 Seconds CONTROL GROUPS • Restrict usage (place quotas) • cpu,cpuacct: used to cap CPU usage and apply CPU shares • docker run --cpus --cpu-shares • memory: used to cap user and kernel memory usage • docker run --memory --kernel-memory • blkio: used to cap IOPS and throughput per block device, and to assign weights (shares) NAMESPACES • Restrict visibility • PID: container gets its own PIDs • mnt: container gets its own / • net: container gets its own network interfaces • user: container gets its own user and group ids • Etc.
  5. 5. USE Checklist for Linux Systems http://www.brendangregg.com/USEmethod/use-linux.html CPU Core Core LLC RAM Memory controller GFX I/O controller SSD E1000 FSB PCIe U: perf U: mpstat -P 0 S: vmstat 1 E: perf U: sar -n DEV 1 S: ifconfig E: ifconfig U: free -m S: sar -B E: dmesg U: iostat 1 S: iostat -xz 1 E: …/ioerr_cntU: perf
  6. 6. Retrieving Container Resource Utilization • High-level, Docker-specific: docker stats • htop with cgroup column (highly unwieldy) • systemd-cgtop • Execute a command in the container : • nsenter -t $PID -p -m top • docker exec -it $CID top • Control group metrics • E.g. /sys/fs/cgroup/cpu,cpuacct/*/*/* • Third-party monitoring solutions: cAdvisor, Intel Snap, PCP, collectd 6
  7. 7. But Beware Of: • /proc/loadavg showing host statistics • free showing systemwide memory usage • iostat showing host block I/O 7
  8. 8. Tool Deployment Strategies 8
  9. 9. Tool Deployment: On The Host 9 User Kernel perf_events node:6 openjdk:8 ubuntu:xenial # perf record -G … -g -F 97
  10. 10. Tool Deployment: In Container 10 User Kernel perf_events node:6 openjdk:8 ubuntu:xenial # perf record -G … -g -F 97 ☢฀
  11. 11. Problems • On the host: • Need privileged access to the host (most container orchestrators try to abstract this away from you) • Tools need to understand container pid namespace, mount namespace, and other details (discussed later) • In the container: • Need to run the container with extra privileges (e.g. for perf: enable perf_event_open syscall, perf_event_paranoid sysctl) • Bloats container images with performance tools that are not always used, and increases attack surface • Performance tools may be throttled by quotas placed on the container 11
  12. 12. Tool Deployment: Sidecar Container 12 User Kernel perf_events node:6 openjdk:8 ubuntu:xenial ubuntu:xenial # perf record -G … -g -F 97 ☢฀
  13. 13. Profiling Containers with perf 13
  14. 14. perf_events Architecture 14 UserKernel tcp_msgsend Page fault LLC miss sched_switch perf_events libc:malloc mmap buffer mmap buffer Real-time analysis perf script…perf.data file
  15. 15. Recording CPU Stacks With perf • To find a CPU bottleneck, record stacks at timed intervals: # system-wide perf record -ag -F 97 # specific process perf record -p 188 -g -F 97 # specific cgroup perf record -G docker-1ae… -g -F 97 Legend -a all CPUs -p specific process -G specific cgroup -g capture call stacks -F frequency of samples (Hz) -c # of events in each sample
  16. 16. Symbol Resolution Challenges • Address to module and symbol resolution, dynamic instrumentation require access to debug information • Because of mount namespace, container’s binaries and debuginfo are not visible to host (/lib64/libc.so.6 – what?) • Need to enter the container’s namespace or share the binaries 16
  17. 17. Perf Maps: Containerized Java/Node/.NET… • For interpreted or JIT-compiled languages, map files need to be generated at runtime $ cat /tmp/perf-1882.map 7f2cd1108880 1e8 Ljava/lang/System;::arraycopy 7f2cd1108c00 200 Ljava/lang/String;::hashCode 7f2cd1109120 2e0 Ljava/lang/String;::indexOf • When profiling from the host: • PID namespace – always /tmp/perf-1.map in the container, not on the host • Mount namespace – container’s /tmp/perf-1.map not visible to host • perf handles this automatically in Linux 4.13+, as do the BCC tools (discussed next) 17
  18. 18. Tracing Containers with BPF 18
  19. 19. BPF Tracing kernel BPF runtimekprobes tracepoints user control program BPF program map application USDT uprobesperf output installs BPF program and attaches to events events invoke the BPF program perf_events BPF program updates a map or pushes a new event to a buffer shared with user-space user-space program is invoked with data from the shared buffer user-space program reads statistics from the map and clears it if necessary control program 19
  20. 20. Java/Node/… Syscall interface Block I/O Ethernet Scheduler Mem Device drivers Filesystem TCP/IP CPU Applications System libraries profile llcstat hardirqs softirqs ttysnoop runqlat cpudist offcputime offwaketime cpuunclaimed memleak oomkill slabratetoptcptop tcplife tcpconnect tcpaccept biotop biolatency biosnoop bitesize filetop filelife fileslower vfscount vfsstat cachestat cachetop mountsnoop *fsslower *fsdist dcstat dcsnoop mdflush execsnoop opensnoop killsnoop statsnoop syncsnoop setuidsnoop mysqld_qslower bashreadline dbslower dbstat mysqlsniff memleak sslsniff gethostlatency deadlock_detector ustat ugc ucalls uthreads uobjnew uflow argdist trace funccount funclatency stackcount 20
  21. 21. Summary • We have seen: To retrieve basic resource utilization per-container Which deployment strategies to use with container performance tools To dig into high-CPU issues using perf To make symbol resolution work across namespaces To use BPF-based tools with containers 21
  22. 22. References • Container performance analysis • https://www.slideshare.net/brendangregg/container-performance-analysis • http://batey.info/docker-jvm-flamegraphs.html • Performance tooling support • https://lkml.org/lkml/2017/7/5/771 • https://github.com/iovisor/bcc/pull/1051 • Monitoring tools • https://github.com/intelsdi-x/snap • http://pcp.io/docs/guide.html • https://github.com/bobrik/collectd-docker • https://github.com/ivotron/docker-perf 22
  23. 23. Thank You! Slides: https://s.sashag.net/ktlv1217 Demos & labs: https://github.com/goldshtn/linux-tracing-workshop ✁ Sasha Goldshtein CTO, Sela Group goldshtn goldshtn

×