If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
Agenda:
The Linux kernel has multiple "tracers" built-in, with various degrees of support for aggregation, dynamic probes, parameter processing, filtering, histograms, and other features. Starting from the venerable ftrace, introduced in kernel 2.6, all the way through eBPF, which is still under development, there are many options to choose from when you need to statically instrument your software with probes, or diagnose issues in the field using the system's dynamic probes. Modern tools include SystemTap, Sysdig, ktap, perf, bcc, and others. In this talk, we will begin by reviewing the modern tracing landscape -- ftrace, perf_events, kprobes, uprobes, eBPF -- and what insight into system activity these tools can offer. Then, we will look at specific examples of using tracing tools for diagnostics: tracing a memory leak using low-overhead kmalloc/kfree instrumentation, diagnosing a CPU caching issue using perf stat, probing network and block I/O latency distributions under load, or merely snooping user activities by capturing terminal input and output.
Speaker:
Sasha is the CTO of Sela Group, a training and consulting company based in Israel that employs over 400 developers world-wide. Most of Sasha's work revolves around performance optimization, production debugging, and low-level system diagnostics, but he also dabbles in mobile application development on iOS and Android. Sasha is the author of two books and three Pluralsight courses, and a contributor to multiple open-source projects. He blogs at http://blog.sashag.net.
USENIX ATC 2017 Performance Superpowers with Enhanced BPFBrendan Gregg
Talk for USENIX ATC 2017 by Brendan Gregg
"The Berkeley Packet Filter (BPF) in Linux has been enhanced in very recent versions to do much more than just filter packets, and has become a hot area of operating systems innovation, with much more yet to be discovered. BPF is a sandboxed virtual machine that runs user-level defined programs in kernel context, and is part of many kernels. The Linux enhancements allow it to run custom programs on other events, including kernel- and user-level dynamic tracing (kprobes and uprobes), static tracing (tracepoints), and hardware events. This is finding uses for the generation of new performance analysis tools, network acceleration technologies, and security intrusion detection systems.
This talk will explain the BPF enhancements, then discuss the new performance observability tools that are in use and being created, especially from the BPF compiler collection (bcc) open source project. These tools provide new insights for file system and storage performance, CPU scheduler performance, TCP performance, and much more. This is a major turning point for Linux systems engineering, as custom advanced performance instrumentation can be used safely in production environments, powering a new generation of tools and visualizations.
Because these BPF enhancements are only in very recent Linux (such as Linux 4.9), most companies are not yet running new enough kernels to be exploring BPF yet. This will change in the next year or two, as companies including Netflix upgrade their kernels. This talk will give you a head start on this growing technology, and also discuss areas of future work and unsolved problems."
Nadav Markus goes over the path from a simple crash POC provided by Google Project Zero (for CVE-2015-7547), to a fully weaponized exploit.
He explores how an attacker can utilize the behavior of the Linux kernel in order to bypass ASLR, allowing an attacker to remotely execute code on vulnerable targets.
The Linux Block Layer - Built for Fast StorageKernel TLV
The arrival of flash storage introduced a radical change in performance profiles of direct attached devices. At the time, it was obvious that Linux I/O stack needed to be redesigned in order to support devices capable of millions of IOPs, and with extremely low latency.
In this talk we revisit the changes the Linux block layer in the
last decade or so, that made it what it is today - a performant, scalable, robust and NUMA-aware subsystem. In addition, we cover the new NVMe over Fabrics support in Linux.
Sagi Grimberg
Sagi is Principal Architect and co-founder at LightBits Labs.
A 2015 performance study by Brendan Gregg, Nitesh Kant, and Ben Christensen. Original is in https://github.com/Netflix-Skunkworks/WSPerfLab/tree/master/test-results
Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions.
Agenda:
The Linux kernel has multiple "tracers" built-in, with various degrees of support for aggregation, dynamic probes, parameter processing, filtering, histograms, and other features. Starting from the venerable ftrace, introduced in kernel 2.6, all the way through eBPF, which is still under development, there are many options to choose from when you need to statically instrument your software with probes, or diagnose issues in the field using the system's dynamic probes. Modern tools include SystemTap, Sysdig, ktap, perf, bcc, and others. In this talk, we will begin by reviewing the modern tracing landscape -- ftrace, perf_events, kprobes, uprobes, eBPF -- and what insight into system activity these tools can offer. Then, we will look at specific examples of using tracing tools for diagnostics: tracing a memory leak using low-overhead kmalloc/kfree instrumentation, diagnosing a CPU caching issue using perf stat, probing network and block I/O latency distributions under load, or merely snooping user activities by capturing terminal input and output.
Speaker:
Sasha is the CTO of Sela Group, a training and consulting company based in Israel that employs over 400 developers world-wide. Most of Sasha's work revolves around performance optimization, production debugging, and low-level system diagnostics, but he also dabbles in mobile application development on iOS and Android. Sasha is the author of two books and three Pluralsight courses, and a contributor to multiple open-source projects. He blogs at http://blog.sashag.net.
USENIX ATC 2017 Performance Superpowers with Enhanced BPFBrendan Gregg
Talk for USENIX ATC 2017 by Brendan Gregg
"The Berkeley Packet Filter (BPF) in Linux has been enhanced in very recent versions to do much more than just filter packets, and has become a hot area of operating systems innovation, with much more yet to be discovered. BPF is a sandboxed virtual machine that runs user-level defined programs in kernel context, and is part of many kernels. The Linux enhancements allow it to run custom programs on other events, including kernel- and user-level dynamic tracing (kprobes and uprobes), static tracing (tracepoints), and hardware events. This is finding uses for the generation of new performance analysis tools, network acceleration technologies, and security intrusion detection systems.
This talk will explain the BPF enhancements, then discuss the new performance observability tools that are in use and being created, especially from the BPF compiler collection (bcc) open source project. These tools provide new insights for file system and storage performance, CPU scheduler performance, TCP performance, and much more. This is a major turning point for Linux systems engineering, as custom advanced performance instrumentation can be used safely in production environments, powering a new generation of tools and visualizations.
Because these BPF enhancements are only in very recent Linux (such as Linux 4.9), most companies are not yet running new enough kernels to be exploring BPF yet. This will change in the next year or two, as companies including Netflix upgrade their kernels. This talk will give you a head start on this growing technology, and also discuss areas of future work and unsolved problems."
Nadav Markus goes over the path from a simple crash POC provided by Google Project Zero (for CVE-2015-7547), to a fully weaponized exploit.
He explores how an attacker can utilize the behavior of the Linux kernel in order to bypass ASLR, allowing an attacker to remotely execute code on vulnerable targets.
The Linux Block Layer - Built for Fast StorageKernel TLV
The arrival of flash storage introduced a radical change in performance profiles of direct attached devices. At the time, it was obvious that Linux I/O stack needed to be redesigned in order to support devices capable of millions of IOPs, and with extremely low latency.
In this talk we revisit the changes the Linux block layer in the
last decade or so, that made it what it is today - a performant, scalable, robust and NUMA-aware subsystem. In addition, we cover the new NVMe over Fabrics support in Linux.
Sagi Grimberg
Sagi is Principal Architect and co-founder at LightBits Labs.
A 2015 performance study by Brendan Gregg, Nitesh Kant, and Ben Christensen. Original is in https://github.com/Netflix-Skunkworks/WSPerfLab/tree/master/test-results
Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Have you ever heard of FreeBSD? Probably.
Have you ever interacted with its kernel? Probably not.
In this talk, Gili Yankovitch (nyxsecuritysolutions.com) will talk about the FreeBSD operating system, its network stack and how to write network drivers for it.
The talk will cover the following topics:
* Kernel/User interation in FreeBSD
* The FreeBSD Network Stack
* Network Buffers API
* L2 and L3 Hooking
Kernel Recipes 2015: Kernel packet capture technologiesAnne Nicolas
Sniffing through the ages
Capturing packets running on the wire to send them to a software doing analysis seems at first sight a simple tasks. But one has not to forget that with current network this can means capturing 30M packets per second. The objective of this talk is to show what methods and techniques have been implemented in Linux and how they have evolved over time.
The talk will cover AF_PACKET capture as well as PF_RING, dpdk and netmap. It will try to show how the various evolution of hardware and software have had an impact on the design of these technologies. Regarding software a special focus will be made on Suricata IDS which is implementing most of these capture methods.
Eric Leblond, Stamus Networks
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedAnne Nicolas
Ftrace’s most powerful feature is the function tracer (and function graph tracer which is built from it). But to have this enabled on production systems, it had to have its overhead be negligible when disabled. As the function tracer uses gcc’s profiling mechanism, which adds a call to “mcount” (or more recently fentry, don’t worry if you don’t know what this is, it will all be explained) at the start of almost all functions, it had to do something about the overhead that causes. The solution was to turn those calls into “nops” (an instruction that the CPU simply ignores). But this was no easy feat. It took a lot to come up with a solution (and also turning a few network cards into bricks). This talk will explain the history of how ftrace came about implementing the function tracer, and brought with it the possibility of static branches and soon static calls!
Steven Rostedt
Emerging Persistent Memory Hardware and ZUFS - PM-based File Systems in User ...Kernel TLV
In this talk, Dr. Amit Golander looks into emerging PM/NVDIMM devices, the value they bring to applications and most importantly how they revolutionize the storage stack.
In the second part, Boaz Harrosh and Shachar Sharon dive into new opportunities to develop memory-based filesystems in user space, leveraging a new open source project called ZUFS. ZUFS was presented in the last Linux Plumbers conference and unlike FUSE it focuses on delivering low latency and zero copy.
Dr. Amit Golander was the CTO of Plexistor, which developed the first enterprise-grade PM-based file system, and which was acquired earlier this year by NetApp.
Boaz Harrosh and Shachar Sharon are ZUFS maintainers and longtime Storage and Linux veterans.
Spying on the Linux kernel for fun and profitAndrea Righi
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
Video: https://www.facebook.com/atscaleevents/videos/1693888610884236/ . Talk by Brendan Gregg from Facebook's Performance @Scale: "Linux performance analysis has been the domain of ancient tools and metrics, but that's now changing in the Linux 4.x series. A new tracer is available in the mainline kernel, built from dynamic tracing (kprobes, uprobes) and enhanced BPF (Berkeley Packet Filter), aka, eBPF. It allows us to measure latency distributions for file system I/O and run queue latency, print details of storage device I/O and TCP retransmits, investigate blocked stack traces and memory leaks, and a whole lot more. These lead to performance wins large and small, especially when instrumenting areas that previously had zero visibility. This talk will summarize this new technology and some long-standing issues that it can solve, and how we intend to use it at Netflix."
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
Kernel Recipes 2015: Solving the Linux storage scalability bottlenecksAnne Nicolas
lash devices introduced a sudden shift in the performance profile of direct attached storage. With IOPS rates orders of magnitude higher than rotating storage, it became clear that Linux needed a re-design of its storage stack to properly support and get the most out of these new devices.
This talk will detail the architecture of blk-mq, the redesign of the core of the Linux storage stack, and the later set of changes made to adapt the SCSI stack to this new queuing model. Early results of running Facebook infrastructure production workloads on top of the new stack will also be shared.
Jense Axboe, Facebook
Kernel Recipes 2015 - So you want to write a Linux driver frameworkAnne Nicolas
Writing a new driver framework in Linux is hard. There are many pitfalls along the way; this talk hopes to point out some of those pitfalls and hard lessons learned through examples, advice and humorous anecdotes in the hope that it will aid those adventurous enough to take on the task of writing a new driver framework. The scope of the talk includes internal framework design as well as external API design exposed to drivers and consumers of the framework. This presentation pulls directly from the Michael Turquette’s experience authoring the Common Clock
Framework and maintaining that code for the last four years.
Additionally Mike has solicited tips and advice from other subsystem maintainers, for a well-rounded overview. Be prepared to learn some winning design patterns and hear some embarrassing stories of framework design gone wrong.
Mike Turquette, BayLibre
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...OpenShift Origin
Extending OpenShift Origin: Build Your Own Cartridge
Presenters: Bill DeCoste
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
An Introduction to eBPF (and cBPF). Topics covered include history, implementation, program types & maps. Also gives a brief introduction to XDP and DPDK
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Have you ever heard of FreeBSD? Probably.
Have you ever interacted with its kernel? Probably not.
In this talk, Gili Yankovitch (nyxsecuritysolutions.com) will talk about the FreeBSD operating system, its network stack and how to write network drivers for it.
The talk will cover the following topics:
* Kernel/User interation in FreeBSD
* The FreeBSD Network Stack
* Network Buffers API
* L2 and L3 Hooking
Kernel Recipes 2015: Kernel packet capture technologiesAnne Nicolas
Sniffing through the ages
Capturing packets running on the wire to send them to a software doing analysis seems at first sight a simple tasks. But one has not to forget that with current network this can means capturing 30M packets per second. The objective of this talk is to show what methods and techniques have been implemented in Linux and how they have evolved over time.
The talk will cover AF_PACKET capture as well as PF_RING, dpdk and netmap. It will try to show how the various evolution of hardware and software have had an impact on the design of these technologies. Regarding software a special focus will be made on Suricata IDS which is implementing most of these capture methods.
Eric Leblond, Stamus Networks
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedAnne Nicolas
Ftrace’s most powerful feature is the function tracer (and function graph tracer which is built from it). But to have this enabled on production systems, it had to have its overhead be negligible when disabled. As the function tracer uses gcc’s profiling mechanism, which adds a call to “mcount” (or more recently fentry, don’t worry if you don’t know what this is, it will all be explained) at the start of almost all functions, it had to do something about the overhead that causes. The solution was to turn those calls into “nops” (an instruction that the CPU simply ignores). But this was no easy feat. It took a lot to come up with a solution (and also turning a few network cards into bricks). This talk will explain the history of how ftrace came about implementing the function tracer, and brought with it the possibility of static branches and soon static calls!
Steven Rostedt
Emerging Persistent Memory Hardware and ZUFS - PM-based File Systems in User ...Kernel TLV
In this talk, Dr. Amit Golander looks into emerging PM/NVDIMM devices, the value they bring to applications and most importantly how they revolutionize the storage stack.
In the second part, Boaz Harrosh and Shachar Sharon dive into new opportunities to develop memory-based filesystems in user space, leveraging a new open source project called ZUFS. ZUFS was presented in the last Linux Plumbers conference and unlike FUSE it focuses on delivering low latency and zero copy.
Dr. Amit Golander was the CTO of Plexistor, which developed the first enterprise-grade PM-based file system, and which was acquired earlier this year by NetApp.
Boaz Harrosh and Shachar Sharon are ZUFS maintainers and longtime Storage and Linux veterans.
Spying on the Linux kernel for fun and profitAndrea Righi
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
Video: https://www.facebook.com/atscaleevents/videos/1693888610884236/ . Talk by Brendan Gregg from Facebook's Performance @Scale: "Linux performance analysis has been the domain of ancient tools and metrics, but that's now changing in the Linux 4.x series. A new tracer is available in the mainline kernel, built from dynamic tracing (kprobes, uprobes) and enhanced BPF (Berkeley Packet Filter), aka, eBPF. It allows us to measure latency distributions for file system I/O and run queue latency, print details of storage device I/O and TCP retransmits, investigate blocked stack traces and memory leaks, and a whole lot more. These lead to performance wins large and small, especially when instrumenting areas that previously had zero visibility. This talk will summarize this new technology and some long-standing issues that it can solve, and how we intend to use it at Netflix."
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
Kernel Recipes 2015: Solving the Linux storage scalability bottlenecksAnne Nicolas
lash devices introduced a sudden shift in the performance profile of direct attached storage. With IOPS rates orders of magnitude higher than rotating storage, it became clear that Linux needed a re-design of its storage stack to properly support and get the most out of these new devices.
This talk will detail the architecture of blk-mq, the redesign of the core of the Linux storage stack, and the later set of changes made to adapt the SCSI stack to this new queuing model. Early results of running Facebook infrastructure production workloads on top of the new stack will also be shared.
Jense Axboe, Facebook
Kernel Recipes 2015 - So you want to write a Linux driver frameworkAnne Nicolas
Writing a new driver framework in Linux is hard. There are many pitfalls along the way; this talk hopes to point out some of those pitfalls and hard lessons learned through examples, advice and humorous anecdotes in the hope that it will aid those adventurous enough to take on the task of writing a new driver framework. The scope of the talk includes internal framework design as well as external API design exposed to drivers and consumers of the framework. This presentation pulls directly from the Michael Turquette’s experience authoring the Common Clock
Framework and maintaining that code for the last four years.
Additionally Mike has solicited tips and advice from other subsystem maintainers, for a well-rounded overview. Be prepared to learn some winning design patterns and hear some embarrassing stories of framework design gone wrong.
Mike Turquette, BayLibre
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...OpenShift Origin
Extending OpenShift Origin: Build Your Own Cartridge
Presenters: Bill DeCoste
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
An Introduction to eBPF (and cBPF). Topics covered include history, implementation, program types & maps. Also gives a brief introduction to XDP and DPDK
State of Containers and the Convergence of HPC and BigDatainside-BigData.com
In this deck from 2018 Swiss HPC Conference, Christian Kniep from Docker Inc. presents: State of Containers and the Convergence of HPC and BigData.
"This talk will recap the history of and what constitutes Linux Containers, before laying out how the technology is employed by various engines and what problems these engines have to solve. Afterward Christian will elaborate on why the advent of standards for images and runtimes moved the discussion from building and distributing containers to orchestrating containerized applications at scale. In conclusion attendees will get an update on how containers foster the convergence of Big Data and HPC workloads and the state of native HPC containers."
Learn more: http://docker.com
and
http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The OpenCSD library for decoding CoreSight traces has reached the point where it is ready to be integrated into applications. This session will present an overview of the state of the library, its interfaces and explore and demonstrate a sample integration with perf.
Customize and Secure the Runtime and Dependencies of Your Procedural Language...VMware Tanzu
Customize and Secure the Runtime and Dependencies of Your Procedural Languages Using PL/Container
Greenplum Summit at PostgresConf US 2018
Hubert Zhang and Jack Wu
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce Diane Mueller
Presenters: Jhon Honce
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...OpenShift Origin
Extending OpenShift Origin: Build Your Own Cartridge
Presenters: Jhon Honce
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
Historically, sharing a Linux server entailed all kinds of untenable compromises. In addition to the security concerns, there was simply no good way to keep one application from hogging resources and messing with the others. The classic “noisy neighbor” problem made shared systems the bargain-basement slums of the Internet, suitable only for small or throwaway projects.
Serious use-cases traditionally demanded dedicated systems. Over the past decade virtualization (in conjunction with Moore’s law) has democratized the availability of what amount to dedicated systems, and the result is hundreds of thousands of websites and applications deployed into VPS or cloud instances. It’s a step in the right direction, but still has glaring flaws.
Most of these websites are just piles of code sitting on a server somewhere. How did that code got there? How can it can be scaled? Secured? Maintained? It’s anybody’s guess. There simply isn’t enough SysAdmin talent in the world to meet the demands of managing all these apps with anything close to best practices without a better model.
Containers are a whole new ballgame. Unlike VMs, you skip the overhead of running an entire OS for every application environment. There’s also no need to provision a whole new machine to have a place to deploy, meaning you can spin up or scale your application with orders of magnitude more speed and accuracy.
This work presents a P4 compiler backend targeting XDP, the eXpress Data Path. P4 is a domain-specific language describing how packets are processed by the data plane of a programmable network elements. XDP is designed for users who want programmability as well as performance.
https://github.com/williamtu/p4c-xdp/
Similar to Make Your Containers Faster: Linux Container Performance Tools (20)
DPDK (Data Plane Development Kit) Overview by Rami Rosen
* Background and short history
* Advantages and disadvantages
- Very High speed networking acceleration in L2
- How this acceleration is achieved (hugepages, optimizations)
- rte_kni (and KCP)
- VPP (and FD.io project) , providing routing and switching.
- TLDK (Transport Layer Development Kit, TCP/UDP)
* Anatomy of a simple DPDK application.
* Development and governance model
* Testpmd: DPDK CLI tool
* DDP - Dynamic Device Profiles
Rami Rosen is a Linux Kernel expert, the author of "Linux Kernel Networking", Apress, 2014.
Rami had published two articles about DPDK in the last year:
"Network acceleration with DPDK"
https://lwn.net/Articles/725254/
"Userspace Networking with DPDK"
https://www.linuxjournal.com/content/userspace-networking-dpdk
Building Network Functions with eBPF & BCCKernel TLV
eBPF (Extended Berkeley Packet Filter) is an in-kernel virtual machine that allows running user-supplied sandboxed programs inside of the kernel. It is especially well-suited to network programs and it's possible to write programs that filter traffic, classify traffic and perform high-performance custom packet processing.
BCC (BPF Compiler Collection) is a toolkit for creating efficient kernel tracing and manipulation programs. It makes use of eBPF.
BCC provides an end-to-end workflow for developing eBPF programs and supplies Python bindings, making eBPF programs much easier to write.
Together, eBPF and BCC allow you to develop and deploy network functions safely and easily, focusing on your application logic (instead of kernel datapath integration).
In this session, we will introduce eBPF and BCC, explain how to implement a network function using BCC, discuss some real-life use-cases and show a live demonstration of the technology.
About the speaker
Shmulik Ladkani, Chief Technology Officer at Meta Networks,
Long time network veteran and kernel geek.
Shmulik started his career at Jungo (acquired by NDS/Cisco) implementing residential gateway software, focusing on embedded Linux, Linux kernel, networking and hardware/software integration.
Some billions of forwarded packets later, Shmulik left his position as Jungo's lead architect and joined Ravello Systems (acquired by Oracle) as tech lead, developing a virtual data center as a cloud-based service, focusing around virtualization systems, network virtualization and SDN.
Recently he co-founded Meta Networks where he's been busy architecting secure, multi-tenant, large-scale network infrastructure as a cloud-based service.
Intel trusted execution environment, SGX, offers an attractive solution for protecting one's private data in the public cloud environment, even in the presence of a malicious OS or VMM.
In this talk, we will:
* explore how SGX mitigates various attack surfaces and the caveats of naively using the technology to protect applications,
* discuss the performance implications of SGX on common applications and understand the new bottlenecks created by SGX, which may lead to a 5X performance degradation.
* describe an optimized SGX interface, HotCalls, that provides a 13-27x speedup compared to the built-in mechanism supplied by the SGX SDK.
* discuss how it is possible for the OS to manage secure memory without having access to it.
* explore various attack surfaces and published attacks which require collusion with the OS. Specifically, page-fault and page-fault-less “controlled channel attacks”, branch-shadowing attacks and potential mitigations.
Ofir Weisse is a Researcher PhD Student at University of Michigan.
Video available at: https://www.youtube.com/watch?v=I3TCctdnOEc
Dima Krasner talks about FUSE, Filesystem in Userspace, its pros and cons, usage, tips and tricks, and more.
Dima is a senior developer at Sam Seamless Network.
Present Absence of Linux Filesystem SecurityKernel TLV
"Present Absence - A character who does not appear for much of or all of the plot, but whose absence is most significant."
Linux filesystem security had appeared before security was an issue. It stagnated for a long time, while the world changed.
A look from the trenches on what is there, what the problems are and what is missing.
Older than he looks, Philip Derbeko has been programming for over 20 years.
He was using Linux since days of Slackware 3.0 with kernel 2.0
Most of the years worked on storage, security systems and machine learning.
Currently, develops and herds a team of Linux, OS X and Windows kernel developers at enSilo.
YouTube: https://www.youtube.com/watch?v=IC-Uf96aUj4
OpenWrt is a Linux distribution for embedded systems that runs on many routers and networking devices today. In this session we'll talk about OpenWrt's origins, architecture and get down to building apps for the platform.
Along the way we will touch on some basic firmware concepts and at last present the final working OpenWrt router and its capabilities.
Anton Lerner, Architect at Sitaro, computer geek, developer and occasional maker.
Sitaro provides total cyber protection for small business and home networks. Sitaro prevents massive scale IoT cyber attacks.
Find out more information in the meetup event page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
Philip Derbeko presents past design decisions that influenced the design of current filesystems, takes a look at how Linux tackles those problems and compares it with other operating systems, and discusses the upcoming revolution in storage and filesystem design.
Older than he looks, Philip Derbeko has been programming for over 20 years.
He was using Linux since days of Slackware 3.0 with kernel 2.0
Most of the years worked on storage, security systems and machine learning.
Currently, develops and herds a team of Linux, OS X and Windows kernel developers at enSilo.
netfilter is a framework provided by the Linux kernel that allows various networking-related operations to be implemented in the form of customized handlers.
iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different netfilter modules) and the chains and rules it stores.
Many systems use iptables/netfilter, Linux's native packet filtering/mangling framework since Linux 2.4, be it home routers or sophisticated cloud network stacks.
In this session, we will talk about the netfilter framework and its facilities, explain how basic filtering and mangling use-cases are implemented using iptables, and introduce some less common but powerful extensions of iptables.
Shmulik Ladkani, Chief Architect at Nsof Networks.
Long time network veteran and kernel geek.
Shmulik started his career at Jungo (acquired by NDS/Cisco) implementing residential gateway software, focusing on embedded Linux, Linux kernel, networking and hardware/software integration.
Some billions of forwarded packets later, Shmulik left his position as Jungo's lead architect and joined Ravello Systems (acquired by Oracle) as tech lead, developing a virtual data center as a cloud-based service, focusing around virtualization systems, network virtualization and SDN.
Recently he co-founded Nsof Networks, where he's been busy architecting network infrastructure as a cloud-based service, gazing at internet routes in astonishment, and playing the chkuku.
Userfaultfd: Current Features, Limitations and Future DevelopmentKernel TLV
userfaultfd is a mechanism that allows user-space paging implementation. Originally designed for post-copy migration of virtual machines, it applies to different use cases, such as implementation of volatile ranges, container migration, improvement of robustness of shared memory, efficient memory snapshotting and more.
This talk covers the current status and supported features of userfaultfd, work currently in progress and future development plans.
Mike Rapoport
Mike has been hacking on the Linux kernel for over a decade. He has added his 2 cents to the mess in arch/arm, contributed to several device drivers and now he is focused on userfaultfd and CRIU.
Mike’s current position is a researcher at IBM.
Linux Kernel Cryptographic API and Use CasesKernel TLV
The Linux kernel has a rich and modular cryptographic API that is used extensively by familiar user facing software such as Android. It's also cryptic, badly documented, subject to change and can easily bite you in unexpected and painful ways.
This talk will describe the crypto API, provide some usage example and discuss some of the more interesting in-kernel users, such as DM-Crypt, DM-Verity and the new fie system encryption code.
Gilad Ben-Yossef is a principal software engineer at ARM. He works on the kernel security sub-system and the ARM CryptCell engine. Open source work done by Gilad includes an experiment in integration of network processors in the networking stack, a patch set for reducing the interference caused to user space processes in large multi-core systems by Linux kernel “maintenance” work and on SMP support for the Synopsys Arc processor among others.
Gilad has co-authored O’Reilly’s “Building Embedded Linux Systems” 2nd edition and presented at such venues as Embedded Linux Conference Europe and the Ottawa Linux Symposium, as well as co-founded Hamakor, an Israeli NGO for the advancement for Open Source and Free Software in Israel. When not hacking on kernel code you can find Gilad meditating and making dad jokes on Twitter.
Ramon Fried covers the following topics:
* What DMA is.
* DMA Buffer Allocations and Management.
* Cache Coherency.
* PCI and DMA.
* dmaengine Framework.
Ramon is an Embedded Linux team leader in TandemG, leading various cutting edge projects in the Linux kernel.
He has years of experience in embedded systems, operating systems and Linux kernel.
Kirill Tsym discusses Vector Packet Processing:
* Linux Kernel data path (in short), initial design, today's situation, optimization initiatives
* Brief overview of DPDK, Netmap, etc.
* Userspace Networking projects comparison: OpenFastPath, OpenSwitch, VPP.
* Introduction to VPP: architecture, capabilities and optimization techniques.
* Basic Data Flow and introduction to vectors.
* VPP Single and Multi-thread modes.
* Router and switch for namespaces example.
* VPP L4 protocol processing - Transport Layer Development Kit.
* VPP Plugins.
Kiril is a software developer at Check Point Software Technologies, part of Next Generation Gateway and Architecture team, developing proof of concept around DPDK and FD.IO VPP. He has years of experience in software, Linux kernel and networking development and has worked for Polycom, Broadcom and Qualcomm before joining Check Point.
In this talk, Ouri Lipner provides an introduction to the WiFi protocol and its implementations in the Linux kernel.
When not busy writing about himself in 3rd person, Ouri is a full time hacker of both cyber and kernel varieties.
Ariel Waizel discusses the Data Plane Development Kit (DPDK), an API for developing fast packet processing code in user space.
* Who needs this library? Why bypass the kernel?
* How does it work?
* How good is it? What are the benchmarks?
* Pros and cons
Ariel worked on kernel development at the IDF, Ben Gurion University, and several companies. He is interested in networking, security, machine learning, and basically everything except UI development. Currently a Solution Architect at ConteXtream (an HPE company), which specializes in SDN solutions for the telecom industry.
Specializing the Data Path - Hooking into the Linux Network StackKernel TLV
Ever needed to add your custom logic into the network stack?
Ever hacked the network stack but wasn't certain you're doing it right?
Shmulik Ladkani talks about various mechanisms for customizing packet processing logic to the network stack's data path.
He covers covering topics such as packet sockets, netfilter hooks, traffic control actions and ebpf. We will discuss their applicable use-cases, advantages and disadvantages.
Shmulik Ladkani is a Tech Lead at Ravello Systems.
Shmulik started his career at Jungo (acquired by NDS/Cisco) implementing residential gateway software, focusing on embedded Linux, Linux kernel, networking and hardware/software integration.
51966 coffees and billions of forwarded packets later, with millions of homes running his software, Shmulik left his position as Jungo’s lead architect and joined Ravello Systems (acquired by Oracle) as tech lead, developing a virtual data center as a cloud service. He's now focused around virtualization systems, network virtualization and SDN.
In this talk Liran will discuss interrupt management in Linux, effective handling, how to defer work using tasklets, workqueues and timers. We'll learn how to handle interrupts in userspace and talk about the performance and latency aspects of each method as well as look at some examples from the kernel source.
Liran is the CTO at Mabel technology and co-founder of DiscoverSDK - Software Libraries directory and DiscoverCloud - Business Apps directory.
More than 20 years of training experience including courses in: Linux, Android, Real-time and Embedded systems, and many more.
userfaultfd is a (relatively) new set of kernel features that allows us to handle page faults in userspace. It is used to implement migration in KVM/Qemu and CRIU. In this talk Mike Rapoport describes the userfaultfd mechanisms and their usage for implementation of post-copy migration. In addition, Mike addresses current limitations of userfaultfd, particularly when it is used in non-cooperative mode.
In this talk Jiří Pírko discusses the design and evolution of the VLAN implementation in Linux, the challenges and pitfalls as well as hardware acceleration and alternative implementations.
Jiří Pírko is a major contributor to kernel networking and the creator of libteam for link aggregation.
In this talk Elad Raz from Mellanox Technologies talks about his effort to open source switch development and transition from proprietary SDKs to a standard, in-kernel solution.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
2. The Plan
• This talk explains how to analyze Linux container performance and
which challenges you’ll encounter along the way
• You’ll learn:
To retrieve basic resource utilization per-container
Which deployment strategies to use with container performance tools
To dig into high-CPU issues using perf
To make symbol resolution work across namespaces
To use BPF-based tools with containers
2
4. Containers Under The Hood in 30 Seconds
CONTROL GROUPS
• Restrict usage (place quotas)
• cpu,cpuacct: used to cap CPU
usage and apply CPU shares
• docker run --cpus --cpu-shares
• memory: used to cap user and
kernel memory usage
• docker run --memory
--kernel-memory
• blkio: used to cap IOPS and
throughput per block device,
and to assign weights (shares)
NAMESPACES
• Restrict visibility
• PID: container gets its own PIDs
• mnt: container gets its own /
• net: container gets its own
network interfaces
• user: container gets its own user
and group ids
• Etc.
5. USE Checklist for Linux Systems
http://www.brendangregg.com/USEmethod/use-linux.html
CPU
Core Core
LLC
RAM
Memory
controller
GFX
I/O
controller
SSD
E1000
FSB
PCIe
U: perf
U: mpstat -P 0
S: vmstat 1
E: perf
U: sar -n DEV 1
S: ifconfig
E: ifconfig
U: free -m
S: sar -B
E: dmesg
U: iostat 1
S: iostat -xz 1
E: …/ioerr_cntU: perf
6. Retrieving Container Resource Utilization
• High-level, Docker-specific: docker stats
• htop with cgroup column (highly unwieldy)
• systemd-cgtop
• Execute a command in the container :
• nsenter -t $PID -p -m top
• docker exec -it $CID top
• Control group metrics
• E.g. /sys/fs/cgroup/cpu,cpuacct/*/*/*
• Third-party monitoring solutions: cAdvisor, Intel Snap, PCP, collectd
6
9. Tool Deployment: On The Host
9
User
Kernel
perf_events
node:6 openjdk:8 ubuntu:xenial
# perf record -G …
-g -F 97
10. Tool Deployment: In Container
10
User
Kernel
perf_events
node:6 openjdk:8
ubuntu:xenial
# perf record -G … -g -F 97
☢
11. Problems
• On the host:
• Need privileged access to the host (most container orchestrators try to
abstract this away from you)
• Tools need to understand container pid namespace, mount namespace, and
other details (discussed later)
• In the container:
• Need to run the container with extra privileges (e.g. for perf: enable
perf_event_open syscall, perf_event_paranoid sysctl)
• Bloats container images with performance tools that are not always used, and
increases attack surface
• Performance tools may be throttled by quotas placed on the container
11
15. Recording CPU Stacks With perf
• To find a CPU bottleneck, record stacks at timed intervals:
# system-wide
perf record -ag -F 97
# specific process
perf record -p 188 -g -F 97
# specific cgroup
perf record -G docker-1ae… -g -F 97
Legend
-a all CPUs
-p specific process
-G specific cgroup
-g capture call stacks
-F frequency of samples (Hz)
-c # of events in each sample
16. Symbol Resolution Challenges
• Address to module and symbol resolution, dynamic instrumentation
require access to debug information
• Because of mount namespace, container’s binaries and debuginfo are not
visible to host (/lib64/libc.so.6 – what?)
• Need to enter the container’s namespace or share the binaries
16
17. Perf Maps: Containerized Java/Node/.NET…
• For interpreted or JIT-compiled languages, map files need to be
generated at runtime
$ cat /tmp/perf-1882.map
7f2cd1108880 1e8 Ljava/lang/System;::arraycopy
7f2cd1108c00 200 Ljava/lang/String;::hashCode
7f2cd1109120 2e0 Ljava/lang/String;::indexOf
• When profiling from the host:
• PID namespace – always /tmp/perf-1.map in the container, not on the host
• Mount namespace – container’s /tmp/perf-1.map not visible to host
• perf handles this automatically in Linux 4.13+, as do the BCC tools
(discussed next)
17
19. BPF Tracing
kernel
BPF runtimekprobes
tracepoints
user
control program
BPF program
map
application
USDT
uprobesperf output
installs BPF program and attaches to events
events invoke the BPF program
perf_events
BPF program updates a map or pushes a
new event to a buffer shared with user-space
user-space program is invoked with
data from the shared buffer
user-space program reads statistics
from the map and clears it if necessary
control
program
19
21. Summary
• We have seen:
To retrieve basic resource utilization per-container
Which deployment strategies to use with container performance tools
To dig into high-CPU issues using perf
To make symbol resolution work across namespaces
To use BPF-based tools with containers
21