Debugging is an essential part of Linux kernel development. In
user-space we have the support of the kernel and many debugging tools, tracking down a kernel bug, instead, can be very difficult if you don't know the proper methodologies. This talk will cover some techniques to understand how the kernel works, hunt down and fix kernel bugs in order to become a better kernel developer.
Make Your Containers Faster: Linux Container Performance ToolsKernel TLV
If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
An Introduction to the Formalised Memory Model for Linux KernelSeongJae Park
Linux kernel provides executable and formalized memory model. These slides describe the nature of parallel programming in the Linux kernel and what memory model is and why it is necessary and important for kernel programmers. The slides were used at KOSSCON 2018 (https://kosscon.kr/).
Agenda:
The Linux kernel has multiple "tracers" built-in, with various degrees of support for aggregation, dynamic probes, parameter processing, filtering, histograms, and other features. Starting from the venerable ftrace, introduced in kernel 2.6, all the way through eBPF, which is still under development, there are many options to choose from when you need to statically instrument your software with probes, or diagnose issues in the field using the system's dynamic probes. Modern tools include SystemTap, Sysdig, ktap, perf, bcc, and others. In this talk, we will begin by reviewing the modern tracing landscape -- ftrace, perf_events, kprobes, uprobes, eBPF -- and what insight into system activity these tools can offer. Then, we will look at specific examples of using tracing tools for diagnostics: tracing a memory leak using low-overhead kmalloc/kfree instrumentation, diagnosing a CPU caching issue using perf stat, probing network and block I/O latency distributions under load, or merely snooping user activities by capturing terminal input and output.
Speaker:
Sasha is the CTO of Sela Group, a training and consulting company based in Israel that employs over 400 developers world-wide. Most of Sasha's work revolves around performance optimization, production debugging, and low-level system diagnostics, but he also dabbles in mobile application development on iOS and Android. Sasha is the author of two books and three Pluralsight courses, and a contributor to multiple open-source projects. He blogs at http://blog.sashag.net.
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017OpenEBS
The slides were presented by Jeffry Molanus who is the CTO of OpenEBS in Golang Meetup. OpenEBS is an open source cloud native storage. OpenEBS delivers storage and storage services to containerized environments. OpenEBS allows stateful workloads to be managed more like stateless containers. OpenEBS storage services include: per container (or pod) QoS SLAs, tiering and replica policies across AZs and environments, and predictable and scalable performance.Our vision is simple: let’s let storage and storage services for persistent workloads be so fully integrated into the environment and hence managed automatically that is almost disappears into the background as just yet another infrastructure service that works.
Andrea Righi - Spying on the Linux kernel for fun and profitlinuxlab_conf
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
This talk explores methodologies that allow to take a look “live” at kernel internal operations, from a network perspective, to I/O paths, CPU usage, memory allocations, etc., using in-kernel technologies, like eBPF and ftrace. Understanding such kernel internals can be really helpful to track down performance bottlenecks, debug system failures and it can be also a very effective way to approach to kernel development.
“p4alu” is a P4 program who would parse UDP packet with payload in "p4alu header format" and apply calculation.
This program is tested using BMv2 simple_switch P4 target.
Replacing iptables with eBPF in Kubernetes with CiliumMichal Rostecki
Cilium is an open source project which provides networking, security and load balancing for application services that are deployed using Linux container technologies by using the native eBPF technology in the Linux kernel. In this presentation we talked about:
- The evolution of the BPF filters and explained the advantages of eBPF Filters and its use cases today in Linux especially on how Cilium networking utilizes the eBPF Filters to secure the Kubernetes workload with increased performance when compared to legacy iptables.
- How Cilium uses SOCKMAP for layer 7 policy enforcement - How Cilium integrates with Istio and handles L7 Network Policies with Envoy Proxies.
- The new features since the last release such as running Kubernetes cluster without kube-proxy, providing clusterwide NetworkPolicies, providing fully distributed networking and security observability platform for cloud native workloads etc.
Make Your Containers Faster: Linux Container Performance ToolsKernel TLV
If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
An Introduction to the Formalised Memory Model for Linux KernelSeongJae Park
Linux kernel provides executable and formalized memory model. These slides describe the nature of parallel programming in the Linux kernel and what memory model is and why it is necessary and important for kernel programmers. The slides were used at KOSSCON 2018 (https://kosscon.kr/).
Agenda:
The Linux kernel has multiple "tracers" built-in, with various degrees of support for aggregation, dynamic probes, parameter processing, filtering, histograms, and other features. Starting from the venerable ftrace, introduced in kernel 2.6, all the way through eBPF, which is still under development, there are many options to choose from when you need to statically instrument your software with probes, or diagnose issues in the field using the system's dynamic probes. Modern tools include SystemTap, Sysdig, ktap, perf, bcc, and others. In this talk, we will begin by reviewing the modern tracing landscape -- ftrace, perf_events, kprobes, uprobes, eBPF -- and what insight into system activity these tools can offer. Then, we will look at specific examples of using tracing tools for diagnostics: tracing a memory leak using low-overhead kmalloc/kfree instrumentation, diagnosing a CPU caching issue using perf stat, probing network and block I/O latency distributions under load, or merely snooping user activities by capturing terminal input and output.
Speaker:
Sasha is the CTO of Sela Group, a training and consulting company based in Israel that employs over 400 developers world-wide. Most of Sasha's work revolves around performance optimization, production debugging, and low-level system diagnostics, but he also dabbles in mobile application development on iOS and Android. Sasha is the author of two books and three Pluralsight courses, and a contributor to multiple open-source projects. He blogs at http://blog.sashag.net.
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017OpenEBS
The slides were presented by Jeffry Molanus who is the CTO of OpenEBS in Golang Meetup. OpenEBS is an open source cloud native storage. OpenEBS delivers storage and storage services to containerized environments. OpenEBS allows stateful workloads to be managed more like stateless containers. OpenEBS storage services include: per container (or pod) QoS SLAs, tiering and replica policies across AZs and environments, and predictable and scalable performance.Our vision is simple: let’s let storage and storage services for persistent workloads be so fully integrated into the environment and hence managed automatically that is almost disappears into the background as just yet another infrastructure service that works.
Andrea Righi - Spying on the Linux kernel for fun and profitlinuxlab_conf
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
This talk explores methodologies that allow to take a look “live” at kernel internal operations, from a network perspective, to I/O paths, CPU usage, memory allocations, etc., using in-kernel technologies, like eBPF and ftrace. Understanding such kernel internals can be really helpful to track down performance bottlenecks, debug system failures and it can be also a very effective way to approach to kernel development.
“p4alu” is a P4 program who would parse UDP packet with payload in "p4alu header format" and apply calculation.
This program is tested using BMv2 simple_switch P4 target.
Replacing iptables with eBPF in Kubernetes with CiliumMichal Rostecki
Cilium is an open source project which provides networking, security and load balancing for application services that are deployed using Linux container technologies by using the native eBPF technology in the Linux kernel. In this presentation we talked about:
- The evolution of the BPF filters and explained the advantages of eBPF Filters and its use cases today in Linux especially on how Cilium networking utilizes the eBPF Filters to secure the Kubernetes workload with increased performance when compared to legacy iptables.
- How Cilium uses SOCKMAP for layer 7 policy enforcement - How Cilium integrates with Istio and handles L7 Network Policies with Envoy Proxies.
- The new features since the last release such as running Kubernetes cluster without kube-proxy, providing clusterwide NetworkPolicies, providing fully distributed networking and security observability platform for cloud native workloads etc.
The Linux Block Layer - Built for Fast StorageKernel TLV
The arrival of flash storage introduced a radical change in performance profiles of direct attached devices. At the time, it was obvious that Linux I/O stack needed to be redesigned in order to support devices capable of millions of IOPs, and with extremely low latency.
In this talk we revisit the changes the Linux block layer in the
last decade or so, that made it what it is today - a performant, scalable, robust and NUMA-aware subsystem. In addition, we cover the new NVMe over Fabrics support in Linux.
Sagi Grimberg
Sagi is Principal Architect and co-founder at LightBits Labs.
A 2015 performance study by Brendan Gregg, Nitesh Kant, and Ben Christensen. Original is in https://github.com/Netflix-Skunkworks/WSPerfLab/tree/master/test-results
DPDK (Data Plane Development Kit) Overview by Rami Rosen
* Background and short history
* Advantages and disadvantages
- Very High speed networking acceleration in L2
- How this acceleration is achieved (hugepages, optimizations)
- rte_kni (and KCP)
- VPP (and FD.io project) , providing routing and switching.
- TLDK (Transport Layer Development Kit, TCP/UDP)
* Anatomy of a simple DPDK application.
* Development and governance model
* Testpmd: DPDK CLI tool
* DDP - Dynamic Device Profiles
Rami Rosen is a Linux Kernel expert, the author of "Linux Kernel Networking", Apress, 2014.
Rami had published two articles about DPDK in the last year:
"Network acceleration with DPDK"
https://lwn.net/Articles/725254/
"Userspace Networking with DPDK"
https://www.linuxjournal.com/content/userspace-networking-dpdk
Spying on the Linux kernel for fun and profitAndrea Righi
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
Mirko Damiani - An Embedded soft real time distributed system in Golinuxlab_conf
An embedded system usually involves low level languages like C and highly customized hardware. In this talk we will see a use case of a soft real time system which was developed taking a very different approach, written in Go. We will see what are the advantages of this choice, along with its limits.
Kernel Recipes 2016 - Landlock LSM: Unprivileged sandboxingAnne Nicolas
Linux has multiple access-control features, which help to contain the damage from a malicious process. However, it is difficult and complex, especially for unprivileged users, to create a sandboxed application because of the currently administrator-oriented security.
seccomp-bpf was a big step forward in empowering any user with the ability to filter syscalls and therefore limit access to some resources. Nevertheless, it lacks the ability to create a full standalone sandbox (e.g. restrict access to a set of files), unlike Seatbelt/XNU Sandbox or OpenBSD Pledge.
In this talk, we present Landlock, a new Linux Security Module for unprivileged users. This brings some interesting challenges, from architecture design to userland API definition.
Mickaël Salaün
This talk will provide several examples of how Facebook engineers use BPF to scale the networking, prevent denial of service, secure containers, analyze performance. It’s suitable for BPF newbies and experts.
Alexei Starovoitov, Facebook
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
In the networking world there are a number of ways to increase performance over naive use of basic Berkeley sockets. These techniques have ranged from polling blocking sockets, non-blocking sockets controlled by Epoll, all the way through completely bypassing the Linux kernel for maximum network performance where you talk directly to the network interface card by using something like DPDK or Netmap. All these tools have their place, and generally occupy a space from convenience to performance. But in recent years, that landscape has changed massively.. The tools available to the average Linux systems developer have improved from the creation of io_uring, to the expansion of bpf from a simple filtering language to a full-on programming environment embedded directly in the kernel. Along with that came something called XDP (express datapath). This was Linux kernel's answer to kernel-bypass networking. AF_XDP is the new socket type created by this feature, and generally works very similarly to something like DPDK. History lessons out of the way, this talk will look into, and discuss the merits of this technology, it's place in the broader ecosystem and how it can be used to attain the highest level of performance possible. This talk will dive into crucial details, such as how AF_XDP works, how it can be integrated into a larger system and finally more advanced topics such as request sharding/load balancing. There will be detailed look at the design of AF_XDP, the eBpf code used, as well as the userspace code required to drive it all. It will also include performance numbers from this setup compared to regular kernel networking. And most importantly how to put all this together to handle as much data as possible on a single modern multi-core system.
FBTFTP: an opensource framework to build dynamic tftp serversAngelo Failla
Talk given at EuroPython2016, Bilbao:
https://ep2016.europython.eu/conference/talks/fbtftp-facebooks-python3-framework-for-tftp-servers
TFTP was first standardized in ’81 (same year I was born!) and one of its primary uses is in the early stage of network booting. TFTP is very simple to implement, and one of the reasons it is still in use is that its small footprint allows engineers to fit the code into very low resource, single board computers, system-on-a-chip implementations and mainboard chipsets, in the case of modern hardware.
It is therefore a crucial protocol deployed in almost every data center environment. It is used, together with DHCP, to chain load Network Boot Programs (NBPs), like Grub2 and iPXE. They allow machines to bootstrap themselves and install operating systems off of the network, downloading kernels and initrds via HTTP and starting them up.
At Facebook, we have been using the standard in.tftpd daemon for years, however, we started to reach its limitations. Limitations that were partially due to our scale and the way TFTP was deployed in our infrastructure, but also to the protocol specifications based on requirements from the 80’s.
To address those limitations we ended up writing our own framework for creating dynamic TFTP servers in Python3, and we decided to open source it.
I will take you thru the framework and the features it offers. I’ll discuss the specific problems that motivated us to create it. We will look at practical examples of how touse it, along with a little code, to build your own server that are tailored to your own infra needs.
Emerging Persistent Memory Hardware and ZUFS - PM-based File Systems in User ...Kernel TLV
In this talk, Dr. Amit Golander looks into emerging PM/NVDIMM devices, the value they bring to applications and most importantly how they revolutionize the storage stack.
In the second part, Boaz Harrosh and Shachar Sharon dive into new opportunities to develop memory-based filesystems in user space, leveraging a new open source project called ZUFS. ZUFS was presented in the last Linux Plumbers conference and unlike FUSE it focuses on delivering low latency and zero copy.
Dr. Amit Golander was the CTO of Plexistor, which developed the first enterprise-grade PM-based file system, and which was acquired earlier this year by NetApp.
Boaz Harrosh and Shachar Sharon are ZUFS maintainers and longtime Storage and Linux veterans.
Although we don't use it for the core web application, most other places in Launchpad that have to deal with concurrency issues do it using Twisted. This talk will survey these areas and talk about issues we've found and design patterns we've found helpful.
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Igalia
By Katerina Barone-Adesi.
Driven by the needs of scientific computing, rapid rises in memory bandwidth have made it possible to implement high-performance network functions in a radically simpler way. Snabb Switch rides this wave, bypassing the kernel to process network packets in terse Lua, leaving the programmer free to focus on the essence of their problem. This talk presents our experiences delivering a carrier-grade implementation of "lightweight 4 over 6", an IPv4-as-a-service architecture that tunnels access to the IPv4 internet through specialized Snabb appliances.
We report on our recent experience implementing a carrier-grade virtualized network function, with observations on what it is like to build real-world, high-performance Snabb applications. (and kernel bypass). Each instance runs at essentially line speed on two ten-gigabit Ethernet cards.
Lightweight 4-over-6 (lw4o6) defines an IPv4-as-a-service architecture that allows ISPs to internally operate an IPv6-only network, tunneling IPv4 connections between lw4o6-aware endpoints installed at the customer's site (e.g. in OpenWRT) and an internet-facing "lwAFTR". Lw4o6 was specified in 2015 as RFC 7596 and has the architectural advantage that the carrier-side lwAFTR only needs per-customer state, not per-flow state. An lw4o6 system can also be configured to share IPv4 addresses between multiple customers as part of an IPv4 exhaustion strategy. It allows IPv4 networks to interoperate smoothly, while a carrier between them runs a pure-IPv6 network.
Igalia has built an open source "lwAFTR" implementation that is ready to deploy in production. We describe the joys of hacking with Snabb, giving a quick intro to Snabb, modern x86, and lw4o6 along the way.
(c) 2016 FOSDEM VZW
CC BY 2.0 BE
https://archive.fosdem.org/2016/
All of Your Network Monitoring is (probably) Wrongice799
Monitorama 2016 talk about network monitoring covering topics like network device drivers, ethtool, and some interesting bugs/features.
For more information about monitoring and tuning the entire Linux network stack, see: blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
Dataplane programming with eBPF: architecture and toolsStefano Salsano
eBPF is definitely a complex technology. Developing complex systems based on eBPF is challenging due to the intrinsic limitations of the model and the known shortcomings of the tool chain.
The learning curve of this technology is very steep and needs continuous coaching from experts. This tutorial will investigate:
What is eBPF and why it has gained a prominent position among the solutions to improve the packet processing performance in Linux/x86 nodes. We will shortly present some important use case scenarios for eBPF, like Kubernetes’ Cilium
The architecture of eBPF and its programming toolchain (e.g. bcc
What are the frameworks for eBPF programming, such as Polycube and InKeV.
How to make eBPF programming easier, more flexible and modular with HIKe/eCLAT
How to implement a custom application logic in eBPF with eCLAT using a python-like script
How to extend the framework and develop new modules
Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions.
The Linux Block Layer - Built for Fast StorageKernel TLV
The arrival of flash storage introduced a radical change in performance profiles of direct attached devices. At the time, it was obvious that Linux I/O stack needed to be redesigned in order to support devices capable of millions of IOPs, and with extremely low latency.
In this talk we revisit the changes the Linux block layer in the
last decade or so, that made it what it is today - a performant, scalable, robust and NUMA-aware subsystem. In addition, we cover the new NVMe over Fabrics support in Linux.
Sagi Grimberg
Sagi is Principal Architect and co-founder at LightBits Labs.
A 2015 performance study by Brendan Gregg, Nitesh Kant, and Ben Christensen. Original is in https://github.com/Netflix-Skunkworks/WSPerfLab/tree/master/test-results
DPDK (Data Plane Development Kit) Overview by Rami Rosen
* Background and short history
* Advantages and disadvantages
- Very High speed networking acceleration in L2
- How this acceleration is achieved (hugepages, optimizations)
- rte_kni (and KCP)
- VPP (and FD.io project) , providing routing and switching.
- TLDK (Transport Layer Development Kit, TCP/UDP)
* Anatomy of a simple DPDK application.
* Development and governance model
* Testpmd: DPDK CLI tool
* DDP - Dynamic Device Profiles
Rami Rosen is a Linux Kernel expert, the author of "Linux Kernel Networking", Apress, 2014.
Rami had published two articles about DPDK in the last year:
"Network acceleration with DPDK"
https://lwn.net/Articles/725254/
"Userspace Networking with DPDK"
https://www.linuxjournal.com/content/userspace-networking-dpdk
Spying on the Linux kernel for fun and profitAndrea Righi
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
Mirko Damiani - An Embedded soft real time distributed system in Golinuxlab_conf
An embedded system usually involves low level languages like C and highly customized hardware. In this talk we will see a use case of a soft real time system which was developed taking a very different approach, written in Go. We will see what are the advantages of this choice, along with its limits.
Kernel Recipes 2016 - Landlock LSM: Unprivileged sandboxingAnne Nicolas
Linux has multiple access-control features, which help to contain the damage from a malicious process. However, it is difficult and complex, especially for unprivileged users, to create a sandboxed application because of the currently administrator-oriented security.
seccomp-bpf was a big step forward in empowering any user with the ability to filter syscalls and therefore limit access to some resources. Nevertheless, it lacks the ability to create a full standalone sandbox (e.g. restrict access to a set of files), unlike Seatbelt/XNU Sandbox or OpenBSD Pledge.
In this talk, we present Landlock, a new Linux Security Module for unprivileged users. This brings some interesting challenges, from architecture design to userland API definition.
Mickaël Salaün
This talk will provide several examples of how Facebook engineers use BPF to scale the networking, prevent denial of service, secure containers, analyze performance. It’s suitable for BPF newbies and experts.
Alexei Starovoitov, Facebook
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
In the networking world there are a number of ways to increase performance over naive use of basic Berkeley sockets. These techniques have ranged from polling blocking sockets, non-blocking sockets controlled by Epoll, all the way through completely bypassing the Linux kernel for maximum network performance where you talk directly to the network interface card by using something like DPDK or Netmap. All these tools have their place, and generally occupy a space from convenience to performance. But in recent years, that landscape has changed massively.. The tools available to the average Linux systems developer have improved from the creation of io_uring, to the expansion of bpf from a simple filtering language to a full-on programming environment embedded directly in the kernel. Along with that came something called XDP (express datapath). This was Linux kernel's answer to kernel-bypass networking. AF_XDP is the new socket type created by this feature, and generally works very similarly to something like DPDK. History lessons out of the way, this talk will look into, and discuss the merits of this technology, it's place in the broader ecosystem and how it can be used to attain the highest level of performance possible. This talk will dive into crucial details, such as how AF_XDP works, how it can be integrated into a larger system and finally more advanced topics such as request sharding/load balancing. There will be detailed look at the design of AF_XDP, the eBpf code used, as well as the userspace code required to drive it all. It will also include performance numbers from this setup compared to regular kernel networking. And most importantly how to put all this together to handle as much data as possible on a single modern multi-core system.
FBTFTP: an opensource framework to build dynamic tftp serversAngelo Failla
Talk given at EuroPython2016, Bilbao:
https://ep2016.europython.eu/conference/talks/fbtftp-facebooks-python3-framework-for-tftp-servers
TFTP was first standardized in ’81 (same year I was born!) and one of its primary uses is in the early stage of network booting. TFTP is very simple to implement, and one of the reasons it is still in use is that its small footprint allows engineers to fit the code into very low resource, single board computers, system-on-a-chip implementations and mainboard chipsets, in the case of modern hardware.
It is therefore a crucial protocol deployed in almost every data center environment. It is used, together with DHCP, to chain load Network Boot Programs (NBPs), like Grub2 and iPXE. They allow machines to bootstrap themselves and install operating systems off of the network, downloading kernels and initrds via HTTP and starting them up.
At Facebook, we have been using the standard in.tftpd daemon for years, however, we started to reach its limitations. Limitations that were partially due to our scale and the way TFTP was deployed in our infrastructure, but also to the protocol specifications based on requirements from the 80’s.
To address those limitations we ended up writing our own framework for creating dynamic TFTP servers in Python3, and we decided to open source it.
I will take you thru the framework and the features it offers. I’ll discuss the specific problems that motivated us to create it. We will look at practical examples of how touse it, along with a little code, to build your own server that are tailored to your own infra needs.
Emerging Persistent Memory Hardware and ZUFS - PM-based File Systems in User ...Kernel TLV
In this talk, Dr. Amit Golander looks into emerging PM/NVDIMM devices, the value they bring to applications and most importantly how they revolutionize the storage stack.
In the second part, Boaz Harrosh and Shachar Sharon dive into new opportunities to develop memory-based filesystems in user space, leveraging a new open source project called ZUFS. ZUFS was presented in the last Linux Plumbers conference and unlike FUSE it focuses on delivering low latency and zero copy.
Dr. Amit Golander was the CTO of Plexistor, which developed the first enterprise-grade PM-based file system, and which was acquired earlier this year by NetApp.
Boaz Harrosh and Shachar Sharon are ZUFS maintainers and longtime Storage and Linux veterans.
Although we don't use it for the core web application, most other places in Launchpad that have to deal with concurrency issues do it using Twisted. This talk will survey these areas and talk about issues we've found and design patterns we've found helpful.
Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Igalia
By Katerina Barone-Adesi.
Driven by the needs of scientific computing, rapid rises in memory bandwidth have made it possible to implement high-performance network functions in a radically simpler way. Snabb Switch rides this wave, bypassing the kernel to process network packets in terse Lua, leaving the programmer free to focus on the essence of their problem. This talk presents our experiences delivering a carrier-grade implementation of "lightweight 4 over 6", an IPv4-as-a-service architecture that tunnels access to the IPv4 internet through specialized Snabb appliances.
We report on our recent experience implementing a carrier-grade virtualized network function, with observations on what it is like to build real-world, high-performance Snabb applications. (and kernel bypass). Each instance runs at essentially line speed on two ten-gigabit Ethernet cards.
Lightweight 4-over-6 (lw4o6) defines an IPv4-as-a-service architecture that allows ISPs to internally operate an IPv6-only network, tunneling IPv4 connections between lw4o6-aware endpoints installed at the customer's site (e.g. in OpenWRT) and an internet-facing "lwAFTR". Lw4o6 was specified in 2015 as RFC 7596 and has the architectural advantage that the carrier-side lwAFTR only needs per-customer state, not per-flow state. An lw4o6 system can also be configured to share IPv4 addresses between multiple customers as part of an IPv4 exhaustion strategy. It allows IPv4 networks to interoperate smoothly, while a carrier between them runs a pure-IPv6 network.
Igalia has built an open source "lwAFTR" implementation that is ready to deploy in production. We describe the joys of hacking with Snabb, giving a quick intro to Snabb, modern x86, and lw4o6 along the way.
(c) 2016 FOSDEM VZW
CC BY 2.0 BE
https://archive.fosdem.org/2016/
All of Your Network Monitoring is (probably) Wrongice799
Monitorama 2016 talk about network monitoring covering topics like network device drivers, ethtool, and some interesting bugs/features.
For more information about monitoring and tuning the entire Linux network stack, see: blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
Dataplane programming with eBPF: architecture and toolsStefano Salsano
eBPF is definitely a complex technology. Developing complex systems based on eBPF is challenging due to the intrinsic limitations of the model and the known shortcomings of the tool chain.
The learning curve of this technology is very steep and needs continuous coaching from experts. This tutorial will investigate:
What is eBPF and why it has gained a prominent position among the solutions to improve the packet processing performance in Linux/x86 nodes. We will shortly present some important use case scenarios for eBPF, like Kubernetes’ Cilium
The architecture of eBPF and its programming toolchain (e.g. bcc
What are the frameworks for eBPF programming, such as Polycube and InKeV.
How to make eBPF programming easier, more flexible and modular with HIKe/eCLAT
How to implement a custom application logic in eBPF with eCLAT using a python-like script
How to extend the framework and develop new modules
Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions.
eBPF is an exciting new technology that is poised to transform Linux performance engineering. eBPF enables users to dynamically and programatically trace any kernel or user space code path, safely and efficiently. However, understanding eBPF is not so simple. The goal of this talk is to give audiences a fundamental understanding of eBPF, how it interconnects existing Linux tracing technologies, and provides a powerful aplatform to solve any Linux performance problem.
An Introduction to eBPF (and cBPF). Topics covered include history, implementation, program types & maps. Also gives a brief introduction to XDP and DPDK
Not breaking userspace: the evolving Linux ABIAlison Chaiken
Don't break userspace" is famously one of the core tenets of Linux kernel development. The rule makes it possible for distros to upgrade the kernel independently of userspace applications. But what does the phrase actually mean in 2022? What portions of the kernel's interface are guaranteed stable and are safe for applications to rely on?
Kernel contributors try to make only backward-compatible changes to the procfs and sysfs filesystems. These virtual filesystems (VFS) constitute the primary applications binary interface (ABI). System calls in a C library like glibc send commands to and receive information from the kernel via the VFS. Linux userspace applications can then rely on the syscalls to interact with the OS or access the VFS directly.
Linux OS's have many artifacts besides procfs and sysfs, however. Which of them are part of the stable ABI? The kernel's documentation calls out the fact that kernel configuration parameters are not stable, but what about devicetree formats, the dmesg log, the BPF validator interface, /dev files, kernel command-line parameters, tracepoints and filesystem metadata? Changes to not explicitly stable parameters have often been the source of controversy.
The difficulty of adding features and fixing bugs without changing the stable ABI is universal in software development. Sometimes there is a stark choice between implementing an upstream fix and "breaking" the library's users. When inevitably the kernel ABI is modified for bug fixes, C-library developers try to balance support for the changes with customer stability expectations. For example, a recent fork of libpthread with an ABI break was created by the realtime Linux community in order to fix problems in time-critical systems like robotics. In 2021, the kernel added a "printk indexing" feature that will make it easier for applications to parse error messages in the dmesg log. The presentation should equip attendees to better understand the Linux ABI and feel better prepared to deal with changes to them that affect their own projects.
Linux Kernel Platform Development: Challenges and InsightsGlobalLogic Ukraine
This presentation is about the main tasks which Linux kernel platform engineers take care of. The talk includes real-life cases which help understand the role of respective specialists and might be helpful to those who consider such change in their careers.
The talk was delivered by Sam Protsenko (Software Engineer, Consultant, GlobalLogic) at GlobalLogic Embedded Career Day #2 on February 10, 2018.
More about GlobalLogic Embedded Career Day #2: https://www.globallogic.com/ua/events/globallogic-kyiv-embedded-career-day-2-materials
BPF & Cilium - Turning Linux into a Microservices-aware Operating SystemThomas Graf
Container runtimes cause Linux to return to its original purpose: to serve applications interacting directly with the kernel. At the same time, the Linux kernel is traditionally difficult to change and its development process is full of myths. A new efficient in-kernel programming language called eBPF is changing this and allows everyone to extend existing kernel components or glue them together in new forms without requiring to change the kernel itself.
The Linux kernel is undergoing the most fundamental architecture evolution in history and is becoming a microkernel. Why is the Linux kernel evolving into a microkernel? The potentially biggest fundamental change ever happening to the Linux kernel. This talk covers how companies like Facebook and Google use BPF to patch 0-day exploits, how BPF will change the way features are added to the kernel forever, and how BPF is introducing a new type of application deployment method for the Linux kernel.
DevopsItalia2015 - DHCP at Facebook - Evolution of an infrastructureAngelo Failla
Facebook e' uno dei piu' grandi siti nel mondo, con datacenter e POP in giro per il mondo, e una grande quantita' di macchine.
In questo talk useremo DHCP come un esempio per discutere perche' e' buono progettare sistemi stateless e discutere la sottile linea di separazione tra utilizzare un prodotto OpenSource o prendere un approccio "Not Invented here".
It's been three years since Netflix's Brendan Gregg described the Berkeley Packet Filter as "Superpowers for Linux". Since then there has been an explosion of capabilities and tools based on eBPF, so you've probably heard the term, but do you know what it is and how to use it? In this demo-rich talk we'll explore some of the powerful things we can do with this technology, especially in the context of containers.
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
In this presentation I talk about our motivation to converting our microservices to run on Kubernetes. I discuss many of the technical challenges we encountered along the way, including networking issues, Java issues, monitoring and alerting, and managing all of our resources!
Debugging is an essential part of the Linux kernel development, but without the proper knowledge it can be quite challenging. In this talk, we will cover some techniques that will help you to study how the kernel works “under the hood”. We will see how to crash the kernel in many spectacular ways, how to analyze the crashes and how to hunt down new bugs, fix them and become a better kernel developer.
Linux kernel tracing superpowers in the cloudAndrea Righi
The Linux 4.x series introduced a new powerful engine of programmable tracing (BPF) that allows to actually look inside the kernel at runtime. This talk will show you how to exploit this engine in order to debug problems or identify performance bottlenecks in a complex environment like a cloud. This talk will cover the latest Linux superpowers that allow to see what is happening “under the hood” of the Linux kernel at runtime. I will explain how to exploit these “superpowers” to measure and trace complex events at runtime in a cloud environment. For example, we will see how we can measure latency distribution of filesystem I/O, details of storage device operations, like individual block I/O request timeouts, or TCP buffer allocations, investigating stack traces of certain events, identify memory leaks, performance bottlenecks and a whole lot more.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
6. Kernel debugging hands-on
●
Virtualization can help to track down kernel bugs
●
virtme
●
Run the kernel inside a qemu/kvm instance, virtualizing the running
system
●
Generate crash dump
●
Analyze system data offline (after the crash)
●
crash test kernel module
●
https://github.com/arighi/crashtest
●
Simple scripts to speed up kernel development
(wrappers around virtme/qemu/crash):
●
https://github.com/arighi/kernel-crash-tools
8. Profiling vs tracing
●
Profiling
●
Create a periodic timed interrupt that collects the current
program counter, function address and the entire stack
back trace
●
Tracing
●
Record times and invocations of specific events
10. Tracing example: strace
●
strace(1): system call tracer in Linux
●
It uses the ptrace() system call that pauses the target
process for each syscall so that the debugger can read
the state
●
And it’s doing this twice: when the syscall begins and when
it ends!
11. strace overhead
### Regular execution ###
$ dd if=/dev/zero of=/dev/null bs=1 count=500k
512000+0 records in
512000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 0,501455 s, 1.0 MB/s
### Strace execution (tracing a syscall that is never called) ###
$ strace -e trace=accept dd if=/dev/zero of=/dev/null bs=1 count=500k
512000+0 records in
512000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 44.0216 s, 11,6 kB/s
+++ exited with 0 +++
14. eBPF history
●
Initially it was BPF: Berkeley Packet Filter
●
It has its roots in BSD in the very early 1990’s
●
Originally designed as a mechanism for fast filtering network packets
●
3.15: Linux introduced eBPF: extended Berkeley Packet Filter
●
More efficient / more generic than the original BPF
●
3.18: eBPF VM exposed to user-space
●
4.9: eBPF programs can be attached to perf_events
●
4.10: eBPF programs can be attached to cgroups
●
4.15: eBPF LSM hooks
15. eBPF features
●
Highly efficient VM that lives in the kernel
●
Inject safe sanboxed bytecode into the kernel
●
Attach code to kernel functions / events
●
In-kernel JIT compiler
●
Dynamically translate eBPF bytecode into native opcodes
●
eBPF makes kernel programmable without having to
cross kernel/user-space boundaries
●
Access in-kernel data structures directly without the risk
of crashing, hanging or breaking the kernel in any way
16. eBPF as a VM
●
Example assembly of a simple
eBPF filter
●
Load 16-bit quantity from offset
12 in the packet to the
accumulator (ethernet type)
●
Compare the value to see if the
packet is an IP packet
●
If the packet is IP, return TRUE
(packet is accepted)
●
otherwise return 0 (packet is
rejected)
●
Only 4 VM instructions to filter
IP packets!
ldh [12]
jeq #ETHERTYPE_IP, l1, l2
l1: ret #TRUE
l2: ret #0
20. BCC tracing tools
●
BPF Compiler Collection https://github.com/iovisor/bcc
●
Front-end to eBPF
●
BCC makes eBPF programs easier to write
●
Include C wrapper around LLVM
●
Python
●
Lua
●
C++
●
C helper libs
●
Lots of pre-defined tools available
21. Example #1: trace exec()
●
Intercept all the processes executed in the system
23. Example #3: ping
●
Identify where ICMP packets (ECHO_REQUEST /
ECHO_REPLY) are received and processed by the kernel
24. Example #4: task wait / wakeup
●
Determine the stack trace
of a sleeping process and
the stack trace of the
process that wakes up a
sleeping process
26. eBPF: performance overhead - use case #1
●
user-space deamon using an eBPF program attached to a
function (via kprobe)
●
kernel is updated, function doesn’t exist anymore
●
daemon starts to use an older/slower non-BPF method
●
5% performance regression
27. eBPF: performance overhead - use case #2
●
kernel function mapped to a 2MB huge page
●
eBPF program attached to that function (via kprobe)
●
setting the kprobe causes the function to be remapped to a
regular 4KB page
●
increased TLB misses
●
2% performance regression
28. eBPF: compile once, run everywhere?
●
… not exactly! :-(
●
eBPF programs are compiled on the target system
immediately before they are loaded
●
Linux headers are needed to understand kernel data
structures
●
structure randomization is a problem
●
BTF (BPF type format) has been created
●
kernel data description embedded in the kernel (no longer
any need to ship kernel headers around!)
29. Conclusion
●
Virtualization is your friend to speed up kernel
development
●
Real-time tracing can be an effective way to study and
understand how the kernel works
●
Kernel development can be challenging... but fun! :)