Ftrace is the official tracer of the Linux kernel. It has been apart of Linux since 2.6.31, and has grown tremendously ever since. Ftrace’s name comes from its most powerful feature: function tracing. But the ftrace infrastructure is much more than that. It also encompasses the trace events that are used by perf, as well as kprobes that can dynamically add trace events that the user defines.
This talk will focus on learning how the kernel works by using the ftrace infrastructure. It will show how to see what happens within the kernel during a system call; learn how interrupts work; see how ones processes are being scheduled, and more. A quick introduction to some tools like trace-cmd and KernelShark will also be demonstrated.
Steven Rostedt, VMware
Page cache mechanism in Linux kernel.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Vmlinux: anatomy of bzimage and how x86 64 processor is bootedAdrian Huang
This slide deck describes the Linux booting flow for x86_64 processors.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Anatomy of the loadable kernel module (lkm)Adrian Huang
Talk about how Linux kernel invokes your module's init function.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedAnne Nicolas
Ftrace’s most powerful feature is the function tracer (and function graph tracer which is built from it). But to have this enabled on production systems, it had to have its overhead be negligible when disabled. As the function tracer uses gcc’s profiling mechanism, which adds a call to “mcount” (or more recently fentry, don’t worry if you don’t know what this is, it will all be explained) at the start of almost all functions, it had to do something about the overhead that causes. The solution was to turn those calls into “nops” (an instruction that the CPU simply ignores). But this was no easy feat. It took a lot to come up with a solution (and also turning a few network cards into bricks). This talk will explain the history of how ftrace came about implementing the function tracer, and brought with it the possibility of static branches and soon static calls!
Steven Rostedt
Page cache mechanism in Linux kernel.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Vmlinux: anatomy of bzimage and how x86 64 processor is bootedAdrian Huang
This slide deck describes the Linux booting flow for x86_64 processors.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Anatomy of the loadable kernel module (lkm)Adrian Huang
Talk about how Linux kernel invokes your module's init function.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedAnne Nicolas
Ftrace’s most powerful feature is the function tracer (and function graph tracer which is built from it). But to have this enabled on production systems, it had to have its overhead be negligible when disabled. As the function tracer uses gcc’s profiling mechanism, which adds a call to “mcount” (or more recently fentry, don’t worry if you don’t know what this is, it will all be explained) at the start of almost all functions, it had to do something about the overhead that causes. The solution was to turn those calls into “nops” (an instruction that the CPU simply ignores). But this was no easy feat. It took a lot to come up with a solution (and also turning a few network cards into bricks). This talk will explain the history of how ftrace came about implementing the function tracer, and brought with it the possibility of static branches and soon static calls!
Steven Rostedt
Process Address Space: The way to create virtual address (page table) of user...Adrian Huang
Process Address Space: The way to create virtual address (page table) of userspace application.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Decompressed vmlinux: linux kernel initialization from page table configurati...Adrian Huang
Talk about how Linux kernel initializes the page table.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
New Ways to Find Latency in Linux Using TracingScyllaDB
Ftrace is the official tracer of the Linux kernel. It originated from the real-time patch (now known as PREEMPT_RT), as developing an operating system for real-time use requires deep insight and transparency of the happenings of the kernel. Not only was tracing useful for debugging, but it was critical for finding areas in the kernel that was causing unbounded latency. It's no wonder why the ftrace infrastructure has a lot of tooling for seeking out latency. Ftrace was introduced into mainline Linux in 2008, and several talks have been done on how to utilize its tracing features. But a lot has happened in the past few years that makes the tooling for finding latency much simpler. Other talks at P99 will discuss the new ftrace tracers "osnoise" and "timerlat", but this talk will focus more on the new flexible and dynamic aspects of ftrace that facilitates finding latency issues which are more specific to your needs. Some of this work may still be in a proof of concept stage, but this talk will give you the advantage of knowing what tools will be available to you in the coming year.
Virtual File System in Linux Kernel
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...Adrian Huang
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg
Keynote for Ubuntu Masters 2019 by Brendan Gregg, Netflix. Video https://www.youtube.com/watch?v=7pmXdG8-7WU&feature=youtu.be . "Extended BPF is a new type of software, and the first fundamental change to how kernels are used in 50 years. This new type of software is already in use by major companies: Netflix has 14 BPF programs running by default on all of its cloud servers, which run Ubuntu Linux. Facebook has 40 BPF programs running by default. Extended BPF is composed of an in-kernel runtime for executing a virtual BPF instruction set through a safety verifier and with JIT compilation. So far it has been used for software defined networking, performance tools, security policies, and device drivers, with more uses planned and more we have yet to think of. It is changing how we use and think about systems. This talk explores the past, present, and future of BPF, with BPF performance tools as a use case."
USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.
Linux Kernel Booting Process (2) - For NLKBshimosawa
Describes the bootstrapping part in Linux, and related architectural mechanisms and technologies.
This is the part two of the slides, and the succeeding slides may contain the errata for this slide.
BPF of Berkeley Packet Filter mechanism was first introduced in linux in 1997 in version 2.1.75. It has seen a number of extensions of the years. Recently in versions 3.15 - 3.19 it received a major overhaul which drastically expanded it's applicability. This talk will cover how the instruction set looks today and why. It's architecture, capabilities, interface, just-in-time compilers. We will also talk about how it's being used in different areas of the kernel like tracing and networking and future plans.
Memory Mapping Implementation (mmap) in Linux KernelAdrian Huang
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Kernel Recipes 2017 - Linux Kernel Release Model - Greg KHAnne Nicolas
This talk describes how the Linux kernel development model works, what a long term supported kernel is, and why all Linux-based systems devices should be using all of the stable releases and not attempting to pick and choose random patches.
It also goes into how the kernel community approaches “security” patches, and the reasoning why they do it in the manner they do.
This is an expanded talk based on a lighting talk presented at Kernel Recipes 2015, and the speaker’s interactions with companies reaction to the supposedly confusing development model on which they rely on.
Greg KH, Linux Foundation
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea ArcangeliAnne Nicolas
Andrea will provide a short high level view of the most notable milestones in the evolution of the Linux Virtual Memory over the years. He will then focus on the various Memory Management features such as Transparent Huge Pages(THP), automatic NUMA balancing and userfaultd/postcopy live migration of Kernel Virtual Machines (KVM). Andrea will cover best practices, providing the audience with an understanding of when and how to leverage these features in their environments.
Andrea Arcangeli, Red Hat
Process Address Space: The way to create virtual address (page table) of user...Adrian Huang
Process Address Space: The way to create virtual address (page table) of userspace application.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Decompressed vmlinux: linux kernel initialization from page table configurati...Adrian Huang
Talk about how Linux kernel initializes the page table.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
New Ways to Find Latency in Linux Using TracingScyllaDB
Ftrace is the official tracer of the Linux kernel. It originated from the real-time patch (now known as PREEMPT_RT), as developing an operating system for real-time use requires deep insight and transparency of the happenings of the kernel. Not only was tracing useful for debugging, but it was critical for finding areas in the kernel that was causing unbounded latency. It's no wonder why the ftrace infrastructure has a lot of tooling for seeking out latency. Ftrace was introduced into mainline Linux in 2008, and several talks have been done on how to utilize its tracing features. But a lot has happened in the past few years that makes the tooling for finding latency much simpler. Other talks at P99 will discuss the new ftrace tracers "osnoise" and "timerlat", but this talk will focus more on the new flexible and dynamic aspects of ftrace that facilitates finding latency issues which are more specific to your needs. Some of this work may still be in a proof of concept stage, but this talk will give you the advantage of knowing what tools will be available to you in the coming year.
Virtual File System in Linux Kernel
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...Adrian Huang
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg
Keynote for Ubuntu Masters 2019 by Brendan Gregg, Netflix. Video https://www.youtube.com/watch?v=7pmXdG8-7WU&feature=youtu.be . "Extended BPF is a new type of software, and the first fundamental change to how kernels are used in 50 years. This new type of software is already in use by major companies: Netflix has 14 BPF programs running by default on all of its cloud servers, which run Ubuntu Linux. Facebook has 40 BPF programs running by default. Extended BPF is composed of an in-kernel runtime for executing a virtual BPF instruction set through a safety verifier and with JIT compilation. So far it has been used for software defined networking, performance tools, security policies, and device drivers, with more uses planned and more we have yet to think of. It is changing how we use and think about systems. This talk explores the past, present, and future of BPF, with BPF performance tools as a use case."
USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.
Linux Kernel Booting Process (2) - For NLKBshimosawa
Describes the bootstrapping part in Linux, and related architectural mechanisms and technologies.
This is the part two of the slides, and the succeeding slides may contain the errata for this slide.
BPF of Berkeley Packet Filter mechanism was first introduced in linux in 1997 in version 2.1.75. It has seen a number of extensions of the years. Recently in versions 3.15 - 3.19 it received a major overhaul which drastically expanded it's applicability. This talk will cover how the instruction set looks today and why. It's architecture, capabilities, interface, just-in-time compilers. We will also talk about how it's being used in different areas of the kernel like tracing and networking and future plans.
Memory Mapping Implementation (mmap) in Linux KernelAdrian Huang
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Kernel Recipes 2017 - Linux Kernel Release Model - Greg KHAnne Nicolas
This talk describes how the Linux kernel development model works, what a long term supported kernel is, and why all Linux-based systems devices should be using all of the stable releases and not attempting to pick and choose random patches.
It also goes into how the kernel community approaches “security” patches, and the reasoning why they do it in the manner they do.
This is an expanded talk based on a lighting talk presented at Kernel Recipes 2015, and the speaker’s interactions with companies reaction to the supposedly confusing development model on which they rely on.
Greg KH, Linux Foundation
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea ArcangeliAnne Nicolas
Andrea will provide a short high level view of the most notable milestones in the evolution of the Linux Virtual Memory over the years. He will then focus on the various Memory Management features such as Transparent Huge Pages(THP), automatic NUMA balancing and userfaultd/postcopy live migration of Kernel Virtual Machines (KVM). Andrea will cover best practices, providing the audience with an understanding of when and how to leverage these features in their environments.
Andrea Arcangeli, Red Hat
Thanks to a partnership with Jumpshot, Moz is presenting data about Google's search growth, click distribution, and more via a panel of millions of US web users.
Lee Rainie, director of internet and technology research at Pew Research Center, presented these findings at the International Monetary Fund/World Bank’s Youth Dialogue and its program, “A World Without Work?” The findings tie to several pieces of research at the Center, including reports on the state of American jobs, automation in everyday life, and the future of jobs training programs.
AI and Machine Learning Demystified by Carol Smith at Midwest UX 2017Carol Smith
What is machine learning? Is UX relevant in the age of artificial intelligence (AI)? How can I take advantage of cognitive computing? Get answers to these questions and learn about the implications for your work in this session. Carol will help you understand at a basic level how these systems are built and what is required to get insights from them. Carol will present examples of how machine learning is already being used and explore the ethical challenges inherent in creating AI. You will walk away with an awareness of the weaknesses of AI and the knowledge of how these systems work.
Kafka Summit SF 2017 - One Day, One Data Hub, 100 Billion Messages: Kafka at ...confluent
LINE is a messaging service with 200+ million active users. I will introduce why we feed 100+ billion daily messages into Kafka and how various systems such as data sync, abuse detection and analysis are depending on and leveraging it. It will be also introduced how we leverage dynamic tracing tools like SystemTap to inspect broker’s performance on production system, which led me to fix KAFKA-4614.
Presented by Yuto Kawamura, LINE Corporation
Systems Track
A short overview of current technologies plucked from the Texas Linux Fest schedule for 2014. Includes overviews of systemd, popular configuration management tools, docker, distributed log collection, and openstack.
Presented at LISA18: https://www.usenix.org/conference/lisa18/presentation/babrou
This is a technical dive into how we used eBPF to solve real-world issues uncovered during an innocent OS upgrade. We'll see how we debugged 10x CPU increase in Kafka after Debian upgrade and what lessons we learned. We'll get from high-level effects like increased CPU to flamegraphs showing us where the problem lies to tracing timers and functions calls in the Linux kernel.
The focus is on tools what operational engineers can use to debug performance issues in production. This particular issue happened at Cloudflare on a Kafka cluster doing 100Gbps of ingress and many multiple of that egress.
You have a system with an advanced programmatic tracer: do you know what to do with it? Brendan has used numerous tracers in production environments, and has published hundreds of tracing-based tools. In this talk he will share tips and know-how for creating CLI tracing tools and GUI visualizations, to solve real problems effectively. Programmatic tracing is an amazing superpower, and this talk will show you how to wield it!
Embedded Recipes 2018 - Finding sources of Latency In your system - Steven Ro...Anne Nicolas
Having just an RTOS is not enough for a real-time system. The hardware must be deterministic as well as the applications that run on the system. When you are missing deadlines, the first thing that must be done is to find what is the source of the latency that caused the issue. It could be the hardware, the operating system or the application, or even a combination of the above. This talk will discuss how to determine where the latency is using tools that come with the Linux Kernel, and will explain a few cases that caused issues.
OSSNA 2017 Performance Analysis Superpowers with Linux BPFBrendan Gregg
Talk by Brendan Gregg for OSSNA 2017. "Advanced performance observability and debugging have arrived built into the Linux 4.x series, thanks to enhancements to Berkeley Packet Filter (BPF, or eBPF) and the repurposing of its sandboxed virtual machine to provide programmatic capabilities to system tracing. Netflix has been investigating its use for new observability tools, monitoring, security uses, and more. This talk will be a dive deep on these new tracing, observability, and debugging capabilities, which sooner or later will be available to everyone who uses Linux. Whether you’re doing analysis over an ssh session, or via a monitoring GUI, BPF can be used to provide an efficient, custom, and deep level of detail into system and application performance.
This talk will also demonstrate the new open source tools that have been developed, which make use of kernel- and user-level dynamic tracing (kprobes and uprobes), and kernel- and user-level static tracing (tracepoints). These tools provide new insights for file system and storage performance, CPU scheduler performance, TCP performance, and a whole lot more. This is a major turning point for Linux systems engineering, as custom advanced performance instrumentation can be used safely in production environments, powering a new generation of tools and visualizations."
I’ve been keeping a collection of Linux commands that are particularly useful; some are from websites I’ve visited, others from experience
I hope you find these are useful as I have. I’ll periodically add to the list, so check back occasionally.
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
Kernel Recipes 2019 - Driving the industry toward upstream firstAnne Nicolas
Wanting to avoid the Android experience, Google developers always aimed to make their Chrome OS Linux kernels as close to mainline as possible. However, when Chromebooks were first created, Google was left with no choice, the mainline kernel, in some subsystems, still did not have all the functionalities needed by Chromebooks. Hence, similarly to Android, Chrome OS had to develop their own out-of-tree code for the kernel and maintain that for a few different kernel versions.
Luckily, over the last few years a strong and consistent effort has been happening to bring Chromebook devices closer to mainline. It has led to significant improvements that now make it possible to run mainline on Chrome OS devices. And not only Chromebooks, as these significant strides are also improving Arm-based SOCs and other key components of the rich Chromebook hardware ecosystem. In this talk, we will look at how and why upstream support for Chromebooks improved, the current status of various models, and what we expect in the future.
Enric Balletbò i Serra
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMIAnne Nicolas
As the name would suggest, a Non-Maskable Interrupt (NMI) is an interrupt-like feature that is unaffected by the disabling of classic interrupts. In Linux, NMIs are involved in some features such as performance event monitoring, hard-lockup detector, on demand state dumping, etc… Their potential to fire when least expected can fill the most seasoned kernel hackers with dread.
AArch64 (aka arm64 in the Linux tree) does not provide architected NMIs, a consequence being that features benefiting from NMIs see their use limited on AArch64. However, the Arm Generic Interrupt Controller (GIC) supports interrupt prioritization and masking, which, among other things, provides a way to control whether or not a set of interrupts can be signaled to a CPU.
This talk will cover how, using the GIC interrupt priorities, we provide a way to configure some interrupts to behave in an NMI-like manner on AArch64. We’ll discuss the implementation, some of the complications that ensued and also some of the benefits obtained from it.
Julien Thierry
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelAnne Nicolas
At a rate of almost 9 changes per hour (24/7), the Linux kernel is definitely a scary beast. Bugs are introduced on a daily basis and, through the use of multiple code analyzers, *some* of them are detected and fixed before they hit mainline. Over the course of the last few years, Gustavo has been fixing such bugs and many different issues in every corner of the Linux kernel. Recently, he was in charge of leading the efforts to globally enable -Wimplicit-fallthrough; which appears by default in Linux v5.3. This presentation is a report on all the stuff Gustavo has found and fixed in the kernel with the support of the Core Infrastructure Initiative.
Gustavo A.R. Silva
Kernel Recipes 2019 - Metrics are moneyAnne Nicolas
In I.T. we all use all kinds of metrics. Operations teams rely heavily on these, especially when things go south. These metrics are sometimes overrated. Let’s dive into a few real life stories together.
Aurélien Rougemont
Kernel Recipes 2019 - Kernel documentation: past, present, and futureAnne Nicolas
The Linux kernel project includes a huge amount of documentation, but that information has seen little in the way of care over the
years. The amount of care has increased significantly recently, though, and things are improving quickly. Listen as the kernel’s documentation maintainer discusses the current state of the kernel’s docs, how we got here, where we’re trying to go, and how you can help.
Jonathan Corbet
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...Anne Nicolas
Modern SoC designs incorporate technologies from numerous vendors, each with their own inconsistent, confusing, undocumented and even contradictory terminology. The result is a mess of acronyms and product names which have a surprising impact on the ability to develop reusable, modular code thanks to the nature of the underlying IP being obscured.
This presentation will dive into some of the misnomers plaguing the Arm ecosystem, with the aim of explaining why things are like they are, how they fit together under the architectural umbrella and how you, as a developer, can decipher the baffling ingredients list of your next SoC design!
Will Deacon
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary dataAnne Nicolas
GNU poke is a new interactive editor for binary data. Not limited to editing basic ntities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them. Once a user has defined a structure for binary data (usually matching some file format) she can search, inspect, create, shuffle and modify abstract entities such as ELF relocations, MP3 tags, DWARF expressions, partition table entries, and so on, with primitives resembling simple editing of bits and bytes. The program comes with a library of already written descriptions (or “pickles” in poke parlance) for many binary formats.
GNU poke is useful in many domains. It is very well suited to aid in the development of programs that operate on binary files, such as assemblers and linkers. This was in fact the primary inspiration that brought me to write it: easily injecting flaws into ELF files in order to reproduce toolchain bugs. Also, due to its flexibility, poke is also very useful for reverse engineering, where the real structure of the data being edited is discovered by experiment, interactively. It is also good for the fast development of prototypes for programs like linkers, compressors or filters, and it provides a convenient foundation to write other utilities such as diff and patch tools for binary files.
This talk (unlike Gaul) is divided into four parts. First I will introduce the program and show what it does: from simple bits/bytes editing to user-defined structures. Then I will show some of the internals, and how poke is implemented. The third block will cover the way of using Poke to describe user data, which is to say the art of writing “pickles”. The presentation ends with a status of the project, a call for hackers, and a hint at future works.
Jose E. Marchesi
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...Anne Nicolas
Operating system distributors often face challenges that are somewhat different from that of upstream kernel developers. For instance, some kernel updates often need to stay at least binary compatible with modules that might be “out of tree” for some time.
In that context, being able to automatically detect and analyze changes to the binary interface exposed by the kernel to its module does have some noticeable value.
The Libabigail framework is capable of analyzing ELF binaries along with their accompanying debug info in the DWARF format, detect and report changes in types, functions, variables and ELF symbols. It has historically supported that for user space shared libraries and application so we worked to make it understand the Linux kernel
binaries.
In this presentation, we are going to present the current support of ABI analysis for Linux Kernel binaries, the challenges we face, how we address them and the plans we have for the future.
Dodji Seketeli, Jessica Yu, Matthias Männich
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and BareboxAnne Nicolas
Different upgrade and update strategies exist when it comes to embedded Linux system. If at development time none of these strategies have been chosen, adding them afterwards can be tedious task.
Even harder it gets when the system is already deployed in the field and only accessible via a 3G connection.
This talk is a developer experience of putting in place exactly that. Giving a return of experience on one way of doing it on a system running Barebox and a Yocto-based distribution.
Patrick Boettcher
Embedded Recipes 2019 - Making embedded graphics less specialAnne Nicolas
Traditionally graphics drivers were one of the last hold-outs of proprietary software in an embedded Linux system. This situation is changing with open-source graphics drivers showing up for almost all of the graphics acceleration peripherals on the market right now. This talk will show how open-source graphics drivers are making embedded systems less special, as well as trying to provide an overview of the Linux graphics stack, de-mystifying what is often seen as black magic GPU stuff from outside observers.
Lucas Stach
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre SiliconAnne Nicolas
This talk will explore Open Source Hardware projects relevant to Linux, including boards like BeagleBone, Olimex OLinuXino, Giant board and more. Looking at the benefits and challenges of designing Open Source Hardware for a Linux system, along with BeagleBoard.org’s experience of working with community, manufacturers, and distributors to create an Open Source Hardware platform. In closing also looking at the future, Libre Silicon like RISC-V designs, and where this might take Linux.
Drew Fustini
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) pictureAnne Nicolas
The I2C subsystem is not the shiniest part of the Linux Kernel. For embedded devices, though, it is one of the many puzzle pieces which just have to work. Wolfram Sang has the experience of maintaining this subsystem for nearly 7 years now. This talk gives a short overview of how maintaining works in general and specifically in this subsystem. But mainly, it will highlight noteworthy points in the timeline and lessons learnt from that. It will present trends, not so much regarding I2C but more the Linux Kernel and the embedded ecosystem in general. And of course, there will be plenty of anecdotes and bits from behind the scenes for your entertainment.
Wolfram Sang
Embedded Recipes 2019 - Testing firmware the devops wayAnne Nicolas
ITRenew is selling recertified OCP servers under the Sesame brand, those servers come either with their original UEFI BIOS or with LinuxBoot. The LinuxBoot project is pushing the Linux kernel inside bios flash and using userland programs as bootloader.
To achieve quality on our software stack, as any project, we need to test it. Traditional BIOS are tested by hand, this is 2019 we need to do it automatically! We already presented the hardware setup behind the LinuxBoot CI, this talk will focus on the software.
We use u-root for our userland bootloader; this software is written in Go so we naturally choose to use Go for our testing too. We will present how we are using and extending the Go native test framework `go test` for testing embedded systems (serial console) and improving the report format for integration to a CI.
Julien Viard de Galbert
Embedded Recipes 2019 - Herd your socs become a matchmakerAnne Nicolas
About 60% of the Linux kernel source tree is devoted to drivers for a large variety of supported hardware components. Especially in the embedded world, the number of different SoC families, versions, and revisions, integrating a myriad of “IP cores”, keeps on growing.
In this presentation, Geert will explain how to match drivers against hardware, and how to support a wide variety of (dis)similar devices, without turning platform and driver code into an entangled bowl of spaghetti.tra
Starting with a brief history of driver matching in Linux, he will fast-forward to device-tree based matching. He will discuss ways to handle slight variations of the same hardware devices, and different SoC revisions, each with their own quirks and bugs. Finally, Geert will show best practices for evolving device drivers in a maintainable way, based on his experiences as an embedded Linux kernel developer and maintainer.
Geert Uytterhoeven
Embedded Recipes 2019 - LLVM / Clang integrationAnne Nicolas
Buildroot is a popular and easy to use embedded Linux build system. It generates, in few minutes, lightweight and customized Linux systems, including the cross-compilation toolchain, kernel and bootloader images, as well as a wide variety of userspace libraries and programs.
This talk is about the integration of LLVM/clang into Buildroot.
In 2018, Valentin Korenblit, supervised by Romain Naour, worked on this topic during his internship at Smile ECS. After a short introduction about llvm/clang and Buildroot, this talk will go through the numerous issues discovered while adding llvm/clang componants and how these issues were fixed. Romain will also detail the work in progress and the work to be done based on llvm/clang libraries (OpenCL, Compiler-rt, BCC. Chromium, ldd).
Romain Naour
Embedded Recipes 2019 - Introduction to JTAG debuggingAnne Nicolas
This talk introduces JTAG debugging capabilities, both for debugging hardware and software. Marek first explains what the JTAG stands for and explains the operation of the JTAG state machine. This is followed by an introduction to free software JTAG tools, OpenOCD and urJTAG. Marek shortly explains how to debug software using those tools and how that ties into the JTAG state machine. However, JTAG was designed for testing hardware. Marek explains what boundary scan testing (BST) is, what are BSDL files and their format, and practically demonstrates how to blink an LED using BST and only free software tools.
Marek Vasut
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimediaAnne Nicolas
PipeWire is an open source project that aims to greatly improve audio and video handling under Linux. Utilising a fresh design, it bridges use cases that have been previously addressed by different tools – or not addressed at all -, providing ground for building complex, yet secure and efficient, multimedia systems.
In this talk, Julien is going to present the PipeWire project and the concepts that make up its design. In addition, he is going to give an update of the current and future work going on around PipeWire, both upstream and in Automotive Grade Linux, an early adopter that Julien is actively working on.
Julian Bouzas
Kernel Recipes 2019 - Suricata and XDPAnne Nicolas
Suricata is a network threat detection engine using network packets capture to reconstruct the traffic till the application layer and find threats on the network using rules that define behavior to detect. This task is really CPU intensive and discarding non interesting traffic is a solution to enable a scaling of Suricata to 40gbps and other.
This talk will present the latest evolution of Suricata that knows uses eBPF and XDP to bypass traffic. Suricata 5.0 is supporting the hardware XDP to provide ypass with network card such as Netronome. It also takes advantage of pinned maps to get persistance of the bypassed flows. This talk will cover the different usage of XDP and eBPF in Suricata and shows how it impact performance and usability. If development time permit, the talk will also cover AF_XDP and the impact on this new capture method on Suricata.
Eric Leblond
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)Anne Nicolas
System memory configuration is a transparent operation nowadays, something that we all came to expect to just work out of the box. Still, it does happen behind the scenes every single time we boot our computers. This requires the cooperation of hardware components on the mainboard and on memory modules themselves, as well as firmware code to drive these. While it is possible to just let it happen, having a deeper understanding of how it works makes it possible to access valuable information from the operating system at run-time.
I will take you through the history of system memory configuration from the mid 70s to now. We will explore the different types of memory modules, how their configuration data is stored and how the firmware can access them. We will see which problems had to be solved along the way and how they were solved. Lastly we will see how Linux supports reading the memory configuration information and what you can do with that information.
Jean Delvare
Kernel Recipes 2019 - Formal modeling made easyAnne Nicolas
Modeling parts of Linux has become a recurring topic. For instance, the memory model, the model for PREEMPT_RT synchronization, and so on. But the term "formal model" causes panic for most of the developers. Mainly because of the complex notations and reasoning that involves formal languages. It seems to be a very theoretical thing, far from our day-by-day reality.
Believe me. Modeling can be more practical than you might guess!
This talk will discuss the challenges and benefits of modeling, based on the experience of developing the PREEMPT_RT model. It will present a methodology for modeling the Linux behavior as Finite-State Machines (automata), using terms that are very known by kernel developers: tracing events! With the particular focus on how to use models for the formal verification of Linux kernel, at runtime, with low overhead, and in many cases, without even modifying Linux kernel!
Daniel Bristot de Oliveira
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
4. Disclaimer
●
I talk fast
●
I will be giving a 120 minute presentation in 40 minutes
– You need to watch this 3 times to get the full understanding (3 x 40 = 120)
5. Disclaimer
●
I talk fast
●
I will be giving a 120 minute presentation in 40 minutes
– You need to watch this 3 times to get the full understanding (3 x 40 = 120)
●
Don’t try to understand everything I present here
– I’ll try to point out what is important and what you can come back to
7. What is ftrace?
●
The official tracer of the Linux Kernel
●
Added in 2.6.31 (2009)
●
Infrastructure with several features
– events
– individual functions
– latency tracing
8. How to get to it?
●
Probably already in your kernel
●
tracefs (also debugfs)
– /sys/kernel/tracing
– /sys/kernel/debug/tracing
(all files in this document are in here)
●
Can use it with just echo and cat
– Normal use of shell commands
10. The tracers
# cat available_tracers
hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup
function nop
# cat available_tracers
blk mmiotrace function_graph function nop
My custom kernel
Debian 4.9.0-3-amd64 kernel
11. The nop tracer
●
echo “nop” > current_tracer
●
A way to disable other tracers
●
Has no actual function
●
Is the default tracer
●
Can be used with events
13. The Function Graph Tracer
# echo function_graph > current_tracer
# cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) 8.183 us | } /* ep_scan_ready_list.constprop.12 */
3) ! 273.670 us | } /* ep_poll */
3) 0.074 us | fput();
3) ! 276.267 us | } /* SyS_epoll_wait */
3) ! 278.559 us | } /* do_syscall_64 */
3) | do_syscall_64() {
3) | syscall_trace_enter() {
3) | __secure_computing() {
3) | __seccomp_filter() {
3) 0.824 us | __bpf_prog_run();
3) 1.700 us | }
3) 2.434 us | }
3) 3.292 us | }
3) | SyS_read() {
3) | __fdget_pos() {
3) | __fget_light() {
3) 0.201 us | __fget();
3) 1.005 us | }
3) 1.753 us | }
3) | vfs_read() {
3) | rw_verify_area() {
14. Temporarily enabling and disabling tracing
●
tracing_on
– echo 0 > tracing_on
●
disables writing to the ring buffer (does not stop the functionality)
– echo 0 > tracing_on
●
Enables writing to the ring buffer
●
Be ware of the “echo 0> tracing_on”!
15. Temporarily enabling and disabling tracing
●
tracing_on
– echo 0 > tracing_on
●
disables writing to the ring buffer (does not stop the functionality)
– echo 0 > tracing_on
●
Enables writing to the ring buffer
●
Be ware of the “echo 0> tracing_on”!
– Hint: there's no space between the zero and the greater than sign
16. Temporarily enabling and disabling tracing
●
tracing_on
– echo 0 > tracing_on
●
disables writing to the ring buffer (does not stop the functionality)
– echo 0 > tracing_on
●
Enables writing to the ring buffer
●
Be ware of the “echo 0> tracing_on”!
– Hint: there's no space between the zero and the greater than sign
– You just wrote standard input into “tracing_on”
21. set_ftrace_pid
●
Only traces functions executed by a task with the given pid
– For threads, it is the thread id (in the kernel, threads are just tasks)
●
Neat little trick to trace only what you want:
– echo 0 > tracing_on
– echo function > current_tracer
– # sh -c 'echo $$ > set_ftrace_pid; echo 1 > tracing_on; exec my_prog'
23. System Call Functions (hack mode)
●
System calls like read(), write() and open()
●
The kernel has special wrappers for them
– SYSCALL_DEFINE[0-6]
– For sign extension of system call arguments
●
Appends “sys_”, sys_read(), sys_write() and sys_open()
●
But also uses an alias SyS_read(), SyS_write() and SyS_open()
SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count)
# grep -i ' sys_read$' /proc/kallsyms
ffffffffbd4032a0 T SyS_read
ffffffffbd4032a0 T sys_read
24. set_ftrace_filter with System Calls
●
Unfortunately, ftrace just picks one
●
And it usually picks the wrong one :-(
# grep -i '^sys_read$' available_filter_functions
SyS_read
# echo sys_read > set_ftrace_filter
bash: echo: write error: Invalid argument
# echo SyS_read > set_ftrace_filter
# cat set_ftrace_filter
SyS_read
25. Let's look at the code for the read() system call (v4.12)
SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count)
{
struct fd f = fdget_pos(fd);
ssize_t ret = -EBADF;
if (f.file) {
loff_t pos = file_pos_read(f.file);
ret = vfs_read(f.file, buf, count, &pos);
if (ret >= 0)
file_pos_write(f.file, pos);
fdput_pos(f);
}
return ret;
}
26. Let's look at the code for the read() system call (v4.12)
ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos)
{
ssize_t ret;
if (!(file->f_mode & FMODE_READ))
return -EBADF;
if (!(file->f_mode & FMODE_CAN_READ))
return -EINVAL;
if (unlikely(!access_ok(VERIFY_WRITE, buf, count)))
return -EFAULT;
ret = rw_verify_area(READ, file, pos, count);
if (!ret) {
if (count > MAX_RW_COUNT)
count = MAX_RW_COUNT;
ret = __vfs_read(file, buf, count, pos);
if (ret > 0) {
fsnotify_access(file);
add_rchar(current, ret);
}
inc_syscr(current);
}
return ret;
}
27. Let's look at the code for the read() system call (v4.12)
ssize_t __vfs_read(struct file *file, char __user *buf, size_t count,
loff_t *pos)
{
if (file->f_op->read)
return file->f_op->read(file, buf, count, pos);
else if (file->f_op->read_iter)
return new_sync_read(file, buf, count, pos);
else
return -EINVAL;
}
28. Let's look at the code for the read() system call (v4.12)
ssize_t __vfs_read(struct file *file, char __user *buf, size_t count,
loff_t *pos)
{
if (file->f_op->read)
return file->f_op->read(file, buf, count, pos);
else if (file->f_op->read_iter)
return new_sync_read(file, buf, count, pos);
else
return -EINVAL;
}
29. Let's look at the code for the read() system call (v4.12)
ssize_t __vfs_read(struct file *file, char __user *buf, size_t count,
loff_t *pos)
{
if (file->f_op->read)
return file->f_op->read(file, buf, count, pos);
else if (file->f_op->read_iter)
return new_sync_read(file, buf, count, pos);
else
return -EINVAL;
}
30. Let's look at the code for the read() system call (v4.12)
struct file {
union {
struct llist_node fu_llist;
struct rcu_head fu_rcuhead;
} f_u;
struct path f_path;
struct inode *f_inode; /* cached value */
const struct file_operations *f_op;
31. Let's look at the code for the read() system call (v4.12)
# git grep 'struct file_operations' | wc -l
2885
32. set_graph_function
●
Similar to set_ftrace_filter
– same way to set it (with globs)
– same way to disable it (write nothing to it)
– same issues with syscalls (SyS_read not sys_read)
●
Picks a function to graph
●
See what a function does
46. trace options
●
Modifies the way tracers may work
●
Modifies output format
●
Some tracers have their own options
●
Two ways to modify the options
– file trace_options
●
shows only global or current tracer options
●
to enable: echo option_name > trace_options
●
to disable: echo nooption_name > trace_options
– options/ directory
●
shows global options
●
shows current tracer options
●
starting with v4.4 - shows all tracer options
●
to enable: echo 1 > options/option_name
●
to disable: echo 0 > options/option_name
47. func_stack_trace option
●
Creates a stack trace of all functions traced
●
WARNING: Can cause live lock if all functions are traced!!!!
– Must use set_ftrace_filter - BEFORE enabling
– Nothing will prevent you from shooting yourself in the foot
– You have been warned!
●
echo schedule > set_ftrace_filter
●
echo 1 > options/func_stack_trace
●
echo function > current_tracer
●
Do stuff
●
echo 0 > tracing_on
●
echo 0 > options/func_stack_trace
53. Filter specific modules
# echo function_graph > current_tracer
# cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) | ieee80211_rx_napi [mac80211]() {
3) 0.160 us | remove_monitor_info [mac80211]();
3) 0.321 us | ieee80211_scan_rx [mac80211]();
3) 0.993 us | sta_info_get_bss [mac80211]();
3) | ieee80211_prepare_and_rx_handle [mac80211]() {
3) 0.132 us | ieee80211_get_bssid [mac80211]();
3) | ieee80211_rx_handlers [mac80211]() {
3) 0.126 us | ieee80211_sta_rx_notify [mac80211]();
3) 0.173 us | ieee80211_get_mmie_keyidx [mac80211]();
3) 0.133 us | ieee80211_rx_h_michael_mic_verify [mac80211]();
3) 1.533 us | ieee80211_queue_work [mac80211]();
3) 0.117 us | ieee80211_rx_handlers_result [mac80211]();
3) 9.483 us | }
3) + 12.299 us | }
3) + 20.847 us | }
2) | ieee80211_iface_work [mac80211]() {
2) | ieee80211_sta_rx_queued_mgmt [mac80211]() {
2) | ieee80211_rx_mgmt_beacon [mac80211]() {
2) 0.601 us | ieee80211_sta_reset_beacon_monitor [mac80211]();
2) + 17.092 us | ieee802_11_parse_elems_crc [mac80211]();
2) + 21.018 us | }
2) + 22.399 us | }
2) 0.514 us | ieee80211_sta_work [mac80211]();
2) + 27.349 us | }
54. Function triggers
●
Attach a trigger (execution) to a specific function
– There are several, but this talk will only mention
●
traceoff
●
traceon
●
stacktrace
●
traceoff
– Stop tracing when the function is hit
●
traceon
– Start tracing when a function is hit
●
stacktrace
– Much safer than using the func_stack_trace option
– Do not need to filter
– Only give a stack trace for the function you care about
– Can use in conjunction with function_graph tracer
55. Filter specific modules
# echo function_graph > current_tracer
# cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) | ieee80211_rx_napi [mac80211]() {
3) 0.160 us | remove_monitor_info [mac80211]();
3) 0.321 us | ieee80211_scan_rx [mac80211]();
3) 0.993 us | sta_info_get_bss [mac80211]();
3) | ieee80211_prepare_and_rx_handle [mac80211]() {
3) 0.132 us | ieee80211_get_bssid [mac80211]();
3) | ieee80211_rx_handlers [mac80211]() {
3) 0.126 us | ieee80211_sta_rx_notify [mac80211]();
3) 0.173 us | ieee80211_get_mmie_keyidx [mac80211]();
3) 0.133 us | ieee80211_rx_h_michael_mic_verify [mac80211]();
3) 1.533 us | ieee80211_queue_work [mac80211]();
3) 0.117 us | ieee80211_rx_handlers_result [mac80211]();
3) 9.483 us | }
3) + 12.299 us | }
3) + 20.847 us | }
2) | ieee80211_iface_work [mac80211]() {
2) | ieee80211_sta_rx_queued_mgmt [mac80211]() {
2) | ieee80211_rx_mgmt_beacon [mac80211]() {
2) 0.601 us | ieee80211_sta_reset_beacon_monitor [mac80211]();
2) + 17.092 us | ieee802_11_parse_elems_crc [mac80211]();
2) + 21.018 us | }
2) + 22.399 us | }
2) 0.514 us | ieee80211_sta_work [mac80211]();
2) + 27.349 us | }
56. Filter specific modules
# echo function_graph > current_tracer
# cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) | ieee80211_rx_napi [mac80211]() {
3) 0.160 us | remove_monitor_info [mac80211]();
3) 0.321 us | ieee80211_scan_rx [mac80211]();
3) 0.993 us | sta_info_get_bss [mac80211]();
3) | ieee80211_prepare_and_rx_handle [mac80211]() {
3) 0.132 us | ieee80211_get_bssid [mac80211]();
3) | ieee80211_rx_handlers [mac80211]() {
3) 0.126 us | ieee80211_sta_rx_notify [mac80211]();
3) 0.173 us | ieee80211_get_mmie_keyidx [mac80211]();
3) 0.133 us | ieee80211_rx_h_michael_mic_verify [mac80211]();
3) 1.533 us | ieee80211_queue_work [mac80211]();
3) 0.117 us | ieee80211_rx_handlers_result [mac80211]();
3) 9.483 us | }
3) + 12.299 us | }
3) + 20.847 us | }
2) | ieee80211_iface_work [mac80211]() {
2) | ieee80211_sta_rx_queued_mgmt [mac80211]() {
2) | ieee80211_rx_mgmt_beacon [mac80211]() {
2) 0.601 us | ieee80211_sta_reset_beacon_monitor [mac80211]();
2) + 17.092 us | ieee802_11_parse_elems_crc [mac80211]();
2) + 21.018 us | }
2) + 22.399 us | }
2) 0.514 us | ieee80211_sta_work [mac80211]();
2) + 27.349 us | }
61. funcgraph-irqs option
●
Enables showing interrupts in function graph tracer
●
Default on
●
Sometimes they clutter the trace
●
echo 0 > options/funcgraph-irqs
62. funcgraph-irqs option enabled
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 3.982 us | }
------------------------------------------
1) chrome-3204 => <idle>-0
------------------------------------------
1) | finish_task_switch() {
1) ==========> |
1) | smp_apic_timer_interrupt() {
1) | irq_enter() {
1) 0.114 us | rcu_irq_enter();
1) | tick_irq_enter() {
1) 0.083 us | tick_check_oneshot_broadcast_this_cpu();
1) 1.277 us | }
1) | _local_bh_enable() {
1) 0.067 us | __local_bh_enable();
1) 0.885 us | }
1) 4.434 us | }
1) 0.090 us | exit_idle();
1) | local_apic_timer_interrupt() {
1) | hrtimer_interrupt() {
1) 0.065 us | _raw_spin_lock();
1) 0.130 us | ktime_get_update_offsets_now();
1) | __hrtimer_run_queues() {
1) 0.358 us | __remove_hrtimer();
1) | tick_sched_timer() {
1) 0.107 us | ktime_get();
1) | tick_sched_do_timer() {
63. funcgraph-irqs option enabled
1) 0.070 us | idle_cpu();
1) 0.748 us | }
1) 0.070 us | __msecs_to_jiffies();
1) + 26.917 us | }
1) + 27.645 us | }
1) | rcu_process_callbacks() {
1) | note_gp_changes() {
1) 0.121 us | _raw_spin_trylock();
1) | __note_gp_changes() {
1) 0.110 us | rcu_advance_cbs();
1) 0.848 us | }
1) 0.086 us | _raw_spin_unlock_irqrestore();
1) 2.803 us | }
1) 0.073 us | cpu_needs_another_gp();
1) 0.167 us | note_gp_changes();
1) 0.174 us | cpu_needs_another_gp();
1) 5.830 us | }
1) 0.094 us | rcu_bh_qs();
1) 0.070 us | __local_bh_enable();
1) + 41.106 us | }
1) 0.073 us | idle_cpu();
1) 0.094 us | tick_nohz_irq_exit();
1) 0.093 us | rcu_irq_exit();
1) + 44.029 us | }
1) + 97.119 us | }
1) <========== |
1) + 98.748 us | }
1) ! 225.374 us | } /* schedule */
64. max_graph_depth
●
Usually set to just 1
●
Shows where user space enters the kernel
●
Made for NO_HZ_FULL
●
strace on steroids (all tasks!)
65. max_graph_depth
# echo 1 > max_graph_depth
# cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) # 3066.150 us | } /* call_cpuidle */
1) 0.719 us | cpuidle_reflect();
1) 0.417 us | rcu_idle_exit();
1) 0.565 us | arch_cpu_idle_exit();
1) 1.273 us | tick_nohz_idle_exit();
1) 0.103 us | sched_ttwu_pending();
1) | schedule_preempt_disabled() {
------------------------------------------
1) <idle>-0 => Xorg-948
------------------------------------------
1) @ 189902.5 us | } /* SyS_epoll_wait */
1) 4.738 us | SyS_setitimer();
1) + 13.246 us | SyS_recvmsg();
1) 7.513 us | SyS_ioctl();
1) 2.519 us | SyS_ioctl();
1) 2.529 us | SyS_ioctl();
1) 1.473 us | SyS_ioctl();
1) 2.706 us | SyS_ioctl();
66. max_graph_depth
# echo 0 > tracing_on
# echo 1 > max_graph_depth
# echo function_graph > current_tracer
# sh -c 'echo $$ > set_ftrace_pid; echo 1 > tracing_on; exec echo hello'
# cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) 0.508 us | mutex_unlock();
3) 0.292 us | __fsnotify_parent();
3) 0.231 us | fsnotify();
3) 0.167 us | __sb_end_write();
3) 1.638 us | SyS_dup2();
3) 3.325 us | exit_to_usermode_loop();
3) 0.835 us | SyS_close();
3) + 21.778 us | do_syscall_64();
3) + 12.394 us | do_syscall_64();
3) + 11.289 us | do_syscall_64();
3) + 10.945 us | do_syscall_64();
3) + 10.028 us | do_syscall_64();
3) ! 688.935 us | do_syscall_64();
3) + 16.727 us | __do_page_fault();
3) 4.429 us | __do_page_fault();
3) 8.208 us | __do_page_fault();
3) 6.877 us | __do_page_fault();
3) + 10.033 us | __do_page_fault();
3) 8.013 us | __do_page_fault();
3) 0.644 us | SyS_brk();
3) 4.904 us | __do_page_fault();
3) 4.702 us | __do_page_fault();
3) + 12.393 us | __do_page_fault();
3) 3.459 us | __do_page_fault();
3) 7.670 us | __do_page_fault();
3) 4.145 us | __do_page_fault();
3) 9.870 us | SyS_access();
3) 7.549 us | SyS_mmap();
3) 7.867 us | __do_page_fault();
3) 6.820 us | SyS_access();
3) + 13.364 us | SyS_open();
3) 3.289 us | SyS_newfstat();
3) 9.849 us | SyS_mmap();
67. Events
●
Function tracing is very limited
– Only shows enter and exit of functions
– Shows the function and the function that called it
– No parameters
– No exit code
●
Events located in /sys/kernel/tracing/events/
●
Events show data similar to a printk()
●
The Debian 4.9.0-3-amd64 kernel has 1530 trace events defined
– with my modules loaded
68. Events - continued
●
Broken up by systems
– interrupts
– scheduling
– timer
– block
– file systems
– system calls
– devices
– various other subsystems
●
May enable/disable all events at once
●
May enable/disable a set of system specific events at once
●
May just enable/disable a single event
69. Events
# ls events
block filemap jbd2 net sched v4l2
cfg80211 ftrace kmem nfsd scsi vb2
cgroup gpio kvm nmi signal vmscan
clk hda kvmmmu oom skb vsyscall
compaction hda_controller libata page_isolation sock workqueue
cpuhp hda_intel mac80211 pagemap spi writeback
drm header_event mce power sunrpc x86_fpu
enable header_page mei printk swiotlb xen
exceptions huge_memory migrate random syscalls xfs
ext4 i2c mmc ras task xhci-hcd
fence i915 module raw_syscalls thermal
fib iommu mpx rcu timer
fib6 irq msr regmap tlb
filelock irq_vectors napi rpm udp
# ls events/irq
enable irq_handler_entry softirq_entry softirq_raise
filter irq_handler_exit softirq_exit
# ls events/irq/irq_hander_entry
enable filter format id trigger
70. Events
●
Most common subsystems
– sched - for scheduling events
– irq - for interrupts
– timer - for timers set by user and kernel
– exceptions - like page faults
●
echo 1 > events/sched/enable
●
Can be used along with any tracer
●
Use “nop” tracer to just show events
74. Tracing children of a task
●
Two trace options
– event-fork - added v4.7
●
When enabled, all tasks in set_event_pid have their children added on fork
– function-fork - added v4.12
●
When enabled, all tasks in set_ftrace_pid have theiri children added on fork
●
All functionality was there in v4.7 but I wanted to test with events first
75. Tracing children of a task
●
Two trace options
– event-fork - added v4.7
●
When enabled, all tasks in set_event_pid have their children added on fork
– function-fork - added v4.12
●
When enabled, all tasks in set_ftrace_pid have theiri children added on fork
●
All functionality was there in v4.7 but I wanted to test with events first
●
But I forgot about it after the 4.7 merge window (Doh!)
76. Tracing children of a task
●
Two trace options
– event-fork - added v4.7
●
When enabled, all tasks in set_event_pid have their children added on fork
– function-fork - added v4.12
●
When enabled, all tasks in set_ftrace_pid have theiri children added on fork
●
All functionality was there in v4.7 but I wanted to test with events first
●
But I forgot about it after the 4.7 merge window (Doh!)
●
Namhyung Kim sent a simple patch to make functions trace children
80. Introducing trace-cmd
●
Can do most everything that can be done via tracefs
– as root only
●
No need to know about tracefs
– It will mount it for you if not done already (remember, you are root)
●
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
●
make; make doc
●
sudo make install; sudo make install_doc