SystemTap is a dynamic tracing tool for Linux systems. It allows users to easily gather information about the running Linux system by defining probe points in a script. The script is compiled into a kernel module which can then be loaded to monitor the specified probe points. Some examples of useful probe points include functions, system calls, and kernel statements. SystemTap scripts can be used to trace execution, profile performance, monitor kernel functions and debug problems by printing at probe points. It provides a safe way to observe a live system without needing to recompile the kernel.
Talk by Brendan Gregg for YOW! 2021. "The pursuit of faster performance in computing is the driving reason for many new technologies and updates. This talk discusses performance improvements now underway that you will likely be adopting soon, for processors (including 3D stacking and cloud vendor CPUs), memory (including DDR5 and high-bandwidth memory [HBM]), disks (including 3D Xpoint as a 3D NAND accelerator), networking (including QUIC and eXpress Data Path [XDP]), runtimes, hypervisors, and more. The future of performance is increasingly cloud-based, with hardware hypervisors and custom processors, meaningful observability of everything down to cycle stalls (even as cloud guests), and high-speed syscall-avoiding applications that use eBPF, FPGAs, and io_uring. The talk also discusses where future performance improvements might be expected, with predictions for new technologies."
Comparing Next-Generation Container Image Building ToolsAkihiro Suda
http://sched.co/EaYe
Until recently, running `docker build` against Dockerfile had been the only way to build container images.
However, lots of opensource software are being proposed as successors/alternatives to `docker build`:
- BuildKit (Moby Project / Docker)
- img (Jessica Frazelle / Microsoft)
- Buildah (Project Atomic / Red Hat)
- umoci & Orca (SUSE)
- Bazel (Google)
- OpenShift S2I (Red Hat)
Akihiro Suda compares these new tools' advantages and disadvantages.
His evaluation basis would include but not be limited to:
- Performance (Cache efficiency, Concurrency, Distributed Execution)
- Secret management, e.g. SSH and AWS keys
- Support for non-Dockerfile
- Non-root execution
- UI & UX
- Governance of the community
He also proposes a unified interface for using these tools with Kubernetes in a vendor-neutral way.
SlideTeam presents Kubernetes Docker Container Implementation Ppt PowerPoint Presentation Slide Templates. This PPT slideshow is an ideal virtual expression of the fundamentals of Kubernetes. The smart data-visualizations make this PowerPoint presentation easy-to-understand and perfect to introduce your audience to the container orchestration system. Use our PPT theme to communicate the definition and need for containers or virtual private servers. Communicate the container, and microservices architecture using cutting-edge graphics. Explain the need for and benefits of Kubernetes for an organization. Elucidate the features, architecture, use cases, installation roadmap, and the 30-60-90 day plan in Kubernetes. Use the neat tabular format to compare Kubernetes with docker swarm based on various parameters. Familiarize your viewers with the various components of Kubernetes. Elaborate on what is Kubelet, Kubectl, and Kubeadm with the help of labeled diagrams. This presentation acquaints your audience with the significance of Kubernetes in management, scaling, automating, and deploying computer applications. Hit the download icon and start personalization. https://bit.ly/2L0Ojdu
Talk by Brendan Gregg for YOW! 2021. "The pursuit of faster performance in computing is the driving reason for many new technologies and updates. This talk discusses performance improvements now underway that you will likely be adopting soon, for processors (including 3D stacking and cloud vendor CPUs), memory (including DDR5 and high-bandwidth memory [HBM]), disks (including 3D Xpoint as a 3D NAND accelerator), networking (including QUIC and eXpress Data Path [XDP]), runtimes, hypervisors, and more. The future of performance is increasingly cloud-based, with hardware hypervisors and custom processors, meaningful observability of everything down to cycle stalls (even as cloud guests), and high-speed syscall-avoiding applications that use eBPF, FPGAs, and io_uring. The talk also discusses where future performance improvements might be expected, with predictions for new technologies."
Comparing Next-Generation Container Image Building ToolsAkihiro Suda
http://sched.co/EaYe
Until recently, running `docker build` against Dockerfile had been the only way to build container images.
However, lots of opensource software are being proposed as successors/alternatives to `docker build`:
- BuildKit (Moby Project / Docker)
- img (Jessica Frazelle / Microsoft)
- Buildah (Project Atomic / Red Hat)
- umoci & Orca (SUSE)
- Bazel (Google)
- OpenShift S2I (Red Hat)
Akihiro Suda compares these new tools' advantages and disadvantages.
His evaluation basis would include but not be limited to:
- Performance (Cache efficiency, Concurrency, Distributed Execution)
- Secret management, e.g. SSH and AWS keys
- Support for non-Dockerfile
- Non-root execution
- UI & UX
- Governance of the community
He also proposes a unified interface for using these tools with Kubernetes in a vendor-neutral way.
SlideTeam presents Kubernetes Docker Container Implementation Ppt PowerPoint Presentation Slide Templates. This PPT slideshow is an ideal virtual expression of the fundamentals of Kubernetes. The smart data-visualizations make this PowerPoint presentation easy-to-understand and perfect to introduce your audience to the container orchestration system. Use our PPT theme to communicate the definition and need for containers or virtual private servers. Communicate the container, and microservices architecture using cutting-edge graphics. Explain the need for and benefits of Kubernetes for an organization. Elucidate the features, architecture, use cases, installation roadmap, and the 30-60-90 day plan in Kubernetes. Use the neat tabular format to compare Kubernetes with docker swarm based on various parameters. Familiarize your viewers with the various components of Kubernetes. Elaborate on what is Kubelet, Kubectl, and Kubeadm with the help of labeled diagrams. This presentation acquaints your audience with the significance of Kubernetes in management, scaling, automating, and deploying computer applications. Hit the download icon and start personalization. https://bit.ly/2L0Ojdu
Performance Wins with BPF: Getting StartedBrendan Gregg
Keynote by Brendan Gregg for the eBPF summit, 2020. How to get started finding performance wins using the BPF (eBPF) technology. This short talk covers the quickest and easiest way to find performance wins using BPF observability tools on Linux.
Seven Habits of Highly Effective Jenkins Users (2014 edition!)Andrew Bayer
What plugins, tools and behaviors can help you get the most out of your Jenkins setup without all of the pain? We'll find out as we go over a set of Jenkins power tools, habits and best practices that will help with any Jenkins setup.
GitHub Actions (Nakov at RuseConf, Sept 2022)Svetlin Nakov
Building a CI/CD System with GitHub Actions
Dr. Svetlin Nakov
September 2022
Intro to Continuous Integration (CI), Continuous Delivery (CD), Continuous Deployment (CD), and CI/CD Pipelines
Intro to GitHub Actions
Building CI workflow
Building CD workflow
Live Demo: Build CI System for JS and .NET Apps
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
Talk for SCaLE13x. Video: https://www.youtube.com/watch?v=_Ik8oiQvWgo . Profiling can show what your Linux kernel and appliacations are doing in detail, across all software stack layers. This talk shows how we are using Linux perf_events (aka "perf") and flame graphs at Netflix to understand CPU usage in detail, to optimize our cloud usage, solve performance issues, and identify regressions. This will be more than just an intro: profiling difficult targets, including Java and Node.js, will be covered, which includes ways to resolve JITed symbols and broken stacks. Included are the easy examples, the hard, and the cutting edge.
When your whole system is unresponsive, how to investigate on this failure ?
We'll see how to get a memory dump for offline analysis with kdump system.
Then how to analyze it with crash utility.
And finally, how to use crash on a running system to modify the kernel memory (at your own risks !)
Performance Wins with BPF: Getting StartedBrendan Gregg
Keynote by Brendan Gregg for the eBPF summit, 2020. How to get started finding performance wins using the BPF (eBPF) technology. This short talk covers the quickest and easiest way to find performance wins using BPF observability tools on Linux.
Seven Habits of Highly Effective Jenkins Users (2014 edition!)Andrew Bayer
What plugins, tools and behaviors can help you get the most out of your Jenkins setup without all of the pain? We'll find out as we go over a set of Jenkins power tools, habits and best practices that will help with any Jenkins setup.
GitHub Actions (Nakov at RuseConf, Sept 2022)Svetlin Nakov
Building a CI/CD System with GitHub Actions
Dr. Svetlin Nakov
September 2022
Intro to Continuous Integration (CI), Continuous Delivery (CD), Continuous Deployment (CD), and CI/CD Pipelines
Intro to GitHub Actions
Building CI workflow
Building CD workflow
Live Demo: Build CI System for JS and .NET Apps
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
Talk for SCaLE13x. Video: https://www.youtube.com/watch?v=_Ik8oiQvWgo . Profiling can show what your Linux kernel and appliacations are doing in detail, across all software stack layers. This talk shows how we are using Linux perf_events (aka "perf") and flame graphs at Netflix to understand CPU usage in detail, to optimize our cloud usage, solve performance issues, and identify regressions. This will be more than just an intro: profiling difficult targets, including Java and Node.js, will be covered, which includes ways to resolve JITed symbols and broken stacks. Included are the easy examples, the hard, and the cutting edge.
When your whole system is unresponsive, how to investigate on this failure ?
We'll see how to get a memory dump for offline analysis with kdump system.
Then how to analyze it with crash utility.
And finally, how to use crash on a running system to modify the kernel memory (at your own risks !)
Kernel Recipes 2015: Kernel packet capture technologiesAnne Nicolas
Sniffing through the ages
Capturing packets running on the wire to send them to a software doing analysis seems at first sight a simple tasks. But one has not to forget that with current network this can means capturing 30M packets per second. The objective of this talk is to show what methods and techniques have been implemented in Linux and how they have evolved over time.
The talk will cover AF_PACKET capture as well as PF_RING, dpdk and netmap. It will try to show how the various evolution of hardware and software have had an impact on the design of these technologies. Regarding software a special focus will be made on Suricata IDS which is implementing most of these capture methods.
Eric Leblond, Stamus Networks
We'll discuss our experiences with tooling aimed at finding and fixing performance problems in a production Rust application, as experienced through the eyes of somebody who's more familiar with the Go ecosystem but grew to love Rust. We'll cover CPU and Heap profiling, and also briefly touch causal profiling.
Linux Server Deep Dives (DrupalCon Amsterdam)Amin Astaneh
Over the past few years the Linux kernel has gained features that allow us to learn more about what's really happening on our servers and the applications that run on them.
This talk will explore how these new features, particularly perf_events and ebpf, enable us to answer questions about what a Drupal site is doing in real time beyond what the standard logs, server performance tools, and even strace will reveal. Attendees will be provided a brief introduction to example uses of these tools to diagnose performance problems.
This talk is intended for attendees that are familiar with Linux, the command line, and have used host observability tools in the past (top, netstat, etc).
DTrace and SystemTap are dynamic tracing frameworks available for Solaris and Linux respectively. This session will give an overview of the static DTrace probes available in both Drizzle and MySQL and show numerous examples of scripts that utilize these probes. Mixing dynamic and static probes will also be discussed.
Slides for JavaOne 2015 talk by Brendan Gregg, Netflix (video/audio, of some sort, hopefully pending: follow @brendangregg on twitter for updates). Description: "At Netflix we dreamed of one visualization to show all CPU consumers: Java methods, GC, JVM internals, system libraries, and the kernel. With the help of Oracle this is now possible on x86 systems using system profilers (eg, Linux perf_events) and the new JDK option -XX:+PreserveFramePointer. This lets us create Java mixed-mode CPU flame graphs, exposing all CPU consumers. We can also use system profilers to analyze memory page faults, TCP events, storage I/O, and scheduler events, also with Java method context. This talk describes the background for this work, instructions generating Java mixed-mode flame graphs, and examples from our use at Netflix where Java on x86 is the primary platform for the Netflix cloud."
Linux Tracing Superpowers by Eugene PirogovPivorak MeetUp
For a long time Linux was far behind operating systems of Unix family from the perspective of debuggability, specifically in a live production systems.
However, over the course of 2016 Linux saw a series of patches that brought it on par with Unix world: an old Linux tool called BPF has risen and extended into powerful new one – eBPF. Some say that eBPF marks the begining of true DTrace for Linux.
During the presentation I'm going to talk about tracing basics, cover a series of events that led to the development of eBPF and will compare eBPF with DTrace from Unix world. Current state of affairs of Linux tracing tools will be explored. Finally, together we'll look at some of the exciting examples of eBPF application.
***
Eugene is well known in our Ruby (and Elixir) communities. Last time when he was at #pivorak he made a very light and interesting intro to the Elixir. You can check his speech out here - http://bit.ly/2evCd9R
O'Reilly Velocity New York 2016 presentation on modern Linux tracing tools and technology. Highlights the available tracing data sources on Linux (ftrace, perf_events, BPF) and demonstrates some tools that can be used to obtain traces, including DebugFS, the perf front-end, and most importantly, the BCC/BPF tool collection.
3. SystemTap 是什么?
According to http://sourceware.org/systemtap/
SystemTap provides free software (GPL) infrastructure to
simplify the gathering of information about the running
Linux system. This assists diagnosis of a performance or
functional problem. SystemTap eliminates the need for
the developer to go through the tedious and disruptive
instrument, recompile, install, and reboot sequence that
may be otherwise required to collect data.
观察活体系统最佳工具,前提是你懂得如何观察!
4. SystemTap 是如何工作的
1. write or choose a script describing what you want to observe
2. stap translates it into a kernel module
3. stap loads the module and communicates with it
4. just wait for your data
5. 五步走
# stap -uv test.stp
Pass 1: parsed user script and 74 library script(s) using
86868virt/20488res/1792shr kb, in 190usr/20sys/209real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using
87264virt/21148res/1976shr kb, in 10usr/0sys/7real ms.
Pass 3: translated to C into
"/tmp/stapz2iv97/stap_aef621603e006af62084b361e0a0c981_553.c" using
87264virt/21332res/2144shr kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_aef621603e006af62084b361e0a0c981_553.ko" in
1230usr/160sys/1384real ms.
Pass 5: starting run.
Pass 5: run completed in 10usr/20sys/12331real ms.
6. SystemTap 探测点例子
SystemTap is all about executing certain actions when hitting
certain probe points.
• syscall.read
when entering read() system call
• syscall.close.return
when returning from the close() system call
• module("floppy").function("*")
when entering any function from the "floppy" module
• kernel.function("*@net/socket.c").return
when returning from any function in le net/socket.c
• kernel.statement("*@kernel/sched.c:2917")
when hitting line 2917 of le kernel/sched.c
8. SystemTap 编程语言
• mostly C-style syntax with a feeling of awk
• builtin associative arrays
• builtin aggregates of statistical data
very easy to collect data and do statistics on it (average, min,
max, count,. . . )
• many helper functions (builtin and in tapsets)
RTFM: SystemTap Language Reference shipped with SystemTap
(langref.pdf)
9. Performances and safety
• language-level safety features
o no pointers
o no unbounded loops
o type inference
o you can also write probe handlers in C (with -g) but don't complain if you
break stuff
• runtime safety features
o stap enforces maximum run time for each probe handler
o various concurrency constraints are enforced
o overload processing (don't allow stap to take up all the CPU time)
o many things can be overriden manually if you really want
o see SAFETY AND SECURITY section of stap(1)
The overhead depends a lot of what you are trying to do but in
general stap will try to stop you from doing something stupid (but
then you can still force it to do it).
10. Some helper functions you'll see a lot
pid() which process is this?
uid() which user is running this?
execname() what is the name of this process?
tid() which thread is this?
gettimeofday_s() epoch time in seconds
probefunc() what function are we in?
print_backtrace() figure out how we ended up here
There are many many more. RTFM (man stapfuncs) and explore
/usr/share/systemtap/tapset/.
11. Some cool stap options
-x trace only speci ed PID (only for userland probing)
-c run given command and only trace it and its children
(will still trace all threads for kernel probes)
-L list probe points matching given pattern along with
available variables
-d load given module debuginfo to help with symbol resolution in
backtraces
-g embed C code in stap script unsafe, dangerous and fun
31. Example 6: 修改程序的行为
global line
function alert_line(line:long) %{ /* pure */
strcpy((char*)THIS->line, "abcdefgn");
%}
probe process("a.out").function("reverse_line"){
line = user_string($line);
}
probe process("a.out").function("reverse_line").return{
if(isinstr(line, "tac")) $return = $haha;
else if (isinstr(line, "hello")) alert_line($return);
}
37. 结论
SystemTap is often described as "DTrace for Linux".
OProfile takes sample every $N CPU cycles so you can try to fi gureout
what each CPU is spending its time on.
SystemTap ,居家必备!!!