Always-on Profiling of All Linux Threads, On-CPU and Off-CPU, with ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

HOSTED BY
Always-on Profiling of Linux Threads,
On-CPU and Off-CPU, with eBPF and
Context Enrichment
Tanel Põder
Consultan...

Tanel Põder
A long time computer performance geek & consultant
■ Built low-tech tools for OS process/thread & DB
connectio...

Concepts & Motivation

Systematic performance troubleshooting
For systematic, deterministic troubleshooting drilldown, you need:
■ Avoid guesswor...

System level metrics vs. thread state sampling
Let's
sample
thread
states!

How to sample what threads are doing? (Linux)
Options:
■ Attach with ptrace() / pstack / gdb – not practical in production...

Sampling thread states via /proc

/proc/1984

What can /proc sampling tools give you?
■ 0x.tools is a suite of low-tech Linux performance troubleshooting tools
● https:...

psn – default output mode (it's like top that shows wait/sleep activity too)

Sample all threads of "sync|kworker" procs, group time spent by syscall, whcan

Measure threads of a single process, group activity also by syscall, filename

Can I have always-on /proc sampling?
■ psn is meant for interactive troubleshooting of currently ongoing problems
● psn sa...

Sample all threads from /proc every second, print out threads in R & D state

Sample all threads including the ones Sleeping state, print more fields & kstack

Sampling thread states with eBPF

Can we get the same (and more) with eBPF?
Yes!!!
■ We will not be tracing every single event to output
● Unrealistic amoun...

Populating & sampling the thread state "array"

Populating & sampling the thread state "array"
Time
tid 10
tid 11
tid 42
10 11 42 N
...
BPF_HASH(syscall_id) 10
10
10
trac...

Populating & sampling the thread state "array"
Time
tid 10
tid 11
tid 42
10 11 42 N
...
BPF_HASH(syscall_id) 10 11
11
11
1...

Populating & sampling the thread state "array"
Time
tid 10
tid 11
tid 42
10 11 42 N
...
BPF_HASH(syscall_id) 10 42
42
42
4...

Populating & sampling the thread state "array"
Time
tid 10
tid 11
tid 42
10 11 42 N
...
BPF_HASH(syscall_id)
tracepoint:ra...

Populating & sampling the thread state "array"
Time
tid 10
tid 11
tid 42
10 11 42 N
...
BPF_HASH(syscall_id)
tracepoint:ra...

Demo

Demo!

Demo! (No need to read this )

TODO

TODO
■ This is a PoC prototype script, not a production ready tool or a product 
■ Rewrite using bcc or libbpf for flexib...

Links & resources
■ 0x.tools
● https://0x.tools 
■ Processes as files (1984)
● https://lucasvr.gobolinux.org/etc/Killian8...

Tanel Põder
tanel@tanelpoder.com
@tanelpoder
tanelpoder.com
Thank you! Let’s connect.

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1024759 views
Storytelling For The Web: Integrate... by Chiara Aliotta 934642 views
Artificial Intelligence, Data and C... by OECD Directorate ... 848885 views
How to Leverage AI to Boost Employe... by SocialHRCamp 355199 views
2024 State of Marketing Report – by... by Marius Sescu 202124 views
Everything You Need To Know About C... by Expeed Software 223622 views

View on Slideshare

1 of 30 Ad

View on Slideshare

1 of 30 Ad