SlideShare a Scribd company logo
Performance Analysis Tools for
Linux Kernel
openSUSE Asia Summit 2017
Gary Lin
Software Engineer, SUSE Labs
glin@suse.com
perf
3
perf
• A collection of performance analysis tools
• Sampling and profiling the system
• Providing some tracing mechanisms
• Included in the linux kernel (tools/perf)
4
Before using perf...
• Narrow down the scope with other tools
– dmesg/syslog
– top
– iostat/vmstat
– iotop
– pidstat
– strace
– latencytop
5
perf events
6
Events
# perf list
List of pre-defined events (to be used in -e):
branch-instructions OR branches [Hardware event]
branch-misses [Hardware event]
bus-cycles [Hardware event]
...
alignment-faults [Software event]
bpf-output [Software event]
context-switches OR cs [Software event]
...
block:block_bio_backmerge [Tracepoint event]
block:block_bio_bounce [Tracepoint event]
block:block_bio_complete [Tracepoint event]
...
7
Hardware Events
8
“hardware performance counters …
are a set of special-purpose registers …
to store the counts of hardware-related
activities”
– Wikipedia
9
Hardware Events
• Hardware events are CPU-specific.
Check the CPU manual for more details.
• The common events:
– branch-instructions
– bus-cycles
– cache-misses
– cache-references
– cpu-cycles
– Instructions
10
Multiplexing and Scaling Events
• The hardware resources are limited.
• When there are more enabled events than the available
counters, the events are managed in round-robin
fashion.
• The event count may be scaled.
final_count = raw_count * time_enabled / time_running
11
Software Events
12
Software Events
• Special “software” counters provided by the kernel,
even if the hardware does not support performance
counters.
• The software events:
– cpu-clock
– task-clock
– context-switches
– page-faults, major-faults, and minor-faults
– alignment-faults
• The available events are defined in “perf_sw_ids” in
include/uapi/linux/perf_event.h.
13
Page Fault
• Major Fault
The page is not loaded at the time the fault is generated. The
additional disk latency is expected.
• Minor Fault
The page is already in the physical memory but not marked in MMU at
the time the fault is generated. No disk latency occurs.
14
Tracepoint Events
15
Tracepoint Events
• In-kernel static trace events
• Tracepoint events depend on the kernel and loaded
modules.
• The tracepoint event list:
# perf list tracepoint
# less /sys/kernel/debug/tracing/avaliable_events
16
Tracepoint Events – Example
static long do_wait(struct wait_opts *wo)
{
struct task_struct *tsk;
int retval;
trace_sched_process_wait(wo->wo_pid);
init_waitqueue_func_entry(&wo->child_wait,
child_wait_callback);
wo->child_wait.private = current;
add_wait_queue(&current->signal->wait_chldexit,
&wo->child_wait);
repeat:
...
17
Dynamic Tracing
18
perf probe
• Define new dynamic tracepoints events
• kprobe
The kernel symbols are listed in /proc/kallsyms.
• uprobe
The symbols can be read with “readelf”.
• Define a new kprobe event
# perf probe <kprobe event>
• Delete a kprobe event
# perf probe --del=<kprobe event>
• Use the kprobe event
# perf trace --event probe:<kprobe event>
# perf record -e probe:<kprobe event>
19
perf probe (cont’d)
• The following commands need kernel debuginfo and
debugsource.
• Show the arguments of the probed function
# perf probe -V <func>
• Show the source code of the probed function
# perf probe -L <func>
• Probe the line 12 of a function
# perf probe <func>:12
• Probe a member of a struct in the arguments (ifindex in
struct net of icmp_out_count())
# perf probe ‘icmp_out_count net->ifindex’
• Do a verbose dry run
# perf probe -nv <probe>
NOTE: You may encounter build-id mismatching. See bsc#964063.
20
System Profiling and Tracing
21
Real-time System Profiling
22
perf top
23
24
perf top – Tip
Press ‘z’ to zero all counts for a few seconds to
avoid recording activity from “perf top” itself.
25
Offline System Profiling
26
perf record
27
perf record
Sample the events and store them in perf.data
28
perf record – Examples
Sample on-CPU functions for the specified PID, until Ctrl-C:
# perf record -p PID
Sample CPU stack traces for the PID, using dwarf (dbg info) to
unwind stacks, for 10 seconds
# perf record -p PID --call-graph dwarf sleep 10
Sample CPU stack traces for the entire system, for 10 seconds:
# perf record -ag -- sleep 10
Trace all block completions, of size at least 100 Kbytes
# perf record -e block:block_rq_complete 
--filter 'nr_sector > 200'
29
perf report
30
31
perf script
32
Process PID CPU Timestamp CPU Cycles
audioPipe:src 31884 [000] 620396.493554: 141988 cycles:
3ed40 [unknown] (/usr/lib64/gstreamer-1.0/libgstcoreeleme
8f [unknown] ([unknown])
55b5f048f100 [unknown] ([unknown])
0 [unknown] ([unknown])
offlineimap 32210 [000] 620396.507109: 156215 cycles:
2529 [unknown] (/usr/lib64/python2.7/lib-dynload/time.so
c87200016b0004 [unknown] ([unknown])
alsa-sink-ALC32 3300 [002] 620396.512810: 419889 cycles:
39a17 [unknown] (/usr/lib64/pulseaudio/libpulsecommon-10.
55ba449bb190 [unknown] ([unknown])
threaded-ml 31876 [000] 620396.512898: 156215 cycles:
7fff8173d763 __sched_text_start ([kernel.kallsyms])
7fff8173e04d schedule ([kernel.kallsyms])
7fff81741f9b schedule_hrtimeout_range_clock ([kernel.kallsyms])
7fff8124bae1 poll_schedule_timeout ([kernel.kallsyms])
7fff8124cf2a do_sys_poll ([kernel.kallsyms])
e418d [unknown] (/lib64/libc-2.25.so)
6 [unknown] ([unknown])
backtrace
33
Performance Counter Statistics
34
perf stat
35
“perf stat” is much lighter weight than “perf record”.
36
perf stat – Output
# perf stat -a sleep 10
Performance counter stats for 'system wide':
20032.666323 task-clock (msec) # 2.000 CPUs utilized (100.00%)
41074 context-switches # 0.002 M/sec (100.00%)
2152 cpu-migrations # 0.107 K/sec (100.00%)
6252 page-faults # 0.312 K/sec
6838201211 cycles # 0.341 GHz (50.01%)
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
4157856056 instructions # 0.61 insns per cycle (74.99%)
871168471 branches # 43.487 M/sec (75.03%)
38497517 branch-misses # 4.42% of all branches (74.98%)
10.016014696 seconds time elapsed
count
scale
37
perf stat – Examples
CPU counter statistics for the entire system, for 10 seconds:
# perf stat -a sleep 10
Various CPU level 1 data cache statistics for the specified command:
# perf stat -e L1-dcache-loads,L1-dcache-load-misses,
L1-dcache-stores command
Count scheduler events for the specified PID, until Ctrl-C:
# perf stat -e 'sched:*' -p PID
Measure “make” 10 times:
# perf stat -r 10 --sync --pre make clean -- make
38
System Event Tracing
39
perf trace
40
Timestamp Duration Process PID Syscall
16412.718 ( 0.002 ms): InputThread/2933 epoll_wait(epfd: 35<anon_inode:[event
16412.723 ( 0.003 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf:
16412.727 ( 0.001 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf:
16412.733 ( 0.001 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf:
16412.735 ( 0.001 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf:
16412.745 ( 0.004 ms): InputThread/2933 write(fd: 31<pipe:[21362]>, buf: 0x7f
16394.158 (18.593 ms): X/2874 ... [continued]: epoll_wait()) = 1
16412.750 ( 0.004 ms): InputThread/2933 epoll_wait(epfd: 34<anon_inode:[event
16412.753 ( 0.002 ms): X/2874 read(fd: 30<pipe:[21362]>, buf: 0x7fffae2c9890,
16412.765 ( 0.005 ms): X/2874 writev(fd: 53, vec: 0x7fffae2c87d0, vlen: 1
16394.152 (18.621 ms): QXcbEventReade/3136 ... [continued]: poll()) = 1
16412.775 ( 0.003 ms): QXcbEventReade/3136 recvmsg(fd: 3<socket:[31188]>, msg
16412.783 ( 0.003 ms): QXcbEventReade/3136 write(fd: 5<anon_inode:[eventfd]>,
16396.068 (16.723 ms): kwin_x11/3131 ... [continued]: ppoll()) = 1
16412.787 ( 0.006 ms): QXcbEventReade/3136 poll(ufds: 0x7f90bc8e6bc8, nfds: 1
16412.793 ( 0.001 ms): kwin_x11/3131 read(fd: 5<anon_inode:[eventfd]>, buf: 0
16412.814 ( 0.004 ms): konsole/13778 poll(ufds: 0x55db78c35440, nfds: 16, tim
16412.820 ( 0.001 ms): konsole/13778 ioctl(fd: 28</dev/ptmx>, cmd: FIONREAD,
16412.822 ( 0.003 ms): konsole/13778 read(fd: 28</dev/ptmx>, buf: 0x55db78afd
41
perf trace – Examples
Only display the events had duration longer than 0.2 for the specified
command (also allocate 64MB for perf buffer):
# perf trace -m 64M --duration 0.2 command
Trace all syscalls except write
# perf trace -m 64M -e !write
Trace only block_rq_issue and block_rq_complete
# perf trace --no-syscalls 
--event block:block_rq_issue,block:block_rq_complete
42
Choose Your perf command
• Real-time monitoring
perf top
• Offline profiling
perf record, perf report, perf script
• Event counting
perf stat
• System event tracing
perf trace
43
And more...
• perf sched – scheduler properties
• perf kmem – kernel memory properties
• perf mem – profiling memory accesses
• perf lock – analyzing lock events
• perf kvm – tracing/measuring kvm guest os
• perf timechart – visualizing the system behavior
• perf bench – benchmark suites
...
44
Choose Your Tool
• For CPU-bound issues: perf
• For IO-bound issues: perf trace, or other tools
45
For more perf examples, search
“Brendan Gregg perf examples”
46
Question?
48
References
• Linux perf Examples
http://www.brendangregg.com/perf.html
• Linux kernel profiling with perf
https://perf.wiki.kernel.org/index.php/Tutorial
49
Backup Slides
50
Raw PMU
●
Show the available PMUs
# showevtinfo |less
IDX : 23068702
PMU name : core (Intel Core)
Name : L2_LINES_IN
Desc : L2 cache misses
●
Look up the raw number from the PMU name
# evt2raw L2_LINES_IN
r537024
●
Use the event
# perf stat -e r537024 -a sleep 10
51
Event Modifiers
u - user-space counting
k - kernel counting
h - hypervisor counting
I - non idle counting
G - guest counting (in KVM guests)
H - host counting (not in KVM guests)
p - precise level
P - use maximum detected precise level
S - read sample value (PERF_SAMPLE_READ)
D - pin the event to the PMU
52
Event Modifiers – Examples
• Request 0 skid on cpu-cycles
# perf stat -e cpu-cycles:pp -a sleep 10
• Only count user cache misses
# perf record -e cache-misses:u -a sleep 10
53
perf top – Options
-C, --cpu
Focuses on specific CPUs
-p, --pid, or -t, --tid
Focuses on specific process or thread IDs
-d, --delay
Number of seconds to delay between refreshes
-n
Show the number of samples
--dsos, --comms
Limits the information to particular DSO’s and commands
-s, --sort
Sorts the samples
-g
enable call-graph (stack chain/backtrace) recording
54
perf record – Options
-d, --delay N
Wait N msecs before the profiling starts
-F N
Sample every N times a second
-c N
Sample every N events
-s, --stat
Record per-thread event counts
-i, --no-inhert
Child tasks do not inherit counters
-a, --all-cpus
System-wide collection from all CPUs
55
perf record – Options (cont’d)
-g
Enable call-graph recording
--call-graph <type>
Setup and enable call-graph recording.
“fp” (frame pointer) is the default for “-g”. Try “dwarf” if
“--fomit-frame-pointer” is used
-e, --event=<event>
Select the event
--filter=<filter>
Use the event filter
-m
Specify the buffer size
56
perf stat – Options
--pre, --post
Pre and post measurement hooks
--sync
Call sync every iteration
-d, --detailed
Print more detailed statistics, can be specified up to 3 times
-r, --repeat=<n>
Repeat command and print average + stddev
-I msecs, --interval-print msecs
Print count deltas every N msecs
-e, --event=<event>
Select the event
-a, --all-cpus
System-wide collection from all CPUs
57
perf trace – Options
-p, --pid, -t, --tid, or, -u, --uid
Focuses on specific process, thread, or user IDs
-e, --expr
List of syscalls to show
--event=<event>
Trace other events
--no-syscalls
Don’t trace syscalls
-m
Specify the buffer size
--duration
Show only events that had a duration greater than N.M msecs
58
perf trace – Options (cont’d)
-s, --summary
Show only a summary of syscalls
-S, --with-summary
Show all syscalls followed by a summary
-o, --output=
Output file name
-F=[all|min|maj]
Trace pagefaults
59
perf trace vs strace
• perf trace is inspired by strace.
• perf trace uses tracepoints instead of ptrace.
perf trace is faster in general.
• perf trace outputs to a buffer first while strace dumps
the result directly.
• perf trace may lose events if the ring buffer fills while
strace never loses events.
It can be moderated by increasing the buffer size.

More Related Content

What's hot

Profiling your Applications using the Linux Perf Tools
Profiling your Applications using the Linux Perf ToolsProfiling your Applications using the Linux Perf Tools
Profiling your Applications using the Linux Perf Tools
emBO_Conference
 
Qemu Pcie
Qemu PcieQemu Pcie
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
Brendan Gregg
 
Linux Initialization Process (2)
Linux Initialization Process (2)Linux Initialization Process (2)
Linux Initialization Process (2)
shimosawa
 
Page cache in Linux kernel
Page cache in Linux kernelPage cache in Linux kernel
Page cache in Linux kernel
Adrian Huang
 
EBPF and Linux Networking
EBPF and Linux NetworkingEBPF and Linux Networking
EBPF and Linux Networking
PLUMgrid
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting Started
Brendan Gregg
 
malloc & vmalloc in Linux
malloc & vmalloc in Linuxmalloc & vmalloc in Linux
malloc & vmalloc in Linux
Adrian Huang
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
Adrien Mahieux
 
Process Address Space: The way to create virtual address (page table) of user...
Process Address Space: The way to create virtual address (page table) of user...Process Address Space: The way to create virtual address (page table) of user...
Process Address Space: The way to create virtual address (page table) of user...
Adrian Huang
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and Monitoring
Georg Schönberger
 
Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!
Ray Jenkins
 
Fun with Network Interfaces
Fun with Network InterfacesFun with Network Interfaces
Fun with Network Interfaces
Kernel TLV
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
SUSE Labs Taipei
 
Boosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringBoosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uring
ShapeBlue
 
Reverse Mapping (rmap) in Linux Kernel
Reverse Mapping (rmap) in Linux KernelReverse Mapping (rmap) in Linux Kernel
Reverse Mapping (rmap) in Linux Kernel
Adrian Huang
 
Kdump and the kernel crash dump analysis
Kdump and the kernel crash dump analysisKdump and the kernel crash dump analysis
Kdump and the kernel crash dump analysis
Buland Singh
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
Brendan Gregg
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Adrian Huang
 

What's hot (20)

Profiling your Applications using the Linux Perf Tools
Profiling your Applications using the Linux Perf ToolsProfiling your Applications using the Linux Perf Tools
Profiling your Applications using the Linux Perf Tools
 
Qemu Pcie
Qemu PcieQemu Pcie
Qemu Pcie
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
 
Linux Initialization Process (2)
Linux Initialization Process (2)Linux Initialization Process (2)
Linux Initialization Process (2)
 
Page cache in Linux kernel
Page cache in Linux kernelPage cache in Linux kernel
Page cache in Linux kernel
 
EBPF and Linux Networking
EBPF and Linux NetworkingEBPF and Linux Networking
EBPF and Linux Networking
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting Started
 
malloc & vmalloc in Linux
malloc & vmalloc in Linuxmalloc & vmalloc in Linux
malloc & vmalloc in Linux
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
 
Process Address Space: The way to create virtual address (page table) of user...
Process Address Space: The way to create virtual address (page table) of user...Process Address Space: The way to create virtual address (page table) of user...
Process Address Space: The way to create virtual address (page table) of user...
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and Monitoring
 
Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!
 
Fun with Network Interfaces
Fun with Network InterfacesFun with Network Interfaces
Fun with Network Interfaces
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
 
Boosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringBoosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uring
 
Reverse Mapping (rmap) in Linux Kernel
Reverse Mapping (rmap) in Linux KernelReverse Mapping (rmap) in Linux Kernel
Reverse Mapping (rmap) in Linux Kernel
 
Kdump and the kernel crash dump analysis
Kdump and the kernel crash dump analysisKdump and the kernel crash dump analysis
Kdump and the kernel crash dump analysis
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
 

Similar to Performance Analysis Tools for Linux Kernel

Kernel Recipes 2017 - Using Linux perf at Netflix - Brendan Gregg
Kernel Recipes 2017 - Using Linux perf at Netflix - Brendan GreggKernel Recipes 2017 - Using Linux perf at Netflix - Brendan Gregg
Kernel Recipes 2017 - Using Linux perf at Netflix - Brendan Gregg
Anne Nicolas
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
Brendan Gregg
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
Brendan Gregg
 
HKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightHKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with Coresight
Linaro
 
Introduction to Perf
Introduction to PerfIntroduction to Perf
Introduction to Perf
Wang Hsiangkai
 
The New Systems Performance
The New Systems PerformanceThe New Systems Performance
The New Systems Performance
Brendan Gregg
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
Brendan Gregg
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
Brendan Gregg
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
Sasha Goldshtein
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
Brendan Gregg
 
Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)
Amin Astaneh
 
Android Boot Time Optimization
Android Boot Time OptimizationAndroid Boot Time Optimization
Android Boot Time Optimization
Kan-Ru Chen
 
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysisKernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
Anne Nicolas
 
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
Jarod Wang
 
Debug generic process
Debug generic processDebug generic process
Debug generic process
Vipin Varghese
 
bcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challengesbcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challenges
IO Visor Project
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
Brendan Gregg
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
Brendan Gregg
 
Linux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudLinux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloud
Andrea Righi
 
PerfUG 3 - perfs système
PerfUG 3 - perfs systèmePerfUG 3 - perfs système
PerfUG 3 - perfs système
Ludovic Piot
 

Similar to Performance Analysis Tools for Linux Kernel (20)

Kernel Recipes 2017 - Using Linux perf at Netflix - Brendan Gregg
Kernel Recipes 2017 - Using Linux perf at Netflix - Brendan GreggKernel Recipes 2017 - Using Linux perf at Netflix - Brendan Gregg
Kernel Recipes 2017 - Using Linux perf at Netflix - Brendan Gregg
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
HKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with CoresightHKG18-TR14 - Postmortem Debugging with Coresight
HKG18-TR14 - Postmortem Debugging with Coresight
 
Introduction to Perf
Introduction to PerfIntroduction to Perf
Introduction to Perf
 
The New Systems Performance
The New Systems PerformanceThe New Systems Performance
The New Systems Performance
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
 
Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)
 
Android Boot Time Optimization
Android Boot Time OptimizationAndroid Boot Time Optimization
Android Boot Time Optimization
 
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysisKernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
 
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
 
Debug generic process
Debug generic processDebug generic process
Debug generic process
 
bcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challengesbcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challenges
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
 
Linux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudLinux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloud
 
PerfUG 3 - perfs système
PerfUG 3 - perfs systèmePerfUG 3 - perfs système
PerfUG 3 - perfs système
 

Recently uploaded

Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Sven Peters
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
What is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdfWhat is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdf
kalichargn70th171
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
ppt on the brain chip neuralink.pptx
ppt  on   the brain  chip neuralink.pptxppt  on   the brain  chip neuralink.pptx
ppt on the brain chip neuralink.pptx
Reetu63
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
Maitrey Patel
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
safelyiotech
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
Benefits of Artificial Intelligence in Healthcare!
Benefits of  Artificial Intelligence in Healthcare!Benefits of  Artificial Intelligence in Healthcare!
Benefits of Artificial Intelligence in Healthcare!
Prestware
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
Yara Milbes
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Jhone kinadey
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 

Recently uploaded (20)

Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
What is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdfWhat is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdf
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
ppt on the brain chip neuralink.pptx
ppt  on   the brain  chip neuralink.pptxppt  on   the brain  chip neuralink.pptx
ppt on the brain chip neuralink.pptx
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
Benefits of Artificial Intelligence in Healthcare!
Benefits of  Artificial Intelligence in Healthcare!Benefits of  Artificial Intelligence in Healthcare!
Benefits of Artificial Intelligence in Healthcare!
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 

Performance Analysis Tools for Linux Kernel

  • 1. Performance Analysis Tools for Linux Kernel openSUSE Asia Summit 2017 Gary Lin Software Engineer, SUSE Labs glin@suse.com
  • 3. 3 perf • A collection of performance analysis tools • Sampling and profiling the system • Providing some tracing mechanisms • Included in the linux kernel (tools/perf)
  • 4. 4 Before using perf... • Narrow down the scope with other tools – dmesg/syslog – top – iostat/vmstat – iotop – pidstat – strace – latencytop
  • 6. 6 Events # perf list List of pre-defined events (to be used in -e): branch-instructions OR branches [Hardware event] branch-misses [Hardware event] bus-cycles [Hardware event] ... alignment-faults [Software event] bpf-output [Software event] context-switches OR cs [Software event] ... block:block_bio_backmerge [Tracepoint event] block:block_bio_bounce [Tracepoint event] block:block_bio_complete [Tracepoint event] ...
  • 8. 8 “hardware performance counters … are a set of special-purpose registers … to store the counts of hardware-related activities” – Wikipedia
  • 9. 9 Hardware Events • Hardware events are CPU-specific. Check the CPU manual for more details. • The common events: – branch-instructions – bus-cycles – cache-misses – cache-references – cpu-cycles – Instructions
  • 10. 10 Multiplexing and Scaling Events • The hardware resources are limited. • When there are more enabled events than the available counters, the events are managed in round-robin fashion. • The event count may be scaled. final_count = raw_count * time_enabled / time_running
  • 12. 12 Software Events • Special “software” counters provided by the kernel, even if the hardware does not support performance counters. • The software events: – cpu-clock – task-clock – context-switches – page-faults, major-faults, and minor-faults – alignment-faults • The available events are defined in “perf_sw_ids” in include/uapi/linux/perf_event.h.
  • 13. 13 Page Fault • Major Fault The page is not loaded at the time the fault is generated. The additional disk latency is expected. • Minor Fault The page is already in the physical memory but not marked in MMU at the time the fault is generated. No disk latency occurs.
  • 15. 15 Tracepoint Events • In-kernel static trace events • Tracepoint events depend on the kernel and loaded modules. • The tracepoint event list: # perf list tracepoint # less /sys/kernel/debug/tracing/avaliable_events
  • 16. 16 Tracepoint Events – Example static long do_wait(struct wait_opts *wo) { struct task_struct *tsk; int retval; trace_sched_process_wait(wo->wo_pid); init_waitqueue_func_entry(&wo->child_wait, child_wait_callback); wo->child_wait.private = current; add_wait_queue(&current->signal->wait_chldexit, &wo->child_wait); repeat: ...
  • 18. 18 perf probe • Define new dynamic tracepoints events • kprobe The kernel symbols are listed in /proc/kallsyms. • uprobe The symbols can be read with “readelf”. • Define a new kprobe event # perf probe <kprobe event> • Delete a kprobe event # perf probe --del=<kprobe event> • Use the kprobe event # perf trace --event probe:<kprobe event> # perf record -e probe:<kprobe event>
  • 19. 19 perf probe (cont’d) • The following commands need kernel debuginfo and debugsource. • Show the arguments of the probed function # perf probe -V <func> • Show the source code of the probed function # perf probe -L <func> • Probe the line 12 of a function # perf probe <func>:12 • Probe a member of a struct in the arguments (ifindex in struct net of icmp_out_count()) # perf probe ‘icmp_out_count net->ifindex’ • Do a verbose dry run # perf probe -nv <probe> NOTE: You may encounter build-id mismatching. See bsc#964063.
  • 23. 23
  • 24. 24 perf top – Tip Press ‘z’ to zero all counts for a few seconds to avoid recording activity from “perf top” itself.
  • 27. 27 perf record Sample the events and store them in perf.data
  • 28. 28 perf record – Examples Sample on-CPU functions for the specified PID, until Ctrl-C: # perf record -p PID Sample CPU stack traces for the PID, using dwarf (dbg info) to unwind stacks, for 10 seconds # perf record -p PID --call-graph dwarf sleep 10 Sample CPU stack traces for the entire system, for 10 seconds: # perf record -ag -- sleep 10 Trace all block completions, of size at least 100 Kbytes # perf record -e block:block_rq_complete --filter 'nr_sector > 200'
  • 30. 30
  • 32. 32 Process PID CPU Timestamp CPU Cycles audioPipe:src 31884 [000] 620396.493554: 141988 cycles: 3ed40 [unknown] (/usr/lib64/gstreamer-1.0/libgstcoreeleme 8f [unknown] ([unknown]) 55b5f048f100 [unknown] ([unknown]) 0 [unknown] ([unknown]) offlineimap 32210 [000] 620396.507109: 156215 cycles: 2529 [unknown] (/usr/lib64/python2.7/lib-dynload/time.so c87200016b0004 [unknown] ([unknown]) alsa-sink-ALC32 3300 [002] 620396.512810: 419889 cycles: 39a17 [unknown] (/usr/lib64/pulseaudio/libpulsecommon-10. 55ba449bb190 [unknown] ([unknown]) threaded-ml 31876 [000] 620396.512898: 156215 cycles: 7fff8173d763 __sched_text_start ([kernel.kallsyms]) 7fff8173e04d schedule ([kernel.kallsyms]) 7fff81741f9b schedule_hrtimeout_range_clock ([kernel.kallsyms]) 7fff8124bae1 poll_schedule_timeout ([kernel.kallsyms]) 7fff8124cf2a do_sys_poll ([kernel.kallsyms]) e418d [unknown] (/lib64/libc-2.25.so) 6 [unknown] ([unknown]) backtrace
  • 35. 35 “perf stat” is much lighter weight than “perf record”.
  • 36. 36 perf stat – Output # perf stat -a sleep 10 Performance counter stats for 'system wide': 20032.666323 task-clock (msec) # 2.000 CPUs utilized (100.00%) 41074 context-switches # 0.002 M/sec (100.00%) 2152 cpu-migrations # 0.107 K/sec (100.00%) 6252 page-faults # 0.312 K/sec 6838201211 cycles # 0.341 GHz (50.01%) <not supported> stalled-cycles-frontend <not supported> stalled-cycles-backend 4157856056 instructions # 0.61 insns per cycle (74.99%) 871168471 branches # 43.487 M/sec (75.03%) 38497517 branch-misses # 4.42% of all branches (74.98%) 10.016014696 seconds time elapsed count scale
  • 37. 37 perf stat – Examples CPU counter statistics for the entire system, for 10 seconds: # perf stat -a sleep 10 Various CPU level 1 data cache statistics for the specified command: # perf stat -e L1-dcache-loads,L1-dcache-load-misses, L1-dcache-stores command Count scheduler events for the specified PID, until Ctrl-C: # perf stat -e 'sched:*' -p PID Measure “make” 10 times: # perf stat -r 10 --sync --pre make clean -- make
  • 40. 40 Timestamp Duration Process PID Syscall 16412.718 ( 0.002 ms): InputThread/2933 epoll_wait(epfd: 35<anon_inode:[event 16412.723 ( 0.003 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf: 16412.727 ( 0.001 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf: 16412.733 ( 0.001 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf: 16412.735 ( 0.001 ms): InputThread/2933 read(fd: 40</dev/input/event1>, buf: 16412.745 ( 0.004 ms): InputThread/2933 write(fd: 31<pipe:[21362]>, buf: 0x7f 16394.158 (18.593 ms): X/2874 ... [continued]: epoll_wait()) = 1 16412.750 ( 0.004 ms): InputThread/2933 epoll_wait(epfd: 34<anon_inode:[event 16412.753 ( 0.002 ms): X/2874 read(fd: 30<pipe:[21362]>, buf: 0x7fffae2c9890, 16412.765 ( 0.005 ms): X/2874 writev(fd: 53, vec: 0x7fffae2c87d0, vlen: 1 16394.152 (18.621 ms): QXcbEventReade/3136 ... [continued]: poll()) = 1 16412.775 ( 0.003 ms): QXcbEventReade/3136 recvmsg(fd: 3<socket:[31188]>, msg 16412.783 ( 0.003 ms): QXcbEventReade/3136 write(fd: 5<anon_inode:[eventfd]>, 16396.068 (16.723 ms): kwin_x11/3131 ... [continued]: ppoll()) = 1 16412.787 ( 0.006 ms): QXcbEventReade/3136 poll(ufds: 0x7f90bc8e6bc8, nfds: 1 16412.793 ( 0.001 ms): kwin_x11/3131 read(fd: 5<anon_inode:[eventfd]>, buf: 0 16412.814 ( 0.004 ms): konsole/13778 poll(ufds: 0x55db78c35440, nfds: 16, tim 16412.820 ( 0.001 ms): konsole/13778 ioctl(fd: 28</dev/ptmx>, cmd: FIONREAD, 16412.822 ( 0.003 ms): konsole/13778 read(fd: 28</dev/ptmx>, buf: 0x55db78afd
  • 41. 41 perf trace – Examples Only display the events had duration longer than 0.2 for the specified command (also allocate 64MB for perf buffer): # perf trace -m 64M --duration 0.2 command Trace all syscalls except write # perf trace -m 64M -e !write Trace only block_rq_issue and block_rq_complete # perf trace --no-syscalls --event block:block_rq_issue,block:block_rq_complete
  • 42. 42 Choose Your perf command • Real-time monitoring perf top • Offline profiling perf record, perf report, perf script • Event counting perf stat • System event tracing perf trace
  • 43. 43 And more... • perf sched – scheduler properties • perf kmem – kernel memory properties • perf mem – profiling memory accesses • perf lock – analyzing lock events • perf kvm – tracing/measuring kvm guest os • perf timechart – visualizing the system behavior • perf bench – benchmark suites ...
  • 44. 44 Choose Your Tool • For CPU-bound issues: perf • For IO-bound issues: perf trace, or other tools
  • 45. 45 For more perf examples, search “Brendan Gregg perf examples”
  • 47.
  • 48. 48 References • Linux perf Examples http://www.brendangregg.com/perf.html • Linux kernel profiling with perf https://perf.wiki.kernel.org/index.php/Tutorial
  • 50. 50 Raw PMU ● Show the available PMUs # showevtinfo |less IDX : 23068702 PMU name : core (Intel Core) Name : L2_LINES_IN Desc : L2 cache misses ● Look up the raw number from the PMU name # evt2raw L2_LINES_IN r537024 ● Use the event # perf stat -e r537024 -a sleep 10
  • 51. 51 Event Modifiers u - user-space counting k - kernel counting h - hypervisor counting I - non idle counting G - guest counting (in KVM guests) H - host counting (not in KVM guests) p - precise level P - use maximum detected precise level S - read sample value (PERF_SAMPLE_READ) D - pin the event to the PMU
  • 52. 52 Event Modifiers – Examples • Request 0 skid on cpu-cycles # perf stat -e cpu-cycles:pp -a sleep 10 • Only count user cache misses # perf record -e cache-misses:u -a sleep 10
  • 53. 53 perf top – Options -C, --cpu Focuses on specific CPUs -p, --pid, or -t, --tid Focuses on specific process or thread IDs -d, --delay Number of seconds to delay between refreshes -n Show the number of samples --dsos, --comms Limits the information to particular DSO’s and commands -s, --sort Sorts the samples -g enable call-graph (stack chain/backtrace) recording
  • 54. 54 perf record – Options -d, --delay N Wait N msecs before the profiling starts -F N Sample every N times a second -c N Sample every N events -s, --stat Record per-thread event counts -i, --no-inhert Child tasks do not inherit counters -a, --all-cpus System-wide collection from all CPUs
  • 55. 55 perf record – Options (cont’d) -g Enable call-graph recording --call-graph <type> Setup and enable call-graph recording. “fp” (frame pointer) is the default for “-g”. Try “dwarf” if “--fomit-frame-pointer” is used -e, --event=<event> Select the event --filter=<filter> Use the event filter -m Specify the buffer size
  • 56. 56 perf stat – Options --pre, --post Pre and post measurement hooks --sync Call sync every iteration -d, --detailed Print more detailed statistics, can be specified up to 3 times -r, --repeat=<n> Repeat command and print average + stddev -I msecs, --interval-print msecs Print count deltas every N msecs -e, --event=<event> Select the event -a, --all-cpus System-wide collection from all CPUs
  • 57. 57 perf trace – Options -p, --pid, -t, --tid, or, -u, --uid Focuses on specific process, thread, or user IDs -e, --expr List of syscalls to show --event=<event> Trace other events --no-syscalls Don’t trace syscalls -m Specify the buffer size --duration Show only events that had a duration greater than N.M msecs
  • 58. 58 perf trace – Options (cont’d) -s, --summary Show only a summary of syscalls -S, --with-summary Show all syscalls followed by a summary -o, --output= Output file name -F=[all|min|maj] Trace pagefaults
  • 59. 59 perf trace vs strace • perf trace is inspired by strace. • perf trace uses tracepoints instead of ptrace. perf trace is faster in general. • perf trace outputs to a buffer first while strace dumps the result directly. • perf trace may lose events if the ring buffer fills while strace never loses events. It can be moderated by increasing the buffer size.