Embedded Recipes 2018 - Finding sources of Latency In your system -...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

© 2016 VMware Inc. All rights
reserved.
© 2016 VMware Inc. All rights
reserved.
Finding sources of Latency
In your system ...

What is Latency?
2

What is Latency?
3
“Latency is a time interval between the stimulation and
response, or, from a more general point of view...

What is Latency?
●
The time between
– When something needs to be done
– When that something actually gets done
4
“Latency ...

What is Latency?
●
The time between
– When something needs to be done
– When that something actually gets done
●
The time ...

There is ALWAYS Latency
●
Nothing happens instantaneously
●
There’s always going to be some “lag”
●
Real Time systems avoi...

Why do we care about Latency?
●
For Real Time tasks, it is critical
– Need to know worse case latency
– Guarantee that it ...

Why do we care about Latency?
●
Practically everyone/everything cares about Latency
– How long would you wait for ‘a’ to s...

Where is that Latency?
●
Latency adds up
– A system’s latency is the combination of its sub-components
●
It can be hard fi...

Where is that Latency?
HARDWARE
Kernel
Library
Application
BIOS

Hardware Latency
●
Can’t get better than what the hardware gives you
●
Sources of HW latency
– System Management Interrupt...

Cyclictest
●
A tool to measure latency
●
Runs a simple loop (in a high priority task)
– Sleep for a specified time
– Get t...

Cyclictest
start = gettimeofday()
Sleep 250us
user-space
kernel
Put task to sleep
Set interrupt timer for 250us
Timer Inte...

Cyclictest
start = gettimeofday()
Sleep 250us
user-space
kernel
Put task to sleep
Set interrupt timer for 250us
Timer Inte...

Cyclictest
start = gettimeofday()
Sleep 250us
user-space
kernel
Put task to sleep
Set interrupt timer for 250us
Timer Inte...

Cyclictest
start = gettimeofday()
Sleep 250us
user-space
kernel
Put task to sleep
Set interrupt timer for 250us
Timer Inte...

Cyclictest
●
The options I am using here
– “-p80” : set the starting priority to 80
– “-i250” : set the starting interval ...

The tools
●
Measuring latency
– cyclictest
●
Using Tracing tools can help find what’s happening
– Ftrace
●
hwlat
●
functio...

Hardware Latency
●
Can’t get better than what the hardware gives you
●
Sources of HW latency
– System Management Interrupt...

Hardware Latency Detector
●
Part of Ftrace
– CONFIG_HWLAT_TRACER (in kernel .config build file)
●
cd /sys/kernel/tracing (...

Hardware Latency Detector
# cd /sys/kernel/tracing
# echo hwlat > current_tracer
# sleep 10
# cat trace
# tracer: hwlat
#
...

Hardware Latency Detector
# cd /sys/kernel/tracing
# echo hwlat > current_tracer
# sleep 10
# cat trace
# tracer: hwlat
#
...

Hardware Latency Detector
# cd /sys/kernel/tracing
# echo hwlat > current_tracer
# sleep 10
# cat trace
# tracer: hwlat
#
...

Hardware Latency Algorithm
●
Runs a kernel thread
●
Does a tight loop for a period of time
●
Interrupts are disabled (duri...

t1 = time_get()
t2 = time_get()
last_t2 == 0
last_t2 = 0
diff = t1 - last_t2
diff > outer
diff = t2 - t1
diff > inner
oute...

Hardware Latency Detector
# cd /sys/kernel/tracing
# cat hwlat_detector/width
500000
# cat hwlat_detector/window
1000000
#...

Hardware Latency Detector (with NMIs)
●
It checks if an NMI triggered
●
NMIs are not hardware, but software controlled (in...

Hardware Latency Detector (With NMIs)
# cd /sys/kernel/tracing
# echo hwlat > current_tracer
# sleep 10
# cat trace
# trac...

If that’s not the problem?
●
What if the hardware is fine?
●
What else could it be?
●
The Kernel?
●
Your application?

Kernel Entry Tracing (more than strace)
# echo 1 > /sys/kernel/tracing/options/function-fork
# trace-cmd record -F -c -p f...

Kernel Entry Tracing (more than strace)
# echo 1 > /sys/kernel/tracing/options/function-fork
# trace-cmd record -F -c -p f...

More Efficient Trace
# trace-cmd report -F 'funcgraph_entry'| cut -d'|' -f 2 | cut -d'(' -f1 | sort -u
cpus=4
do_IRQ
__do_...

More Efficient Trace (I need to make this easier)
# trace-cmd record -F -c -p function_graph --max-graph-depth 1 -e syscal...

More Accurate Trace
# trace-cmd report | less
cpus=4
cyclictest-10668 [003] 16938.997362: funcgraph_entry: | mutex_unlock(...

$More Accurate Trace cyclictest-10670 [001] 16939.002227: funcgraph_entry: | do_syscall_64() { cyclictest-10670 [001] 16939...$

Wakeup Latency
●
The time a task wakes up, to the time it is scheduled
●
“wakeup” tracer
– Traces the time of the highest ...

Wakeup Latency
# trace-cmd start -p wakeup_rt -d -e all
# ./cyclictest -p80 -i250 -n -a -t -q -d 0
# /dev/cpu_dma_latency ...

Wakeup Latency
# trace-cmd show | less
# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 4.19.0-rc4-test+
# ------...

Wakeup Latency
<...>-42 2dN.1 102us : irq_disable: caller=_raw_spin_lock_irq+0x15/0x40 parent= (null)
<...>-42 2dN.2 102us...

Wakeup Latency
# trace-cmd start -p wakeup_rt -l ‘*spin*’ -e all
# ./cyclictest -p80 -i250 -n -a -t -q -d 0
# /dev/cpu_dma...

Wakeup Latency
Xorg-747 2.N.2 274us : _raw_spin_lock_irqsave <-pagevec_lru_move_fn
Xorg-747 2dN.3 274us : irq_enable: call...

IRQ and Preemption Latency
●
When interrupts or preemption is disabled
– Other tasks can not be scheduled in
●
Tracers:
– ...

Preempt and IRQs Off Latency
# trace-cmd start -p preemptirqsoff -d
# trace-cmd show | less
# tracer: preemptirqsoff
#
# p...

Preempt and IRQs Off Latency
# echo 1 > /sys/kernel/debug/tracing/options/sym-offset
# trace-cmd show | less
# tracer: pre...

IRQ and Preemption Latency
●
When interrupts or preemption is disabled
– Other tasks can not be scheduled in
●
Tracers:
– ...

The “trace_marker” and “tracing_on” files
●
/sys/kernel/tracing/trace_marker
– Let’s userspace write into the kernel ring ...

Using trace_marker and tracing_on in C code
static int marker_fd = -1;
static int tracing_fd = -1;
static __thread char bu...

Having cyclictest stop on max threshold
●
cyclictest is aware of the tracing infrastructure
– “-b 500” : break on if laten...

Preempt and IRQs Off Latency
# ./cyclictest -p80 -i250 -n -a -t -q -d 0 -b 500
# /dev/cpu_dma_latency set to 0us
INFO: deb...

The issue has been record (we hope)
●
The trace buffer has the issue (with a marker tag)
●
Save the text files (just to ke...

For full power of trace-cmd
●
trace-cmd extract retrieves the kernel buffer into a trace.dat file
●
Can now do all sorts o...

For full power of trace-cmd
●
trace-cmd extract retrieves the kernel buffer into a trace.dat file
●
Can now do all sorts o...

# trace-cmd report -l --cpu 3 -O parent | less
cyclicte-15129 3.... 23346.433176: hrtimer_init: hrtimer=0xffffb45f4290be70...

Lock Stats (based off of Lockdep)
●
lockdep - checks correctness of locking
– Avoids various deadlock scenarios
●
lockstat...

The /proc/lockstat file
●
The headers
– con-bounces : How many times contended across CPUs
– contentions : Number of locks...

trylock(lock) record lock contetion(&lock)
record lock acquisition(&lock)
lock(&lock)
How lockstats work

lock_stat version 0.4
----------------------------------------------------------------------------------------------------...

Questions?