This document proposes a new IRQ tracer to measure interrupt latency in real-time Linux systems. The current methods are intrusive and not portable. The new tracer would combine features of existing tracers with less overhead as a standalone kernel module. It would measure all interrupts, not just simulated ones, and allow more customization and post-processing in user space.

1. Real-Time Linux and Interrupts
Related Latency Measurements
Propose A New IRQ Tracer
Steven S. Chen

2. Interrupt Latency
• Interrupt Latency - Duration between receiving an interrupt and processing the
interrupt
• Source: Preemption latency of real-time Linux systems
• The standard method to determine the preemption latency of a system is to
trigger preemption and to measure how long it takes until the stimulated
preemption is completed and the user-space program under consideration
continues execution after returning from the waiting system call. The tool that
was created for this purpose is called cyclictest

3. Existing method to calculate Interrupt Latency
Nokia driver - Adding a periodic timer interrupt
Record the maximum difference between (actual time of invocation -
expected time of invocation)
Record the maximum skid in cycles from the time the interrupt was
executed last to the time the ISR is executed this time
Problem with existing method
Intrusive - The act of measuring changes the application behavior
Not portable – depends on hardware timer

4. /sys/kernel/debug/latency_hist

5. New IRQ Tracer
• Measure isr entry/exit time can determine if there are any abnormal
events in ISR processing in realtime systems
• Most problems are in user added driver code not in the kernel source
• Existing methods
• ftrace
• Redhat irq tracer patch https://lwn.net/Articles/320336/
• Broadcom irq trace driver
https://android.googlesource.com/kernel/bcm/+/android-bcm-tetra-3.10-
kitkat-wear/drivers/char/broadcom/trace_log.c
• Hw_latency detector https://lwn.net/Articles/696460/
• https://github.com/XaF/hwlat-detector

6. New Irq Tracer (2)
• Combine the best of above feature
• Less overhead
• As standalone kernel module
• Can be used in real time in production code
• Measures ALL real interrupts not just simulated timer interrupts
• Allow more customization start/stop/duration/window etc
• And post processing in user space

7. References
• Frequently Asked Questions