Downloaded 141 times
![Preemptible Kernel
● Kernel preemption could happen when
– return from irq handler
__irq_svc:
svc_entry
irq_handler
#ifdef CONFIG_PREEMPT
get_thread_info tsk
ldr r8, [tsk, #TI_PREEMPT] @ get preempt count
teq r8, #0 @ if preempt count != 0
bne 1f @ return from exeption
ldr r0, [tsk, #TI_FLAGS] @ get flags
tst r0, #_TIF_NEED_RESCHED @ if NEED_RESCHED is set
blne svc_preempt @ preempt!
...
#endif](https://image.slidesharecdn.com/preempt-rt-150601103137-lva1-app6891/85/Linux-Preempt-RT-Internals-15-320.jpg)
![Trace Log Format
● tracecmd1061 [000] 142.334403: irq_handler_entry:
irq=21 name=critical_irq
Current
Task
CPU# Time
Stamp
Event
Name
Message
trace-cmd-1061 [000] 142.334403 irq_handler_entry Irq=21
name=critical_irq](https://image.slidesharecdn.com/preempt-rt-150601103137-lva1-app6891/85/Linux-Preempt-RT-Internals-35-320.jpg)
![Trace Log
tracecmd1061 [000] 142.334403: irq_handler_entry: irq=29
name=critical_irq
tracecmd1061 [000] 142.334437: sched_wakeup:
critical_task:1056 [9] success=1 CPU:000
tracecmd1061 [000] 142.334456: irq_handler_exit: irq=29
ret=handled
tracecmd1061 [000] 142.334480: sched_wakeup:
ksoftirqd/0:3 [98] success=1 CPU:000
tracecmd1061 [000] 142.334526: sched_switch: trace
cmd:1061 [120] R ==> critical_task:1056 [9]](https://image.slidesharecdn.com/preempt-rt-150601103137-lva1-app6891/85/Linux-Preempt-RT-Internals-37-320.jpg)
![● Trace Log
tracecmd1079 [000] 1074.488916: irq_handler_entry: irq=29
name=critical_irq
tracecmd1079 [000] 1074.488970: sched_wakeup: critical_task:1056
[9] success=1 CPU:000
tracecmd1079 [000] 1074.488992: irq_handler_exit: irq=29 ret=handled
tracecmd1079 [000] 1074.489018: sched_wakeup: ksoftirqd/0:3 [98]
success=1 CPU:000
tracecmd1079 [000] 1074.489046: irq_handler_entry: irq=59
name=ohci_hcd:usb1
tracecmd1079 [000] 1074.489056: irq_handler_exit: irq=59 ret=handled
tracecmd1079 [000] 1074.489075: sched_wakeup: irq/59ohci_hcd:979
[49] success=1 CPU:000
tracecmd1079 [000] 1074.489113: sched_stat_runtime: comm=tracecmd
pid=1079 runtime=694958 [ns] vruntime=30984068262 [ns]
tracecmd1079 [000] 1074.489139: sched_switch: tracecmd:1079 [120]
R ==> critical_task:1056 [9]
critical_task1056 [000] 1074.489233: irq_handler_entry: irq=21
name=clockevent
critical_task1056 [000] 1074.489361: irq_handler_exit: irq=21
ret=handled
critical_task1056 [000] 1074.489424: sched_switch:
critical_task:1056 [9] D ==> irq/59ohci_hcd:979 [49]
Interrupt Preemption](https://image.slidesharecdn.com/preempt-rt-150601103137-lva1-app6891/85/Linux-Preempt-RT-Internals-41-320.jpg)
Uploaded by哲豪 康哲豪
Linux Preempt-RT Internals
Linux PREEMPT_RT improves the preemptiveness of the Linux kernel by allowing preemption everywhere except when preemption is disabled or interrupts are disabled. This reduces latency from preemption, critical sections, and interrupts. However, non-deterministic external interrupt events and timing as well as interrupt collisions can still cause unpredictable latency. Tracing tools can help analyze latency but practical issues remain in fully guaranteeing hard real-time behavior.
More Related Content
Similar to Linux Preempt-RT Internals
Linux Preempt-RT Internals
- 1. Linux PREEMPT_RT Internals George Kang( 康哲豪 ) <georgekang03@gmail.com> May,30 2015/MOSUT
- 2. ● Deadline ● HardReal-time – Meet deadline Deterministically – Guaranteed worst case ● Soft Real-time – Best Effort Real-time System
- 3. Real-time System ● Assumption –critical task has highest priority – task workload is fixed ● Deadline miss – Undetermined latency – Unpredictable event Critical TaskLatency? Critical Event Deadline Events?
- 4.
- 5. Latency ● Time intervalfrom event trigger till handler task react this event Critical TaskLatency? Critical Event Deadline Events?
- 6. From Event toReceived Interrupt Taken Interrupt Received in User/Thread Context Critical section with interrupts disabled HW Excepti on “Top Half” / ISR Exit from IRQ Reschedule Context SwitchSignal/ Wakeup Resource Conflicts
- 7. Latency results from... ●Preemption ● Critical Section ● Interrupt
- 8. Preemption ● Re-schedule whenhigh priority task is ready ● Increase responsibility ● Decrease throughput
- 9. Preemption in Linux ●Preempt configurations – NONE, Voluntary, Basic RT, RT_FULL
- 10. Toward full preemption ●Most important aspects of Real-time – Controlling latency by allowing kernel to be preemptible everywhere
- 11. Non-Preemptive ● CONFIG_PREEMPT_NONE ● Preemptionis not allowed in Kernel Mode ● Preemption could happen upon returning to user space
- 12. Non-Preemptive Issue ● Latencyof Non-Preemptive configuration Task A (user mode) Interrupt handler Wakes up Task B Task B (user mode) System call Task A (kernel mode) Task A (kernel mode) Interrupt ? Return from syscall Unbound Latency
- 13. Preemption Point ● CONFIG_PREEMPT_VOLUNTARY ●Insert explicit preemption point in Kernel – might_sleep → __cond_resched ● Kernel can be preempted only at preemption point
- 14. Preemptible Kernel ● CONFIG_PREEMPT ●Implicit preemption in Kernel ● preempt_count – Member of thread_info – Preemption could happen when preempt_count == 0
- 15. Preemptible Kernel ● Kernelpreemption could happen when – return from irq handler __irq_svc: svc_entry irq_handler #ifdef CONFIG_PREEMPT get_thread_info tsk ldr r8, [tsk, #TI_PREEMPT] @ get preempt count teq r8, #0 @ if preempt count != 0 bne 1f @ return from exeption ldr r0, [tsk, #TI_FLAGS] @ get flags tst r0, #_TIF_NEED_RESCHED @ if NEED_RESCHED is set blne svc_preempt @ preempt! ... #endif
- 16. Full Preemptive ● Goal:Preempt Everywhere except – Preempt disable – Interrupt disable ● Reduce non-preemptible cases in kernel – spin_lock – Interrupt
- 17. Latency Issue ● Preemption ●Critical Section ● Interrupt
- 18. Spinlock ● Task willbe busy waiting until it acquires the lock ● Spinlock in Preemptible Linux (CONFIG_PREEMPT) – Uniprocessor ● preempt_disable – SMP ● preempt_disable ● Lock acquire, busy waiting – No sleeping ● Convert spinlock to rt-mutex – Sleeping
- 19. Priority Inversion Acquires a lock Priority Time preempted Triesto get the same lock waits ● High priority task is preempted by medium priority task ● Unbound waiting for high priority task
- 20. Priority Inheritance Acquires a lock Priority Ti me preempted Triesto get the lock waits Executes with the priority of the waiting process Releases the lock Acquires the lock preempted
- 21. Latency Issue ● Preemption ●Critical Section ● Interrupt
- 22. Task triggered byIRQ interrupt latency Interrupt handler Scheduler Interrupt handler duration scheduler latency scheduler duration Process context Interrupt context Makes the task runnable Total Latency Task Sleep Task wakeup
- 23. Task interrupted by IRQ TaskRun Interrupt handler Task Resume Interrupt Process context Interrupt context Interrupt handler ... ... Interrupt Preempt Latency Unbound??
- 24. Original Linux Kernel UserSpace User Context Interrupt Handlers Kernel Space Interrupt Context SoftIRQsHiprio tasklets Network Stack Timers Regular tasklets ... Scheduling Points Process Thread Kernel Thread Priority of interrupt context is always higher than others
- 25. Non-determined Issue ● Interruptcontext can always preempt others ● Interrupt as an external event – Interrupt number of a time interval is non-determinated – Nature of interrupt, can not be avoided ● Behavior of interrupt handler is not well defined – Non-determined interrupt handler
- 26. IRQ in PREEMPT_RT ●Threaded IRQ – IRQ handler is actually a kernel thread by default – Hard IRQ handler only wakes up IRQ handler thread ● Behavior of hard IRQ handler is well defined – Original IRQ handler (No Delayed) is reserved by IRQF_NO_THREAD flag. ● Remove softirq – ksoftirqd as a normal kernel thread, handles all softirqs
- 27. PREEMPT_RT User Space NODELAY InterruptHandlers Kernel SpaceNetworkStack Timers Tasklets Scheduling Points Process Thread Kernel Threads
- 28. Experiments on ARM ●OMAP-L138 (ARM11; 375/456 MHz) ● Cyclictest – measuring accuracy of sleep and wake operations of highly prioritized realtime threads ● cyclictest -t1 -p 80 -n -i 1000 -l 600000 -h 1000 ● Ping target with inverval 0.004s
- 29.
- 30.
- 31.
- 32. RT-Basic vs. RT-Full(syscall) Basic Full
- 33.
- 34. Tool for Observation ●Linux Kernel Tracing Tool ● Event tracing – Tracing kernel events mount t debugfs debugfs /sys/kernel/debug – Event: irq_handler_entry, irq_handler_exit, sched:* (/sys/kernel/debug/tracing/events/) tracecmd record e irq_handler_entry e irq_handler_exit e sched:* tracecmd report
- 35. Trace Log Format ●tracecmd1061 [000] 142.334403: irq_handler_entry: irq=21 name=critical_irq Current Task CPU# Time Stamp Event Name Message trace-cmd-1061 [000] 142.334403 irq_handler_entry Irq=21 name=critical_irq
- 36. Trace Log interrupt latency Interrupt handler Scheduler Interrupt handler duration scheduler latency scheduler duration Process context Interrupt context Makesthe task runnable trace-cmd critical_task irq_handler_entry irq_handler_exit sched_wakeup* sched_wakeup** sched_switch sched_wakeup*: wakeup critical_task sched_wakeup**: wakeup ksoftirqd
- 37.
- 38. Practical Issues ● Interrupt –Non-determined external events – Collision: Several Interrupt happens almost at the same time – Preemption: Interrupt takes place when task is woke up but not starts yet.
- 39.
- 40. Critical Handler Critical TaskHandler1 Critical IRQ IRQ1 IRQ2... Handler2 ... Critical Task Latency = Critical IRQ Latency + Critical Handler Cost + IRQ1 Latency + Handler1 Cost + … + + Schedule Latency Interrupt Preemption
- 41.
- 42. ● Defer OtherInterrupt Critical Handler Critical TaskHandler1 Critical IRQ IRQ1 IRQ2 ... Handler2 ... Critical Handler Critical Task Handler1 Critical IRQ IRQ1 IRQ2 ... Handler2 ... Interrupt Preemption
- 43. ● Disable otherinterrupts when critical interrupt occurs ● Schedule a tasklet to re-enable other interrupts. – Critical Task's priority could be higher than tasklet. – Ensure determination of critical task. Critical Handler Critical Task Handler1 IRQ1 IRQ2 ... Handler2 ... Critical IRQ tasklet Interrupt Preemption
- 44. Critical Handler Critical TaskHandler1 IRQ1 Critical IRQ ●Several Interrupt happens almost at the same time ● Set Critical IRQ with highest priority ● Depended on hardware Interrupt Collision Handler2 IRQ2 Almost at the same time !!!
- 45. ● High ResolutionTimer – Non threaded Interrupt ● Non-determined IRQ handler – Heavy and variable jitter ● 100us ~ 160us (375MHZ ARM926EJS) – Important and complex component of Linux Kernel ● Tick timer ● Timer for RR scheduler ● nanosleep, clock_nanosleep ● Futex, rtmux ● Others – Can not be disabled Many Unpredicted Timer IRQ sources !! HRT Impact
- 46. Simplify Timer Interrupt ●Move to Periodic Timer – Tick is the only one timer source – However, the behavior of timer interrupt is still non- determined – Timer is not precise enough ● Periodic task could not be Real-time task
- 47. Conclusion ● PREEMPT_RT patchsignificantly improves the preemptiveness of Linux kernel. – However we should keep in mind that these results are statistical in nature rather than being conclusive. ● Using PREEMPT_RT will not guarantee that your system is hard real-time. – Your system and applications will also have to be designed properly including correct priorities, use of deterministic APIs, allocation of critical resources.
- 48. Reference ● Inside TheRT Patch, Steven Rostedt, Darren V Hart ● A real time preemption overview, Paul Mckenny ● Making Linux do Hard Real Time, Jim Hung