The document presents an overview of real-time Linux (RT-Linux) and its implementation approaches, particularly focusing on the preempt_rt patch and Xenomai framework. It discusses the differences between hard and soft real-time requirements, critical sections, and various strategies for achieving determinism in Linux. The conclusion emphasizes the advantages of preempt_rt and Xenomai in terms of stability, usability, and performance for real-time applications.

1. Title: RT Linux
Seyed Navid Ashrafi
Telecommunication Networks Group
Technische Universität Berlin
Summer Semester 2018
35-minute presentation
NES SS2018 1

2. Over view
>> Introduction to RTOS
- Definitions and history
- Usages & requirements
NES SS2018 2
>> PREEMPT_RT patch
- Overview
- Features
>> Latency measurement test
>> Conclusion
>> Different approaches
- Xenomai
- PREEMPT_RT
- Cons & pros

3. What is real time?
• It is about fast operation time
• It is about performance
• Its about DETERMINISM
- You need to execute an operation in a specific time frame and you need a
guarantee for that.
- As fast as specified, not as fast as possible
- RT-Linux: Deterministic timing behavior in Linux.
- Depends on the system
NES SS2018 3

4. Usage and requirements of RT-Linux
• Usages:
- Industrial applications
- Multimedia systems
- Aerospace system
- Automative
- Financial services
• Requirements:
- Deterministic timing behavior
- Preemption
- priority inheritance
NES SS2018 4

5. Soft Realtime??
• It is in contrast with deterministic behavior
• We concentrate on 100% and 95% hard RT.
• 100% hard RT: Realtime requirements should be met 100% otherwise
it will lead to an error condition (industry)
• 95% hard RT: Realtime requirements should be met 95% (data
collection)
NES SS2018 5

6. Critical section
• Concurrent access to a shared resource can lead to
unexpected or error condition
• There are over 10,000 individual critical section in a
typical Linux 2.6 kernel
• In the non-preemptible design of Linux 2.6
kernel, any critical section can block preemption
• In the Preemptible design the effort was to reduce the total size and
number of non-preemptible critical sections.
NES SS2018 6

7. Approaches
Dual-kernel / Co-kernel
Single-kernel / In-kernel
NES SS2018 7

8. Dual-kernel approach
• First approaches which are still existing.
• Linux is running on top of a microkernel as a low priority real time task.
• The microkernel does the real time stuff.
• You need to maintain the microkernel and a hardware abstraction layer on Linux
to make it run on the microkernel:
- Big, time consuming effort.
- Reason why most of these approaches
are always one step behind
Linux fast development
NES SS2018 8

9. Single-kernel approach
• A way to make Linux it self real-time capable.
• You would have to touch every single file in the kernel.
• No microkernel or hardware abstraction layer (you can just work with
Linux standard tools).
NES SS2018 9

10. Xenomai
Xenomai 1.0
• Announced in 2001 – as portability framework for RTOS applications
• Required a real-time basis
• Development of ADEOS layer for Linux and RTAI
• Merged with RTAI
Xenomai 2.0
• Departed from RTAI in 2005 – incompatible design goals
• Evolved ADEOS to I-pipe layer (also used by RTAI)
• Ported to 6 architectures
Xenomai 3.0
• Released in 2015 after 5 years of development
• Rework of in-kernel core (now POSIX-centric)
• Support for native Linux
NES SS2018 10

11. Xenomai
NES SS2018 11
microkernel preempts Linux

12. Xenomai architecture
NES SS2018 12

13. Migration between schedulers
• creating realtime task in application, over a normal Linux task
• 2 threads for one task which share a lot of states
• only one of them can run at the same time
• Migration to RT: suspend Linux task, resume Cobalt thread
• Migration to Linux (on syscall, fault/trap, signal):
suspend Cobalt thread, resume Linux task
• Every things that we do not handle in Xenomai will be directed
to Linux
NES SS2018 13

14. Hardware abstraction layer
I-pipe Core
NES SS2018 14

15. Issues of dual-kernel approach
• Special API
• Special tools and libraries
• Microkernel needs to be ported for new HW and new Linux versions
• Bad scaling on big platforms
• Various number of not palatable patches
• Changes to critical subsystems regularly cause regressions
• Porting efforts consume core developer resources
-Most work done by Philippe and Gilles so far
-Time would be better spent on feature improvements...
NES SS2018 15

16. Xenomai application
• Machine control systems, PLCs
• Printing machines (manroland)
• Printers / copying machines
• Network switches (e.g. Ruggedcom)
• Magnetic resonance tomographs (Siemens Healthcare)
• OROCOS (OSS robotics framework)
• Robotic research projects
• … (many, many incognito applications)
NES SS2018 16

17. Preempt_RT
• Founded 14 years ago by: Thomas Gleixner & Ingo Molnar
• Huge community
• In-kernel approach
• Main idea: only code that, absolutely must be non-preemptible, is allowed
to be non-preemptible
• Most of the features already made it into the “mainline”
• Interrupt handlers run in a kernel thread which has two advantages:
- The kernel thread is interruptible
- It shows up in a process list with a PID
• First mainline integration 2006
NES SS2018 17

18. Preempt_RT Goals
• 100% Preemptible kernel
- not possible yet, but let’s try regardless
- Removal of interrupts and preemption disabling
- Fast “worst case” time.
• Quick reaction time
- Bring latencies down to a minimum
• Minimize non-Preemptible kernel code
- at the meantime Minimize code changing amount
NES SS2018 18

19. Evolution of RT-Linux
NES SS2018 19

20. PREEMPT_RT architecture
NES SS2018 20

21. Fully Preemptible kernel (the RT patch)
• Preemption almost everywhere
• Spinlocks turn into mutexes
• No hard interrupt context
• Preemptible critical section
• Preemptible interrupt handlers
• Preemptible “interrupt disable” code sequence
• Priority inheritance
• Deferred operation
• Latency reduction
NES SS2018 21

22. Mutex
• Critical sections are still protected without disabling preemption
• A Mutex guarantees data integrity while extending the operation of the
spinlock
- we have priority inheritance
- All locks are assumed to be preemptible
- This design eliminates the possibility that an inefficient locking
implementation in a driver can introduce un-detected latency into
the system.
- Only a small subset of locks stay unpreemptible which are manageable.
NES SS2018 22

23. Priority inversion
• When a high priority task wants to run but it can not because a lower
priority task is holding that lock
• It is bad but we can not get rid of it, what we can get rid of is
unbounded priority inversion
NES SS2018 23

24. Unbounded priority inversion
NES SS2018 24
A
B
C

25. Priority inheritance
NES SS2018 25
A
B
C

26. local_irq_disable
asmlinkage void
do_entInt(unsigned long type, unsigned long vector,
unsigned long la_ptr, struct pt_regs *regs)
{
struct pt_regs *old_regs;
local_irq_disable();
switch (type) {
case 0:
#ifdef CONFIG_SMP
NES SS2018 26
handle_ipi(regs);
return;
#else
irq_err_count++;
printk(KERN_CRIT "Interprocessor interrupt?
"You must be kidding!n");
#endif
break;
1 2

27. Preempt_disable
NES SS2018 27
• Local_irq_disable younger sibling
• Also does not give a hint to what does it do!
• Has the exact same problems of local_irq_disable
• Preempt_enable_no_resched
- only should be used within preempt_disable locations

28. Why do we still need Co-kernel?
Functional limitations of In-kernel
• Emulation of RTOS scheduling behavior limited
by Linux scheduler
• Not all kernel+libc code paths used by In-kernel approaches
are necessarily hard real-time under PREEMPT-RT
• No detection of non-RT behavior of your application
Performance limitations of In-kernel
• Co-kernel is usually more light-weight on low-end platforms
(limited caches vs. code path lengths)
• PREEMPT-RT can have unwanted impact
co-located non-RT workloads
Xenomai adds value to Linux
• it keeps the realtime stuff away from the complexities
of Linux kernel
• Co-kernel approach can be beneficial
for low latencies and real-time application architecture
NES SS2018 28

29. Latency measurement on xenomai & preempt_RT
• Made by Jan Altenberg from Linutronix GmbH
• On two ARM Cortex A9 SOC platforms
• IRQ test with 10KHz frequency with the latency box
• 100% CPU load with “hackbench”
• They did the test in usespace and kernelspace
• 12-hour duration
NES SS2018 29

30. Hackbench
• Starts n groups of 20 clients and 20 servers
• Each client sends 100 messages to each server via a socket
connection
NES SS2018 30

31. Latency box
NES SS2018 31

32. Xenomai latency userspace task
NES SS2018 32

33. PREEMPT_RT latency userspace task
NES SS2018 33

34. PREEMPT_RT latency userspace task (isolated CPU)
NES SS2018 34

35. Latency: userspac- Comparison
NES SS2018 35

36. Xenomai latency in kernelspace
NES SS2018 36

37. PREEMPT_RT latency in kernelspace
NES SS2018 37

38. PREEMPT_RT latency in kernelspace (isolated)
NES SS2018 38

39. PREEMPT_RT latency in kernelspace (using FIQ)
NES SS2018 39

40. Latency: Kernel - Comparison
NES SS2018 40

41. Test results
• Which approach to choose?
- If timing is vital, choose Xenomai, it separates the realtime interrupt
handling from complexities of Linux kernel
• You can go with PREEMPT_RT using FIQ
• Xenomai kernelspace is the best for 100% hard performance
• Xenomai userspace is slower than it’s kernelspace
• Tests only can help you in the field of latency which is dependent on
the hardware
• No guarantee for results repeatation
NES SS2018 41

42. Conclusion
• PREEMPT_RT is getting integrated in Linux mainline
- Download here: https://www.kernel.org/
• PREEMPT_RT Increases Linux stability and quality
• PREEMPT_RT is simple to use
• Xenomai has it’s own advantages over PREEMPT_RT
• Microkernels are hard to handle
• For the most use cases the latency of the both approaches is almost
the same
• FIQ offers very fast latency but brings limitations
NES SS2018 42

43. Thank you
NES SS2018 43