Real time Linux programming
Mark Veltzer
mailto:mark@veltzer.net
Definitions
What is “real time”?
● From wikipedia: “real-time computing (RTC), or reactive
computing, is the study of hardware and sof...
“Real time” hardware
● A hardware may guarantee real time
performance by utilizing a “hardware clock”
which is a crystal w...
Terms
● “Hard real time” means a piece of software that must meet
time requirements and never miss even a single dead line...
Designing real time software
●
There are methodologies designed specifically for the design of
real time software such as ...
Real time and programming
language
● Real time systems may be written in any
language (including Java and Python contrary
...
Traditional real time operating
systems
● In traditional real time operating systems the
programmers write threads which a...
Where latency/non determinism
come from?
● Hardware. Examples can be self monitoring
interrupts, Caches.
● Operating syste...
The art of real time programming
● All this means that to create a real time
system one must master all levels of the
syst...
Linux and real time
Linux and real time
●
Linux was never designed from the beginning to be a real time system and
some would claim that UNIX ...
What does real time Linux give me?
● Interrupts are handled by the kernel in threads with a given priority.
● The ability ...
Page faults and their cost
● In a Linux system going to the kernel is
expensive (1usec region).
● The system is lazy by de...
Real time programming on Linux
Monitoring page faults
● The proc filesystem: /proc/[pid]/stat
● From the command line: ps -o min_flt
● Link with libproc....
Avoiding page faults
● Stack: allocate a big buffer on the stack and
zero it or call a recursive function a lot.
● Malloc:...
Avoiding latency
● If you want a really clean CPU all to yourself:
– Put your thread on a CPU that does not get
hardware i...
Avoiding latency (cont)
● On larger systems (Redhat,...) you usually have
a /usr/sbin/irqbalance distributing the interrup...
Measuring the entire systems
latency
● Is done using cyclictest(1) which part of the rt-
tests package.
● If you don't hav...
Upcoming SlideShare
Loading in …5
×

Realtime

1,104 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,104
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Realtime

  1. 1. Real time Linux programming Mark Veltzer mailto:mark@veltzer.net
  2. 2. Definitions
  3. 3. What is “real time”? ● From wikipedia: “real-time computing (RTC), or reactive computing, is the study of hardware and software systems that are subject to a "real-time constraint"— e.g. operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints.[1] Often real-time response times are understood to be in the order of milliseconds and sometimes microseconds. In contrast, a non- real-time system is one that cannot guarantee a response time in any situation, even if a fast response is the usual result.”
  4. 4. “Real time” hardware ● A hardware may guarantee real time performance by utilizing a “hardware clock” which is a crystal which delivers a pulse at a constant rate. ● The engineers will make sure that the hardware concludes it's business before the clock strikes again. ● What you get is a guarantee for how long it takes you to do a certain operation.
  5. 5. Terms ● “Hard real time” means a piece of software that must meet time requirements and never miss even a single dead line. ● “Soft real time” is the same but for a system that will continue to perform even if the deadlines are missed. ● “High performance” is a piece of software that has to utilize the hardware to the maximum but where each operation is not constrained. ● A piece of software may fall into more than one category (think chess playing software).
  6. 6. Designing real time software ● There are methodologies designed specifically for the design of real time software such as MASCOT, HOOD, AADL, Ravenscar, profile and others. ● These methodologies deal with how you design your system: decide on numbers of threads or process, the roles of each process, the priorities of the threads and more. ● They also deal with how to debug a multi-threaded piece of software that must meat the strict deadlines. ● We will not discuss them further.
  7. 7. Real time and programming language ● Real time systems may be written in any language (including Java and Python contrary to public opinion). ● Because care is needed to disable non- deterministic aspects of the programming language (e.g. garbage collector) then programmers prefer more deterministic languages like C and C++. ● No, C++ is not less deterministic than C.
  8. 8. Traditional real time operating systems ● In traditional real time operating systems the programmers write threads which are part of the operating system kernel (operate in the same space). ● These systems are kept simple by design so as not to be non deterministic. ● These systems are fast and deterministic. ● They are also very hard to debug once the software grows because of the single space.
  9. 9. Where latency/non determinism come from? ● Hardware. Examples can be self monitoring interrupts, Caches. ● Operating system. Examples can be operating system threads, swap, long interrupt handlers, page faults, laziness. ● Programming language. Examples can be malloc/free(3), garbage collectors. ● The actual software. Examples can be “if” statements, algorithm bugs, locking.
  10. 10. The art of real time programming ● All this means that to create a real time system one must master all levels of the system: from the hardware, through the operating system, through the programming language to the software. ● That is also why hard real time programming is hard.
  11. 11. Linux and real time
  12. 12. Linux and real time ● Linux was never designed from the beginning to be a real time system and some would claim that UNIX philosophy is contrary to real time. ● But: Linux has been developing real time capabilities for over a decade. ● The kernel patch has been written by Ingo Molnar and Thomas Gleixner. ● It is almost fully integrated into the mainline kernel (3.6). ● Is expected to be fully integrated in the next year or so. ● Is usually applied as a patch to the kernel by the people building the real time OS for you. ● See https://rt.wiki.kernel.org/index.php/Main_Page
  13. 13. What does real time Linux give me? ● Interrupts are handled by the kernel in threads with a given priority. ● The ability to run threads at OS interrupt handler priority or higher. ● Hard Real time scheduling - separate from the soft scheduling that user space usually uses. ● Priority inversion. ● Fine granularity of CPU affinity of user space and operating system threads. ● New real time clock API for better precision. ● Less interrupts for real time threads – the scheduler does not use regular tick interrupts to check if real time threads need to stop running and be replaced by other threads. ● Low latency of the kernel (kernel was reviewed and locations where interrupts were disabled were improved) ● What is the worst time latency? Around 30usec for dedicated systems. More if you put more drivers, less if you use less.
  14. 14. Page faults and their cost ● In a Linux system going to the kernel is expensive (1usec region). ● The system is lazy by default and so uses the MMU to fetch pages it has promised but has not attached to the process. ● This happens in fetching pages for the executable, data segment, stack, libraries, mmap(2) and malloc(3) (because of mmap(2)). ● See examples.
  15. 15. Real time programming on Linux
  16. 16. Monitoring page faults ● The proc filesystem: /proc/[pid]/stat ● From the command line: ps -o min_flt ● Link with libproc.so and use it's API to monitor your own page faults. ● Use top(1) and configure page faults as one of the fields you see.
  17. 17. Avoiding page faults ● Stack: allocate a big buffer on the stack and zero it or call a recursive function a lot. ● Malloc: zero the memory when you get it. ● Mmap: zero the memory or pass a flag to mmap. ● Data segment: zero the memory. ● For everything including all of the above there is also mlock(2).
  18. 18. Avoiding latency ● If you want a really clean CPU all to yourself: – Put your thread on a CPU that does not get hardware interrupts. – Do this using the pthread affinity API. – On embedded systems hardware interrupts usually go to CPU 0. – Use cat /proc/interrupts to make sure. – Tune where interrupts go using echo to /proc/irq/ [irq_number]/smp_affinity
  19. 19. Avoiding latency (cont) ● On larger systems (Redhat,...) you usually have a /usr/sbin/irqbalance distributing the interrupts evenly between the cores. ● Either get rid of it (don't run it on boot in /etc/init.d). ● Or configure it not to send interrupts to the CPU you want to keep clean (use IRQBALANCE_BANNED_CPUS in /etc/default/irqbalance).
  20. 20. Measuring the entire systems latency ● Is done using cyclictest(1) which part of the rt- tests package. ● If you don't have it in your system vendors repository then install it from source http://git.kernel.org/cgit/linux/kernel/git/clrkwllms/rt-t ● It contains not only cyclicttest(1) but other very useful tools.

×