Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Real time Linux programming
Mark Veltzer
mailto:mark@veltzer.net
Definitions
What is “real time”?
● From wikipedia: “real-time computing (RTC), or reactive
computing, is the study of hardware and sof...
“Real time” hardware
● A hardware may guarantee real time
performance by utilizing a “hardware clock”
which is a crystal w...
Terms
● “Hard real time” means a piece of software that must meet
time requirements and never miss even a single dead line...
Designing real time software
●
There are methodologies designed specifically for the design of
real time software such as ...
Real time and programming
language
● Real time systems may be written in any
language (including Java and Python contrary
...
Traditional real time operating
systems
● In traditional real time operating systems the
programmers write threads which a...
Where latency/non determinism
come from?
● Hardware. Examples can be self monitoring
interrupts, Caches.
● Operating syste...
The art of real time programming
● All this means that to create a real time
system one must master all levels of the
syst...
Linux and real time
Linux and real time
●
Linux was never designed from the beginning to be a real time system and
some would claim that UNIX ...
What does real time Linux give me?
● Interrupts are handled by the kernel in threads with a given priority.
● The ability ...
Page faults and their cost
● In a Linux system going to the kernel is
expensive (1usec region).
● The system is lazy by de...
Real time programming on Linux
Monitoring page faults
● The proc filesystem: /proc/[pid]/stat
● From the command line: ps -o min_flt
● Link with libproc....
Avoiding page faults
● Stack: allocate a big buffer on the stack and
zero it or call a recursive function a lot.
● Malloc:...
Avoiding latency
● If you want a really clean CPU all to yourself:
– Put your thread on a CPU that does not get
hardware i...
Avoiding latency (cont)
● On larger systems (Redhat,...) you usually have
a /usr/sbin/irqbalance distributing the interrup...
Measuring the entire systems
latency
● Is done using cyclictest(1) which part of the rt-
tests package.
● If you don't hav...
Upcoming SlideShare
Loading in …5
×

Realtime

1,128 views

Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Realtime

  1. 1. Real time Linux programming Mark Veltzer mailto:mark@veltzer.net
  2. 2. Definitions
  3. 3. What is “real time”? ● From wikipedia: “real-time computing (RTC), or reactive computing, is the study of hardware and software systems that are subject to a "real-time constraint"— e.g. operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints.[1] Often real-time response times are understood to be in the order of milliseconds and sometimes microseconds. In contrast, a non- real-time system is one that cannot guarantee a response time in any situation, even if a fast response is the usual result.”
  4. 4. “Real time” hardware ● A hardware may guarantee real time performance by utilizing a “hardware clock” which is a crystal which delivers a pulse at a constant rate. ● The engineers will make sure that the hardware concludes it's business before the clock strikes again. ● What you get is a guarantee for how long it takes you to do a certain operation.
  5. 5. Terms ● “Hard real time” means a piece of software that must meet time requirements and never miss even a single dead line. ● “Soft real time” is the same but for a system that will continue to perform even if the deadlines are missed. ● “High performance” is a piece of software that has to utilize the hardware to the maximum but where each operation is not constrained. ● A piece of software may fall into more than one category (think chess playing software).
  6. 6. Designing real time software ● There are methodologies designed specifically for the design of real time software such as MASCOT, HOOD, AADL, Ravenscar, profile and others. ● These methodologies deal with how you design your system: decide on numbers of threads or process, the roles of each process, the priorities of the threads and more. ● They also deal with how to debug a multi-threaded piece of software that must meat the strict deadlines. ● We will not discuss them further.
  7. 7. Real time and programming language ● Real time systems may be written in any language (including Java and Python contrary to public opinion). ● Because care is needed to disable non- deterministic aspects of the programming language (e.g. garbage collector) then programmers prefer more deterministic languages like C and C++. ● No, C++ is not less deterministic than C.
  8. 8. Traditional real time operating systems ● In traditional real time operating systems the programmers write threads which are part of the operating system kernel (operate in the same space). ● These systems are kept simple by design so as not to be non deterministic. ● These systems are fast and deterministic. ● They are also very hard to debug once the software grows because of the single space.
  9. 9. Where latency/non determinism come from? ● Hardware. Examples can be self monitoring interrupts, Caches. ● Operating system. Examples can be operating system threads, swap, long interrupt handlers, page faults, laziness. ● Programming language. Examples can be malloc/free(3), garbage collectors. ● The actual software. Examples can be “if” statements, algorithm bugs, locking.
  10. 10. The art of real time programming ● All this means that to create a real time system one must master all levels of the system: from the hardware, through the operating system, through the programming language to the software. ● That is also why hard real time programming is hard.
  11. 11. Linux and real time
  12. 12. Linux and real time ● Linux was never designed from the beginning to be a real time system and some would claim that UNIX philosophy is contrary to real time. ● But: Linux has been developing real time capabilities for over a decade. ● The kernel patch has been written by Ingo Molnar and Thomas Gleixner. ● It is almost fully integrated into the mainline kernel (3.6). ● Is expected to be fully integrated in the next year or so. ● Is usually applied as a patch to the kernel by the people building the real time OS for you. ● See https://rt.wiki.kernel.org/index.php/Main_Page
  13. 13. What does real time Linux give me? ● Interrupts are handled by the kernel in threads with a given priority. ● The ability to run threads at OS interrupt handler priority or higher. ● Hard Real time scheduling - separate from the soft scheduling that user space usually uses. ● Priority inversion. ● Fine granularity of CPU affinity of user space and operating system threads. ● New real time clock API for better precision. ● Less interrupts for real time threads – the scheduler does not use regular tick interrupts to check if real time threads need to stop running and be replaced by other threads. ● Low latency of the kernel (kernel was reviewed and locations where interrupts were disabled were improved) ● What is the worst time latency? Around 30usec for dedicated systems. More if you put more drivers, less if you use less.
  14. 14. Page faults and their cost ● In a Linux system going to the kernel is expensive (1usec region). ● The system is lazy by default and so uses the MMU to fetch pages it has promised but has not attached to the process. ● This happens in fetching pages for the executable, data segment, stack, libraries, mmap(2) and malloc(3) (because of mmap(2)). ● See examples.
  15. 15. Real time programming on Linux
  16. 16. Monitoring page faults ● The proc filesystem: /proc/[pid]/stat ● From the command line: ps -o min_flt ● Link with libproc.so and use it's API to monitor your own page faults. ● Use top(1) and configure page faults as one of the fields you see.
  17. 17. Avoiding page faults ● Stack: allocate a big buffer on the stack and zero it or call a recursive function a lot. ● Malloc: zero the memory when you get it. ● Mmap: zero the memory or pass a flag to mmap. ● Data segment: zero the memory. ● For everything including all of the above there is also mlock(2).
  18. 18. Avoiding latency ● If you want a really clean CPU all to yourself: – Put your thread on a CPU that does not get hardware interrupts. – Do this using the pthread affinity API. – On embedded systems hardware interrupts usually go to CPU 0. – Use cat /proc/interrupts to make sure. – Tune where interrupts go using echo to /proc/irq/ [irq_number]/smp_affinity
  19. 19. Avoiding latency (cont) ● On larger systems (Redhat,...) you usually have a /usr/sbin/irqbalance distributing the interrupts evenly between the cores. ● Either get rid of it (don't run it on boot in /etc/init.d). ● Or configure it not to send interrupts to the CPU you want to keep clean (use IRQBALANCE_BANNED_CPUS in /etc/default/irqbalance).
  20. 20. Measuring the entire systems latency ● Is done using cyclictest(1) which part of the rt- tests package. ● If you don't have it in your system vendors repository then install it from source http://git.kernel.org/cgit/linux/kernel/git/clrkwllms/rt-t ● It contains not only cyclicttest(1) but other very useful tools.

×