What is “real time”?
● From wikipedia: “real-time computing (RTC), or reactive
computing, is the study of hardware and software systems that
are subject to a "real-time constraint"— e.g. operational
deadlines from event to system response. Real-time programs
must guarantee response within strict time constraints. Often
real-time response times are understood to be in the order of
milliseconds and sometimes microseconds. In contrast, a non-
real-time system is one that cannot guarantee a response time
in any situation, even if a fast response is the usual result.”
“Real time” hardware
● A hardware may guarantee real time
performance by utilizing a “hardware clock”
which is a crystal which delivers a pulse at a
● The engineers will make sure that the hardware
concludes it's business before the clock strikes
● What you get is a guarantee for how long it
takes you to do a certain operation.
● “Hard real time” means a piece of software that must meet
time requirements and never miss even a single dead line.
● “Soft real time” is the same but for a system that will
continue to perform even if the deadlines are missed.
● “High performance” is a piece of software that has to
utilize the hardware to the maximum but where each
operation is not constrained.
● A piece of software may fall into more than one category
(think chess playing software).
Designing real time software
There are methodologies designed specifically for the design of
real time software such as MASCOT, HOOD, AADL, Ravenscar,
profile and others.
These methodologies deal with how you design your system:
decide on numbers of threads or process, the roles of each
process, the priorities of the threads and more.
● They also deal with how to debug a multi-threaded piece of
software that must meat the strict deadlines.
We will not discuss them further.
Real time and programming
● Real time systems may be written in any
language (including Java and Python contrary
to public opinion).
● Because care is needed to disable non-
deterministic aspects of the programming
language (e.g. garbage collector) then
programmers prefer more deterministic
languages like C and C++.
● No, C++ is not less deterministic than C.
Traditional real time operating
● In traditional real time operating systems the
programmers write threads which are part of the
operating system kernel (operate in the same
● These systems are kept simple by design so as
not to be non deterministic.
● These systems are fast and deterministic.
● They are also very hard to debug once the
software grows because of the single space.
Where latency/non determinism
● Hardware. Examples can be self monitoring
● Operating system. Examples can be operating
system threads, swap, long interrupt handlers,
page faults, laziness.
● Programming language. Examples can be
malloc/free(3), garbage collectors.
● The actual software. Examples can be “if”
statements, algorithm bugs, locking.
The art of real time programming
● All this means that to create a real time
system one must master all levels of the
system: from the hardware, through the
operating system, through the programming
language to the software.
● That is also why hard real time programming
Linux and real time
Linux was never designed from the beginning to be a real time system and
some would claim that UNIX philosophy is contrary to real time.
● But: Linux has been developing real time capabilities for over a decade.
The kernel patch has been written by Ingo Molnar and Thomas Gleixner.
● It is almost fully integrated into the mainline kernel (3.6).
● Is expected to be fully integrated in the next year or so.
Is usually applied as a patch to the kernel by the people building the real
time OS for you.
What does real time Linux give me?
● Interrupts are handled by the kernel in threads with a given priority.
● The ability to run threads at OS interrupt handler priority or higher.
● Hard Real time scheduling - separate from the soft scheduling that user space usually
● Priority inversion.
● Fine granularity of CPU affinity of user space and operating system threads.
● New real time clock API for better precision.
● Less interrupts for real time threads – the scheduler does not use regular tick
interrupts to check if real time threads need to stop running and be replaced by other
● Low latency of the kernel (kernel was reviewed and locations where interrupts were
disabled were improved)
● What is the worst time latency? Around 30usec for dedicated systems. More if you put
more drivers, less if you use less.
Page faults and their cost
● In a Linux system going to the kernel is
expensive (1usec region).
● The system is lazy by default and so uses the
MMU to fetch pages it has promised but has not
attached to the process.
● This happens in fetching pages for the
executable, data segment, stack, libraries,
mmap(2) and malloc(3) (because of mmap(2)).
● See examples.
Monitoring page faults
● The proc filesystem: /proc/[pid]/stat
● From the command line: ps -o min_flt
● Link with libproc.so and use it's API to monitor
your own page faults.
● Use top(1) and configure page faults as one of
the fields you see.
Avoiding page faults
● Stack: allocate a big buffer on the stack and
zero it or call a recursive function a lot.
● Malloc: zero the memory when you get it.
● Mmap: zero the memory or pass a flag to
● Data segment: zero the memory.
● For everything including all of the above there
is also mlock(2).
● If you want a really clean CPU all to yourself:
– Put your thread on a CPU that does not get
– Do this using the pthread affinity API.
– On embedded systems hardware interrupts
usually go to CPU 0.
– Use cat /proc/interrupts to make sure.
– Tune where interrupts go using echo to /proc/irq/
Avoiding latency (cont)
● On larger systems (Redhat,...) you usually have
a /usr/sbin/irqbalance distributing the interrupts
evenly between the cores.
● Either get rid of it (don't run it on boot in
● Or configure it not to send interrupts to the CPU
you want to keep clean (use
Measuring the entire systems
● Is done using cyclictest(1) which part of the rt-
● If you don't have it in your system vendors
repository then install it from source
● It contains not only cyclicttest(1) but other very