Real Time Systems

Real Time Systems

© 2010 Anil Kumar Pugalia <email@sarika-pugs.com>
All Rights Reserved.

What to Expect?
W's of Real Time & RTOS?
Design Methodologies of RTOS
Linux → Real Time Linux
Various Latencies & Schedulers
The “Real Time Patch” Path
Real Time Applications
Real Time Kernel Debugging
Peek into a Co-Kernel
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 2

Real Time Definition
Correctness depends on
Functional Accuracy, and
Timings of the Result produced
Examples
MPEG Decoder in your DVD player
Call Response in your Mobile Phone


Real Time Types
Hard Real Time
Guaranteed to meet the response reqs
Examples: Defense Systems, Vehicle Control Systems,
Satellite Systems
Soft Real Time
Once in a while may not meet the response reqs
Examples: Multimedia Devices, VoIP, CE devices
An intermediate one: Firm Real Time
Check: What is our Desktop?


Is Real Time needed?
Deadline of Applications
Reduces Quality or Unacceptable
External Device Interactions with Apps
Handling of Messages – Time-bound or not
Application Task Priority
Could it be needed higher than OS services
Expressing delays & timeouts
Is it needed to be fine-grained, say in usecs

Real Time OS
RTOS Phylum
Non Real-time
Soft Real-time
Hard Real-time

Conflicting Requirements
Real Time vs Rich Features of GPOS


Expected Features of an RTOS
RTOS Requirements
Strictly Enforced Task Priorities
Fine-grained Preemption
External Event Handling in Bounded time frame
Which entails
Predictable Low Interrupt Latency
Predictable Low Scheduler Latency
Priority Inheritance
But wanted with Rich Features of
Multi Tasking / Processing
Interprocess Communication & Synchronization

RTOS Design Methodologies
Expand an RTOS
Make a GPOS Real Time capable
Linux → Real Time Linux (Separate Session)
The Co-Kernel Approach
RT Apps in User Space
Open Source Xenomai (Separate Session)
Open Source RTAI
RT Apps in Kernel Space
RTLinux from Windriver

Expand an RTOS
Difficult to ensure Determinism
As all core capabilities must be fully preemptive

Drivers for Hardware becomes very complex

Existing Software cannot be used without changes

High Maintenance Costs for Developers and Customers

So, let's understand the other one w.r.t. Linux


Why Linux is non Real Time?
High Interrupt Latency
High Scheduler Latency
OS services with indeterministic timing behaviour
IPC, Memory Allocation, ...
Linux is in general undeterministic in its response
Virtual Memory, System Calls, …
Later two are more of a better Coding Principles
However, the first two are major Design Changes
Let's take a closer look


Real Time related Concepts
Latency
Interrupt
Scheduling
Both are related to
Preemption
And Critical Sections


Latency Visualization

Interrupt ISR Runs ISR Signals RT Process
Event RT Process Runs

Time
t0 t1 t2 t3

Interrupt Interrupt Scheduling
Latency Scheduling Latency

Interrupt to Process Latency


Latency Details
Why Interrupt Latency?
Attributes of ISR
Real Time Process
Typically, high priority
Otherwise, anyway it is a problem
All 3 could potentially pose problems
For Hard Real Time


Possible Solutions
Key Requirement
Schedule RT Process, within a given time
Possible Helpers
Remove / Reduce interrupt-disabled code
(critical sections)
Write ISRs really really minimal in execution
time
Real Time Apps with the highest priority
May be higher than interrupts, softirqs, etc


Preemption Visualization

Time

User Space

System Call
Interface

Kernel Space
Process A
Process B


Preemption Details
Higher Preemption makes it more Real
Time feasible
Higher Priority (RT Tasks) could be
scheduled in a more predictable time
Sources of Preemption Latency
Critical Section Processing
Interrupt Context Processing


Preemption & Critical Sections
Higher Preemption at the cost of reducing Critical Sections
Isolate & Minimize the critical sections
Protect them by disabling preemption
spin_lock / spin_unlock; preempt_disable / preempt_enable
Preemption Models for reducing Critical Sections
Preempt only safe locations
Entry & Exit of system calls, Return from interrupt processing, …
Put explicit “allow preempt” code at these places
Overall not an excellent approach
Preempt everywhere except only the Critical Sections
lock-breaking mechanism
Getting achieved by instrumenting kernel for latency measurements
And by fixing the longest latency code paths


SMP World & Critical Sections
Uniprocessor case needs Protection from
Interrupt Processing
Exception Processing
Multiprocessor with additional
Multiple Kernel Threads
Posed a bigger need than Real Time for
reduced Critical Sections
Real Time has gained benefits from SMP
development, as well

Real Time Kernel Patch
Critical Section Control is a major step
We have achieved good Soft Real Time Performance
Since 2.6.12, in single-digit milliseconds
On reasonably fast x86 processor
But without solving the Interrupt Latency
Hard Real Time is not possible
Recent Developments have majorly focused on this aspect
A major Real Time Kernel Patch evolved
Maintained by Ingo Molnar
Available @ http://people.redhat.com/~mingo/realtime-preempt
Could be downloaded using say, ketchup -G 2.6.22.1-rt9
And takes care of many more issues

W's & How's of RT Kernel Patch?
Lot's of Questions
What all does it contain?
How to configure it?
What should be used & when?
And many more
Before answering these, let's understand
Schedulers in main-stream Kernel
Preemption Support in main-stream Kernel


Linux Schedulers
Provides multi-tasking capabilities by
Time Slicing
Preemption
Based on various task priorities
Specified by its scheduling policies
Understands the following execution instances
Kernel Thread
User Process
User Thread
Linux Basic Scheduling
Normal (SCHED_OTHER) – Fairness Scheduling
Other Advanced Scheduling supported
Round Robin (SCHED_RR)
FIFO (SCHED_FIFO)
All Schedulers are O(1)

Linux Kernel Preemption Levels
None (PREEMPT_NONE)
No forced preemption
Overall throughput, on average, is good
Voluntary (PREEMPT_VOLUNTARY)
First stage of latency reduction
Explicit preemption points are placed at strategic locations
Standard (PREEMPT_DESKTOP)
Preemption enabled everywhere except within critical sections
Good for soft real-time applications, like audio, multimedia, …
Kernel Parameter: preempt


Preemption Level from RT Patch
Complete (PREEMPT_RT)
Spin locks replaced with preembtable
mutexes
Preemption enabled everywhere except
where protected by preempt_disable()
Smoothes out variation in latency
Allows a low & predictable latency for time-
critical real-time applications


Additional Real Time Features
ISRs as Kernel Tasks
CONFIG_PREEMPT_HARDIRQ
Schedulable & Preemptible
Priority Assignment as Required
Control Window: /proc/sys/kernel/hardirq_preemption
Kernel Parameter: hardirq-preempt
Preemptable Softirqs
CONFIG_PREEMPT_SOFTIRQ
All ran in the context of ksoftirqd → Now as individual threads
A proper Linux task configured for real time
Control Window: /proc/sys/kernel/softirq_preemption
Kernel Parameter: softirq-preempt
Certain Read-Copy-Update (RCU) sections preemptible
CONFIG_PREEMPT_RCU
RCU is synchronization primitive for frequently read data


Creating a Real Time Process
#include <sched.h>
#define MY_RT_PRIO 1
int main(...) {
int old_policy;
struct sched_param params = { .sched_priority = MY_RT_PRIO };
...
old_policy = sched_getscheduler(0);
if (sched_setscheduler(0, SCHED_RR, &params) == -1)
handle_error();
…
}

RT Application: Do's & Don'ts
Have very few Real Time Processes
Keep the RT Applications Short & Efficient
Do not do heavy duty operations
Like Memory Allocation, etc
If essential, move to Initialization Sections
Check for the working of the needed OS
services
Especially, if any RT Application has priority higher
than that OS Service
Test for the Desired Results

Debugging Real Time Kernel
Soft Lockup
CONFIG_DETECT_SOFTLOCKUP
Unsafe Preemption Use Logging
CONFIG_DEBUG_PREEMPT
Deadlock Conditions Detection & Reporting
CONFIG_DEBUG_DEADLOCK
Deadlocks due to semaphores & spinlocks
Runtime Control of Locking Mode
CONFIG_DEBUG_RT_LOCKING_MODE
Mutex back to Spinlock

Tracing Real Time Kernel
Latency Tracing
CONFIG_LATENCY_TRACE
For data associated with last maximum preemption
latency
Kernel Window: /proc/latency_trace
Function Call Tracing
Inserts hooks to every function via gcc
Gives function level details through
Kernel Window: /proc/latency_trace
Soon getting outdated by Ftrace

Further Tracing RT Kernel
High Priority Process Wakeup
CONFIG_WAKEUP_TIMING
CONFIG_WAKEUP_LATENCY_HIST
Or, Interrupt Off Timing
CONFIG_CRITICAL_IRQSOFF_TIMING
CONFIG_INTERRUPT_OFF_HIST
Kernel Windows
/proc/sys/kernel/preempt_max_latency
/proc/latency_hist/interrupt_off_latency/CPU0 (0-10000us)
Time spent with Preemption disabled (in critical sections)
CONFIG_CRITICAL_PREEMPT_TIMING
CONFIG_PREEMPT_OFF_HIST

Real Time Linux
Myths & Realities
With all these understood
Is Linux really Hard Real Time?
Can it be used in all Real Time Embedded activities?
Can it be used in a Nuclear Reactor?
Answers
Can be used in most Real Time Embedded Applications, such as Robotics
But cannot be “one-size-fits-it-all” approach to Real Time
Is still a patch and hence not that extensibly tested
Not all Device Drivers have yet been verified as Real Time safe
More & more users (such as multimedia users), using the patch would verify more
drivers, as they are the users noticing longer latency delays caused by these devices
But even if they are, it is after all a GPOS
Harder to verify a lack of bugs
Could have potential bugs (from real time perspective)


Co-Kernel Approach


W's of Co-Kernel Approach
Do not convert the GPOS into an RTOS
Place a small Real Time Kernel besides it
On the same Hardware
This would be the Co-Kernel
Taking care of all Real Time needs
Non-RT tasks are done by the GPOS
So,


What is a Co-Kernel?

Co-Kernel is a Subsystem

Integrated with the main Kernel

Handling the Real Time part


Types of Co-Kernel Approaches
Two Prevalent Approaches
Support running of RT Apps in User Space
RT Apps to be embodied into Kernel Space
Our Focus of Discussion is the First One
With Xenomai in consideration


Benefits of Xenomai
Co-Kernel or Core running under Kernel, licensed under GPL 2
Kernel Support
Both 2.4 & 2.6 for with & without MMU systems
Provides generic building blocks to implement RT APIs (skins)
RT API Support for
VxWorks, pSOS+, VRTX, uITRON, POSIX 1003.1b
User Space i/f libraries: LGPL 2.1
Architecture Support
PowerPC*, Blackfin, ARM, x86*
Excellent Resources
Documentation, Technical Articles, …
Website: http://www.xenomai.org

Xenomai Architecture
User Space User Space User Space User Space
Application Application Application Application

Linux syscall interface

VxWorks pSOS VRTX POSIX
... Kernel-based
Skin Skin Skin Skin Applications

Abstract RTOS Core

SAL / HAL
Portability
Layers
I-Pipe


Xenomai Components
Interrupt Pipeline
Hardware & System Abstraction Layers
Xenomai Core & Nucleus
Xenomai Skins


How Xenomai Works?
Real Time Shadow
New Sets of System Calls
Sharing Kernel Features
By Domain Migration
Real Time Driver Model Mediation


What all have we learnt?
W's of Real Time & RTOS?
Design Methodologies of RTOS
Various Latencies & Schedulers
The “Real Time Patch” Path
Real Time Applications
Real Time Kernel Debugging
Peek into a Co-Kernel

Any Queries?


Real Time Systems

More Related Content

What's hot

Similar to Real Time Systems

More from Anil Kumar Pugalia

Recently uploaded

Real Time Systems