Your SlideShare is downloading. ×
Real Time Systems© 2010 Anil Kumar Pugalia <email@sarika-pugs.com>               All Rights Reserved.
What to Expect?Ws of Real Time & RTOS?Design Methodologies of RTOSLinux → Real Time Linux  Various Latencies & Schedulers ...
Real Time DefinitionCorrectness depends on Functional Accuracy, and Timings of the Result producedExamples MPEG Decoder in...
Real Time TypesHard Real Time  Guaranteed to meet the response reqs  Examples: Defense Systems, Vehicle Control Systems,  ...
Is Real Time needed?Deadline of Applications  Reduces Quality or UnacceptableExternal Device Interactions with Apps  Handl...
Real Time OSRTOS Phylum  Non Real-time  Soft Real-time  Hard Real-timeConflicting Requirements  Real Time vs Rich Features...
Expected Features of an RTOSRTOS Requirements  Strictly Enforced Task Priorities  Fine-grained Preemption  External Event ...
RTOS Design MethodologiesExpand an RTOSMake a GPOS Real Time capable Linux → Real Time Linux (Separate Session)The Co-Kern...
Expand an RTOSDifficult to ensure Determinism  As all core capabilities must be fully preemptiveDrivers for Hardware becom...
Linux → Real Time Linux© 2010 Anil Kumar Pugalia <email@sarika-pugs.com>   10               All Rights Reserved.
Why Linux is non Real Time?High Interrupt LatencyHigh Scheduler LatencyOS services with indeterministic timing behaviour  ...
Real Time related ConceptsLatency  Interrupt  SchedulingBoth are related to  Preemption  And Critical Sections          © ...
Latency VisualizationInterrupt      ISR Runs         ISR Signals      RT ProcessEvent                           RT Process...
Latency DetailsWhy Interrupt Latency?Attributes of ISRReal Time Process  Typically, high priority  Otherwise, anyway it is...
Possible SolutionsKey Requirement  Schedule RT Process, within a given timePossible Helpers  Remove / Reduce interrupt-dis...
Preemption Visualization                        Time                                                        User Space    ...
Preemption DetailsHigher Preemption makes it more RealTime feasibleHigher Priority (RT Tasks) could bescheduled in a more ...
Preemption & Critical SectionsHigher Preemption at the cost of reducing Critical Sections  Isolate & Minimize the critical...
SMP World & Critical SectionsUniprocessor case needs Protection from  Interrupt Processing  Exception ProcessingMultiproce...
Real Time Kernel PatchCritical Section Control is a major stepWe have achieved good Soft Real Time Performance  Since 2.6....
Ws & Hows of RT Kernel Patch? Lots of Questions   What all does it contain?   How to configure it?   What should be used &...
Linux SchedulersProvides multi-tasking capabilities by  Time Slicing  Preemption  Based on various task priorities  Specif...
Linux Kernel Preemption LevelsNone (PREEMPT_NONE) No forced preemption Overall throughput, on average, is goodVoluntary (P...
Preemption Level from RT Patch Complete (PREEMPT_RT)  Spin locks replaced with preembtable  mutexes  Preemption enabled ev...
Additional Real Time FeaturesISRs as Kernel Tasks  CONFIG_PREEMPT_HARDIRQ  Schedulable & Preemptible  Priority Assignment ...
Creating a Real Time Process#include <sched.h>#define MY_RT_PRIO 1int main(...) {    int old_policy;    struct sched_param...
RT Application: Dos & DontsHave very few Real Time ProcessesKeep the RT Applications Short & EfficientDo not do heavy duty...
Debugging Real Time KernelSoft Lockup  CONFIG_DETECT_SOFTLOCKUPUnsafe Preemption Use Logging  CONFIG_DEBUG_PREEMPTDeadlock...
Tracing Real Time KernelLatency Tracing  CONFIG_LATENCY_TRACE  For data associated with last maximum preemption  latency  ...
Further Tracing RT KernelHigh Priority Process Wakeup  CONFIG_WAKEUP_TIMING    CONFIG_WAKEUP_LATENCY_HISTOr, Interrupt Off...
Real Time Linux                  Myths & RealitiesWith all these understood  Is Linux really Hard Real Time?  Can it be us...
Co-Kernel Approach© 2010 Anil Kumar Pugalia <email@sarika-pugs.com>   32               All Rights Reserved.
Ws of Co-Kernel ApproachDo not convert the GPOS into an RTOSPlace a small Real Time Kernel besides it  On the same Hardwar...
What is a Co-Kernel?   Co-Kernel is a SubsystemIntegrated with the main Kernel Handling the Real Time part    © 2010 Anil ...
Types of Co-Kernel ApproachesTwo Prevalent Approaches  Support running of RT Apps in User Space  RT Apps to be embodied in...
Benefits of XenomaiCo-Kernel or Core running under Kernel, licensed under GPL 2Kernel Support  Both 2.4 & 2.6 for with & w...
Xenomai Architecture   User Space       User Space        User Space     User Space   Application      Application       A...
Xenomai ComponentsInterrupt PipelineHardware & System Abstraction LayersXenomai Core & NucleusXenomai Skins        © 2010 ...
How Xenomai Works?Real Time ShadowNew Sets of System CallsSharing Kernel Features  By Domain MigrationReal Time Driver Mod...
What all have we learnt?Ws of Real Time & RTOS?Design Methodologies of RTOSLinux → Real Time Linux  Various Latencies & Sc...
Any Queries?© 2010 Anil Kumar Pugalia <email@sarika-pugs.com>   41               All Rights Reserved.
Upcoming SlideShare
Loading in...5
×

Real Time Systems

12,262

Published on

Published in: Technology, Business

Transcript of "Real Time Systems"

  1. 1. Real Time Systems© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> All Rights Reserved.
  2. 2. What to Expect?Ws of Real Time & RTOS?Design Methodologies of RTOSLinux → Real Time Linux Various Latencies & Schedulers The “Real Time Patch” Path Real Time Applications Real Time Kernel DebuggingPeek into a Co-Kernel © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 2 All Rights Reserved.
  3. 3. Real Time DefinitionCorrectness depends on Functional Accuracy, and Timings of the Result producedExamples MPEG Decoder in your DVD player Call Response in your Mobile Phone © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 3 All Rights Reserved.
  4. 4. Real Time TypesHard Real Time Guaranteed to meet the response reqs Examples: Defense Systems, Vehicle Control Systems, Satellite SystemsSoft Real Time Once in a while may not meet the response reqs Examples: Multimedia Devices, VoIP, CE devicesAn intermediate one: Firm Real TimeCheck: What is our Desktop? © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 4 All Rights Reserved.
  5. 5. Is Real Time needed?Deadline of Applications Reduces Quality or UnacceptableExternal Device Interactions with Apps Handling of Messages – Time-bound or notApplication Task Priority Could it be needed higher than OS servicesExpressing delays & timeouts Is it needed to be fine-grained, say in usecs © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 5 All Rights Reserved.
  6. 6. Real Time OSRTOS Phylum Non Real-time Soft Real-time Hard Real-timeConflicting Requirements Real Time vs Rich Features of GPOS © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 6 All Rights Reserved.
  7. 7. Expected Features of an RTOSRTOS Requirements Strictly Enforced Task Priorities Fine-grained Preemption External Event Handling in Bounded time frameWhich entails Predictable Low Interrupt Latency Predictable Low Scheduler Latency Priority InheritanceBut wanted with Rich Features of Multi Tasking / Processing Interprocess Communication & Synchronization © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 7 All Rights Reserved.
  8. 8. RTOS Design MethodologiesExpand an RTOSMake a GPOS Real Time capable Linux → Real Time Linux (Separate Session)The Co-Kernel Approach RT Apps in User Space Open Source Xenomai (Separate Session) Open Source RTAI RT Apps in Kernel Space RTLinux from Windriver © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 8 All Rights Reserved.
  9. 9. Expand an RTOSDifficult to ensure Determinism As all core capabilities must be fully preemptiveDrivers for Hardware becomes very complexExisting Software cannot be used without changesHigh Maintenance Costs for Developers and CustomersSo, lets understand the other one w.r.t. Linux © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 9 All Rights Reserved.
  10. 10. Linux → Real Time Linux© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 10 All Rights Reserved.
  11. 11. Why Linux is non Real Time?High Interrupt LatencyHigh Scheduler LatencyOS services with indeterministic timing behaviour IPC, Memory Allocation, ...Linux is in general undeterministic in its response Virtual Memory, System Calls, …Later two are more of a better Coding PrinciplesHowever, the first two are major Design ChangesLets take a closer look © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 11 All Rights Reserved.
  12. 12. Real Time related ConceptsLatency Interrupt SchedulingBoth are related to Preemption And Critical Sections © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 12 All Rights Reserved.
  13. 13. Latency VisualizationInterrupt ISR Runs ISR Signals RT ProcessEvent RT Process Runs Time t0 t1 t2 t3 Interrupt Interrupt Scheduling Latency Scheduling Latency Interrupt to Process Latency © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 13 All Rights Reserved.
  14. 14. Latency DetailsWhy Interrupt Latency?Attributes of ISRReal Time Process Typically, high priority Otherwise, anyway it is a problemAll 3 could potentially pose problems For Hard Real Time © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 14 All Rights Reserved.
  15. 15. Possible SolutionsKey Requirement Schedule RT Process, within a given timePossible Helpers Remove / Reduce interrupt-disabled code (critical sections) Write ISRs really really minimal in execution time Real Time Apps with the highest priority May be higher than interrupts, softirqs, etc © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 15 All Rights Reserved.
  16. 16. Preemption Visualization Time User Space System Call Interface Kernel Space Process A Process B © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 16 All Rights Reserved.
  17. 17. Preemption DetailsHigher Preemption makes it more RealTime feasibleHigher Priority (RT Tasks) could bescheduled in a more predictable timeSources of Preemption Latency Critical Section Processing Interrupt Context Processing © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 17 All Rights Reserved.
  18. 18. Preemption & Critical SectionsHigher Preemption at the cost of reducing Critical Sections Isolate & Minimize the critical sections Protect them by disabling preemption spin_lock / spin_unlock; preempt_disable / preempt_enablePreemption Models for reducing Critical Sections Preempt only safe locations Entry & Exit of system calls, Return from interrupt processing, … Put explicit “allow preempt” code at these places Overall not an excellent approach Preempt everywhere except only the Critical Sections lock-breaking mechanism Getting achieved by instrumenting kernel for latency measurements And by fixing the longest latency code paths © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 18 All Rights Reserved.
  19. 19. SMP World & Critical SectionsUniprocessor case needs Protection from Interrupt Processing Exception ProcessingMultiprocessor with additional Multiple Kernel ThreadsPosed a bigger need than Real Time forreduced Critical SectionsReal Time has gained benefits from SMPdevelopment, as well © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 19 All Rights Reserved.
  20. 20. Real Time Kernel PatchCritical Section Control is a major stepWe have achieved good Soft Real Time Performance Since 2.6.12, in single-digit milliseconds On reasonably fast x86 processorBut without solving the Interrupt Latency Hard Real Time is not possibleRecent Developments have majorly focused on this aspectA major Real Time Kernel Patch evolved Maintained by Ingo Molnar Available @ http://people.redhat.com/~mingo/realtime-preempt Could be downloaded using say, ketchup -G 2.6.22.1-rt9 And takes care of many more issues © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 20 All Rights Reserved.
  21. 21. Ws & Hows of RT Kernel Patch? Lots of Questions What all does it contain? How to configure it? What should be used & when? And many more Before answering these, lets understand Schedulers in main-stream Kernel Preemption Support in main-stream Kernel © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 21 All Rights Reserved.
  22. 22. Linux SchedulersProvides multi-tasking capabilities by Time Slicing Preemption Based on various task priorities Specified by its scheduling policiesUnderstands the following execution instances Kernel Thread User Process User ThreadLinux Basic Scheduling Normal (SCHED_OTHER) – Fairness SchedulingOther Advanced Scheduling supported Round Robin (SCHED_RR) FIFO (SCHED_FIFO)All Schedulers are O(1) © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 22 All Rights Reserved.
  23. 23. Linux Kernel Preemption LevelsNone (PREEMPT_NONE) No forced preemption Overall throughput, on average, is goodVoluntary (PREEMPT_VOLUNTARY) First stage of latency reduction Explicit preemption points are placed at strategic locationsStandard (PREEMPT_DESKTOP) Preemption enabled everywhere except within critical sections Good for soft real-time applications, like audio, multimedia, …Kernel Parameter: preempt © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 23 All Rights Reserved.
  24. 24. Preemption Level from RT Patch Complete (PREEMPT_RT) Spin locks replaced with preembtable mutexes Preemption enabled everywhere except where protected by preempt_disable() Smoothes out variation in latency Allows a low & predictable latency for time- critical real-time applications © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 24 All Rights Reserved.
  25. 25. Additional Real Time FeaturesISRs as Kernel Tasks CONFIG_PREEMPT_HARDIRQ Schedulable & Preemptible Priority Assignment as Required Control Window: /proc/sys/kernel/hardirq_preemption Kernel Parameter: hardirq-preemptPreemptable Softirqs CONFIG_PREEMPT_SOFTIRQ All ran in the context of ksoftirqd → Now as individual threads A proper Linux task configured for real time Control Window: /proc/sys/kernel/softirq_preemption Kernel Parameter: softirq-preemptCertain Read-Copy-Update (RCU) sections preemptible CONFIG_PREEMPT_RCU RCU is synchronization primitive for frequently read data © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 25 All Rights Reserved.
  26. 26. Creating a Real Time Process#include <sched.h>#define MY_RT_PRIO 1int main(...) { int old_policy; struct sched_param params = { .sched_priority = MY_RT_PRIO }; ... old_policy = sched_getscheduler(0); if (sched_setscheduler(0, SCHED_RR, &params) == -1) handle_error(); …} © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 26 All Rights Reserved.
  27. 27. RT Application: Dos & DontsHave very few Real Time ProcessesKeep the RT Applications Short & EfficientDo not do heavy duty operations Like Memory Allocation, etc If essential, move to Initialization SectionsCheck for the working of the needed OSservices Especially, if any RT Application has priority higher than that OS ServiceTest for the Desired Results © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 27 All Rights Reserved.
  28. 28. Debugging Real Time KernelSoft Lockup CONFIG_DETECT_SOFTLOCKUPUnsafe Preemption Use Logging CONFIG_DEBUG_PREEMPTDeadlock Conditions Detection & Reporting CONFIG_DEBUG_DEADLOCK Deadlocks due to semaphores & spinlocksRuntime Control of Locking Mode CONFIG_DEBUG_RT_LOCKING_MODE Mutex back to Spinlock © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 28 All Rights Reserved.
  29. 29. Tracing Real Time KernelLatency Tracing CONFIG_LATENCY_TRACE For data associated with last maximum preemption latency Kernel Window: /proc/latency_traceFunction Call Tracing Inserts hooks to every function via gcc Gives function level details through Kernel Window: /proc/latency_traceSoon getting outdated by Ftrace © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 29 All Rights Reserved.
  30. 30. Further Tracing RT KernelHigh Priority Process Wakeup CONFIG_WAKEUP_TIMING CONFIG_WAKEUP_LATENCY_HISTOr, Interrupt Off Timing CONFIG_CRITICAL_IRQSOFF_TIMING CONFIG_INTERRUPT_OFF_HISTKernel Windows /proc/sys/kernel/preempt_max_latency /proc/latency_hist/interrupt_off_latency/CPU0 (0-10000us)Time spent with Preemption disabled (in critical sections) CONFIG_CRITICAL_PREEMPT_TIMING CONFIG_PREEMPT_OFF_HIST © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 30 All Rights Reserved.
  31. 31. Real Time Linux Myths & RealitiesWith all these understood Is Linux really Hard Real Time? Can it be used in all Real Time Embedded activities? Can it be used in a Nuclear Reactor?Answers Can be used in most Real Time Embedded Applications, such as Robotics But cannot be “one-size-fits-it-all” approach to Real Time Is still a patch and hence not that extensibly tested Not all Device Drivers have yet been verified as Real Time safe More & more users (such as multimedia users), using the patch would verify more drivers, as they are the users noticing longer latency delays caused by these devices But even if they are, it is after all a GPOS Harder to verify a lack of bugs Could have potential bugs (from real time perspective) © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 31 All Rights Reserved.
  32. 32. Co-Kernel Approach© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 32 All Rights Reserved.
  33. 33. Ws of Co-Kernel ApproachDo not convert the GPOS into an RTOSPlace a small Real Time Kernel besides it On the same HardwareThis would be the Co-Kernel Taking care of all Real Time needsNon-RT tasks are done by the GPOSSo, © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 33 All Rights Reserved.
  34. 34. What is a Co-Kernel? Co-Kernel is a SubsystemIntegrated with the main Kernel Handling the Real Time part © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 34 All Rights Reserved.
  35. 35. Types of Co-Kernel ApproachesTwo Prevalent Approaches Support running of RT Apps in User Space RT Apps to be embodied into Kernel SpaceOur Focus of Discussion is the First One With Xenomai in consideration © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 35 All Rights Reserved.
  36. 36. Benefits of XenomaiCo-Kernel or Core running under Kernel, licensed under GPL 2Kernel Support Both 2.4 & 2.6 for with & without MMU systemsProvides generic building blocks to implement RT APIs (skins)RT API Support for VxWorks, pSOS+, VRTX, uITRON, POSIX 1003.1bUser Space i/f libraries: LGPL 2.1Architecture Support PowerPC*, Blackfin, ARM, x86*Excellent Resources Documentation, Technical Articles, …Website: http://www.xenomai.org © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 36 All Rights Reserved.
  37. 37. Xenomai Architecture User Space User Space User Space User Space Application Application Application Application Linux syscall interfaceVxWorks pSOS VRTX POSIX ... Kernel-based Skin Skin Skin Skin Applications Abstract RTOS Core SAL / HAL Portability Layers I-Pipe © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 37 All Rights Reserved.
  38. 38. Xenomai ComponentsInterrupt PipelineHardware & System Abstraction LayersXenomai Core & NucleusXenomai Skins © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 38 All Rights Reserved.
  39. 39. How Xenomai Works?Real Time ShadowNew Sets of System CallsSharing Kernel Features By Domain MigrationReal Time Driver Model Mediation © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 39 All Rights Reserved.
  40. 40. What all have we learnt?Ws of Real Time & RTOS?Design Methodologies of RTOSLinux → Real Time Linux Various Latencies & Schedulers The “Real Time Patch” Path Real Time Applications Real Time Kernel DebuggingPeek into a Co-Kernel © 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 40 All Rights Reserved.
  41. 41. Any Queries?© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 41 All Rights Reserved.

×