Successfully reported this slideshow.
Your SlideShare is downloading. ×

More Related Content

Real Time Systems

  1. 1. Real Time Systems © 2010 Anil Kumar Pugalia <> All Rights Reserved.
  2. 2. What to Expect? W's of Real Time & RTOS? Design Methodologies of RTOS Linux → Real Time Linux Various Latencies & Schedulers The “Real Time Patch” Path Real Time Applications Real Time Kernel Debugging Peek into a Co-Kernel © 2010 Anil Kumar Pugalia <> 2 All Rights Reserved.
  3. 3. Real Time Definition Correctness depends on Functional Accuracy, and Timings of the Result produced Examples MPEG Decoder in your DVD player Call Response in your Mobile Phone © 2010 Anil Kumar Pugalia <> 3 All Rights Reserved.
  4. 4. Real Time Types Hard Real Time Guaranteed to meet the response reqs Examples: Defense Systems, Vehicle Control Systems, Satellite Systems Soft Real Time Once in a while may not meet the response reqs Examples: Multimedia Devices, VoIP, CE devices An intermediate one: Firm Real Time Check: What is our Desktop? © 2010 Anil Kumar Pugalia <> 4 All Rights Reserved.
  5. 5. Is Real Time needed? Deadline of Applications Reduces Quality or Unacceptable External Device Interactions with Apps Handling of Messages – Time-bound or not Application Task Priority Could it be needed higher than OS services Expressing delays & timeouts Is it needed to be fine-grained, say in usecs © 2010 Anil Kumar Pugalia <> 5 All Rights Reserved.
  6. 6. Real Time OS RTOS Phylum Non Real-time Soft Real-time Hard Real-time Conflicting Requirements Real Time vs Rich Features of GPOS © 2010 Anil Kumar Pugalia <> 6 All Rights Reserved.
  7. 7. Expected Features of an RTOS RTOS Requirements Strictly Enforced Task Priorities Fine-grained Preemption External Event Handling in Bounded time frame Which entails Predictable Low Interrupt Latency Predictable Low Scheduler Latency Priority Inheritance But wanted with Rich Features of Multi Tasking / Processing Interprocess Communication & Synchronization © 2010 Anil Kumar Pugalia <> 7 All Rights Reserved.
  8. 8. RTOS Design Methodologies Expand an RTOS Make a GPOS Real Time capable Linux → Real Time Linux (Separate Session) The Co-Kernel Approach RT Apps in User Space Open Source Xenomai (Separate Session) Open Source RTAI RT Apps in Kernel Space RTLinux from Windriver © 2010 Anil Kumar Pugalia <> 8 All Rights Reserved.
  9. 9. Expand an RTOS Difficult to ensure Determinism As all core capabilities must be fully preemptive Drivers for Hardware becomes very complex Existing Software cannot be used without changes High Maintenance Costs for Developers and Customers So, let's understand the other one w.r.t. Linux © 2010 Anil Kumar Pugalia <> 9 All Rights Reserved.
  10. 10. Linux → Real Time Linux © 2010 Anil Kumar Pugalia <> 10 All Rights Reserved.
  11. 11. Why Linux is non Real Time? High Interrupt Latency High Scheduler Latency OS services with indeterministic timing behaviour IPC, Memory Allocation, ... Linux is in general undeterministic in its response Virtual Memory, System Calls, … Later two are more of a better Coding Principles However, the first two are major Design Changes Let's take a closer look © 2010 Anil Kumar Pugalia <> 11 All Rights Reserved.
  12. 12. Real Time related Concepts Latency Interrupt Scheduling Both are related to Preemption And Critical Sections © 2010 Anil Kumar Pugalia <> 12 All Rights Reserved.
  13. 13. Latency Visualization Interrupt ISR Runs ISR Signals RT Process Event RT Process Runs Time t0 t1 t2 t3 Interrupt Interrupt Scheduling Latency Scheduling Latency Interrupt to Process Latency © 2010 Anil Kumar Pugalia <> 13 All Rights Reserved.
  14. 14. Latency Details Why Interrupt Latency? Attributes of ISR Real Time Process Typically, high priority Otherwise, anyway it is a problem All 3 could potentially pose problems For Hard Real Time © 2010 Anil Kumar Pugalia <> 14 All Rights Reserved.
  15. 15. Possible Solutions Key Requirement Schedule RT Process, within a given time Possible Helpers Remove / Reduce interrupt-disabled code (critical sections) Write ISRs really really minimal in execution time Real Time Apps with the highest priority May be higher than interrupts, softirqs, etc © 2010 Anil Kumar Pugalia <> 15 All Rights Reserved.
  16. 16. Preemption Visualization Time User Space System Call Interface Kernel Space Process A Process B © 2010 Anil Kumar Pugalia <> 16 All Rights Reserved.
  17. 17. Preemption Details Higher Preemption makes it more Real Time feasible Higher Priority (RT Tasks) could be scheduled in a more predictable time Sources of Preemption Latency Critical Section Processing Interrupt Context Processing © 2010 Anil Kumar Pugalia <> 17 All Rights Reserved.
  18. 18. Preemption & Critical Sections Higher Preemption at the cost of reducing Critical Sections Isolate & Minimize the critical sections Protect them by disabling preemption spin_lock / spin_unlock; preempt_disable / preempt_enable Preemption Models for reducing Critical Sections Preempt only safe locations Entry & Exit of system calls, Return from interrupt processing, … Put explicit “allow preempt” code at these places Overall not an excellent approach Preempt everywhere except only the Critical Sections lock-breaking mechanism Getting achieved by instrumenting kernel for latency measurements And by fixing the longest latency code paths © 2010 Anil Kumar Pugalia <> 18 All Rights Reserved.
  19. 19. SMP World & Critical Sections Uniprocessor case needs Protection from Interrupt Processing Exception Processing Multiprocessor with additional Multiple Kernel Threads Posed a bigger need than Real Time for reduced Critical Sections Real Time has gained benefits from SMP development, as well © 2010 Anil Kumar Pugalia <> 19 All Rights Reserved.
  20. 20. Real Time Kernel Patch Critical Section Control is a major step We have achieved good Soft Real Time Performance Since 2.6.12, in single-digit milliseconds On reasonably fast x86 processor But without solving the Interrupt Latency Hard Real Time is not possible Recent Developments have majorly focused on this aspect A major Real Time Kernel Patch evolved Maintained by Ingo Molnar Available @ Could be downloaded using say, ketchup -G And takes care of many more issues © 2010 Anil Kumar Pugalia <> 20 All Rights Reserved.
  21. 21. W's & How's of RT Kernel Patch? Lot's of Questions What all does it contain? How to configure it? What should be used & when? And many more Before answering these, let's understand Schedulers in main-stream Kernel Preemption Support in main-stream Kernel © 2010 Anil Kumar Pugalia <> 21 All Rights Reserved.
  22. 22. Linux Schedulers Provides multi-tasking capabilities by Time Slicing Preemption Based on various task priorities Specified by its scheduling policies Understands the following execution instances Kernel Thread User Process User Thread Linux Basic Scheduling Normal (SCHED_OTHER) – Fairness Scheduling Other Advanced Scheduling supported Round Robin (SCHED_RR) FIFO (SCHED_FIFO) All Schedulers are O(1) © 2010 Anil Kumar Pugalia <> 22 All Rights Reserved.
  23. 23. Linux Kernel Preemption Levels None (PREEMPT_NONE) No forced preemption Overall throughput, on average, is good Voluntary (PREEMPT_VOLUNTARY) First stage of latency reduction Explicit preemption points are placed at strategic locations Standard (PREEMPT_DESKTOP) Preemption enabled everywhere except within critical sections Good for soft real-time applications, like audio, multimedia, … Kernel Parameter: preempt © 2010 Anil Kumar Pugalia <> 23 All Rights Reserved.
  24. 24. Preemption Level from RT Patch Complete (PREEMPT_RT) Spin locks replaced with preembtable mutexes Preemption enabled everywhere except where protected by preempt_disable() Smoothes out variation in latency Allows a low & predictable latency for time- critical real-time applications © 2010 Anil Kumar Pugalia <> 24 All Rights Reserved.
  25. 25. Additional Real Time Features ISRs as Kernel Tasks CONFIG_PREEMPT_HARDIRQ Schedulable & Preemptible Priority Assignment as Required Control Window: /proc/sys/kernel/hardirq_preemption Kernel Parameter: hardirq-preempt Preemptable Softirqs CONFIG_PREEMPT_SOFTIRQ All ran in the context of ksoftirqd → Now as individual threads A proper Linux task configured for real time Control Window: /proc/sys/kernel/softirq_preemption Kernel Parameter: softirq-preempt Certain Read-Copy-Update (RCU) sections preemptible CONFIG_PREEMPT_RCU RCU is synchronization primitive for frequently read data © 2010 Anil Kumar Pugalia <> 25 All Rights Reserved.
  26. 26. Creating a Real Time Process #include <sched.h> #define MY_RT_PRIO 1 int main(...) { int old_policy; struct sched_param params = { .sched_priority = MY_RT_PRIO }; ... old_policy = sched_getscheduler(0); if (sched_setscheduler(0, SCHED_RR, &params) == -1) handle_error(); … } © 2010 Anil Kumar Pugalia <> 26 All Rights Reserved.
  27. 27. RT Application: Do's & Don'ts Have very few Real Time Processes Keep the RT Applications Short & Efficient Do not do heavy duty operations Like Memory Allocation, etc If essential, move to Initialization Sections Check for the working of the needed OS services Especially, if any RT Application has priority higher than that OS Service Test for the Desired Results © 2010 Anil Kumar Pugalia <> 27 All Rights Reserved.
  28. 28. Debugging Real Time Kernel Soft Lockup CONFIG_DETECT_SOFTLOCKUP Unsafe Preemption Use Logging CONFIG_DEBUG_PREEMPT Deadlock Conditions Detection & Reporting CONFIG_DEBUG_DEADLOCK Deadlocks due to semaphores & spinlocks Runtime Control of Locking Mode CONFIG_DEBUG_RT_LOCKING_MODE Mutex back to Spinlock © 2010 Anil Kumar Pugalia <> 28 All Rights Reserved.
  29. 29. Tracing Real Time Kernel Latency Tracing CONFIG_LATENCY_TRACE For data associated with last maximum preemption latency Kernel Window: /proc/latency_trace Function Call Tracing Inserts hooks to every function via gcc Gives function level details through Kernel Window: /proc/latency_trace Soon getting outdated by Ftrace © 2010 Anil Kumar Pugalia <> 29 All Rights Reserved.
  30. 30. Further Tracing RT Kernel High Priority Process Wakeup CONFIG_WAKEUP_TIMING CONFIG_WAKEUP_LATENCY_HIST Or, Interrupt Off Timing CONFIG_CRITICAL_IRQSOFF_TIMING CONFIG_INTERRUPT_OFF_HIST Kernel Windows /proc/sys/kernel/preempt_max_latency /proc/latency_hist/interrupt_off_latency/CPU0 (0-10000us) Time spent with Preemption disabled (in critical sections) CONFIG_CRITICAL_PREEMPT_TIMING CONFIG_PREEMPT_OFF_HIST © 2010 Anil Kumar Pugalia <> 30 All Rights Reserved.
  31. 31. Real Time Linux Myths & Realities With all these understood Is Linux really Hard Real Time? Can it be used in all Real Time Embedded activities? Can it be used in a Nuclear Reactor? Answers Can be used in most Real Time Embedded Applications, such as Robotics But cannot be “one-size-fits-it-all” approach to Real Time Is still a patch and hence not that extensibly tested Not all Device Drivers have yet been verified as Real Time safe More & more users (such as multimedia users), using the patch would verify more drivers, as they are the users noticing longer latency delays caused by these devices But even if they are, it is after all a GPOS Harder to verify a lack of bugs Could have potential bugs (from real time perspective) © 2010 Anil Kumar Pugalia <> 31 All Rights Reserved.
  32. 32. Co-Kernel Approach © 2010 Anil Kumar Pugalia <> 32 All Rights Reserved.
  33. 33. W's of Co-Kernel Approach Do not convert the GPOS into an RTOS Place a small Real Time Kernel besides it On the same Hardware This would be the Co-Kernel Taking care of all Real Time needs Non-RT tasks are done by the GPOS So, © 2010 Anil Kumar Pugalia <> 33 All Rights Reserved.
  34. 34. What is a Co-Kernel? Co-Kernel is a Subsystem Integrated with the main Kernel Handling the Real Time part © 2010 Anil Kumar Pugalia <> 34 All Rights Reserved.
  35. 35. Types of Co-Kernel Approaches Two Prevalent Approaches Support running of RT Apps in User Space RT Apps to be embodied into Kernel Space Our Focus of Discussion is the First One With Xenomai in consideration © 2010 Anil Kumar Pugalia <> 35 All Rights Reserved.
  36. 36. Benefits of Xenomai Co-Kernel or Core running under Kernel, licensed under GPL 2 Kernel Support Both 2.4 & 2.6 for with & without MMU systems Provides generic building blocks to implement RT APIs (skins) RT API Support for VxWorks, pSOS+, VRTX, uITRON, POSIX 1003.1b User Space i/f libraries: LGPL 2.1 Architecture Support PowerPC*, Blackfin, ARM, x86* Excellent Resources Documentation, Technical Articles, … Website: © 2010 Anil Kumar Pugalia <> 36 All Rights Reserved.
  37. 37. Xenomai Architecture User Space User Space User Space User Space Application Application Application Application Linux syscall interface VxWorks pSOS VRTX POSIX ... Kernel-based Skin Skin Skin Skin Applications Abstract RTOS Core SAL / HAL Portability Layers I-Pipe © 2010 Anil Kumar Pugalia <> 37 All Rights Reserved.
  38. 38. Xenomai Components Interrupt Pipeline Hardware & System Abstraction Layers Xenomai Core & Nucleus Xenomai Skins © 2010 Anil Kumar Pugalia <> 38 All Rights Reserved.
  39. 39. How Xenomai Works? Real Time Shadow New Sets of System Calls Sharing Kernel Features By Domain Migration Real Time Driver Model Mediation © 2010 Anil Kumar Pugalia <> 39 All Rights Reserved.
  40. 40. What all have we learnt? W's of Real Time & RTOS? Design Methodologies of RTOS Linux → Real Time Linux Various Latencies & Schedulers The “Real Time Patch” Path Real Time Applications Real Time Kernel Debugging Peek into a Co-Kernel © 2010 Anil Kumar Pugalia <> 40 All Rights Reserved.
  41. 41. Any Queries? © 2010 Anil Kumar Pugalia <> 41 All Rights Reserved.