SlideShare a Scribd company logo
Real time Linux programming
Mark Veltzer
mailto:mark@veltzer.net
Definitions
What is “real time”?
● From wikipedia: “real-time computing (RTC), or reactive
computing, is the study of hardware and software systems that
are subject to a "real-time constraint"— e.g. operational
deadlines from event to system response. Real-time programs
must guarantee response within strict time constraints.[1] Often
real-time response times are understood to be in the order of
milliseconds and sometimes microseconds. In contrast, a non-
real-time system is one that cannot guarantee a response time
in any situation, even if a fast response is the usual result.”
“Real time” hardware
● A hardware may guarantee real time
performance by utilizing a “hardware clock”
which is a crystal which delivers a pulse at a
constant rate.
● The engineers will make sure that the hardware
concludes it's business before the clock strikes
again.
● What you get is a guarantee for how long it
takes you to do a certain operation.
Terms
● “Hard real time” means a piece of software that must meet
time requirements and never miss even a single dead line.
● “Soft real time” is the same but for a system that will
continue to perform even if the deadlines are missed.
● “High performance” is a piece of software that has to
utilize the hardware to the maximum but where each
operation is not constrained.
● A piece of software may fall into more than one category
(think chess playing software).
Designing real time software
●
There are methodologies designed specifically for the design of
real time software such as MASCOT, HOOD, AADL, Ravenscar,
profile and others.
●
These methodologies deal with how you design your system:
decide on numbers of threads or process, the roles of each
process, the priorities of the threads and more.
● They also deal with how to debug a multi-threaded piece of
software that must meat the strict deadlines.
●
We will not discuss them further.
Real time and programming
language
● Real time systems may be written in any
language (including Java and Python contrary
to public opinion).
● Because care is needed to disable non-
deterministic aspects of the programming
language (e.g. garbage collector) then
programmers prefer more deterministic
languages like C and C++.
● No, C++ is not less deterministic than C.
Traditional real time operating
systems
● In traditional real time operating systems the
programmers write threads which are part of the
operating system kernel (operate in the same
space).
● These systems are kept simple by design so as
not to be non deterministic.
● These systems are fast and deterministic.
● They are also very hard to debug once the
software grows because of the single space.
Where latency/non determinism
come from?
● Hardware. Examples can be self monitoring
interrupts, Caches.
● Operating system. Examples can be operating
system threads, swap, long interrupt handlers,
page faults, laziness.
● Programming language. Examples can be
malloc/free(3), garbage collectors.
● The actual software. Examples can be “if”
statements, algorithm bugs, locking.
The art of real time programming
● All this means that to create a real time
system one must master all levels of the
system: from the hardware, through the
operating system, through the programming
language to the software.
● That is also why hard real time programming
is hard.
Linux and real time
Linux and real time
●
Linux was never designed from the beginning to be a real time system and
some would claim that UNIX philosophy is contrary to real time.
● But: Linux has been developing real time capabilities for over a decade.
●
The kernel patch has been written by Ingo Molnar and Thomas Gleixner.
● It is almost fully integrated into the mainline kernel (3.6).
● Is expected to be fully integrated in the next year or so.
●
Is usually applied as a patch to the kernel by the people building the real
time OS for you.
●
See https://rt.wiki.kernel.org/index.php/Main_Page
What does real time Linux give me?
● Interrupts are handled by the kernel in threads with a given priority.
● The ability to run threads at OS interrupt handler priority or higher.
● Hard Real time scheduling - separate from the soft scheduling that user space usually
uses.
● Priority inversion.
● Fine granularity of CPU affinity of user space and operating system threads.
● New real time clock API for better precision.
● Less interrupts for real time threads – the scheduler does not use regular tick
interrupts to check if real time threads need to stop running and be replaced by other
threads.
● Low latency of the kernel (kernel was reviewed and locations where interrupts were
disabled were improved)
● What is the worst time latency? Around 30usec for dedicated systems. More if you put
more drivers, less if you use less.
Page faults and their cost
● In a Linux system going to the kernel is
expensive (1usec region).
● The system is lazy by default and so uses the
MMU to fetch pages it has promised but has not
attached to the process.
● This happens in fetching pages for the
executable, data segment, stack, libraries,
mmap(2) and malloc(3) (because of mmap(2)).
● See examples.
Real time programming on Linux
Monitoring page faults
● The proc filesystem: /proc/[pid]/stat
● From the command line: ps -o min_flt
● Link with libproc.so and use it's API to monitor
your own page faults.
● Use top(1) and configure page faults as one of
the fields you see.
Avoiding page faults
● Stack: allocate a big buffer on the stack and
zero it or call a recursive function a lot.
● Malloc: zero the memory when you get it.
● Mmap: zero the memory or pass a flag to
mmap.
● Data segment: zero the memory.
● For everything including all of the above there
is also mlock(2).
Avoiding latency
● If you want a really clean CPU all to yourself:
– Put your thread on a CPU that does not get
hardware interrupts.
– Do this using the pthread affinity API.
– On embedded systems hardware interrupts
usually go to CPU 0.
– Use cat /proc/interrupts to make sure.
– Tune where interrupts go using echo to /proc/irq/
[irq_number]/smp_affinity
Avoiding latency (cont)
● On larger systems (Redhat,...) you usually have
a /usr/sbin/irqbalance distributing the interrupts
evenly between the cores.
● Either get rid of it (don't run it on boot in
/etc/init.d).
● Or configure it not to send interrupts to the CPU
you want to keep clean (use
IRQBALANCE_BANNED_CPUS in
/etc/default/irqbalance).
Measuring the entire systems
latency
● Is done using cyclictest(1) which part of the rt-
tests package.
● If you don't have it in your system vendors
repository then install it from source
http://git.kernel.org/cgit/linux/kernel/git/clrkwllms/rt-t
● It contains not only cyclicttest(1) but other very
useful tools.

More Related Content

What's hot

Real time operating systems (rtos) concepts 3
Real time operating systems (rtos) concepts 3Real time operating systems (rtos) concepts 3
Real time operating systems (rtos) concepts 3Abu Bakr Ramadan
 
An End to Order (many cores with java, session two)
An End to Order (many cores with java, session two)An End to Order (many cores with java, session two)
An End to Order (many cores with java, session two)Robert Burrell Donkin
 
Real time operating systems (rtos) concepts 9
Real time operating systems (rtos) concepts 9Real time operating systems (rtos) concepts 9
Real time operating systems (rtos) concepts 9Abu Bakr Ramadan
 
RTAI - Earliest Deadline First
RTAI - Earliest Deadline FirstRTAI - Earliest Deadline First
RTAI - Earliest Deadline FirstStefano Bragaglia
 
RTOS- Real Time Operating Systems
RTOS- Real Time Operating Systems RTOS- Real Time Operating Systems
RTOS- Real Time Operating Systems Bayar shahab
 
REAL TIME OPERATING SYSTEM
REAL TIME OPERATING SYSTEMREAL TIME OPERATING SYSTEM
REAL TIME OPERATING SYSTEMprakrutijsh
 
Real time operating system
Real time operating systemReal time operating system
Real time operating systemKhuram Shahzad
 
Real time scheduling - basic concepts
Real time scheduling - basic conceptsReal time scheduling - basic concepts
Real time scheduling - basic conceptsStudent
 
Windows process-scheduling
Windows process-schedulingWindows process-scheduling
Windows process-schedulingTalha Shaikh
 
INTERRUPT LATENCY AND RESPONSE OF THE TASK
INTERRUPT LATENCY AND RESPONSE OF THE TASKINTERRUPT LATENCY AND RESPONSE OF THE TASK
INTERRUPT LATENCY AND RESPONSE OF THE TASKJOLLUSUDARSHANREDDY
 
Real Time Kernels
Real Time KernelsReal Time Kernels
Real Time KernelsArnav Soni
 
Luca Abeni - Real-Time Virtual Machines with Linux and kvm
Luca Abeni - Real-Time Virtual Machines with Linux and kvmLuca Abeni - Real-Time Virtual Machines with Linux and kvm
Luca Abeni - Real-Time Virtual Machines with Linux and kvmlinuxlab_conf
 
Basic functions & types of RTOS ES
Basic functions & types of  RTOS ESBasic functions & types of  RTOS ES
Basic functions & types of RTOS ESJOLLUSUDARSHANREDDY
 
Windows process scheduling presentation
Windows process scheduling presentationWindows process scheduling presentation
Windows process scheduling presentationTalha Shaikh
 

What's hot (20)

Real time operating systems (rtos) concepts 3
Real time operating systems (rtos) concepts 3Real time operating systems (rtos) concepts 3
Real time operating systems (rtos) concepts 3
 
An End to Order
An End to OrderAn End to Order
An End to Order
 
An End to Order (many cores with java, session two)
An End to Order (many cores with java, session two)An End to Order (many cores with java, session two)
An End to Order (many cores with java, session two)
 
Real time operating systems (rtos) concepts 9
Real time operating systems (rtos) concepts 9Real time operating systems (rtos) concepts 9
Real time operating systems (rtos) concepts 9
 
RTDroid_Presentation
RTDroid_PresentationRTDroid_Presentation
RTDroid_Presentation
 
RTAI - Earliest Deadline First
RTAI - Earliest Deadline FirstRTAI - Earliest Deadline First
RTAI - Earliest Deadline First
 
RTOS- Real Time Operating Systems
RTOS- Real Time Operating Systems RTOS- Real Time Operating Systems
RTOS- Real Time Operating Systems
 
REAL TIME OPERATING SYSTEM
REAL TIME OPERATING SYSTEMREAL TIME OPERATING SYSTEM
REAL TIME OPERATING SYSTEM
 
Real time operating system
Real time operating systemReal time operating system
Real time operating system
 
Real time scheduling - basic concepts
Real time scheduling - basic conceptsReal time scheduling - basic concepts
Real time scheduling - basic concepts
 
Windows process-scheduling
Windows process-schedulingWindows process-scheduling
Windows process-scheduling
 
INTERRUPT LATENCY AND RESPONSE OF THE TASK
INTERRUPT LATENCY AND RESPONSE OF THE TASKINTERRUPT LATENCY AND RESPONSE OF THE TASK
INTERRUPT LATENCY AND RESPONSE OF THE TASK
 
RTOS
RTOSRTOS
RTOS
 
RT linux
RT linuxRT linux
RT linux
 
Real Time Kernels
Real Time KernelsReal Time Kernels
Real Time Kernels
 
Real time system tsp
Real time system tspReal time system tsp
Real time system tsp
 
Luca Abeni - Real-Time Virtual Machines with Linux and kvm
Luca Abeni - Real-Time Virtual Machines with Linux and kvmLuca Abeni - Real-Time Virtual Machines with Linux and kvm
Luca Abeni - Real-Time Virtual Machines with Linux and kvm
 
Basic functions & types of RTOS ES
Basic functions & types of  RTOS ESBasic functions & types of  RTOS ES
Basic functions & types of RTOS ES
 
Real time-embedded-system-lec-02
Real time-embedded-system-lec-02Real time-embedded-system-lec-02
Real time-embedded-system-lec-02
 
Windows process scheduling presentation
Windows process scheduling presentationWindows process scheduling presentation
Windows process scheduling presentation
 

Viewers also liked (7)

Gcc
GccGcc
Gcc
 
Volatile
VolatileVolatile
Volatile
 
Gcc opt
Gcc optGcc opt
Gcc opt
 
Effective cplusplus
Effective cplusplusEffective cplusplus
Effective cplusplus
 
Linux logging
Linux loggingLinux logging
Linux logging
 
Streams
StreamsStreams
Streams
 
Multicore
MulticoreMulticore
Multicore
 

Similar to Realtime

Testing real-time Linux. What to test and how
Testing real-time Linux. What to test and how Testing real-time Linux. What to test and how
Testing real-time Linux. What to test and how Chirag Jog
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareQuantum Leaps, LLC
 
Real time system basic concept
Real time system basic conceptReal time system basic concept
Real time system basic conceptMOUMITA GHOSH
 
rtosbyshibu-131026100746-phpapp01.pdf
rtosbyshibu-131026100746-phpapp01.pdfrtosbyshibu-131026100746-phpapp01.pdf
rtosbyshibu-131026100746-phpapp01.pdfreemasajin1
 
Chapter 19 - Real Time Systems
Chapter 19 - Real Time SystemsChapter 19 - Real Time Systems
Chapter 19 - Real Time SystemsWayne Jones Jnr
 
Real Time Operating system (RTOS) - Embedded systems
Real Time Operating system (RTOS) - Embedded systemsReal Time Operating system (RTOS) - Embedded systems
Real Time Operating system (RTOS) - Embedded systemsHariharan Ganesan
 
intro, definitions, basic laws+.pptx
intro, definitions, basic laws+.pptxintro, definitions, basic laws+.pptx
intro, definitions, basic laws+.pptxssuser413a98
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareMiro Samek
 
Four Ways to Improve Linux Performance IEEE Webinar, R2.0
Four Ways to Improve Linux Performance IEEE Webinar, R2.0Four Ways to Improve Linux Performance IEEE Webinar, R2.0
Four Ways to Improve Linux Performance IEEE Webinar, R2.0Michael Christofferson
 

Similar to Realtime (20)

Testing real-time Linux. What to test and how
Testing real-time Linux. What to test and how Testing real-time Linux. What to test and how
Testing real-time Linux. What to test and how
 
Making Linux do Hard Real-time
Making Linux do Hard Real-timeMaking Linux do Hard Real-time
Making Linux do Hard Real-time
 
FreeRTOS introduction
FreeRTOS introductionFreeRTOS introduction
FreeRTOS introduction
 
Making Linux do Hard Real-time
Making Linux do Hard Real-timeMaking Linux do Hard Real-time
Making Linux do Hard Real-time
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
 
Rtos by shibu
Rtos by shibuRtos by shibu
Rtos by shibu
 
Værktøjer udviklet på AAU til analyse af SCJ programmer
Værktøjer udviklet på AAU til analyse af SCJ programmerVærktøjer udviklet på AAU til analyse af SCJ programmer
Værktøjer udviklet på AAU til analyse af SCJ programmer
 
Real time system basic concept
Real time system basic conceptReal time system basic concept
Real time system basic concept
 
rtosbyshibu-131026100746-phpapp01.pdf
rtosbyshibu-131026100746-phpapp01.pdfrtosbyshibu-131026100746-phpapp01.pdf
rtosbyshibu-131026100746-phpapp01.pdf
 
Chapter 19 - Real Time Systems
Chapter 19 - Real Time SystemsChapter 19 - Real Time Systems
Chapter 19 - Real Time Systems
 
Real Time Operating system (RTOS) - Embedded systems
Real Time Operating system (RTOS) - Embedded systemsReal Time Operating system (RTOS) - Embedded systems
Real Time Operating system (RTOS) - Embedded systems
 
Lab3F22.pdf
Lab3F22.pdfLab3F22.pdf
Lab3F22.pdf
 
Os Concepts
Os ConceptsOs Concepts
Os Concepts
 
intro, definitions, basic laws+.pptx
intro, definitions, basic laws+.pptxintro, definitions, basic laws+.pptx
intro, definitions, basic laws+.pptx
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
 
Lecture1
Lecture1Lecture1
Lecture1
 
Rtos 8051
Rtos 8051Rtos 8051
Rtos 8051
 
Autosar Basics hand book_v1
Autosar Basics  hand book_v1Autosar Basics  hand book_v1
Autosar Basics hand book_v1
 
Concept of thread
Concept of threadConcept of thread
Concept of thread
 
Four Ways to Improve Linux Performance IEEE Webinar, R2.0
Four Ways to Improve Linux Performance IEEE Webinar, R2.0Four Ways to Improve Linux Performance IEEE Webinar, R2.0
Four Ways to Improve Linux Performance IEEE Webinar, R2.0
 

Recently uploaded

IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1DianaGray10
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoTAnalytics
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKUXDXConf
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationZilliz
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaRTTS
 
Motion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyMotion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyUXDXConf
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsExpeed Software
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupCatarinaPereira64715
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backElena Simperl
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...CzechDreamin
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
Server-Driven User Interface (SDUI) at Priceline
Server-Driven User Interface (SDUI) at PricelineServer-Driven User Interface (SDUI) at Priceline
Server-Driven User Interface (SDUI) at PricelineUXDXConf
 

Recently uploaded (20)

IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Motion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in TechnologyMotion for AI: Creating Empathy in Technology
Motion for AI: Creating Empathy in Technology
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
In-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT ProfessionalsIn-Depth Performance Testing Guide for IT Professionals
In-Depth Performance Testing Guide for IT Professionals
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Server-Driven User Interface (SDUI) at Priceline
Server-Driven User Interface (SDUI) at PricelineServer-Driven User Interface (SDUI) at Priceline
Server-Driven User Interface (SDUI) at Priceline
 

Realtime

  • 1. Real time Linux programming Mark Veltzer mailto:mark@veltzer.net
  • 3. What is “real time”? ● From wikipedia: “real-time computing (RTC), or reactive computing, is the study of hardware and software systems that are subject to a "real-time constraint"— e.g. operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints.[1] Often real-time response times are understood to be in the order of milliseconds and sometimes microseconds. In contrast, a non- real-time system is one that cannot guarantee a response time in any situation, even if a fast response is the usual result.”
  • 4. “Real time” hardware ● A hardware may guarantee real time performance by utilizing a “hardware clock” which is a crystal which delivers a pulse at a constant rate. ● The engineers will make sure that the hardware concludes it's business before the clock strikes again. ● What you get is a guarantee for how long it takes you to do a certain operation.
  • 5. Terms ● “Hard real time” means a piece of software that must meet time requirements and never miss even a single dead line. ● “Soft real time” is the same but for a system that will continue to perform even if the deadlines are missed. ● “High performance” is a piece of software that has to utilize the hardware to the maximum but where each operation is not constrained. ● A piece of software may fall into more than one category (think chess playing software).
  • 6. Designing real time software ● There are methodologies designed specifically for the design of real time software such as MASCOT, HOOD, AADL, Ravenscar, profile and others. ● These methodologies deal with how you design your system: decide on numbers of threads or process, the roles of each process, the priorities of the threads and more. ● They also deal with how to debug a multi-threaded piece of software that must meat the strict deadlines. ● We will not discuss them further.
  • 7. Real time and programming language ● Real time systems may be written in any language (including Java and Python contrary to public opinion). ● Because care is needed to disable non- deterministic aspects of the programming language (e.g. garbage collector) then programmers prefer more deterministic languages like C and C++. ● No, C++ is not less deterministic than C.
  • 8. Traditional real time operating systems ● In traditional real time operating systems the programmers write threads which are part of the operating system kernel (operate in the same space). ● These systems are kept simple by design so as not to be non deterministic. ● These systems are fast and deterministic. ● They are also very hard to debug once the software grows because of the single space.
  • 9. Where latency/non determinism come from? ● Hardware. Examples can be self monitoring interrupts, Caches. ● Operating system. Examples can be operating system threads, swap, long interrupt handlers, page faults, laziness. ● Programming language. Examples can be malloc/free(3), garbage collectors. ● The actual software. Examples can be “if” statements, algorithm bugs, locking.
  • 10. The art of real time programming ● All this means that to create a real time system one must master all levels of the system: from the hardware, through the operating system, through the programming language to the software. ● That is also why hard real time programming is hard.
  • 12. Linux and real time ● Linux was never designed from the beginning to be a real time system and some would claim that UNIX philosophy is contrary to real time. ● But: Linux has been developing real time capabilities for over a decade. ● The kernel patch has been written by Ingo Molnar and Thomas Gleixner. ● It is almost fully integrated into the mainline kernel (3.6). ● Is expected to be fully integrated in the next year or so. ● Is usually applied as a patch to the kernel by the people building the real time OS for you. ● See https://rt.wiki.kernel.org/index.php/Main_Page
  • 13. What does real time Linux give me? ● Interrupts are handled by the kernel in threads with a given priority. ● The ability to run threads at OS interrupt handler priority or higher. ● Hard Real time scheduling - separate from the soft scheduling that user space usually uses. ● Priority inversion. ● Fine granularity of CPU affinity of user space and operating system threads. ● New real time clock API for better precision. ● Less interrupts for real time threads – the scheduler does not use regular tick interrupts to check if real time threads need to stop running and be replaced by other threads. ● Low latency of the kernel (kernel was reviewed and locations where interrupts were disabled were improved) ● What is the worst time latency? Around 30usec for dedicated systems. More if you put more drivers, less if you use less.
  • 14. Page faults and their cost ● In a Linux system going to the kernel is expensive (1usec region). ● The system is lazy by default and so uses the MMU to fetch pages it has promised but has not attached to the process. ● This happens in fetching pages for the executable, data segment, stack, libraries, mmap(2) and malloc(3) (because of mmap(2)). ● See examples.
  • 16. Monitoring page faults ● The proc filesystem: /proc/[pid]/stat ● From the command line: ps -o min_flt ● Link with libproc.so and use it's API to monitor your own page faults. ● Use top(1) and configure page faults as one of the fields you see.
  • 17. Avoiding page faults ● Stack: allocate a big buffer on the stack and zero it or call a recursive function a lot. ● Malloc: zero the memory when you get it. ● Mmap: zero the memory or pass a flag to mmap. ● Data segment: zero the memory. ● For everything including all of the above there is also mlock(2).
  • 18. Avoiding latency ● If you want a really clean CPU all to yourself: – Put your thread on a CPU that does not get hardware interrupts. – Do this using the pthread affinity API. – On embedded systems hardware interrupts usually go to CPU 0. – Use cat /proc/interrupts to make sure. – Tune where interrupts go using echo to /proc/irq/ [irq_number]/smp_affinity
  • 19. Avoiding latency (cont) ● On larger systems (Redhat,...) you usually have a /usr/sbin/irqbalance distributing the interrupts evenly between the cores. ● Either get rid of it (don't run it on boot in /etc/init.d). ● Or configure it not to send interrupts to the CPU you want to keep clean (use IRQBALANCE_BANNED_CPUS in /etc/default/irqbalance).
  • 20. Measuring the entire systems latency ● Is done using cyclictest(1) which part of the rt- tests package. ● If you don't have it in your system vendors repository then install it from source http://git.kernel.org/cgit/linux/kernel/git/clrkwllms/rt-t ● It contains not only cyclicttest(1) but other very useful tools.