Enea Linux and Light-weight ThreadingReal-time Linux for Next Generation Networking InfrastructureLight-weight Run-time Threading for improved real-time behavior Michael Christofferson Director Marketing, Networking FTF China, 2012
Enea Linux A flexible approach to embedded Linux based on Enea’s 25+ year experience with Tier 1’s and many others in the global telecom market Enea Linux experience – Linux based solutions for last 6 years – Enea LINX, Enea Element, Enea dSPEED, Enea BMPT, Linux Competence Center and Android (services) – And with major silicon vendor Linux’s : Freescale, NetLogic, Cavium, ARM, … – And 3PP integrated Linux solutions – MontaVista, TimeSys, … Q1 2012 But until now, all based on “other” Linux distros … Enea Linux is powered by Yocto open source configuration and build technology Custom tailored solutions that address the specific challenges in the telecom and networking space – Linux packages that “apply” – “Hardened” Linux solution for customer’s platform and rqmts Q3 2012
Enea Linux uses YoctoYocto is not an embedded Linux distribution, it creates a custom ones. The YoctoProject™ is an open source collaboration project that provides templates, toolsand methods to help create custom Linux-based systems for embedded productsregardless of the hardware architecture. Yocto automates how source is fetched from a Yocto packages variety of upstream sources or from local project repositories. Updating to a new version of a Linux package is easy. Linux tools Enea packages Yocto features a powerful customization Yocto architecture that allows the choice of a wide variety of footprint sizes as well as control over Linux Hardened linux the choice or absence of components such as runtime packages graphics subsystems, visualization middleware, networking, and other services. HW vendor packages
Enea Linux Packages - Customization Enea Linux now has over 150 packages that are specially selected for the real Enea Linux Customer X time embedded telecom / networking Available space Packages Middleware Expected to grow to around 200 packages in the future Customer Y Customer driven Supported packages available on request Customer Z Customer Only Gets Desired Packages Integrated and supported on their HW platform
Linux Development Environment Host support Ubuntu, Fedora, Debian and SUSE Graphical configuration and build using Openconf planned Qemu processor emulation Debug, Profiling, Analysis Tools Freeze mode (JTAG from 3PP), but Eclipse LTTng oProfile gcov, & gprof Valgrind More integrated Eclipse based tools planned for future
HW Support in Enea Linux Freescale – Now: P2020/2041, P3041, P4080 – Future: • P5020 (e5500) • T4240 (e6500) • More e5500, e6500 support as well as e500v2 LSI Now: ACP34xx (PPC476) Intel Now: x86 AMCC Now: PPC440 ARM Future: A15
Linux User Space Multi-threading The Enea “LWRT” Innovation Light-weight Runtime Threads
Polling Question 1 Do you think that the Linux implementations that you use now or have studied deliver the kind of real-time performance and behavior that you need? Yes? No? Enea Confidential
3 Ways to Tackle Realtime in Linux: We can add a realtime kernel We can simply avoid using kernel We can rework the internals of underneath Linux: functionality in situations that Linux: causes realtime problems: User Mode Runtime Multi-threading PTREADS Linux Kernel Linux Kernel Linux Kernel Realtime Kernel Examples includes hypervisor, We can partition a single Linux Might require significant changes Xenomai, RTLinux etc. instance and separate realtime compared to “standard” Linux. The end result is two separate from non-realtime. Most common example is the systems/environments. We can configure processes and PREEMPT_RT patch. We cannot completely utilize the interrupts to run with core affinity. Taking 3.0.27 as an example, Linux eco-system (e.g. tools) in the We make minor modifications to PREEMPT_RT patches 500+ realtime domain. the kernel to avoid running kernel locations in the kernel, with threads/timers on realtime cores. 11,500+ new lines of code in total. We can avoid using/calling the kernel, and rely on user-mode functionality instead.
Linux Multi-threading with PthreadsPthreads Model Requires less overhead than "forking" or spawning App App a new process because the system does not initialize a new system virtual memory space and environment for the process. PThread PThreadPthreads “Under the Hood” Threads exist in two separate execution spaces in Linux — user space and kernel space. User-space Process threads are created with the pthread library API and mapped to kernel threads. All thread scheduling therefore must involve a context switch from user space to kernel space Linux Cores So the real question is: can Linux multi-threading be done better?
Why is Multi-threading Performance an Issue? Isn’t Linux is “real time”? What about “Bare Metal” Linux? “Bare Metal” Linux for multicore devices focuses on direct HW access from user space Bare Metal Single thread App App Achieves “bare metal” via a single thread App Good for simple apps like IP Forwarding, transcoding, etc Multi-threading is the “time honored” embedded solution to serious real-time problems wherein an application has multiple PThread PThread PThread purposes Add run-time debug and profiling? Add higher level networking protocols Process Process Add device and run-time configuration and management? Multi-threading has always been a part of Linux Real-time Extensions for Linux have focused on faster and Linux Cores Linux Cores more deterministic process AND thread scheduling among other thingsComplex protocols and applications require multi-threading for max performance and is used widely in Linux
Polling Question 2 Is multithreading in Linux a concern for your applications in terms of performance and/or robustness? Yes? No?
The LWRT Concept• Better to partition the system into separate realtime critical an non- “We improve performance critical domains. and realtime characteristics• It is often the Linux kernel itself that under Linux by partitioning introduces realtime problems. the system into logical• A combination of partitioning, combined with a user-mode domains, and by avoid using environment that allow us to avoid the Linux kernel itself more using the kernel can improve than necessary”. performance and realtime characteristics compared to a standard Linux.
What is LWRT?• LWRT is a “Light-Weight Runtime (Environment)” built upon Linux.• LWRT runs in most part in user-space, and is a essentially a library linked to the application. Application• The LWRT environment provides better performance and realtime characteristics compared to the standard LWRT Environment POSIX/Linux environment. Linux Kernel Core Core 0 N
How does LWRT work?LWRT partitions the system into one realtime domain and one non-realtime domain. Non-realtime Processes Realtime Processes LWRT adds a user-mode runtime environment, including an optimized user-mode scheduler. LWRT Environment Pthread Pthread LWRT migrates some specific kernel functionality (e.g. timers) away from the realtime domain. Linux Kernel LWRT adds a kernel module to catch and forward LWRT Kernel Module interrupts to the user-mode environment. Core Core 0 N
What are the benefits of LWRT?LWRT provides low latency and high throughput. LWRTdoes not depend on the PREEMPT_RT patch, and doesnot affect throughput negatively.LWRT provides a solution that is unencumbered by Non-realtime Processes Realtime ProcessesGPL, even for interrupt driven code which can beplaced in user-space without any major penalty. LWRT EnvironmentLWRT provides optimized APIs for realtimeapplications, and allows the same application to use Pthread Pthreadthe POSIX/Linux APIs when realtime doesn’t matter.LWRT provides very good (i.e. low-latency) interruptresponse time, all the way up to user-mode. Linux Kernel LWRT Kernel ModuleLWRT is an “all-Linux” solution, based on a single LinuxKernel. Thus, almost all tools from the existing Linux Core Coreecosystem will be available. 0 N
So LWRT is a Different Multi-threading Model A Light-Weight Thread Environment (LWT) implemented in user-space over a single pthread A user-mode scheduler in the scope of a Linux process “LWT” “LWT” “LWT” - No kernel thread context switch PThread PThread PThread - Preemptive, priority based scheduler - 100% Linux, utilizes same Linux tools as rest of system Linux Process - Complete access to normal POSIX/libC Support for message-passing between threads (and processes). With simple buffer management (for Core Core Core messages), and interrupt handling So what is the Enea LWRT performance advantage over Pthreads?
LWRT vs Pthreads Demo Core Start GUI 0 Fork Shared Memory Spawn Ping Core Enea LWT LWT 1 Demo Pong LWT Spawn Ping Core PThreads PThread 2 Demo Pong PThread Measurements done on x86 using TSC (time stamp counter) Demo HW: AMD Phenom II N620 Dual-Core processor @ 2.8GHz. Linux: OpenSuSE 12.1, 32 bit
LWRT vs Pthreads - Context Switch Overhead100000 Pthreads LWRT 80000 LWRT has much better 60000 performance i.e. lower scheduling latency 40000 20000 LWRT has much better real-time characteristics, i.e. less variance. Clock cycles 0 0 500 1000 1500 2000 2500 3000 3500 4000 4500
LWRT vs PthreadsInter-thread Communications Ping-Pong benchmark measuring scheduling time/latency Histogram show one-trip latency LWT example based on signaling Pthread example based on semaphores In the demo, the user can interactively play with core affinity and scheduling policy
LWRT vs Pthreads Scheduling Behavior under System LoadScheduling Jitter Due to Load
Comparing LWRT and PREEMPT_RT• PREEMPT_RT applies to the entire system, whereas LWRT partitions the system.• PREEMPT_RT improves latency at the expense of throughput. LWRT on the other hand is built upon the idea that one part of the system might be optimized for latency, and one part might be more focused on throughput.• PREEMPT_RT makes significant modifications to the Linux kernel (e.g. 500+ locations, 11,500+ lines). LWRT makes minimal modifications, and tries to use as much of “standard” kernel as possible.• PREEMPT_RT assumes that all device drivers have been adapted to the patch. LWRT puts no special requirements on device driver.• LWRT achieves same interrupt latency as PREEMP_RT, with better end-to-end performance (time)
Summary - Why LWRT?LWRT combines the best of two worlds – Linux and RTOS Characteristics Linux Linux + LWRT RTOS A “native” all-Linux solution A vast ecosystem Determinism The advantages of the Extensive HW support Low overhead RTOS in Linux user-space Rich toolset Same tools as the rest of the Fine-grained thread model Established programming Linux-based system Priority based scheduling model Access to the normal POSIX/libC. Better performance and real-time characteristics for any multi-threaded application either control plane or data plane
More Good News Remember the three ways to boost Linux Real-time Performance? Enea Can Support All of Them:A realtime kernel based on the Enea Prototypes of LWRT show very good We can offer PREEMPT_RT as partHypervisor offers excellent realtime realtime characteristics with an of Enea Linux: characteristics: almost unmodified kernel. LWRT Linux Kernel Linux Kernel Linux Kernel Realtime Kernel / Hypervisor Examples includes LWRT doesn’t exclude PREEMPT_RT. These technologies are not hypervisor, Xenomai, necessarily competing, but may actually be used in combination. RTLinux etc. However, depending on the situation, an LWRT solution might actually The end result is two achieve better results than PREEMPT_RT. separate systems/environments. We cannot completely utilize the Linux eco- system (e.g. tools) in the realtime domain.
See us in the Exhibit Hallfor an Enea Linux and LWRT Demoalong with other Enea technology: - Packet Acceleration - Linux Base Station Platform - Hypervisor - Element HA Booths 1430-1435