Enea Linux and Light-weight Threading Enea Linux is a customized Linux distribution powered by Yocto that is tailored for telecom and networking applications. It includes over 150 curated packages and supports hardware from various vendors. Enea also introduces Light-weight Runtime Threading (LWRT), which partitions the system into real-time and non-real-time domains to improve performance. LWRT runs most processes in user-space with an optimized scheduler for lower latency compared to standard pthreads. LWRT provides benefits like determinism without depending on the PREEMPT_RT patch and allows both POSIX and real-time APIs.

1. Enea Linux and Light-weight Threading
Real-time Linux for Next Generation Networking Infrastructure
Light-weight Run-time Threading for improved real-time behavior
Michael Christofferson
Director Marketing, Networking
FTF China, 2012

2. Enea Linux
A flexible approach to embedded Linux based on Enea’s 25+ year experience
with Tier 1’s and many others in the global telecom market
 Enea Linux experience
– Linux based solutions for last 6 years – Enea LINX, Enea Element, Enea
dSPEED, Enea BMPT, Linux Competence Center and Android (services)
– And with major silicon vendor Linux’s : Freescale, NetLogic, Cavium,
ARM, …
– And 3PP integrated Linux solutions – MontaVista, TimeSys, …
Q1 2012
But until now, all based on “other” Linux distros …
 Enea Linux is powered by Yocto open source configuration and
build technology
 Custom tailored solutions that address the specific challenges in
the telecom and networking space
– Linux packages that “apply”
– “Hardened” Linux solution for customer’s platform and rqmts
Q3 2012

3. Why Enea Linux?

4. Enea Linux uses Yocto
Yocto is not an embedded Linux distribution, it creates a custom ones. The Yocto
Project™ is an open source collaboration project that provides templates, tools
and methods to help create custom Linux-based systems for embedded products
regardless of the hardware architecture.
 Yocto automates how source is fetched from a Yocto packages
variety of upstream sources or from local
project repositories. Updating to a new version
of a Linux package is easy. Linux
tools Enea packages
 Yocto features a powerful customization Yocto
architecture that allows the choice of a wide
variety of footprint sizes as well as control over Linux Hardened linux
the choice or absence of components such as runtime packages
graphics subsystems, visualization middleware,
networking, and other services.
HW vendor
packages

5. Enea Linux Packages - Customization
 Enea Linux now has over 150 packages
that are specially selected for the real
Enea Linux Customer X
time embedded telecom / networking
Available
space
Packages
Middleware
 Expected to grow to around 200
packages in the future Customer Y
 Customer driven
 Supported packages available on
request Customer Z
Customer Only Gets Desired Packages
Integrated and supported on their HW platform

6. Linux Development Environment
 Host support
 Ubuntu, Fedora, Debian and SUSE
 Graphical configuration and build using Openconf planned
 Qemu processor emulation
 Debug, Profiling, Analysis Tools
 Freeze mode (JTAG from 3PP), but Eclipse
 LTTng
 oProfile
 gcov, & gprof
 Valgrind
 More integrated Eclipse based tools planned for future

7. HW Support in Enea Linux
 Freescale
– Now: P2020/2041, P3041, P4080
– Future:
• P5020 (e5500)
• T4240 (e6500)
• More e5500, e6500 support as well as e500v2
 LSI
 Now: ACP34xx (PPC476)
 Intel
 Now: x86
 AMCC
 Now: PPC440
 ARM
 Future: A15

8. Enea Linux “Add-ons”

9. Linux User Space Multi-threading
The Enea “LWRT” Innovation
Light-weight Runtime Threads

10. Polling Question 1
 Do you think that the Linux implementations that you use
now or have studied deliver the kind of real-time
performance and behavior that you need?
 Yes?
 No?
Enea Confidential

11. 3 Ways to Tackle Realtime in Linux:
We can add a realtime kernel We can simply avoid using kernel We can rework the internals of
underneath Linux: functionality in situations that Linux:
causes realtime problems:
User Mode Runtime
Multi-threading
PTREADS
Linux Kernel Linux Kernel Linux Kernel
Realtime Kernel
 Examples includes hypervisor,  We can partition a single Linux  Might require significant changes
Xenomai, RTLinux etc. instance and separate realtime compared to “standard” Linux.
 The end result is two separate from non-realtime.  Most common example is the
systems/environments.  We can configure processes and PREEMPT_RT patch.
 We cannot completely utilize the interrupts to run with core affinity.  Taking 3.0.27 as an example,
Linux eco-system (e.g. tools) in the  We make minor modifications to PREEMPT_RT patches 500+
realtime domain. the kernel to avoid running kernel locations in the kernel, with
threads/timers on realtime cores. 11,500+ new lines of code in total.
 We can avoid using/calling the
kernel, and rely on user-mode
functionality instead.

12. Linux Multi-threading with Pthreads
Pthreads Model
 Requires less overhead than "forking" or spawning
App App
a new process because the system does not
initialize a new system virtual memory space and
environment for the process.
PThread PThread
Pthreads “Under the Hood”
 Threads exist in two separate execution spaces in
Linux — user space and kernel space. User-space
Process
threads are created with the pthread library API
and mapped to kernel threads.
 All thread scheduling therefore must involve a
context switch from user space to kernel space Linux Cores
So the real question is: can Linux multi-threading be done better?

13. Why is Multi-threading Performance an Issue?
Isn’t Linux is “real time”? What about “Bare Metal” Linux?
“Bare Metal” Linux for multicore devices focuses on direct HW
access from user space Bare Metal
Single thread App App
 Achieves “bare metal” via a single thread App
 Good for simple apps like IP Forwarding, transcoding, etc
Multi-threading is the “time honored” embedded solution to
serious real-time problems wherein an application has multiple PThread PThread PThread
purposes
 Add run-time debug and profiling?
 Add higher level networking protocols Process Process
 Add device and run-time configuration and management?
Multi-threading has always been a part of Linux
 Real-time Extensions for Linux have focused on faster and Linux Cores Linux Cores
more deterministic process AND thread scheduling among
other things
Complex protocols and applications require multi-threading for max performance and is
used widely in Linux

14. Polling Question 2
 Is multithreading in Linux a concern for your applications in
terms of performance and/or robustness?
 Yes?
 No?

15. The LWRT Concept
• Better to partition the system into
separate realtime critical an non- “We improve performance
critical domains. and realtime characteristics
• It is often the Linux kernel itself that under Linux by partitioning
introduces realtime problems.
the system into logical
• A combination of partitioning,
combined with a user-mode domains, and by avoid using
environment that allow us to avoid the Linux kernel itself more
using the kernel can improve than necessary”.
performance and realtime
characteristics compared to a
standard Linux.

16. What is LWRT?
• LWRT is a “Light-Weight Runtime (Environment)” built upon Linux.
• LWRT runs in most part in user-space,
and is a essentially a library linked to the application. Application
• The LWRT environment provides better performance
and realtime characteristics compared to the standard LWRT Environment
POSIX/Linux environment.
Linux Kernel
Core Core
0 N

17. How does LWRT work?
LWRT partitions the system into one realtime domain and
one non-realtime domain.
Non-realtime Processes Realtime Processes
LWRT adds a user-mode runtime environment, including
an optimized user-mode scheduler. LWRT Environment
Pthread Pthread
LWRT migrates some specific kernel functionality (e.g.
timers) away from the realtime domain.
Linux Kernel
LWRT adds a kernel module to catch and forward LWRT Kernel
Module
interrupts to the user-mode environment.
Core Core
0 N

18. What are the benefits of LWRT?
LWRT provides low latency and high throughput. LWRT
does not depend on the PREEMPT_RT patch, and does
not affect throughput negatively.
LWRT provides a solution that is unencumbered by
Non-realtime Processes Realtime Processes
GPL, even for interrupt driven code which can be
placed in user-space without any major penalty.
LWRT Environment
LWRT provides optimized APIs for realtime
applications, and allows the same application to use Pthread Pthread
the POSIX/Linux APIs when realtime doesn’t matter.
LWRT provides very good (i.e. low-latency) interrupt
response time, all the way up to user-mode. Linux Kernel
LWRT Kernel
Module
LWRT is an “all-Linux” solution, based on a single Linux
Kernel. Thus, almost all tools from the existing Linux Core Core
ecosystem will be available. 0 N

19. LWRT Services
LWRT Architecture
 Multiple threads per core  Low inter-thread signaling overhead
 Deterministic scheduling  Cheap thread creation
 Low scheduling overhead  Atomic signal/sleep
 Cheap context switches

20. So LWRT is a Different Multi-threading Model
 A Light-Weight Thread Environment (LWT)
implemented in user-space over a single pthread
 A user-mode scheduler in the scope of a Linux process “LWT” “LWT” “LWT”
 - No kernel thread context switch
PThread PThread PThread
 - Preemptive, priority based scheduler
 - 100% Linux, utilizes same Linux tools as rest
of system Linux Process
 - Complete access to normal POSIX/libC
 Support for message-passing between threads (and
processes). With simple buffer management (for Core Core Core
messages), and interrupt handling
So what is the Enea LWRT performance advantage over Pthreads?

21. LWRT vs Pthreads Demo
Core
Start GUI
0
Fork Shared Memory
Spawn Ping
Core Enea LWT LWT
1 Demo Pong
LWT
Spawn Ping
Core PThreads PThread
2 Demo Pong
PThread
 Measurements done on x86 using TSC (time stamp counter)
 Demo HW: AMD Phenom II N620 Dual-Core processor @ 2.8GHz.
 Linux: OpenSuSE 12.1, 32 bit

22. LWRT vs Pthreads - Context Switch Overhead
100000
Pthreads
LWRT
80000
LWRT has much better
60000 performance i.e. lower
scheduling latency
40000
20000
LWRT has much better real-time
characteristics, i.e. less variance.
Clock cycles
0
0 500 1000 1500 2000 2500 3000 3500 4000 4500

23. LWRT vs Pthreads
Inter-thread Communications
 Ping-Pong benchmark
measuring scheduling
time/latency
 Histogram show one-trip
latency LWT example
based on signaling
 Pthread example based on
semaphores
 In the demo, the user can
interactively play with core
affinity and scheduling
policy

24. LWRT vs Pthreads
Scheduling Behavior under System Load
Scheduling Jitter Due to Load

25. Comparing LWRT and PREEMPT_RT
• PREEMPT_RT applies to the entire system, whereas
LWRT partitions the system.
• PREEMPT_RT improves latency at the expense of
throughput. LWRT on the other hand is built upon the
idea that one part of the system might be optimized
for latency, and one part might be more focused on
throughput.
• PREEMPT_RT makes significant modifications to the
Linux kernel (e.g. 500+ locations, 11,500+ lines). LWRT
makes minimal modifications, and tries to use as much
of “standard” kernel as possible.
• PREEMPT_RT assumes that all device drivers have
been adapted to the patch. LWRT puts no special
requirements on device driver.
• LWRT achieves same interrupt latency as PREEMP_RT,
with better end-to-end performance (time)

26. Summary - Why LWRT?
LWRT combines the best of two worlds – Linux and RTOS Characteristics
Linux Linux + LWRT RTOS
A “native” all-Linux solution
A vast ecosystem Determinism
The advantages of the
Extensive HW support Low overhead
RTOS in Linux user-space
Rich toolset Same tools as the rest of the
Fine-grained thread model
Established programming Linux-based system Priority based scheduling
model
Access to the normal
POSIX/libC.
Better performance and real-time characteristics for any multi-threaded application either
control plane or data plane

27. More Good News
Remember the three ways to boost Linux Real-time
Performance?
Enea Can Support All of Them:
A realtime kernel based on the Enea Prototypes of LWRT show very good We can offer PREEMPT_RT as part
Hypervisor offers excellent realtime realtime characteristics with an of Enea Linux:
characteristics: almost unmodified kernel.
LWRT
Linux Kernel Linux Kernel Linux Kernel
Realtime Kernel / Hypervisor
 Examples includes  LWRT doesn’t exclude PREEMPT_RT. These technologies are not
hypervisor, Xenomai, necessarily competing, but may actually be used in combination.
RTLinux etc.
 However, depending on the situation, an LWRT solution might actually
 The end result is two achieve better results than PREEMPT_RT.
separate
systems/environments.
 We cannot completely
utilize the Linux eco-
system (e.g. tools) in the
realtime domain.

28. See us in the Exhibit Hall
for an Enea Linux and LWRT Demo
along with other Enea technology:
- Packet Acceleration
- Linux Base Station Platform
- Hypervisor
- Element HA
Booths 1430-1435