The document discusses the implementation of the Xen hypervisor on ARM architecture for real-time and automotive use cases by EPAM Systems in Ukraine. It outlines the challenges of ensuring real-time performance in virtualized environments, detailing the evaluation of different scheduling methods and configurations for managing critical tasks. The findings emphasize the importance of dedicated CPU pools for isolation and the potential of the RTDS scheduler for resource efficiency, while also identifying areas for future improvements in scheduling time accounting.

1. Real Time in Xen
on ARM
Andrii Anisov, Lead Systems Engineer,
Epam Systems Inc.

2. Who we are
We are a team of developers at EPAM Systems Inc., based in Kyiv, Ukraine
We are focused on:
• Xen on ARM
• Renesas R-Car Gen3 platform support and maintenance in Xen
• Real-Time use-cases
• Automotive use-cases
• Para-virtual drivers and backends: sound, display, GPU
• SoC’s HW virtualization
• TEE integration
• Power management
• Android HALs
• FuSa ISO 61508/26262 certification
• We are members of GENIVI and AGL and pushing for Xen usage there
• Yocto based build system for multi-domain distributions
• We are upstreaming to Xen Project, Linux Kernel and many OSS projects:
• see us at https://github.com/xen-troops

3. Contents
• Introduction
• What is Real Time operation
• Approaches to run RT domain under XEN
• Empirical evaluation
• Ways to improvement
• Conclusions

4. Introduction

5. Motivation
• We are targeting XEN hypervisor usage in an automotive area
• The required is a mixed criticality of the system. What means that
at least some part of the system is treated as safety-critical and
should meet correspondent requirements
• Safety criticality is toughly coupled with fitting real time
requirements

6. Another level of domains isolation
Adding RT requirements to virtualized systems we can treat as
another level of domains isolation:
Activities in an application domain should not hit timing parameters
of tasks in an RT domain.

7. What is Real Time
operation

8. Types of RT tasks
• Periodic
Tasks with specified worst execution time, comes with some period and have
some deadline.
• Aperiodic
Tasks with unpredictable release time is named as aperiodic.
• Sporadic
Aperiodic tasks which has specified deadline and could be accepted or not
accepted by the system.

9. Periodic tasks scheduling algorithms
• Rate monotonic
• Deadline monotonic
• Early deadline first
• ARINC653
• Other

10. Approaches to run RT
domain under XEN

11. Options to run an RT Domain under XEN
• ARINC653 scheduler
• Dedicated CPU-pool
• RTDS scheduler

12. ARINC653 Scheduler
• Is a periodically repeating fixed timeslice scheduler
• Focused on isolation and partitioning of domains
• Does not support multicore function
• Real-time domains running in specified frames should be aware of
their frames parameters and start moment.
• Only periodic real-time tasks could be scheduled in domains
running under ARINC653 scheduler.

13. RTDS scheduler
• Early Deadline First (EDF) algorithm to schedule VCPU
• Supports multicore operation
• Could be configured both for global EDF and partitioned EDF
VCPU scheduling
• Has an option for work conserving

14. Dedicated CPU-pool with NULL scheduler
• Straight-forward approach
• Inflexible in terms of CPUs resources distribution
• XEN does not care to schedule, its up to Domain to meet RT
requirements
• Domain’s RT scheduling algorithm is not limited to any
• Faces minimal impact from other domains

15. Empirical evaluation

16. Evaluation setup
• Renesas R-Car H3 SoC
• CA57 – 4 cores
• CA53 – 4 cores
• Salvator-X board with two HDMI displays connected
• XEN 4.10 with patches
• Four domains:
• Generic ARMv8 Dom0
• Platform specific driver domain DomD
• Android domain DomA
• Real time domain DomR

17. Evaluation setup: Dom0
• Generic ARMv8 Linux
• Initramfs
• Assigned 4 VCPUs
The purpose of this domain it to run and control other domains only.

18. Evaluation setup: DomD
• Linux with platform drivers
• Runs PV NET, Block, Display and Sound backends
• Sharing GPU with DomA
• Running QT based UI
• Running network services
• Assigned 4 VCPUs
This domain runs hardware drivers, providing them to other
domains, as well as does some part of infotainment functionality.

19. Evaluation setup: DomA
• Android running with PV Block, NET, Display and Sound drivers
• Running platform specific driver for GPU shared with DomD
• With two displays
• Assigned 4 VCPUS
This domain intention is pure infotainment functionality

20. Evaluation setup: DomR
• Generic ARMv8 Linux with LITMUS-RT [1]
• Initramfs
• PV NET driver
This domain provides simple loop based CPU-bound workload
(rtspin) for real-time task emulation and infrastructure for tracing its
runtime characteris (execution time, meeting deadlines, etc).
[1] http://www.litmus-rt.org/index.html

21. Evaluating CPU-pool with NULL scheduler
The approach to evaluate CPU-pool is pretty trivial: just inspect how
activities in application domains affect execution time of some
specific task.
Here we use an rtspin utility, but in a simple loop calibration mode -
interactively observing if calibration error changes depending on
activities in application domains.

22. Evaluating CPU-pool: results
With the CPU-pool set up on the same CPU cluster as application
domains the error is seen up to 4%. This is treated as an impact of
caches sharing with application domains.
With the CPU-pool set up on the other CPU cluster – error is less
than 1%. This is treated as bus contentions between separate CPU
clusters.

23. Evaluating RTDS scheduler
For an RTDS scheduler evaluation setup configuration followed a
correspondent research paper [1]. The idea of this scheduler usage
is to calculate a required RT domain VCPUs parameters in
correspondence to the RT tasks parameters running in this domain.
Such calculations are performed using CARTs tool [2] which
implements compositional scheduling analysis.
[1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf
[2] https://rtg.cis.upenn.edu/carts/

24. Tested RT task and domain configurations
Task p=d=10ms, e=4ms; VCPU p=d=10ms, e=7ms (referred as tp10-
vp10000e7000)
Task p=d=10ms, e=4ms; VCPU p=d=5ms e=3ms (referred as tp10-
vp5000e3000)
Task p=d=10ms, e=4ms; VCPU p=d=2ms e=1ms (referred as tp10-
vp2000e1000)
Task p=d=10ms, e=4ms; VCPU p=d=1ms e=445 us(referred as tp10-
vp1000e445)
Task p=d=100ms, e=40ms; VCPU p=d=10ms e=4445 us (referred as tp100-
vp10000e4445)

25. Tested application domains configurations
• System is idle
• System is load with CPU bound tasks
• System does idle IO
• System is CPU loaded, and idle IO
• System is busy with heavy IO and CPU load

26. Evaluating RTDS: results
Idle CPU bound Idle IO CPU + idle
IO
CPU +
heavy IO
tp10-vp10000e7000 No DL miss No DL miss No DL miss No DL miss DL miss
tp10-vp5000e3000 No DL miss No DL miss No DL miss No DL miss DL miss
tp10-vp2000e1000 No DL miss No DL miss DL miss DL miss DL miss
tp10-vp1000e445 No DL miss No DL miss DL miss DL miss DL miss
tp100-vp10000e4445 No DL miss No DL miss No DL miss No DL miss DL miss

27. Evaluating RTDS: what’s going on?
From the experiment results we can see that the missing deadlines
correlates with amount of IO activity in the system and extra time
provided to VCPU in RT domain. But does not correlate with VCPU
load in domains.

28. Evaluating RTDS: how many IRQs?
Idle CPU bound Idle IO CPU + idle IO CPU + heavy IO
tp10-vp10000e7000 No DL miss No DL miss No DL miss No DL miss DL miss
tp10-vp5000e3000 No DL miss No DL miss No DL miss No DL miss DL miss
tp10-vp2000e1000 No DL miss No DL miss DL miss DL miss DL miss
tp10-vp1000e445 No DL miss No DL miss DL miss DL miss DL miss
tp100-vp10000e4445 No DL miss No DL miss No DL miss No DL miss DL miss
IRQ/sec ~300 ~300 ~400 ~600 ~14000
IPI/sec ~800 ~100 ~2800 ~2300 ~10000

29. What is the real VCPU run time?
So I take following time as the VCPU run time:
• VCPU start time before exiting EL2 (in leave hypervisor_tail())
• VCPU stop time right after entering EL2 (in
enter_hypevisor_head())
• With addition of time to serve guests synchronous exceptions (hvc,
mmio traps)
https://github.com/aanisov/xen/commits/time_accounting

30. What is the real VCPU run time?
So what can we see in a result:
$ xl vcpu-list
Name ID VCPU CPU State Time(s) Real run SYNC IRQ SCHEDULE BEFORE AFTER Affinity
(Hard / Soft)
Domain-0 0 0 7 -b- 756.7 413.1 12.6 1.9 8.9 280.9 747.5 4-7 / all
Domain-0 0 1 4 -b- 642.6 420.8 12.4 2.0 9.4 180.4 428.0 4-7 / all
Domain-0 0 2 5 r-- 665.3 429.0 27.3 2.3 9.8 179.6 464.6 4-7 / all
Domain-0 0 3 6 -b- 645.3 422.2 12.4 2.2 9.4 179.0 566.1 4-7 / all
DomD 1 0 5 r-- 33230.9 27548.7 992.5 1290.5 119.9 2334.7 1695.9 all / all
DomD 1 1 3 -b- 21178.3 19755.0 290.8 180.1 56.5 696.1 733.1 all / all
DomD 1 2 0 r-- 40795.0 40035.6 15.3 210.8 17.9 288.9 422.9 all / all
DomD 1 3 2 r-- 22421.4 20953.7 291.4 178.1 57.0 727.5 734.8 all / all
DomR 2 0 2 -b- 1250.9 912.0 25.1 5.2 18.2 293.1 58.6 all / all
DomA 3 0 3 -b- 21941.5 18594.7 911.4 189.3 85.6 1609.1 1487.1 all / all
DomA 3 1 4 r-- 22080.8 19371.6 768.4 186.3 79.9 1324.7 1236.7 all / all
DomA 3 2 2 -b- 19332.6 16477.6 822.3 152.2 82.8 1443.3 1320.5 all / all
DomA 3 3 5 r-- 19300.0 16450.6 828.7 155.2 81.7 1433.2 1304.6 all / all
Up to 15% overcharging of VPU runtime is possible.

31. Evaluating RTDS: Fixing time accounting issue
With making scheduling decision in RTDS scheduler based on real
VCPU run time improves results for RT loads, but does not
eliminates DL misses for all tested use cases:
Idle CPU bound Idle IO CPU + idle IO CPU + heavy IO
tp10-vp10000e7000 No DL miss No DL miss No DL miss No DL miss No DL miss
tp10-vp5000e3000 No DL miss No DL miss No DL miss No DL miss No DL miss
tp10-vp2000e1000 No DL miss No DL miss No DL miss DL miss DL miss
tp10-vp1000e445 No DL miss No DL miss No DL miss DL miss DL miss
tp100-vp10000e4445 No DL miss No DL miss No DL miss No DL miss No DL miss

32. Conclusion

33. Current status: Dedicated CPU-pool
• Provides best isolation for critical domains, though still affected by
architecture specific cache sharing effects and bus contentions
• Critical domain internals agnostic
But:
• Is not flexible in terms of CPU resources utilization

34. Current status: RTDS scheduler
• Has the best potential for CPU resources conserving
• Is able to reliably schedule RT and non-critical domains together
But:
• Is limited for EDF scheduling algorithm in an RT domain
• Requires the knowledge about RT domain tasks configuration to
be shared to virtualization system for proper configuration
• Suffers from scheduling time accounting issue

35. What’s next?
• Improve scheduling time accounting
• Investigate a possibility for additional restrictions for RT domains
scheduling with RTDS

36. Questions?

37. Thank you!