Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.

100 views

Published on

Currently, several initiatives promote XEN hypervisor into the automotive area as a base of complex virtualized systems. To support those initiatives and plunge into the automotive world XEN should fit at least two requirements: it should be appropriately certified and to be able to host a security domain. Leaving behind certification topic, here we focus on security domain hosting capability of XEN. Particularly on keeping RT guarantees for the specific domain.

This talk is a presentation of the investigation on a XEN hypervisor applicability to building a multi-OS system with real-time guarantees being kept for one of the hosted OSes.

During this presentation following topics would be outlined:
- experimental setup
- experimental use-cases and their motivation
- received results and discovered issues
- solutions and mitigation measures for discovered issues

Published in: Technology
  • Be the first to comment

  • Be the first to like this

XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.

  1. 1. Real Time in Xen on ARM Andrii Anisov, Lead Systems Engineer, Epam Systems Inc.
  2. 2. Who we are We are a team of developers at EPAM Systems Inc., based in Kyiv, Ukraine We are focused on: • Xen on ARM • Renesas R-Car Gen3 platform support and maintenance in Xen • Real-Time use-cases • Automotive use-cases • Para-virtual drivers and backends: sound, display, GPU • SoC’s HW virtualization • TEE integration • Power management • Android HALs • FuSa ISO 61508/26262 certification • We are members of GENIVI and AGL and pushing for Xen usage there • Yocto based build system for multi-domain distributions • We are upstreaming to Xen Project, Linux Kernel and many OSS projects: • see us at https://github.com/xen-troops
  3. 3. Contents • Introduction • What is Real Time operation • Approaches to run RT domain under XEN • Empirical evaluation • Ways to improvement • Conclusions
  4. 4. Introduction
  5. 5. Motivation • We are targeting XEN hypervisor usage in an automotive area • The required is a mixed criticality of the system. What means that at least some part of the system is treated as safety-critical and should meet correspondent requirements • Safety criticality is toughly coupled with fitting real time requirements
  6. 6. Another level of domains isolation Adding RT requirements to virtualized systems we can treat as another level of domains isolation: Activities in an application domain should not hit timing parameters of tasks in an RT domain.
  7. 7. What is Real Time operation
  8. 8. Types of RT tasks • Periodic Tasks with specified worst execution time, comes with some period and have some deadline. • Aperiodic Tasks with unpredictable release time is named as aperiodic. • Sporadic Aperiodic tasks which has specified deadline and could be accepted or not accepted by the system.
  9. 9. Periodic tasks scheduling algorithms • Rate monotonic • Deadline monotonic • Early deadline first • ARINC653 • Other
  10. 10. Approaches to run RT domain under XEN
  11. 11. Options to run an RT Domain under XEN • ARINC653 scheduler • Dedicated CPU-pool • RTDS scheduler
  12. 12. ARINC653 Scheduler • Is a periodically repeating fixed timeslice scheduler • Focused on isolation and partitioning of domains • Does not support multicore function • Real-time domains running in specified frames should be aware of their frames parameters and start moment. • Only periodic real-time tasks could be scheduled in domains running under ARINC653 scheduler.
  13. 13. RTDS scheduler • Early Deadline First (EDF) algorithm to schedule VCPU • Supports multicore operation • Could be configured both for global EDF and partitioned EDF VCPU scheduling • Has an option for work conserving
  14. 14. Dedicated CPU-pool with NULL scheduler • Straight-forward approach • Inflexible in terms of CPUs resources distribution • XEN does not care to schedule, its up to Domain to meet RT requirements • Domain’s RT scheduling algorithm is not limited to any • Faces minimal impact from other domains
  15. 15. Empirical evaluation
  16. 16. Evaluation setup • Renesas R-Car H3 SoC • CA57 – 4 cores • CA53 – 4 cores • Salvator-X board with two HDMI displays connected • XEN 4.10 with patches • Four domains: • Generic ARMv8 Dom0 • Platform specific driver domain DomD • Android domain DomA • Real time domain DomR
  17. 17. Evaluation setup: Dom0 • Generic ARMv8 Linux • Initramfs • Assigned 4 VCPUs The purpose of this domain it to run and control other domains only.
  18. 18. Evaluation setup: DomD • Linux with platform drivers • Runs PV NET, Block, Display and Sound backends • Sharing GPU with DomA • Running QT based UI • Running network services • Assigned 4 VCPUs This domain runs hardware drivers, providing them to other domains, as well as does some part of infotainment functionality.
  19. 19. Evaluation setup: DomA • Android running with PV Block, NET, Display and Sound drivers • Running platform specific driver for GPU shared with DomD • With two displays • Assigned 4 VCPUS This domain intention is pure infotainment functionality
  20. 20. Evaluation setup: DomR • Generic ARMv8 Linux with LITMUS-RT [1] • Initramfs • PV NET driver This domain provides simple loop based CPU-bound workload (rtspin) for real-time task emulation and infrastructure for tracing its runtime characteris (execution time, meeting deadlines, etc). [1] http://www.litmus-rt.org/index.html
  21. 21. Evaluating CPU-pool with NULL scheduler The approach to evaluate CPU-pool is pretty trivial: just inspect how activities in application domains affect execution time of some specific task. Here we use an rtspin utility, but in a simple loop calibration mode - interactively observing if calibration error changes depending on activities in application domains.
  22. 22. Evaluating CPU-pool: results With the CPU-pool set up on the same CPU cluster as application domains the error is seen up to 4%. This is treated as an impact of caches sharing with application domains. With the CPU-pool set up on the other CPU cluster – error is less than 1%. This is treated as bus contentions between separate CPU clusters.
  23. 23. Evaluating RTDS scheduler For an RTDS scheduler evaluation setup configuration followed a correspondent research paper [1]. The idea of this scheduler usage is to calculate a required RT domain VCPUs parameters in correspondence to the RT tasks parameters running in this domain. Such calculations are performed using CARTs tool [2] which implements compositional scheduling analysis. [1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf [2] https://rtg.cis.upenn.edu/carts/
  24. 24. Tested RT task and domain configurations Task p=d=10ms, e=4ms; VCPU p=d=10ms, e=7ms (referred as tp10- vp10000e7000) Task p=d=10ms, e=4ms; VCPU p=d=5ms e=3ms (referred as tp10- vp5000e3000) Task p=d=10ms, e=4ms; VCPU p=d=2ms e=1ms (referred as tp10- vp2000e1000) Task p=d=10ms, e=4ms; VCPU p=d=1ms e=445 us(referred as tp10- vp1000e445) Task p=d=100ms, e=40ms; VCPU p=d=10ms e=4445 us (referred as tp100- vp10000e4445)
  25. 25. Tested application domains configurations • System is idle • System is load with CPU bound tasks • System does idle IO • System is CPU loaded, and idle IO • System is busy with heavy IO and CPU load
  26. 26. Evaluating RTDS: results Idle CPU bound Idle IO CPU + idle IO CPU + heavy IO tp10-vp10000e7000 No DL miss No DL miss No DL miss No DL miss DL miss tp10-vp5000e3000 No DL miss No DL miss No DL miss No DL miss DL miss tp10-vp2000e1000 No DL miss No DL miss DL miss DL miss DL miss tp10-vp1000e445 No DL miss No DL miss DL miss DL miss DL miss tp100-vp10000e4445 No DL miss No DL miss No DL miss No DL miss DL miss
  27. 27. Evaluating RTDS: what’s going on? From the experiment results we can see that the missing deadlines correlates with amount of IO activity in the system and extra time provided to VCPU in RT domain. But does not correlate with VCPU load in domains.
  28. 28. Evaluating RTDS: how many IRQs? Idle CPU bound Idle IO CPU + idle IO CPU + heavy IO tp10-vp10000e7000 No DL miss No DL miss No DL miss No DL miss DL miss tp10-vp5000e3000 No DL miss No DL miss No DL miss No DL miss DL miss tp10-vp2000e1000 No DL miss No DL miss DL miss DL miss DL miss tp10-vp1000e445 No DL miss No DL miss DL miss DL miss DL miss tp100-vp10000e4445 No DL miss No DL miss No DL miss No DL miss DL miss IRQ/sec ~300 ~300 ~400 ~600 ~14000 IPI/sec ~800 ~100 ~2800 ~2300 ~10000
  29. 29. What is the real VCPU run time? So I take following time as the VCPU run time: • VCPU start time before exiting EL2 (in leave hypervisor_tail()) • VCPU stop time right after entering EL2 (in enter_hypevisor_head()) • With addition of time to serve guests synchronous exceptions (hvc, mmio traps) https://github.com/aanisov/xen/commits/time_accounting
  30. 30. What is the real VCPU run time? So what can we see in a result: $ xl vcpu-list Name ID VCPU CPU State Time(s) Real run SYNC IRQ SCHEDULE BEFORE AFTER Affinity (Hard / Soft) Domain-0 0 0 7 -b- 756.7 413.1 12.6 1.9 8.9 280.9 747.5 4-7 / all Domain-0 0 1 4 -b- 642.6 420.8 12.4 2.0 9.4 180.4 428.0 4-7 / all Domain-0 0 2 5 r-- 665.3 429.0 27.3 2.3 9.8 179.6 464.6 4-7 / all Domain-0 0 3 6 -b- 645.3 422.2 12.4 2.2 9.4 179.0 566.1 4-7 / all DomD 1 0 5 r-- 33230.9 27548.7 992.5 1290.5 119.9 2334.7 1695.9 all / all DomD 1 1 3 -b- 21178.3 19755.0 290.8 180.1 56.5 696.1 733.1 all / all DomD 1 2 0 r-- 40795.0 40035.6 15.3 210.8 17.9 288.9 422.9 all / all DomD 1 3 2 r-- 22421.4 20953.7 291.4 178.1 57.0 727.5 734.8 all / all DomR 2 0 2 -b- 1250.9 912.0 25.1 5.2 18.2 293.1 58.6 all / all DomA 3 0 3 -b- 21941.5 18594.7 911.4 189.3 85.6 1609.1 1487.1 all / all DomA 3 1 4 r-- 22080.8 19371.6 768.4 186.3 79.9 1324.7 1236.7 all / all DomA 3 2 2 -b- 19332.6 16477.6 822.3 152.2 82.8 1443.3 1320.5 all / all DomA 3 3 5 r-- 19300.0 16450.6 828.7 155.2 81.7 1433.2 1304.6 all / all Up to 15% overcharging of VPU runtime is possible.
  31. 31. Evaluating RTDS: Fixing time accounting issue With making scheduling decision in RTDS scheduler based on real VCPU run time improves results for RT loads, but does not eliminates DL misses for all tested use cases: Idle CPU bound Idle IO CPU + idle IO CPU + heavy IO tp10-vp10000e7000 No DL miss No DL miss No DL miss No DL miss No DL miss tp10-vp5000e3000 No DL miss No DL miss No DL miss No DL miss No DL miss tp10-vp2000e1000 No DL miss No DL miss No DL miss DL miss DL miss tp10-vp1000e445 No DL miss No DL miss No DL miss DL miss DL miss tp100-vp10000e4445 No DL miss No DL miss No DL miss No DL miss No DL miss
  32. 32. Conclusion
  33. 33. Current status: Dedicated CPU-pool • Provides best isolation for critical domains, though still affected by architecture specific cache sharing effects and bus contentions • Critical domain internals agnostic But: • Is not flexible in terms of CPU resources utilization
  34. 34. Current status: RTDS scheduler • Has the best potential for CPU resources conserving • Is able to reliably schedule RT and non-critical domains together But: • Is limited for EDF scheduling algorithm in an RT domain • Requires the knowledge about RT domain tasks configuration to be shared to virtualization system for proper configuration • Suffers from scheduling time accounting issue
  35. 35. What’s next? • Improve scheduling time accounting • Investigate a possibility for additional restrictions for RT domains scheduling with RTDS
  36. 36. Questions?
  37. 37. Thank you!

×