Dealing with Hardware Heterogeneity Using EmbeddedXEN, a Virtualization Framework Tailored to ARM Based Embedded Systems


Published on

EmbeddedXEN is a particularly efficient virtualization framework tailored to ARM-based core embedded systems.

While security and OS isolation are key features of conventional virtualizuation frameworks, the main concerns for EmbeddedXEN are device heterogeneity and realtime aspects, which are particularly important in the embedded world.

EmbeddedXEN mainly relies on the original XEN architecture but with major differences in the way guest OS are handled: the hypervisor has been simplified, and only two guest OS (dom0 and domU) can run simultaneously; while dom0 is used to manage the native OS with drivers (original and backend splitted drivers), a paravirtualized OS (domU) can be cross-compiled on a different ARM device, and user applications can run seamlessly on the (virtualized) host device. Another important difference is that no user space tools are required to manage the VMs; the framework produces a compact single binary image containing both dom0 and domU guests, which can be easily deployed. The Xenbus architecture has been adapted to that context.

EmbeddedXEN therefore allows the porting of an OS and its applications from an ARM embedded device to last generation ARM hardware, such as HTC Smartphone for example.

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Dealing with Hardware Heterogeneity Using EmbeddedXEN, a Virtualization Framework Tailored to ARM Based Embedded Systems

  1. 1. Dealing with Hardware Heterogeneity Usinga Virtualization Framework Tailored to ARM BasedEmbedded SystemsProf. Daniel Rossier, PhDHEIG-VDInstitut REDS, Reconfigurable & Embedded Digital Systemsrte Cheseaux 1, 1400 Yverdon-les-Bains
  2. 2. Outline• Background• Overview of EmbeddedXEN• Protection & Memory Isolation• Domain Interactions• Device Heterogeneity• Conclusions & Future Work2 REDS@HEIG-VD
  3. 3. Background• HEIG-VD • University of Applied Sciences in Yverdon, Switzerland (CH) • Reconfigurable Embedded Digital System Institute  Hardware Design, FPGA  Embedded Execution Environment, Drivers, BSP, OS/RTOS, etc.• Applied Research & Development • ARM based Microcontrolers (v4, v5, v6, v7) • Low-level interactions between computing cores (CPU, DSP, FPGA, etc.) • Towards Cortex-A15 HVM3 REDS@HEIG-VD
  4. 4. Background• Embedded Virtualization on ARM • XEN: free & easy access to a stable & evolving hypervisor source code • Early port of XEN on ARM in 2007 in the context of a Diploma Project • Necessity to get a simple, thin, fast, robust, easy-to-deployed virtualization framework • Re-use of Linux file organization & build system (Makefiles, scripts, …) • Focus on realtime aspects (Linux/Xenomai as RTOS)• Different sources of inspiration • "Fast Secure Virtualization for the ARM Platform", Daniel Ferstay, Master Thesis, The University of British Columbia, 2006 • XEN ARM port, George G. Davis, MontaVista, 2007 • Secure Xen on ARM Project, Sang-bum SUH, Samsung, 20074 REDS@HEIG-VD
  5. 5. Background• Focus on heterogeneity of embedded devices • Idea to re-use OS & applications from old devices to recent devices • Time-to-market migration to new hardware generation • Dealing with various cross-compiled binaries (ARM v5-v7) • Dealing with different peripherals • Less emphasis on security aspects• Publicly available • REDS@HEIG-VD
  6. 6. Background• Hardware constraints • Low latency, reactivity (response time) • ARM cores do not support virtualization mechanisms  not easy to deal with various levels of execution modes• Para-virtualization remains attractive • About 30 files to be (slightly) adapted • Low execution overhead • Efficient processing of downcalls/upcalls with support of domain interactions  Physical interrupts are quickly processed in dom0.6 REDS@HEIG-VD
  7. 7. Overview of EmbeddedXEN Primary Guest OS Secondary Guest OS known as Dom0 known as DomU Xen-guest Xen-guest (pv) (pv) - Full privilege - Full privilege - Dom0 original drivers - DomU original drivers - Backend drivers - Native drivers - Frontend drivers • Limited use of hypercalls • ARM execution modes (USR/SVC) – pseudo-user mode with double stack handling / downcalls & upcalls are simple jumps to specific (callback) addresses • No subtle use of domain access control (ARM DACR) to protect memory • Memory sharing facilitated between primary and secondary OS Domain Prioritized VCPU Creation and EmbeddedXEN Upcalls sched_flip Setup Handling Migration Hypervisor Hypercalls Manager Scheduler Handling Hardware7 REDS@HEIG-VD
  8. 8. Overview of EmbeddedXEN • Linux-like source tree and build system (Makefiles) Config & build embeddedxen system tools xen-guest environnement.conf linux-2.6.26-domU Common part xenbus core include console include xen-guest arch linux-2.6.32-dom0 hypervisor-4.0.2 include xen-guest arm arch/arm xen-guestmach-msm mach-mx35 xen-guest include arch Secondary mach-mx35 xen OS mx35_fab4.c xen-guest arm Hypervisor arch/arm Primary kernel mm OS mach-msm 8 REDS@HEIG-VD
  9. 9. Overview of EmbeddedXEN• Single binary (multi-kernel) image• Automatic parsing & image relocation during hypervisor bootstrap EmbeddedXEN Boot Head 1 (Dom-U) 0 (Dom-0) Hypervisor EOD EOD DOM-0 DOM-U vmlinux vmlinux.dom0 vmlinux.domUuImage Dom-0 Filesystem /home/root/squeezeos.rootfs.img: • Stored separately in sdcard, for example Dom-U Filesystem9 REDS@HEIG-VD
  10. 10. Protection & Memory Isolation • Memory isolation between domains relies on different address space isolation. • No further advanced mechanisms to protect hypervisor and guest memory • The guest OS kernel runs at the same privilege as the hypervisor. • Each domain receives its own (contiguous) physical memory region during domain set up. • No pagination of the kernel linear address space is performed. • Paravirt of memory management is kept minimal. • Guest OS kernel has access to the whole memory. • No protection mechanism for strong isolation of VM (but is it really necessary?)10 REDS@HEIG-VD
  11. 11. Protection & Memory Isolation• Virtual & Physical Memory Layouts Initial Virtual Address dom0 Virtual Address Space Physical RAM Space Interrupt Vectors Interrupt Vectors 0xffff0000 frame table frame table XEN heap (2 MiB) XEN heap (2 MiB) boot allocator boot allocator domU Hypervisor code Hypervisor code Linear 1st-level page table mapping 1st-level page table 0xFF000000 I/O I/O VMALLOC_END dom0 domU dom0 dom0 Linear PAGE_OFFSET mapping 0xC0000000 Hypervisor 3GiB 3GiB 0x0000000011 REDS@HEIG-VD
  12. 12. Domain Interactions • Virtualization of peripherals in EmbeddedXEN is quite similar to the existing mechanisms in XEN. • Driver split with frontend & backend drivers • Communication with xenbus • Use of grant tables for sharing/copying pages between domains • However, a revisited (simplified) implementation of these mechanisms have been achieved in EmbeddedXEN. • XEN store is dynamically allocated at boot time of guest OSes. • No user space tools are required to manage XEN store entries or peripherals configs. • Hotplugs & dynamic configs of peripherals are less relevant to embedded systems.12 REDS@HEIG-VD
  13. 13. Domain Interactions • domU is passed information during bootstrap under control of dom0. xen-guest OS xen-guest OS xenbus xenstore subsys Subsys subsys Subsys thread thread event_channel xenstore A/B xenbus thread Page Dom0 Page DomU prod prod cons cons backend drivers frontend drivers event_channel C/D Hypervisor via unpause_domU(store_mfn, domU store_evtchn) start_info->store_mfn start_info->store_evtchn13 REDS@HEIG-VD
  14. 14. Domain Interactions• Grant tables are used in a different way • Shared pages are possible only in the vmallocd area. • Kernel linear addresses are not shareable; contents needs to be copied using temporary mappings. Dom0 RAM DomU Hypervisor Hypervisor 0xFF000000 0xFF000000 gnttab(domU) VMALLOC_END has VMALLOC_END has references to references to gnttab(domU) gnttab gnttab_foreign[1] gnttab(domU) domU gnttab(dom0) gnttab_foreign[0] gnttab gnttab(dom0) shared pages has references to has VMALLOC_START VMALLOC_START gnttab(dom0) references to mem_map_foreign mem_map_foreign(domU) dom0 mem_map_foreign(dom0) mem_map_foreign 0xC0000000 0xC0000000 shared pages 3GiB 3GiB 0x00000000 Hypervisor 0x0000000014 REDS@HEIG-VD
  15. 15. Device Heterogeneity• Different levels of heterogeneity • At CPU level: various instructions sets (locks, cache, etc.), various PTEs (MMU) flags, various co-processors, etc.  Compatibility ensured via hypercalls • At peripherals level: not the same hardware  Compatibility ensured via backend driver processing hypervisor-4.0.2 include arch/arm xen-guest mach-msm mach-mx35 xen built-in.o cache-v6.S cache-v7.S Kconfig Makefile mm.h proc-macros.S tlb-v6.S tlb-v7.S mm arch/arm kernel mm 15 REDS@HEIG-VD
  16. 16. Device Heterogeneity • Example of ARMv6 running on ARMv7 CPU (iMX35 -> HTC Desire HD)Original version Paravirt versionlinux-2.6.26-domU/arch/arm/mm/cache-v6.S: linux-2.6.26-domU/arch/arm/mm/cache-v6.S:ENTRY(v6_flush_kern_cache_all) .extern xen_flush_kern_cache_all mov r0, #0 ENTRY(v6_flush_kern_cache_all)#ifdef HARVARD_CACHE b xen_flush_kern_cache_all mcr p15, 0, r0, c7, c14, 0 @ D cache clean+invalidate #if 0 /* paravirt */#ifdef CONFIG_SMP mov r0, #0 mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate#else … b v6_icache_inval_all#endif mov pc, lr#else #endif /* 0 */ mcr p15, 0, r0, c7, c15, 0 @ Cache clean+invalidate#endif mov pc, lrlinux-2.6.26-domU/xen-guest/hypervisor.c: hypervisor-4.0.2/arch/arm/mm/cache_v7.S:void xen_flush_kern_cache_all(void) ENTRY(xen_flush_kern_cache_all){ stmfd sp!, {r4-r5, r7, r9-r11, lr} struct mmuext_op op; bl xen_flush_dcache_all @ much more complex!! mov r0, #0 op.cmd = MMUEXT_FLUSH_CACHE; mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF); ldmfd sp!, {r4-r5, r7, r9-r11, lr}} mov pc, lr ENDPROC(xen_flush_kern_cache_all) 16 REDS@HEIG-VD
  17. 17. Device Heterogeneity• Example of framebuffer device heterogeneity• Configuration of framebuffer is retrieved from Dom0 via xenbus xen-guest Dom0 xen-guest DomU User space applications User spacexenstore (/dev/fb0) xenbus applications thread (/dev/fb0) xenbus thread framebuffer core framebuffer device-specific core framebuffer msmfb_pan_update() Query framebuffer xenfb backend xenfb frontend params fb_event_dom_register domU fb memory dom0 fb memory fb_event_dom_switch FB config is retrieved (allocated by domU) (allocated by dom0) from dom017 REDS@HEIG-VD
  18. 18. Device Heterogeneity • Example of audio device heterogeneity • DomU audio buffers are accessed from Dom0 via shared pages. xen-guest Dom0 xen-guest DomU User space applications User space applications xenstore xenstore /dev/msm_pcm_out /dev/pcm/snd/pcm0c0d0p xenbus - or - xenbus thread /dev/pcm/snd/pcm0c0d0p thread pcm subsystem pcm subsystem xen-audio backend xenvaud_pcm_start xen-audio frontend xenvaud_pcm_stop Audio buffers allocated in domU arch-specific audio driver Sound device (headset, speakers, etc.)18 REDS@HEIG-VD
  19. 19. Conclusions & Future Work • EmbeddedXEN is an embedded virtualization framework which puts emphasis on efficient and heterogeneous hardware. • Application environments can be re-used "as such" on modern platforms (Android-based for example) taking advantage of last generation hardware. • EmbeddedXEN relies on the main principles of XEN, with a revisited lightweight, but less secure architecture. • A single multikernel binary image, easy to deploy on the target platform without additional tools, makes EmbeddedXEN well tailored to embedded systems.19 REDS@HEIG-VD
  20. 20. Conclusions & Future Work • Further investigation projects: • Elaboration of a domU using a graphical desktop for user applications • Support of multicore ARM CPUs (cortex-A9, cortex-A15) • Live migration of domU using remote NFS-filesystem (migration within a cloud) • Support of hard realtime OS (RTEMS-paravirt)20 REDS@HEIG-VD
  21. 21. • Thanks for your attention! • Further information: daniel.rossier@heig-vd.ch21 REDS@HEIG-VD