XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - Chris Patterson & Kyle Temkin, AIS, Inc.

153 Brooks Road, Rome, NY | 315.336.3306 | http://ainfosec.com
Xenbedded:
Xen-based client virtualization for phones and tablets
Kyle Temkin
Chris Patterson
Assured Information Security, Inc.
August 25th, 2016 // XenSummit 2016

Who are we?
Kyle Temkin
‘Hardware Hacker’ and Security Researcher
Twitter: @ktemkin, Freenode: ktemkin
E-mail: temkink@ainfosec.com
Chris Patterson
Senior Computer Engineer
Freenode: cjp256
E-mail: pattersonc@ainfosec.com

Why run Xen on Phones/Tablets?
First, and foremost: it’s cool!
} This is the ‘frontier’ of virtualiztion tech–
there’s lots of new stuff to build!
Mobile virtualization also offers more
pragmatic benefits:
} Separation of critical and non-critical tasks.
◦ Thnk Qubes or OpenXT for mobile: different VMs for
different types of activity.
◦ Potential tools to deal with the whole ‘BYOD’ mess.
Pictured: Google Pixel C
Photo: Google Store
} Potential to extend benefits beyond phones/tablets; e.g. to devices
with integrated touchscreens such as car media centers.

Isn’t ARM support enough?
Q: Xen already supports many ARM processors: shouldn’t it be able to
run on most phones and tablets?
A: Almost. There are still a few things we need:
◦ Hardware support for ARM’s virtualization extensions.
◦ Bootloader support for launching software in hypervisor mode (HYP/EL2).
• It’s easy to find yourself ‘locked out’ of these modes!
First-Stage Bootloader
(usually comes with SoC;
often in ROM)
Second-Stage Bootloader
(e.g. u-boot, android
bootloader)
Final Payload
(Linux, Xen)
All too often, the FSBL drops down
to Supervisor Mode (PL1/EL1)... … leaving no way back into PL2/EL2!

Landing in PL2/EL2
Q: What if the boot chain can’t land me in PL/EL2?
A: If you have to use that device, things will get… interesting.
◦ Best option: convince your vendor to support EL2 on unlocked devices.
◦ Last resort: dust off your hardware hacking skills!
• Samsung CID hacks (developer bootloader)
• Qualcomm TrustZone exploits (chain to get at HVBAR)
- CPU mode All forms of interrupts must be masked in PSTATE.DAIF
(Debug, SError, IRQ and FIQ). The CPU must be in either EL2
(RECOMMENDED in order to have access to the virtualisation extensions)
or non-secure EL1.

Finding the Right Device
Finding a good phone or tablet to work with can be tricky.
You need a device that:
} Supports the ARM Virtualization Extensions, including a full-featured GIC.
} Provides an “unlockable” bootloader entered in NS-HYP or EL2.
You want a device that:
} Is supported by an open-source Linux variant (for dom0)
} Is closely approximated by an available evaluation board.
} Has an accessible UART “serial port”

Identify Devices with HYP/EL2 ‘Payloads’
If your device runs Linux 3.7+ (arm32) or Linux 4.3+ (arm64),
checking for HYP/EL2 is easy:
Otherwise, you can compile a simple kernel module to check:
https://github.com/xenbedded/hyp-mode-checks
# dmesg | grep HYP –A 1 #arm32
CPU: All CPU(s) started in HYP mode.
CPU: Virtualization extensions available.
# dmesg | grep EL2 #arm64
CPU: All CPU(s) started at EL2
# #Note: these only work bare-metal!

Fragmentation of Linux Baselines
Depending on the device, several Linux variants may be available,
including mainline and per-device variants.
Not all Linux forks are created equal:
} If possible, select a device that has reasonable mainline support. (e.g. MSI Primo73)
} Failing that, you may want to look for a fork that deviates less from mainline:
◦ A recent kernel version is usually a good hint.
◦ Another good hint: look at the device tree for your target device. Are many of its compatible
strings supported in mainline?
• Mainline
• Android Variants
• ChromiumOS variants
• linux4tegra
• linux-sunxi
• linux-exynos
• linux-xilinx
• linux-96boards
• … oh, my!

Approximating HW with Dev Boards
≈
Pixel C
image source: Google Store
NVIDIA Jetson TX1
image source: NVIDIA Dev Blogs
Pairing your device with an evaluation board can dramatically speed up
development.
◦ It helps a lot to have a “known good” environment to test your hardware in.
Is the issue in your code, or with the weird tablet bootloader? Compare and find out!
◦ If you can get one, a JTAG debugger make all the difference!

Finding UART Consoles
Most– but not all– phones and tablets have a UART surfaced somewhere.
Friendlier devices have externally accessible UARTs; some devices require soldering.
Externally AccessibleInternally Accessible
The Xen-compatible Primo 73’s UART
is accessible both directly on its PCB…
… or, with some configruation, routed
through its internal microSD slot.
Embedded Linux and Android communities are a good first resource for UART pinouts:
you’ll be surprised how often the community has documented how to get UART access!
Original
image.
Credit:
ssvb

While the Pixel C has no documented external UART, digging through
some of its firmware reveals an internal USB-to-serial-adapter.
} When the USB-C connector’s CC1 and CC2 pins are pulled down with 5.1Ω
resistors, the device enters ‘debug mode’, and the convert is connected as
below:
} Naturally, we went a little overboard, and our intern (@c4757p) developed a full
fledged open-source-hardware debug adapter for these ports:
◦ https://github.com/xenbedded/pixel-c-support/tree/master/hardware/debug_bridge_r2
Example: Pixel C UART

Getting Xen onto Your Device
Once you’ve selected a device, the next step is to get Xen to start.
A basic path forward might be:
} Boot a simple, custom hello-world kernel.
} Boot up the Xen kernel alone– get it to crash!
} Provide Xen with its required images, and see how things behave.

Booting a ‘Hello World’ Kernel
Many devices require their kernels to be encapsulated
in payload-specific image formats.
} ChromiumOS: FIT image with specific contents, prefixed
with signed header
} Many Android Devices: Android-specific boot.img format
} We’ve seen lots of variants on these formats!
Booting a “Hello World” image helps to verify that you
have the particulars of your image format correct.
} Also a great time for a final check for HYP/EL2!
} Good starter example:
https://github.com/andreiw/nexus9_demo

(Some People Just Want to)
Watch Xen Crash
Once you can boot an image on your particular device, try swapping
out your ‘hello-world’ kernel for Xen.
} Be sure to build Xen with EARLYPRINTK set up for your board!
Launching Xen...
- UART enabled –
- CPU 00000000 booting –
- Current EL 00000008 –
- Xen starting at EL2 –
- Zero BSS –
- Setting up control registers –
- Turning on paging –
- Ready –
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) No valid device tree
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
= Victory!

Getting Xen its Images
Once you successfully have Xen crashing, you’re ready to try a real
boot, which means passing in several necessary ‘images’.
This can be tricky. We expect our bootloader to load:
} The Xen kernel image, typically a zImage/Image;
} A flattened device tree (dtb) describing the target device;
} The dom0 kernel, typically also a Linux zImage/Image; and
} An optional ramdisk (initrd/initramfs).
… but most image formats won’t load all of these!
} We’ll need a next-stage bootloader to make our images available.
◦ u-boot is a popular choice.

Discharge Boot Adapter
We created Discharge, a simple
image adapter for phones and tablets.
} Combines the images relevant to a Xen boot into
the single-image formats expected by most
consumer device loaders (Android/CrOS).
} ‘Faster’ and significantly lower-footprint than
u-boot; but also significantly less powerful.
◦ If u-boot better fits your use case, use it!
Full MIT-licensed source code:
http://github.com/xenbedded/discharge

Xen on Your SoC
If things seem broken at first, don’t despair!
You may have to deal with quirks for your particular SoC– or with Xen
behaviors (bugs?) that your new configuration has unmasked.
} This is where things can get complicated. We won’t provide a general method for
moving forward (this talk is too short!), but we will provide examples.
} In particular, see Julien Grall’s slides from Porting Xen on ARM to a new SoC.

Example:
Tegra Legacy Interrupt Controller
On Tegra systems, a set of Legacy Interrupt Controllers must be
programmed in addition to the core Generic Interrupt Controller (GIC).
} Idea is straightforward: implement Xen support for the Tegra LIC.
Figuring out the best way to do this: less straightforward..
} One working implementation is available online:
We’re currently working to get a proper solution mainlined!
https://github.com/xenbedded/xen/tree/tegra_support_v2
Tegra LIC
Legacy
Interrupt
Controllers
GIC
Distributor
GIC
CPU Interface
GIC
CPU Interface
CPU 0
CPU 1
SPI[n:32] SPI IRQ
IRQ
(Very) Simplified View of Tegra Interrupt Routing
The Tegra LIC, by default,
gates all Shared interrupts.

Client Virtualization:
New Peripheral Support for Xen
Tablets and phones present a huge variety of new peripherals we may
want to virtualize for guest domains:
Virtualizing these devices require three components:
} A device driver running in domain zero; ideally provided by the BSP distribution.
} A backend driver to (e.g.) multiplex access to the physical hardware; and
} A frontend driver to deliver events to the guest kernel.
• Display/GPUs
• Light Sensors
• Flash LEDs
• Touch Sensors
• Cameras
• Wireless/Cellular
• Inertial Sensors
• Fingerprint Sensors
• … and more every day.

Display and Input Multiplexing
To provide our guests with a high-performance
display capability, our group has worked to
develop a new display technology for Xen.
} Provides a near-native experience for both mobile and
desktop devices, including backend and frontend
drivers for multi-touch and multi-monitor support.
} Includes a full input stack, including PV drivers.
} We won’t go into detail here. This is covered in full by
the next talk in this room (“Display Handler”).
Baseline: X11 and VNC
Display Handler:
Native Look

Multi-Touch with openxt-kbdfront
Multi-touch input is handled by an
extended version of xen-kbdfront.
} Entirely open; source is GPLv2 and
available on our GitHub.
(http://github.com/xenbedded/openxt-kbdfront)
} Backwards-compatible with xen-kbd,
and thus works with existing backends.
} We may submit our changes as patches
to xen-kbdfront, if well received.

Goal: Android Support
Support for Android guests is not far off—basic functionality is
working on x86—but work towards this support is still underway.
Challenges still to be solved:
} Blocker: Modern Android versions require* hardware-accelerated video;
so we’ll need to implement a solution for multiplexing access to the host GPU.
} Soft req: Android isn’t as useful on a tablet without mobile-centric power mgmt.,
including intelligent use of host sleep states, paralleling Android’s wakelocks.
} Soft req: Many android applications require services from TrustZone ‘Applets’
(Trustlets), but we now need a guest-aware TrustZone, or equivalent.
*At least, according to Google. Software rendering may be an alternative,
but doesn’t work as-is, and likely isn’t worth pursuing for performance reasons.

Android: Accelerated Graphics
Android’s SurfaceFlinger compositor– and
many applications– require OpenGL support.
Potential paths forward:
} On x86, HVM guests can run Android under the
VirGL virtualized GPU, which is heavily tied to its
QEMU implementation.
◦ See Rob Herring’s Android-on-QEMU effort.
◦ To support ARM PVH guests, we’d need to develop a PV
transport for the VirGL.
} Raw “API Sharing” solutions – such as the emugl
library used by the Android emulator – also offer
potential solutions.

Android: TrustZone Implications
In the current implementation, only one trusted enclave is available.
} This is definitely not an optimal architecture for per-guest services, such as
secure password stores, DRM, and cryptographic services.
◦ As a result, we disable guest SMCs: and lose these capabilities altogether! (“No Netflix?”)
} Consider the implications of existing TrustZone escalations (e.g. the QSEE bug)—
do you really want your guests interacting with TrustZone services?
} Potential solution: leverage hypervisor separation to create a service-VM
trusted execution environment (TEE) for per-guest services.
dom0 dom1 dom2
Single TZ-Enabled Enclave
Core System Services
TEE Service VM
dom0
TEE Service VM
dom1
TEE Service VM
dom2
dom0 dom1 dom2
Single TZ-Enabled Enclave
Unused?

Questions and Demo

XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - Chris Patterson & Kyle Temkin, AIS, Inc.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - Chris Patterson & Kyle Temkin, AIS, Inc.

Similar to XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - Chris Patterson & Kyle Temkin, AIS, Inc. (20)

More from The Linux Foundation

More from The Linux Foundation (20)

Recently uploaded

Recently uploaded (20)

XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - Chris Patterson & Kyle Temkin, AIS, Inc.