Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FOSDEM17 - Live patching the xen project hypervisor

20,625 views

Published on

Live patching -- the process of updating software while it is running -- has become a popular topic of late, with several companies offering services providing live patching for Linux culminating with an implementation being contributed to Linux. While Xen supports live migration of VMs which mitigates the need for live patching, there are still some reasons why live patching is useful. For example, to minimize downtime, when running applications with low latency requirements, and when using PCI passthrough.

This talk reviews the state of live patching for Xen. A live patching implementation has recently been contributed to Xen so we will look at the design and implementation in detail, noting how and why it differs from live patching for Linux. We will look at some of the ways that live patches can be created and the pitfalls to watch out for when doing so. There will also be a status report of ongoing and future work for live patching for Xen. Finally, the talk will show a demo of live patching in action by building a live patch for a known security issue and applying it to a vulnerable system.

Xen LivePatch is a new tech preview feature in Xen 4.7 for live patching the hypervisor. It is the result of collaboration between several members in the community guided by the existing live patching efforts for Linux. We will talk about this and how Xen's requirements differ from Linux.

The core idea of Xen LivePatch is to insert a "trampoline" in each buggy function with a jump to a replacement function. This is simple in principle, but there are a number of complexities which make it interesting to look at.

Live patches can be built by hand, by compiling the replacement functions in a separate source file. This is a manual process and can also be unreliable due to the nature of compiler optimizations. So we repurposed kpatch's live patch creation tool for Xen. This computes a binary difference of the output binaries before and after a source patch is applied. This simplifies the process of creating live patches and overcomes issues with compiler optimizations but is still not without difficulties. We look at the process of creating live patches and why it is hard.

Also see https://fosdem.org/2017/schedule/event/iaas_livepatxen/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

FOSDEM17 - Live patching the xen project hypervisor

  1. 1. LIVE PATCHING THE XEN PROJECT HYPERVISOR Ross Lagerwall (Citrix) FOSDEM 2017
  2. 2. Introduction Xen supports live migration — so why live patch? VM is using a pass-through device. Live migration downtime is too long. To avoid host downtime. Not enough spare resources to move all VMs off the host. There are several good reasons to live patch the hypervisor!
  3. 3. Patching basics Inline replacement of hypervisor code would be possible but not necessarily practical. We replace old functions with new functions instead. Xen Live Patch has a function-level granularity.
  4. 4. Payloads A payload is a set of replacement functions plus metadata. It is packaged in a relocatable object le, like a kernel module. Since the module is relocatable, all the symbols need to be resolved. Xen has a simple linker to do this while loading the payload. Plus a whole lot more: perform relocations, apply alternative instructions, and parse the replacement table, hook functions, and bug and exception frames.
  5. 5. Stacking payloads Payloads are speci c to a particular build of the hypervisor. They depend on the exact hypervisor internal ABI — not stable. The hypervisor and each live patch contains a build-id. Each payload is designed to be applied on top of a speci c build-id to prevent applying a patch on the wrong base. This model allows loading a stack of patches.
  6. 6. Applying payloads A loaded payload needs to be applied to take effect. To avoid modifying code while it is being executed, the system is quiesced. We check in the return-to-guest path if any live patches need to be applied. At this point, there is a xed, small subset of functions that could be on the stack.
  7. 7. Hypervisor interface The sysctl hypercall was extended with four new sub- operations for: Uploading a payload Listing payloads Querying a payload state Sending a command to apply, revert or unload a payload The new sub-operations are controllable through XSM. The xen-livepatch tool is used to access this functionality.
  8. 8. Compared with Linux live patching Latency for VMs is not usually that critical so we don't use kGraft's model. Xen's model is largely similar to kPatch but there is no need to perform stack checking. Xen does not have ftrace and is not compiled with -pg to get mcount() calls so the start of the function is overwritten. Can patch any function which is at least 5 bytes long. Overhead is a single unconditional jump.
  9. 9. Building live patches How are these created? Build them by hand? — NO! Enter livepatch-build-tools! livepatch-build-tools is based on kpatch-build http://xenbits.xen.org/gitweb/?p=livepatch-build-tools.git
  10. 10. Building live patches: Inputs $ livepatch­build ­s xen ­c orig.config      ­­depends 55776af8c7377e7191d733797543b87a59631c50      ­p xsa182.patch ­o outdir Takes as input: The exact source tree from the running Xen. The .con g from the original build of Xen. A build-id onto which the livepatch will be applied. A source patch.
  11. 11. Building live patches: Process livepatch-build does: 1. Build Xen 2. Apply the source patch 3. Build Xen with "-ffunction-sections -fdata-sections" 4. Revert the source patch 5. Build Xen again with "-ffunction-sections -fdata-sections" 6. Create a livepatch from the changed object les.
  12. 12. For each pair of changed objects, ‘original’ and ‘patched’, run : Building live patches: Diff create-diff-tool Load objects and check that the headers match. Adjust the ELFs to make them easier to process. Correlate sections, symbols, and static locals. Compare and mark as SAME, CHANGED or NEW. For each CHANGED function or NEW global, include it and its references recursively.
  13. 13. Building live patches: Diff Handle special sections (bug frames, altinstructions, exception tables). For each CHANGED function, create an entry in a special livepatch section (.livepatch.funcs). Write out the new object le. Link each object le together into a relocatable module.
  14. 14. Handling live patches with data New data and read-only data is handled correctly. Changing initialized data or existing data structures is hard so such changes are prevented. Hook functions allow code to be executed at various stages during the patch apply (or revert) process. Allows data to be transformed during patch apply, even if the data is dynamically allocated Allows once-off initializations. Use shadow variables to attach new members to existing data structures.
  15. 15. Compared with Linux live patching Linux live patches are contained in modules. Xen does not support loadable modules which avoids many issues. kGraft does not use a tool to automatically create live patches. kpatch is moving towards not using kpatch-build.
  16. 16. Future Work Support for handling NMIs and MCEs while applying/reverting payloads. Support for signing payloads similarly to how kernel modules can be signed. This would make it more dif cult to load a malicious module. Add OSStest support. Remove the experimental tag.

×