This talk details XenTT, an open-source framework for deterministic replay and systems analysis in development at the University of Utah. The framework consists of two main parts: a set of Xen extensions that implement efficient, deterministic replay, and a powerful analysis engine that extracts information from systems during replay executions.
Deterministic replay promises to change how people analyze and debug software systems. As software stacks grow in complexity, traditional ways of understanding failures, explaining anomalous executions, and analyzing performance are reaching their limits in the face of emergent behavior, unrepeatability, cross-component execution, software aging, and adversarial changes to code. Replay-based, whole-system analyses offer precise solutions to these problems.
XenTT extends Xen with the ability to replay and analyze the execution of VM guests. A number of careful design choices ensure that our implementation, which supports single-CPU, paravirtual, Linux guests, is efficient, maintainable, and extensible. XenTT's run-time checks and offline log-comparison tools enabled us to efficiently scale the recording layer by detecting and debugging errors in the determinism of replay.
Our analysis engine seeks to overcome the semantic gap between an analysis algorithm and the low-level state of a guest. Using debug information to reconstruct functions and data structures within the guest, the engine provides a convenient API for implementing systems analyses. The engine implements a powerful debug-symbol and VM introspection library, which enables an analysis to access the state of the guest through familiar terms. To further simplify the development of new analyses, the engine provides primitives that support common exploration patterns, e.g., breakpoints, watchpoints, and control-flow integrity checking. To enable performance analyses of recorded executions, XenTT provides a performance modeling interface, which faithfully replays performance parameters of the original run.
Beyond describing the design and implementation of XenTT, this talk will present examples of how we have used deterministic replay to implement security and performance analyses.