Tools and techniques to debug systems based on ACRN

1. Debugging ACRN
Yin Fengwei <Fengwei.yin@intel.com>
ACRN vMeet-Up Europe 2021

2. Table of Contents
PART 1: ACRN hypervisor debugging
PART 2: All type guest debugging
PART 3: Post-launched guest debugging

3. ACRN hypervisor debugging – JTAG
➢Trace32 and Intel ITP
➢Need JTAG probe on hardware or DCI interface enabled
➢Only choice for early boot and random system hang debugging
➢Symbol loading
➢ Host tool support
➢ HV relocation brings trouble
➢ Not needed for simple assembly checking

4. ACRN hypervisor debugging – Serial Port
➢More common than JTAG
➢Supported by all OS
➢Supported on all ACRN target hardware
➢Hypervisor serial console
➢ Only enabled with DEBUG build
➢ Hypervisor logs
➢ Loglevel: error, warning, notice, info, debug
➢ Default loglevel: message level lower than default level NOT printed to serial port
➢ Too lower default loglevel: trigger flooding log on serial port
➢ Hypervisor console commands

5. ACRN hypervisor debugging – Serial Port
Example1
timestamp
PCPU num vm: vcpu
Version + build info
Scenario +
Platform
➢ Timestamp could be used to evaluate the ACRN boot time
➢ pcpu number could be used to understand which pcpu log printed
➢ vm:vcpu number could be used to understand which guest trigger log printed
➢ Build version, scenario, platform could be used when report the issue

6. ACRN hypervisor debugging – Serial Port
Example2
• RIP: where the vcpu is
running if guest dead loop
somewhere
• CR0/CR4: vcpu settings like
Paging enabled? Protection
mode enabled?...
• RSP dump could be used to
check the stack of guest

7. All type guest debugging
➢Emulated serial port if guest support
➢ 16550 based IO port serial port emulation
➢ Depends on hypervisor serial port
➢ Mux for different guest and switching by hypervisor command
➢VMEXIT info can be to understand guest behavior
➢ Exit reasons: guest run sequence
➢ VCPU registers:
➢ RIP to locate the code position
➢ RSP for guest backtrace if frame point is enabled in guest

8. All type guest debugging - Continue
➢Hypervisor commands
➢ Dump guest status/register/memory
➢ Dump PT dev info
➢ Special debugging command
➢ Inject exception to guest
➢ GP Trigger windows guest BSOD
➢ NMI Trigger Linux kernel backtrace

9. Post-launched guest debugging
➢Guest itself debugging method support
➢ Windbg with USBC passthru guest
➢ Linux Kernel log/kgdb over ethernet
➢ Android ADB with USBC passthru to guest
➢ Need special device available for guest
➢ USB
➢ Networking
➢ No hardware debugging support exposed to post-launched guest
➢ Guest debugger can’t use hardware debugging infrastructure
➢ Software debugging infrastructure can be used

10. Post-launched guest debugging
➢ACRN extended post-launched guest debugging
➢ post-launched guest <-> ACRN Device Model (acrn-dm) process
➢ Logger for ACRN DM: indirect understand guest behavior
➢ Service VM OS (aka SOS) Linux tools for process:
➢ strace
➢ gdb: process level debugging. Not able to debug guest directly.
➢ trace
➢ Limitation
➢ Can’t work if stuck in hypervisor
➢ Limited function if post-launched guest has higher severity than Service VM

11. Summary
➢Highly depends on physical serial port
➢No hardware based debugging infrastructure to guest
➢Software based debugging infrastructure available to guest
➢Efficient indirect debugging method