This document provides an overview of debugging tools and capabilities for the ACRN hypervisor. It discusses the ACRN hypervisor log (hvlog) and trace tools, as well as logging and debugging options for the Service OS (SOS Linux) including kernel logs, user space loggers, and gdb. Debugging options for User OSs (UOSs) are also covered, which depend on the specific UOS. Finally, some miscellaneous debugging tools are mentioned like platform tracing and JTAG tools.

1. Debugging
Yin Fengwei
Yin, Fengwei<fengwei.yin@intel.com>
04/22/2020

2. Agenda
- Overview
- ACRN Hypervisor
- SOS (Linux)
- UOS
- Misc tool

3. Overview
ACRN
SOS (Linux) UOS
DM APP
HV log
ACRN
trace
UOS logger
UOS profiling tool
SOS logger
(journald/syslogd)
gdb
DM logger
Kernel log/trace
Most debug capabilities available ONLY with DEBUG build

4. ACRN Hypervisor
• Hvlog for function debug
- Serial port
* legacy IO port UART
* PCI UART (needs BIOS initialize UART controller)
- NPK (Hardware dependency. Not available for all platforms)
- log buffer. Exported to SOS. The logs before hvlog setup are lost.
• ACRN trace for profiling
- trace buffer. Exported to SOS
- acrntrace toolset

5. SOS (Linux)
• General kernel log (kmesg) and kernel trace
• User space logger (journald)
- capture the systemd service logger
- constrain: SOS storage capacity. Default journald configuration
- Journald telemetry
• DM logger
- --logger_setting of dm cmdline
- disk logger: save log to disk
- console logger: output log to console
- kmsg logger: output log to SOS kmesg
• Gdb
- very powerful tool to debug DM as a process of SOS
- all BKMs for gdb applied

6. SOS (Clear Linux) - gdb case study
• Check UOS kmesg
- Attach gdb to dm and stop dm and check ctx base address.
- locate UOS kmesg address:
#find ctx->base_addr, +0x200000000, {char[30]}"Supporting XSAVE feature 0x001"
- dump 1M memory range of UOS kmesg buffer
#dump binary memory ~/dump.bin <return_find> <return_find + 1M>
- parse content dumped with attached python script
• Check the ioreq handled in DM
- attach gdb to dm with attached vmexit.txt as init script
- gdb will print the ioreq info
- Very useful when we want to check whether guest is still alive

7. UOS (Linux/Xenomai)
• Depends on UOS

8. Misc tools
• Platform tracing tool
- align the trace from HV(acrntrace), SOS(ftrace) and UOS(UOS trace)
- easy to check the system behavior with aligned timestamp cross HV/SOS/UOS
- very useful to locate system performance bottleneck
https://github.com/conghuic23/acrn_trace
• JTAG tool – powerful lowlevel debugging tool
• Top/htop – CPU usage monitor
• Valgrind – memory leak detection

9. Q & A