Windows guest debugging presentation from KVM Forum 2012


Published on

This presentation will show how to debug Windows virtual machine running on top of open source hypervisor KVM.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Windows guest debugging presentation from KVM Forum 2012

  1. 1. Windows Guest Debugging Yan Vugenfirer, Daynix Computing LTD
  2. 2. Agenda Device manager Logging and DebugView Introduction to BSOD Crash dumps Crash on demand Driver verifier WinDbg Installation and configuration Postmortem debugging !ndiskd Configuring guests Live debugging
  3. 3. Device manager
  4. 4. Device Manager Run -> devmgmt.msc Devices and drivers management and installation Scan for HW changes Effective tool to check resource allocations of the devices, drivers versions and different drivers settings.
  5. 5. Device Manager - Views Change the views of device manager Show non PNP devices
  6. 6. Device Manager - Views View -> Devices by connection This view can be used to check buses and devices relations
  7. 7. Device Manager - Views View -> Resources by type Effective to check IRQ sharing On OSes below Vista virtio devices will use legacy PCI interrupts Virtio-serial and ballon use legacy interrupts also on newer OSes.
  8. 8. Device Manager - Views View -> Resources by connection IO and memory address space with relations between buses and the devices
  9. 9. Device Manager - Show non- present devices Show devices drivers that are installed but the device is curently not present in the system Very useful for removing crashing drivers from the system From command line: SET devmgr_show_nonpresent_devices=1 START devmgmt.msc
  10. 10. Device manager - properties Double click on the device Click on “General” tab You can view device location on the bus If the driver is not started you can see the reason in status text-box. For example - no digital signature on x64 OS from Vista and up
  11. 11. Installing self signed drivers Drivers need to be at least self signed starting from Vista x64 bit To enable test signing: Bcdedit.exe –set TESTSIGNING ON Disable test signing Bcdedit.exe –set TESTSIGNING OFF Install test certificate on the guest Reboot needed after enabling and disabling test siging commands More information on test signing:
  12. 12. Device Manager - resources Double click on the device Click on “Resources” tab Shows device resources (IRQs, IO and memory ranges)
  13. 13. Device manager - driver tab Double click on the device Click on “Driver” tab Get driver version View the driver binaries View the versions of driver binaires
  14. 14. Device manager - details Double click on the device Click on “Details” tab A lot of additional details like: PNP ID of the device, PM policy, location of different device keys in registry and etc
  15. 15. VirtIO - NetKVM parameters Advanced tab in device manager Change logging levels Enable packet statistics printout Other parameters starting with “TestOnly” prefix Full list: page/ Internals_of_NDIS_driver_for_VirtIO _based_network_adapter#List_of_ configurable_parameters
  16. 16. NetKVM - log levels 0 - Basic configuration and unload trace. Critical errors. 1 - Warnings, corner cases 2 - Network packet trace 3 - More verbose trace of packets 4 - Virtio library, DPC 5 - ISR trace 6 - Registers readwrite.
  17. 17. NetKVM parameters from command line Part of WHQL requirements to supply command line utility for drivers that expose parameters through “advanced” tab readme.doc - supplied with driver package explains the usage. Integrated with “netsh” Windows networking tool To register (before the first use): rundll32 netkvmco.dll,RegisterNetKVMNetShHelper Unregistration (removal) rundll32 netkvmco.dll,UnregisterNetKVMNetShHelper
  18. 18. NetKVM parameters from command line Description Command line Output List devices netsh netkvm show devices Names and index for each device Restart device netsh netkvm restart [idx] Should be done after parameters are changed Show parameters netsh netkvm show parameters [idx] List the names of the parameters Show detailed information about parameter netsh netkvm show paraminfo [idx] [param] Current value of the parameter Change parameter value netsh netkvm setparam [idx] [param] [value]
  19. 19. Debug view and tracing
  20. 20. DebugView Shows debug space of Windows (kind of dmesg). Can extract debug trace from crash dump if it was running before the crash Can log debug trace during boot. Download from - sysinternals/bb896647.aspx
  21. 21. DebugView - enable output to debug space To enable debug space outputs from Vista and up: Through registry (using regedit): Add “Default” value of the type DWORD equal to 0xFFFFFFFF under “HKEY_LOCAL_MACHINESYSTEMCurrentControlSet ControlSession ManagerDebug Print Filter” Reboot During live debugging ed nt!Kd_DEFAULT_MASK 0xFFFFFFFF
  22. 22. Debug view - capture kernel Capturing kernel output disabled by default To enable: Capture -> Capture Kernel
  23. 23. Debug view - log boot Run DebugView Enable boot logging Run DebugView after reboot to collect the logs
  24. 24. Debug view - crash dump Run DebugView before crash dump creation After crash and reboot - point DebugView to crash location for processing
  25. 25. Virtio debugging with serial port Uncomment “ENABLE_COM_DEBUG=1” in SOURCES file. Comment out “ENABLE_PRINT_DEBUG” Rebuild the driver On the host side -serial tcp::<port>,server,nowait nc localhost <port>
  26. 26. Blue screens
  27. 27. Bug check System crash called “Bug check” Blue screen Crash dump will be saved Ability to debug the bug check if debugger is connected
  28. 28. Blue screen inside Bug check code Bug check parameters - specific to each bug check Additional information - can be located also below bug check code and parameters Crash dump saving counter
  29. 29. Common bug checks Bug Check 0xA: IRQL_NOT_LESS_OR_EQUAL Bug Check 0x8E: KERNEL_MODE_EXCEPTION_NOT_HANDLED Bug Check 0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL Bug Check 0x9F: DRIVER_POWER_STATE_FAILURE Bug Check 0xC4: DRIVER_VERIFIER_DETECTED_VIOLATION Bug check codes reference: library/hh994433(v=vs.85).aspx
  30. 30. Crash dumps
  31. 31. Crash dump creation During blue screen Windows will save crash dump Minidumps are not always useful Also - remove the option to reboot on crash Right click on “My computer” -> “Properties”->”Advanced system settings”->”Startup and Recovery” -> “Settings” Chose “Kernel memory dump” Remove “Automatically restart” The windows on different OSes can be different but the menu names are the same
  32. 32. Crash dump creation
  33. 33. Crash dump locations After the crash search in: %windir%memory.dmp %windir%MinidumpMinidump<timestamp>.dmp When you have Windows error message after reboot - do not close it. Check minidump path and copy it.
  34. 34. Crash dumps From Windows 7 and up - OS will auto-delete large crash dumps To keep crash dumps: Key: HKLMSystemCurrentControlSetControl CrashControl Value: “AlwaysKeepMemoryDump”:DWORD set to 1
  35. 35. Crash dumps on demand Crash Windows on demand to examine driver status Configure through registry: Key: HKEY_LOCAL_MACHINESystem CurrentControlSetServicesi8042prtParameters, Value: CrashOnCtrlScroll, equal to a REG_DWORD value of 0x01 Key: HKEY_LOCAL_MACHINESystem CurrentControlSetServiceskbdhidParameters, value: CrashOnCtrlScroll, equal to a REG_DWORD value of 0x01
  36. 36. Crash dumps on demand Press Ctrl+ScrollLock+ScrollLock You will get MANUALLY_INITIATED_CRASH (0x000000E2) bug check. Read more (also how to redefine the shortcuts): http://
  37. 37. Crash dumps on demand (2) Issue NMI from QEMU monitor Setup guest: HKLM/System/CurrentControlSet/Control/CrashControl, DWORD value NMICrashDump 1 Restart Enter QEMU monitor (Alt+Ctrl+2, to go back Alt+Ctrl+1) Issue “NMI” command
  38. 38. Driver verifier
  39. 39. Driver verifier Built-in into OS (substitutes OS functions with own layer). To activate run “verifier.exe” from command line Need a reboot after the settings are changed Effective in finding deadlocks, memory leaks, memory corruptions and incorrect parameters to OS functions.
  40. 40. Driver verifier - activation Run from command line: verifier.exe Select “Create custom settings”
  41. 41. Driver verifier - activation
  42. 42. Driver verifier - activation Change selection to “Select driver names from a list”
  43. 43. Driver verifier - activation Select loaded driver from the list Or select driver from its location on files system After clicking finish - reboot is needed to activate the settings.
  44. 44. Driver verifier - bug checks 0x000000C4 Some can be ignored if debugger is connected Provide additional information for debugging
  45. 45. Driver verifier - current settings Run verifier.exe Click on “Display existing settings” radio button Click on “Next” button
  46. 46. Driver verifier - current settings
  47. 47. Driver verifier - removal Don’t forget to reboot
  48. 48. WinDbg
  49. 49. WinDbg We can examine crash dumps with WinDbg Remote live kernel debugging Limited live kernel debugging using one guest only User mode debugging including remote debugging
  50. 50. Installation Install as part of the WDK hardware/hh852362 Install as part of the SDK hardware/hh852363 Download symbols for different Windows OSes hardware/gg463028
  51. 51. Symbols configuration View OS functions and variable names Set Microsoft symbol server and local paths Set environment variable: “_NT_SYMBOL_PATH” Sample symbol path: C:symbolslocal;SRV*C: symbolswebsymbols*http:// download/symbols
  52. 52. Symbol configuration Another way to set the symbols path is through WinDbg menu File ->”Symbol File Path...” You can instruct WinDbg to reload the symbols Also using windbg command: .reload
  53. 53. Post mortem debugging Run WinDbg.exe Load dump file through: File -> “Open Crash Dump...” Might be slow first time
  54. 54. Loading crash dump Information about loaded symbols OS information Hint to use “!analyze -v” Bug check code and parameters
  55. 55. Magic wand - !analyze -v
  56. 56. !analyze -v First command that you should run (or click) Bug check and its parameters explanation Hints for the next steps and how those parameters can be used for further debugging Module name that probably caused the crash Stack trace
  57. 57. WinDbg - help Help -> Contents Switch to “Index” tab for specific command help
  58. 58. WinDbg - views
  59. 59. WinDbg - Views View To display window From command line Memory Alt+5 d <address> Call stack Alt+6 kb Disassembly Alt+7 u <address> Registers Alt+4 r Watch Alt+2
  60. 60. WinDbg - View memory Menu: View -> Memory Command (different variation of “d”): db <address>
  61. 61. Use case - memory leak with verifier
  62. 62. WinDbg - View disassembly
  63. 63. Windbg - Reference symbols <modulename>!<symbol_name> Example: netkvm!DriverEntry
  64. 64. WinDbg - Reference symbols
  65. 65. Useful command Command Description !scsikd Scsi related commands. Use ! to find more .reload Reload symbols lm List loaded modules ~<cpu number> Switch between CPUs. For example ~0 !irpfind Search for IO request packets !irp Display information regarding specific IRP .reboot Restart remote during live debugging
  66. 66. !ndiskd - NDIS debug extensions !ndiskd.ndis Show ndis.sys build information ! Show help !ndiskd.miniports Show NDIS miniports !ndiskd.miniport <addr> Show info about specific mimiport !ndiskd.nbl Show NET_BUFFER_LIST info !ndiskd.oid Show pending OID requests
  67. 67. !ndiskd.miniports
  68. 68. !ndiskd.miniport <miniport handle>
  69. 69. From !ndiskd to NetKVM
  70. 70. Configuring live debugging Host side Run debugget host VM with : “-serial tcp::<port_number>,server,nowait” For example: “-serial tcp::4445,server,nowait” Run target VM with: “-serial tcp:<port_number>”
  71. 71. Configuring live debugging (2) Host side (new format) Run debugget host VM with : “-chardev socket,id=serial0,path=/tmp/debug- serial0,server,nowait -device isa- serial,chardev=serial0,id=serial0 ” Run target VM with: “-chardev socket,id=serial0,path=/tmp/debug-serial0 - device isa-serial,chardev=serial0,id=serial0”
  72. 72. Configuring live debugging Guest side - Windows XP and Windows 2003 Edit c:boot.ini Duplicate the default boot line and add: “/debug /debugport=COM1 /baudrate=115200” Reboot
  73. 73. Configuring live debugging Guest side - Vista and up Use bcdedit.exe bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200 bcdedit /debug ON Reboot Read more: WindowsGuestDrivers/GuestDebugging
  74. 74. WinDbg - Connecting to target VM File -> Kernel Debug... Select previously specified serial port and baud rate
  75. 75. WinDbg - Breaking into target VM Ctrl+C Ctrl+Break Press pause button in the toolbar
  76. 76. WinDbg - break points Command Description bl List breakpoints bp Set breakpoint bu Unresolved breakpoints (will persist between module reloads) ba Breakpoint on access bc Clear breakpoint Read more: windows/hardware/ff552204(v=vs.85).aspx
  77. 77. WinDbg - stepping through the code Command Description g Start executing the given process or thread t Trace - executes single instruction or source line p Step - executes single instruction or source line. Subroutines are treated as single step. Check different variation and parameters of the above commands
  78. 78. Saving the dump file Sometimes the system doesn’t save dump file. What to do? Connect WinDbg Use “.dump /f <local_file_name>” to save dump file Wait for it. It will take a long time
  79. 79. Live debugging on same VM File -> Kernel Debug...
  80. 80. More links More on debugging - hh406283(v=vs.85).aspx Use the Microsoft Symbol Server to obtain debug symbol files - http:// Setting break points with WinDbg - hardware/ff552204(v=vs.85).aspx Working with source files - ff556911(v=vs.85).aspx Live kernel debugging - hh451166(v=vs.85).aspx Debugging techniques - ff541398(v=vs.85).aspx Enabling NDIS debug tracing - hardware/ff542961%28v=vs.85%29.aspx
  81. 81. More tools and links regedit.exe (just in case) Sysinternls tools (home of DbgView) - http:// Wireshark - Extra debugging tools - us/library/windows/hardware/ff543998(v=vs.85).aspx
  82. 82. Q&A