Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Windows Guest Debugging
Yan Vugenfirer, yan@daynix.com
Daynix Computing LTD
Agenda
Device manager
Logging and DebugView
Introduction to BSOD
Crash dumps
Crash on demand
Driver verifier
WinDbg
Install...
Device manager
Device Manager
Run ->
devmgmt.msc
Devices and drivers
management and
installation
Scan for HW
changes
Effective tool to
ch...
Device Manager - Views
Change the
views of
device
manager
Show non
PNP devices
Device Manager - Views
View ->
Devices by
connection
This view
can be used
to check
buses and
devices
relations
Device Manager - Views
View -> Resources
by type
Effective to check
IRQ sharing
On OSes below
Vista virtio devices
will us...
Device Manager - Views
View -> Resources
by connection
IO and memory
address space
with relations
between buses
and the de...
Device Manager - Show non-
present devices
Show devices drivers that are installed but the device is curently not present ...
Device manager - properties
Double click on the device
Click on “General” tab
You can view device
location on the bus
If t...
Installing self signed drivers
Drivers need to be at least self signed starting from Vista x64 bit
To enable test signing:...
Device Manager - resources
Double click on the
device
Click on “Resources”
tab
Shows device
resources (IRQs, IO
and memory...
Device manager - driver tab
Double click on the
device
Click on “Driver” tab
Get driver version
View the driver
binaries
V...
Device manager - details
Double click on the
device
Click on “Details”
tab
A lot of additional
details like: PNP ID
of the...
VirtIO - NetKVM parameters
Advanced tab in device manager
Change logging levels
Enable packet statistics printout
Other pa...
NetKVM - log levels
0 - Basic configuration and unload trace. Critical errors.
1 - Warnings, corner cases
2 - Network packe...
NetKVM parameters from
command line
Part of WHQL requirements to supply command line utility for
drivers that expose param...
NetKVM parameters from
command line
Description Command line Output
List devices netsh netkvm show devices
Names and index...
Debug view and tracing
DebugView
Shows debug space of Windows (kind of dmesg).
Can extract debug trace from crash dump if it was
running before t...
DebugView - enable output
to debug space
To enable debug space outputs from Vista and up:
Through registry (using regedit)...
Debug view - capture kernel
Capturing
kernel output
disabled by
default
To enable:
Capture ->
Capture
Kernel
Debug view - log boot
Run
DebugView
Enable boot
logging
Run
DebugView
after reboot to
collect the
logs
Debug view - crash dump
Run
DebugView
before crash
dump creation
After crash
and reboot -
point
DebugView to
crash locatio...
Virtio debugging with serial port
Uncomment
“ENABLE_COM_DEBUG=1”
in SOURCES file. Comment
out
“ENABLE_PRINT_DEBUG”
Rebuild ...
Blue screens
Bug check
System crash called “Bug check”
Blue screen
Crash dump will be saved
Ability to debug the bug check if debugger ...
Blue screen inside
Bug check code
Bug check parameters
- specific to each bug
check
Additional information
- can be located...
Common bug checks
Bug Check 0xA: IRQL_NOT_LESS_OR_EQUAL
Bug Check 0x8E: KERNEL_MODE_EXCEPTION_NOT_HANDLED
Bug Check 0xD1: ...
Crash dumps
Crash dump creation
During blue screen Windows will save crash dump
Minidumps are not always useful
Also - remove the opti...
Crash dump creation
Crash dump locations
After the crash search in:
%windir%memory.dmp
%windir%MinidumpMinidump<timestamp>.dmp
When you have W...
Crash dumps
From Windows 7 and up - OS will auto-delete large
crash dumps
To keep crash dumps:
Key: HKLMSystemCurrentContr...
Crash dumps on demand
Crash Windows on demand to examine driver status
Configure through registry:
Key: HKEY_LOCAL_MACHINES...
Crash dumps on demand
Press Ctrl+ScrollLock+ScrollLock
You will get MANUALLY_INITIATED_CRASH
(0x000000E2) bug check.
Read ...
Crash dumps on demand (2)
Issue NMI from QEMU monitor
Setup guest:
HKLM/System/CurrentControlSet/Control/CrashControl,
DWO...
Driver verifier
Driver verifier
Built-in into OS (substitutes OS functions with own
layer).
To activate run “verifier.exe” from command line...
Driver verifier - activation
Run from
command line:
verifier.exe
Select “Create
custom
settings”
Driver verifier - activation
Driver verifier - activation
Change
selection to
“Select
driver names
from a list”
Driver verifier - activation
Select loaded
driver from the list
Or select driver
from its location on
files system
After cli...
Driver verifier - bug checks
0x000000C4
Some can be
ignored if
debugger is
connected
Provide
additional
information for
deb...
Driver verifier - current settings
Run verifier.exe
Click on “Display
existing settings”
radio button
Click on “Next”
button
Driver verifier - current settings
Driver verifier - removal
Don’t forget to
reboot
WinDbg
WinDbg
We can examine crash dumps with WinDbg
Remote live kernel debugging
Limited live kernel debugging using one guest o...
Installation
Install as part of the WDK
http://msdn.microsoft.com/en-US/windows/
hardware/hh852362
Install as part of the ...
Symbols configuration
View OS functions and variable
names
Set Microsoft symbol server and
local paths
Set environment vari...
Symbol configuration
Another way to set the
symbols path is through
WinDbg menu
File ->”Symbol File
Path...”
You can instru...
Post mortem debugging
Run
WinDbg.exe
Load dump
file through:
File -> “Open
Crash
Dump...”
Might be slow
first time
Loading crash dump
Information
about loaded
symbols
OS information
Hint to use
“!analyze -v”
Bug check
code and
parameters
Magic wand - !analyze -v
!analyze -v
First command that you should run (or click)
Bug check and its parameters explanation
Hints for the next steps...
WinDbg - help
Help ->
Contents
Switch to
“Index” tab
for specific
command
help
WinDbg - views
WinDbg - Views
View To display window
From command
line
Memory Alt+5 d <address>
Call stack Alt+6 kb
Disassembly Alt+7 u <...
WinDbg - View memory
Menu: View ->
Memory
Command
(different
variation of “d”):
db <address>
Use
case -
memory
leak
with
verifier
WinDbg - View disassembly
Windbg - Reference symbols
<modulename>!<symbol_name>
Example:
netkvm!DriverEntry
WinDbg - Reference symbols
Useful command
Command Description
!scsikd Scsi related commands. Use !scsikd.help
to find more
.reload Reload symbols
lm L...
!ndiskd - NDIS debug extensions
!ndiskd.ndis Show ndis.sys build information
!ndiskd.help Show help
!ndiskd.miniports Show...
!ndiskd.miniports
!ndiskd.miniport <miniport handle>
From !ndiskd to NetKVM
Configuring live debugging
Host side
Run debugget host VM with : “-serial
tcp::<port_number>,server,nowait”
For example: “-...
Configuring live debugging (2)
Host side (new format)
Run debugget host VM with : “-chardev
socket,id=serial0,path=/tmp/deb...
Configuring live debugging
Guest side - Windows XP and Windows 2003
Edit c:boot.ini
Duplicate the default boot line and add...
Configuring live debugging
Guest side - Vista and up
Use bcdedit.exe
bcdedit /dbgsettings SERIAL DEBUGPORT:1
BAUDRATE:11520...
WinDbg - Connecting to target VM
File -> Kernel
Debug...
Select previously
specified serial
port and baud
rate
WinDbg - Breaking into target VM
Ctrl+C
Ctrl+Break
Press pause
button in the
toolbar
WinDbg - break points
Command Description
bl List breakpoints
bp Set breakpoint
bu Unresolved breakpoints (will persist be...
WinDbg - stepping through the code
Command Description
g Start executing the given process or thread
t
Trace - executes si...
Saving the dump file
Sometimes the system doesn’t save dump file. What to
do?
Connect WinDbg
Use “.dump /f <local_file_name>”...
Live debugging on same VM
File -> Kernel
Debug...
More links
More on debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/
hh406283(v=vs.85).aspx
Use the Mi...
More tools and links
regedit.exe (just in case)
Sysinternls tools (home of DbgView) - http://
technet.microsoft.com/en-US/...
Q&A
Upcoming SlideShare
Loading in …5
×

Windows guest debugging presentation from KVM Forum 2012

12,719 views

Published on

This presentation will show how to debug Windows virtual machine running on top of open source hypervisor KVM.

Published in: Technology
  • Be the first to comment

Windows guest debugging presentation from KVM Forum 2012

  1. 1. Windows Guest Debugging Yan Vugenfirer, yan@daynix.com Daynix Computing LTD
  2. 2. Agenda Device manager Logging and DebugView Introduction to BSOD Crash dumps Crash on demand Driver verifier WinDbg Installation and configuration Postmortem debugging !ndiskd Configuring guests Live debugging
  3. 3. Device manager
  4. 4. Device Manager Run -> devmgmt.msc Devices and drivers management and installation Scan for HW changes Effective tool to check resource allocations of the devices, drivers versions and different drivers settings.
  5. 5. Device Manager - Views Change the views of device manager Show non PNP devices
  6. 6. Device Manager - Views View -> Devices by connection This view can be used to check buses and devices relations
  7. 7. Device Manager - Views View -> Resources by type Effective to check IRQ sharing On OSes below Vista virtio devices will use legacy PCI interrupts Virtio-serial and ballon use legacy interrupts also on newer OSes.
  8. 8. Device Manager - Views View -> Resources by connection IO and memory address space with relations between buses and the devices
  9. 9. Device Manager - Show non- present devices Show devices drivers that are installed but the device is curently not present in the system Very useful for removing crashing drivers from the system From command line: SET devmgr_show_nonpresent_devices=1 START devmgmt.msc
  10. 10. Device manager - properties Double click on the device Click on “General” tab You can view device location on the bus If the driver is not started you can see the reason in status text-box. For example - no digital signature on x64 OS from Vista and up
  11. 11. Installing self signed drivers Drivers need to be at least self signed starting from Vista x64 bit To enable test signing: Bcdedit.exe –set TESTSIGNING ON Disable test signing Bcdedit.exe –set TESTSIGNING OFF Install test certificate on the guest Reboot needed after enabling and disabling test siging commands More information on test signing: http://msdn.microsoft.com/en-us/library/bb530195.aspx
  12. 12. Device Manager - resources Double click on the device Click on “Resources” tab Shows device resources (IRQs, IO and memory ranges)
  13. 13. Device manager - driver tab Double click on the device Click on “Driver” tab Get driver version View the driver binaries View the versions of driver binaires
  14. 14. Device manager - details Double click on the device Click on “Details” tab A lot of additional details like: PNP ID of the device, PM policy, location of different device keys in registry and etc
  15. 15. VirtIO - NetKVM parameters Advanced tab in device manager Change logging levels Enable packet statistics printout Other parameters starting with “TestOnly” prefix Full list: http://www.linux-kvm.org/ page/ Internals_of_NDIS_driver_for_VirtIO _based_network_adapter#List_of_ configurable_parameters
  16. 16. NetKVM - log levels 0 - Basic configuration and unload trace. Critical errors. 1 - Warnings, corner cases 2 - Network packet trace 3 - More verbose trace of packets 4 - Virtio library, DPC 5 - ISR trace 6 - Registers readwrite.
  17. 17. NetKVM parameters from command line Part of WHQL requirements to supply command line utility for drivers that expose parameters through “advanced” tab readme.doc - supplied with driver package explains the usage. Integrated with “netsh” Windows networking tool To register (before the first use): rundll32 netkvmco.dll,RegisterNetKVMNetShHelper Unregistration (removal) rundll32 netkvmco.dll,UnregisterNetKVMNetShHelper
  18. 18. NetKVM parameters from command line Description Command line Output List devices netsh netkvm show devices Names and index for each device Restart device netsh netkvm restart [idx] Should be done after parameters are changed Show parameters netsh netkvm show parameters [idx] List the names of the parameters Show detailed information about parameter netsh netkvm show paraminfo [idx] [param] Current value of the parameter Change parameter value netsh netkvm setparam [idx] [param] [value]
  19. 19. Debug view and tracing
  20. 20. DebugView Shows debug space of Windows (kind of dmesg). Can extract debug trace from crash dump if it was running before the crash Can log debug trace during boot. Download from - http://technet.microsoft.com/en-us/ sysinternals/bb896647.aspx
  21. 21. DebugView - enable output to debug space To enable debug space outputs from Vista and up: Through registry (using regedit): Add “Default” value of the type DWORD equal to 0xFFFFFFFF under “HKEY_LOCAL_MACHINESYSTEMCurrentControlSet ControlSession ManagerDebug Print Filter” Reboot During live debugging ed nt!Kd_DEFAULT_MASK 0xFFFFFFFF
  22. 22. Debug view - capture kernel Capturing kernel output disabled by default To enable: Capture -> Capture Kernel
  23. 23. Debug view - log boot Run DebugView Enable boot logging Run DebugView after reboot to collect the logs
  24. 24. Debug view - crash dump Run DebugView before crash dump creation After crash and reboot - point DebugView to crash location for processing
  25. 25. Virtio debugging with serial port Uncomment “ENABLE_COM_DEBUG=1” in SOURCES file. Comment out “ENABLE_PRINT_DEBUG” Rebuild the driver On the host side -serial tcp::<port>,server,nowait nc localhost <port>
  26. 26. Blue screens
  27. 27. Bug check System crash called “Bug check” Blue screen Crash dump will be saved Ability to debug the bug check if debugger is connected
  28. 28. Blue screen inside Bug check code Bug check parameters - specific to each bug check Additional information - can be located also below bug check code and parameters Crash dump saving counter
  29. 29. Common bug checks Bug Check 0xA: IRQL_NOT_LESS_OR_EQUAL Bug Check 0x8E: KERNEL_MODE_EXCEPTION_NOT_HANDLED Bug Check 0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL Bug Check 0x9F: DRIVER_POWER_STATE_FAILURE Bug Check 0xC4: DRIVER_VERIFIER_DETECTED_VIOLATION Bug check codes reference: http://msdn.microsoft.com/en-us/ library/hh994433(v=vs.85).aspx
  30. 30. Crash dumps
  31. 31. Crash dump creation During blue screen Windows will save crash dump Minidumps are not always useful Also - remove the option to reboot on crash Right click on “My computer” -> “Properties”->”Advanced system settings”->”Startup and Recovery” -> “Settings” Chose “Kernel memory dump” Remove “Automatically restart” The windows on different OSes can be different but the menu names are the same
  32. 32. Crash dump creation
  33. 33. Crash dump locations After the crash search in: %windir%memory.dmp %windir%MinidumpMinidump<timestamp>.dmp When you have Windows error message after reboot - do not close it. Check minidump path and copy it.
  34. 34. Crash dumps From Windows 7 and up - OS will auto-delete large crash dumps To keep crash dumps: Key: HKLMSystemCurrentControlSetControl CrashControl Value: “AlwaysKeepMemoryDump”:DWORD set to 1
  35. 35. Crash dumps on demand Crash Windows on demand to examine driver status Configure through registry: Key: HKEY_LOCAL_MACHINESystem CurrentControlSetServicesi8042prtParameters, Value: CrashOnCtrlScroll, equal to a REG_DWORD value of 0x01 Key: HKEY_LOCAL_MACHINESystem CurrentControlSetServiceskbdhidParameters, value: CrashOnCtrlScroll, equal to a REG_DWORD value of 0x01
  36. 36. Crash dumps on demand Press Ctrl+ScrollLock+ScrollLock You will get MANUALLY_INITIATED_CRASH (0x000000E2) bug check. Read more (also how to redefine the shortcuts): http:// msdn.microsoft.com/en-us/library/ff545499.aspx
  37. 37. Crash dumps on demand (2) Issue NMI from QEMU monitor Setup guest: HKLM/System/CurrentControlSet/Control/CrashControl, DWORD value NMICrashDump 1 Restart Enter QEMU monitor (Alt+Ctrl+2, to go back Alt+Ctrl+1) Issue “NMI” command
  38. 38. Driver verifier
  39. 39. Driver verifier Built-in into OS (substitutes OS functions with own layer). To activate run “verifier.exe” from command line Need a reboot after the settings are changed Effective in finding deadlocks, memory leaks, memory corruptions and incorrect parameters to OS functions.
  40. 40. Driver verifier - activation Run from command line: verifier.exe Select “Create custom settings”
  41. 41. Driver verifier - activation
  42. 42. Driver verifier - activation Change selection to “Select driver names from a list”
  43. 43. Driver verifier - activation Select loaded driver from the list Or select driver from its location on files system After clicking finish - reboot is needed to activate the settings.
  44. 44. Driver verifier - bug checks 0x000000C4 Some can be ignored if debugger is connected Provide additional information for debugging
  45. 45. Driver verifier - current settings Run verifier.exe Click on “Display existing settings” radio button Click on “Next” button
  46. 46. Driver verifier - current settings
  47. 47. Driver verifier - removal Don’t forget to reboot
  48. 48. WinDbg
  49. 49. WinDbg We can examine crash dumps with WinDbg Remote live kernel debugging Limited live kernel debugging using one guest only User mode debugging including remote debugging
  50. 50. Installation Install as part of the WDK http://msdn.microsoft.com/en-US/windows/ hardware/hh852362 Install as part of the SDK http://msdn.microsoft.com/en-US/windows/ hardware/hh852363 Download symbols for different Windows OSes http://msdn.microsoft.com/en-us/windows/ hardware/gg463028
  51. 51. Symbols configuration View OS functions and variable names Set Microsoft symbol server and local paths Set environment variable: “_NT_SYMBOL_PATH” Sample symbol path: C:symbolslocal;SRV*C: symbolswebsymbols*http:// msdl.microsoft.com/ download/symbols
  52. 52. Symbol configuration Another way to set the symbols path is through WinDbg menu File ->”Symbol File Path...” You can instruct WinDbg to reload the symbols Also using windbg command: .reload
  53. 53. Post mortem debugging Run WinDbg.exe Load dump file through: File -> “Open Crash Dump...” Might be slow first time
  54. 54. Loading crash dump Information about loaded symbols OS information Hint to use “!analyze -v” Bug check code and parameters
  55. 55. Magic wand - !analyze -v
  56. 56. !analyze -v First command that you should run (or click) Bug check and its parameters explanation Hints for the next steps and how those parameters can be used for further debugging Module name that probably caused the crash Stack trace
  57. 57. WinDbg - help Help -> Contents Switch to “Index” tab for specific command help
  58. 58. WinDbg - views
  59. 59. WinDbg - Views View To display window From command line Memory Alt+5 d <address> Call stack Alt+6 kb Disassembly Alt+7 u <address> Registers Alt+4 r Watch Alt+2
  60. 60. WinDbg - View memory Menu: View -> Memory Command (different variation of “d”): db <address>
  61. 61. Use case - memory leak with verifier
  62. 62. WinDbg - View disassembly
  63. 63. Windbg - Reference symbols <modulename>!<symbol_name> Example: netkvm!DriverEntry
  64. 64. WinDbg - Reference symbols
  65. 65. Useful command Command Description !scsikd Scsi related commands. Use !scsikd.help to find more .reload Reload symbols lm List loaded modules ~<cpu number> Switch between CPUs. For example ~0 !irpfind Search for IO request packets !irp Display information regarding specific IRP .reboot Restart remote during live debugging
  66. 66. !ndiskd - NDIS debug extensions !ndiskd.ndis Show ndis.sys build information !ndiskd.help Show help !ndiskd.miniports Show NDIS miniports !ndiskd.miniport <addr> Show info about specific mimiport !ndiskd.nbl Show NET_BUFFER_LIST info !ndiskd.oid Show pending OID requests
  67. 67. !ndiskd.miniports
  68. 68. !ndiskd.miniport <miniport handle>
  69. 69. From !ndiskd to NetKVM
  70. 70. Configuring live debugging Host side Run debugget host VM with : “-serial tcp::<port_number>,server,nowait” For example: “-serial tcp::4445,server,nowait” Run target VM with: “-serial tcp: 127.0.0.1:<port_number>”
  71. 71. Configuring live debugging (2) Host side (new format) Run debugget host VM with : “-chardev socket,id=serial0,path=/tmp/debug- serial0,server,nowait -device isa- serial,chardev=serial0,id=serial0 ” Run target VM with: “-chardev socket,id=serial0,path=/tmp/debug-serial0 - device isa-serial,chardev=serial0,id=serial0”
  72. 72. Configuring live debugging Guest side - Windows XP and Windows 2003 Edit c:boot.ini Duplicate the default boot line and add: “/debug /debugport=COM1 /baudrate=115200” Reboot
  73. 73. Configuring live debugging Guest side - Vista and up Use bcdedit.exe bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200 bcdedit /debug ON Reboot Read more: http://www.linux-kvm.org/page/ WindowsGuestDrivers/GuestDebugging
  74. 74. WinDbg - Connecting to target VM File -> Kernel Debug... Select previously specified serial port and baud rate
  75. 75. WinDbg - Breaking into target VM Ctrl+C Ctrl+Break Press pause button in the toolbar
  76. 76. WinDbg - break points Command Description bl List breakpoints bp Set breakpoint bu Unresolved breakpoints (will persist between module reloads) ba Breakpoint on access bc Clear breakpoint Read more: http://msdn.microsoft.com/en-us/library/ windows/hardware/ff552204(v=vs.85).aspx
  77. 77. WinDbg - stepping through the code Command Description g Start executing the given process or thread t Trace - executes single instruction or source line p Step - executes single instruction or source line. Subroutines are treated as single step. Check different variation and parameters of the above commands
  78. 78. Saving the dump file Sometimes the system doesn’t save dump file. What to do? Connect WinDbg Use “.dump /f <local_file_name>” to save dump file Wait for it. It will take a long time
  79. 79. Live debugging on same VM File -> Kernel Debug...
  80. 80. More links More on debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/ hh406283(v=vs.85).aspx Use the Microsoft Symbol Server to obtain debug symbol files - http:// support.microsoft.com/kb/311503 Setting break points with WinDbg - http://msdn.microsoft.com/en-us/library/windows/ hardware/ff552204(v=vs.85).aspx Working with source files - http://msdn.microsoft.com/en-us/library/windows/hardware/ ff556911(v=vs.85).aspx Live kernel debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/ hh451166(v=vs.85).aspx Debugging techniques - http://msdn.microsoft.com/en-us/library/windows/hardware/ ff541398(v=vs.85).aspx Enabling NDIS debug tracing - http://msdn.microsoft.com/en-us/library/windows/ hardware/ff542961%28v=vs.85%29.aspx
  81. 81. More tools and links regedit.exe (just in case) Sysinternls tools (home of DbgView) - http:// technet.microsoft.com/en-US/sysinternals Wireshark - http://www.wireshark.org/download.html Extra debugging tools - http://msdn.microsoft.com/en- us/library/windows/hardware/ff543998(v=vs.85).aspx
  82. 82. Q&A

×