Introduction To Debugging ASP.NET applications withWinDBG and Dump Analysis AHMET MİTHAT BOSTANCI MICROSOFT, GTSC IIS/WEB DEVELOPMENT SUPPORT AHMET.BOSTANCI@MICROSOFT.COM
SCOPE This is NOT a full dump analysis workshop. The goal is to introduce the dump analysis as one of the ASP.NET 2.0 web application debugging technics and introduce the tools that help to achieve this task.
AGENDA ASP.NET Debugging Technics What is WinDBG When are WinDBG and dump analysis useful? What are dump files? How to collect dump? Basic commands Toolbox & References
ASP.NET Debugging Old School Debugging Using Response.Write Mostly not useful in production Not useful in hang scenario Usful in development Useful in production if there is a repro Looks easy but not useful Debugging in VS.NET Good for debugging in development Useful in production with remote debugging / attaching to w3wp.exe Not useful in most production env. issues. Debugging with WinDBG Live debugging / attaching to w3wp.exe Offline debugging: Collecting dumps of w3wp.exe and analysing in WinDBG
What is WinDBG A GUI Debugger with commands. Part of Debugging Tools For Windows User mode and kernel mode debugging. Its commands are mainly useful in unmanaged debugging. Extensions are sets of commands to ease debugging. SOS and PSSCOR2 extensions for .NET debugging.
Debugging with WinDBG Offline debugging Collecting dump files with specific tools for different scenarios Analysing in WinDBG. Live debugging Attaching a process Using breakpoints, catching exceptions, dumping the process out to a file, etc... when necessary.
Debugging Scenarios Slow performance Web site is responsive but slow. Hang Web site is unresponsive. Crash Exceptions
Exceptions and ASP.NETExceptions are important objects in debugging. First Chance exceptions Too many exceptions may cause high CPU. Second Chance Exceptions which are not handled Causes crash. First chance exceptions may cause w3wp.exe crashes, if it is thrown in a background thread, such as finalizer threads.
When do we need WinDBG?Hang / Performance Problems – Manuel Web site is running slow / performance issues. Web site is leaking memory - ends with performance problems, hang or crash. Web site is running with high memory but not not leaking. Web site is running with 100% CPU. Web site is running with high CPU or CPU usage spikes. Web site is completely hung - Does not response any (dynamic) requests.
When do we need WinDBG?Crash Web site or a specific page / method is throwing exceptions but not crashing the w3wp.exe process, first chance exceptions. Web site is crashing - w3wp.exe is crashing. Second chance exceptions.
When do we NOT need WinDBG? When there is an easier way for debugging, such as a specific page / method is not functioning as expected In this case, debugging in VS.NET with a repro code might be easier. «Page cannot be displayed» type issues. In this case, troubleshooting IIS or network side is the correct approach.
DumpA dump is a snapshot of a process (user mode) or asnapshot of all of the memory in kernel mode. Usermode dump: Process specific, such as w3wp.exe Kernel mode dump: Snapshot of the kernel memory Complete memory dump: Snapshot of the whole memory including kernel and user mode.
Types of Dump Minidump Contains only selected parts of the memory used by the process. Fulldump Contains the full copy of the process memory. Size of the fulldump is somewhat equal to the virtual memory used by the process. This is the dump type we are interested in most of the times.
What a dump contains All the things in the memory of the process. Information about the objects in heaps, threads, information about the callstack, stack memory, etc... Limited information about the dead objects which are not garbage collected yet.
What a dump does NOT contain Other processes information. What was the CPU or memory usage in time. It is only a snapshot. Also: Minidump is the smallest in size and does not contain some useful information such as stack objects, local variables, etc...
Tools for collecting dumps ADPlus Part of Debugging Tools For Windows Command line adplus -hang -pn w3wp.exe adplus -crash –iis Debug Diagnostic Tool GUI based Easy to configure and use ProcDump Command line Specific for high CPU usage but can be used for collecting manuel dumps as well procdump.exe -c 80 -s 30 -n 5 -ma NNNN >procdump.log Task manager Can be used to collect manuel dump Right click and create userdump Only on Win 7 and Win 2008
32bit vs 64bit If the process is 32bit then the dump should be taken by the 32bit version of the related tools. The dump should be opened with the WinDBG’s 32bit version. Same rule applies to 64bit process.
Which dump in which scenario? Web site is crashing / throwing exceptions Crash rule with Debug Diagnostic Tool, ADPlus Web site is slow, hanging, using high memory Manuel dump(s) with Debug Diagnostic Tool, ADPlus. Web site is using high CPU or CPU is spiking Manuel dump(s) with Debug Diagnostic Tool, ADPlus. ProcDump for CPU spikes Web site is leaking memory Manuel hang dumps. Memory leak rule created in Debug Diagnostic Tool. Full IIS dumps Manuel dump Collects dumps for all of w3wp.exe, all of dllhost.exe and inetinfo.exe process.
How many dump files should I collect? Dump is a snapshot of the memory and does not contain the process history. Most of the crash scenarios One crash dump is enough to start debugging and find the reason of the crash. Performance, hang Consecutive dump files of the same process. For example taking three dumps of w3wp.exe, each are taken one minute apart.
Symbol Files Footprint information of functions in EXE or DLL files Necessary to see useful information in dumps, such as callstacks or variables.
Private vs Public symbols Private symbols contain local variables, structure type information, functions, source line no., etc... Public symbols are the stripped version of the private symbols.
Symbols and WinDBG Symbol server http://msdl.microsoft.com/download/symbols Setting symbol server in WinDBG srv*c:symbolspublic*http://msdl.microsoft.com/download/ symbols Symbols are downloaded to the local folder when needed.
Commands in WinDBG WinDBG commands Mostly used in unmanaged debugging. Extensions DLL files Command sets for specific debugging requirements
Extensions Copied to the WinDBG installation directory .NET related extensions SOS Comes with .NET framework PSSCOR2 Superset of SOS .NET 2.0 specific extension http://www.microsoft.com/downloads/details.aspx?FamilyID=5c 068e9f-ebfe-48a5-8b2f-0ad6ab454ad4&displayLang=en Extract and copy to the WinDBG installation folder. Load from WinDBG: .load psscor2
WinDBG and ASP.NET 2.0 dump analysis Open dump file Check and set symbol server Load the PSSCOR2 extenstion .load psscor2
Some WinDBG commands Vertarget Shows information about the system on which you are debugging Lm Displays loaded module information ~ Lists all OS threads ~Ns Switch to thread #N K Shows unmanaged call stack of the thread switched. .hh Opens help file .reload /f Forces to reload the symbols. !runaway Shows the thread’s CPU time
Some PSSCOR2 commands !DumpAllExceptions (!dae) Shows managed exceptions thrown. !DumpDataTables Shows data tables objects in memory. Useful when debugging memory issues. !DumpColumnNames (!dcn) Prints out the column names for a data table given. !ASPXPages Shows ASPX pages running on all threads. !threadpool Shows ASP.NET thread pool information including the requests in the queue and the current CPU usage of OS. !FindDebugTrue Shows the web applications running in debug mode. !FindDebugModules Shows the modules built in debug mode. !SaveAllModules (!sam) Saves the all modules in the process. Useful when using the reflector to get the code from dump file. !eeversion Prints the .NET framework version. Useful when checking the fixes or service packs installed. !help !help <command>
Toolbox Debugging Tools For Windows (WinDBG and ADPlus) http://www.microsoft.com/whdc/DevTools/Debugging/default.msp x Debug Diagnostic Tool v1.1 http://www.microsoft.com/downloads/details.aspx?FamilyID=28B D5941-C458-46F1-B24D-F60151D875A3&displaylang=en ProcDump http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx PSSCOR2 http://www.microsoft.com/downloads/details.aspx?familyid=5C068 E9F-EBFE-48A5-8B2F-0AD6AB454AD4&displaylang=en