Debugging NET Applications With WinDBG

Debugging .NET Applications with WinDBG
Cory Foy
http://coryfoy.com | @cory_foy

Agenda
.NET Fundamentals
Debugging Crashes
Debugging Memory Leaks
Debugging Hangs
WinDBG Tips and Tricks
Other Tools

.NET Fundamentals
CLR Fundamentals
Memory Management Basics
Debugging Fundamentals

CLR Fundamentals
Managed Assembly Layout
PE Header
CLR Header
IL (Intermediate Language)
Metadata Header
Metadata

CLR Fundamentals
.NET Runtime in a Process

Demo: Viewing Managed Objects
Cory Foy

Viewing Managed Objects
Start up the app and click on the Managed Objects button
Start up WinDBG and attach to the executable (click “No” if it asks you to save the workspace) then load SOS by typing “.loadbysosmscorwks” and hitting enter. Also load symbols by typing “.symfix C:symbols” and hitting enter, then typing “.reload” and hitting enter

View All Threads by typing in “~” and hitting enter. View all managed threads by typing in “!threads” and hitting enter. Note there are 5 native threads, but only 2 managed threads. Also note that thread 2 is marked as (Finalizer) – that’s the thread responsible for finalization
Change to the main thread by typing “~0s” (tilde zero s). This tells WinDBG to switch to thread 0. Then type “!clrstack” to see where we are in the application. Note we are in the Main method right now.

Next, let’s look at the objects on the heap. Type “!dumpheap –stat”. You’ll see many objects. You can filter by typing things like -min or –max as parameters to dumpheap.
Now find all of our objects on the heap by typing “!dumpheap –type WinDBG”. This filters for any object with the text “WinDBG” in it’s name. The top list are the actual objects and their memory locations, while the bottom list are the type names. You can figure out which is which by matching up the MethodTable addresses (MT Column)

We can view a specific object using “!dumpobj” (or “!do” for short). We’ll look at the HelloWorld Object by typing “!do <address>” where address is from the !dumpheap command earlier. We can see we have a string field called “hello” and a world property.
We can then look at the string by typing “!do <address>”. In this example, the address would be 263a000. We can see that field contains the string “Hello”.

When we did a !do on the HelloWorld object, one piece of information was the MethodTable address. We can use the !dumpmt command to see what methods the object exposes. Type “!dumpmt –md <methodtableaddress>”. If the JIT Column has “PreJIT” then the method came from an Ngen’d object. If it has JIT, then the method has been called and JIT’d. If it has NONE, the method hasn’t been called.
You can dissassemble IL that has been JIT’d by passing in the MethodDesc address to “!U”

If we’ve attached to the process doing a live debug (which we’re doing here), then you can set breakpoints using !bpmd. For example, we can have it breakpoint just before the MessageBox shows up by passing in the assembly name and fully qualified type. You can use bl to see the breakpoints.
We then type “g” to release the current break we have on the app, and click the Managed Objects button again.
We’ll then see WinDBG hit the breakpoint. We can run !clrstack to see what led us to that call

CLR Fundamentals
Threads
Managed Threads is an object – it lives on native system threads
CLR Threads
Finalizer
Garbage Collector (Server GC)
Debugger
Timer
Threadpool (I/O and Worker)

CLR Fundamentals
Just-In-Time compilation
Program makes a call to a method
The .NET Runtime checks to see if the method has been called before
If so, it executes the JIT’d code
If not, it compiles the IL code and stores it in memory, updating the MethodDesc

.NET Fundamentals
CLR Fundamentals

Memory Management
Stacks versus Heaps
Stack – First in / First Out
Heap – access by address
Garbage Collector Heap
Where all objects are stored
Broken into 3 generations and one Large Object Heap
Large Object > 85,000 bytes

Memory Management
Garbage Collector Sequence
Suspend Execution Engine
Mark objects without roots
Plan (budgets, fragmentation)
Sweep (delete market objects)
Compact (move leftover objects to back of heap)
Restart Execution Engine

Memory Management
Memory Leaks
Possible in both Managed and Unmanaged Code
Use Perfmon to check for symptoms
Unmanaged Leak
Private Bytes Increase, #Bytes In All Heaps stays flat
Managed Leak
Both Private Bytes and Bytes In All Heaps increase
Need multiple dump files

Memory Management
http://blogs.msdn.com/tess/archive/2008/03/17/net-debugging-demos-lab-6-memory-leak-review.aspx

Memory Management
Why do we leak managed memory?
Objects not being released
“Pinned” memory
Finalized Objects (Destructors on Managed Objects)
Finalized objects require an additional GC cycle to be cleaned since they have to go in the finalizer thread to run

Memory Management
Exception Handling Workflow
Exception Occurs (create an exception object)
Notify Debugger (1st Chance Exception)
Look for a handler by walking up the call stack
If handler found, let it handle exception
If not, throw a 2nd Chance Exception and terminate the process

.NET Fundamentals
CLR Fundamentals

Typical Problems in Production
System hangs or deadlocks
Fatal Exceptions
Data Loss or inconsistency
Performance Problems
Excessive Memory Usage
App Pool Restarts (slow access to ASP.NET pages)

Approaching the problem
Be the application (visualize what could cause the problem)
Use Application and System Logs
Try to reproduce in Development or Staging environments
Create a hypothesis and use WinDBG to validate

Debugging Production Apps
Typically don’t have Visual Studio installed, or access to the remote debugger
Attaching a debugger freezes all threads
Capture memory dumps
At time of crash / exception
Over time to troubleshoot hangs / leaks

Agenda
.NET Fundamentals
Debugging Crashes
Debugging Hangs
WinDBG Tips and Tricks Other Tools

Debugging Crashes
Application seems to work fine, but something happens
Unhandled Exception Dialog
App “disappears”
Steps
Desktop App: Attach and stop on Exceptions
Web App: Use DebugDiag to capture memory dump

Demo: Debugging Crashes
Cory Foy

Debugging Crashes
Open Application and click the “Crash” button”. Unhandled Dialog should appear and kill the app. Start app back up and attach to it with WinDBG, then load SOS. Once SOS is loaded, type “sxeclr” to tell WinDBG to break on all .NET Exceptions, then type “g” and click on the Crash button again.
Because we’ve enabled exception notification, WinDBG breaks on the first-chance exception.

Debugging Crashes
The first place to look is the exception object itself. We can either do a “!do” on the object address listed, or simply type “!pe”. You can also pass an address to !pe if you need to view other than the last exception thrown.
We can see there is a file not found exception. Let’s see where we were at.
So it looks like the app calls “ReadAllText” from a method called “Crash_Click” from the Form1 object. If we have access to Source, we’d start there. If not, we can find the method address and dissassemble

Debugging Crashes
Note that if we click “g” at this point, WinDBG breaks again. This is because the exception we first saw was *not* the crash reason. Like the previous exception, this one is listed as a First Chance Exception.
If we click “g” again, we’ll see WinDBG breakpoint again. Note that it clearly tells us this is a second chance exception – the next thing that will happen is process termination.

Demo: Debugging Crashes on Application Startup
Cory Foy

Debugging Crashes On Startup
Open the App and check the “Crash on Startup” option, then close the app and restart. Notice it immediately crashes.
The challenge is that we won’t have time to attach a debugger. You can set registry keys to automatically launch a debugger, or we could launch the app from the debugger, which we’ll do here.
Start up WinDBG and go to File->Open Executable. Browse to the WinDBGDemo executable and select it.

At this point, we are in the PE load of the application, meaning it hasn’t even decided if this is a native or managed app yet. Which also means no .NET libraries are loaded, so we can’t load SOS.
What we can do is set a breakpoint to fire when the .NET runtime is loaded, at which point we can set the handlers we need. Type “sxe ld mscorwks” which means set WinDBG to break when mscorwks is loaded. We can now load SOS and set the CLR Exception breakpoints. Now type “g” till our app breaks. You can now debug it as a crash.

If you have a Windows Service which is crashing on startup, then you’ll need to modify some registry keys and permissions on the service. For more information, see the following KB article, or the blog post from our site
(Note: When you finish this demo, go to your user directoryAppDataLocalMicrosoftWinDBGDemo.exe1.0.0.0 and modify the user.config file to have CrashOnStartup to be False)
http://support.microsoft.com/kb/824344
http://blog.coryfoy.com/2008/06/debugging-a-net-windows-service-that-is-crashing-on-startup/

Agenda
.NET Fundamentals
Debugging Crashes
Debugging Hangs

Memory Usage Continues to grow in the app
May cause app pool restarts in ASP.NET
Out of Memory exceptions when you still have memory free
Several Reasons
Unreleased objects
“Pinned” memory
You have plenty of memory! (GC only runs when it feels memory pressure)

Diagnosing Memory Leaks
PerfMon to determine if it is a managed leak
Take multiple memory dumps to determine what is happening
For the objects on the heap, find out why they are sticking around
Check the GC Generation for the objects

Demo: Debugging Memory Leaks
Cory Foy

Start the app and task manager, then click on the Memory Leak button. Note that the memory is still increasing.
Fire up Perfmon to see if this is a managed or native leak. Start->Run->Perfmon. You can remove the Processor Time counter and add the following counters: .NET CLR Memory -> #Bytes in all Heaps -> WinDBGDemo and Process -> Private Bytes -> WinDBGDemo.
You should see that both are growing at about the same rate indicating a managed leak

To debug this, we’ll need to take two snapshots of the application far enough apart to see memory difference. You can do this right from Task Manager. Create two, about 15 seconds apart. You can now kill the application.
Open two instances of WinDBG and open each memory dump file you created in a different instance of WinDBG. Load up SOS in each one as well.

What we’re wanting to do is compare the two memory dumps to see if we can spot the culprit. Using !VMStat on each file we can see that memory usage is certainly increasing.
And if we do a “!dumpheap –stat” on each one, we can see something quite startlingly – 230,000 additional objects are on the heap!

Normally you’d have to dig through the heap to find which objects increased, but we see something interesting. An object called “LeakForm” has had 900 instances created. Perhaps those aren’t being cleaned up?
Let’s look at one. We’ll see all the instances by doing a “!dumpheap -type WinDBGDemo.LeakForm” then picking any object at random to do a !do on
With that object address, let’s see what is holding on to it with “!GCRoot <address>”

We see that the LeakForm is being held on to an object, which is of type System.Object[]. If we do a !do on it, we don’t see much of interest
But if we compare the arrays between the first dump and the second dump using the “!da <address>” command, we find a clue. Between the first and second memory dumps, all of the extra LeakForms were added to this array!
At this point, we examine the source code to see what is going on.

Agenda
.NET Fundamentals
Debugging Crashes
Debugging Hangs

Debugging Hangs
Two types of hangs
Low CPU Hang
Generally caused by a deadlock
High CPU Hang
App in a tight loop, or under heavy load

Debugging Hangs
Diagnosing Hangs
High CPU Hang
Multiple memory dumps
Compare which thread is getting processor time
Low CPU Hang
Walk through the waiting threads and match locks
Very tedious process

Demo: Debugging High CPU Hangs
Cory Foy

Debugging High CPU Hangs
Start the app and Task Manager. Click on the High CPU Hang. You should see a single processor spike (100% on 1CPU). In this example, I have a 4CPU system, so a CPU of 25% indicates it is fully utilizing one processor.
Take two memory dumps from Task Manager about 15 seconds apart. You can then kill the application.

Open the two memory dump files in different WinDBG instances and load SOS.
The first thing we need to know is if a single thread is using up all of the time, indicating a possible loop. We can use the command !runaway to see which threads are getting CPU time
We can see from this that thread 0 was using up all of the time in between the two memory dumps

So what is thread 0 doing? By running !clrstack on each memory dump, we can see that the call stacks are the same, possibly indicating that this is the culprit.
I say possibly because if a bunch of calls are happening to different methods, we very well could have gotten “lucky” and seen the same method. Always verify your assumptions.
In this case, looking at the source (either real source, or through reflector) we see the following. I think we have our culprit

Agenda
.NET Fundamentals
Debugging Crashes
Debugging Hangs
Other Tools

Demo: WinDBG Tips and Tricks
Cory Foy

You can get quite a bit of help with both WinDBG and SOS.
For the WinDBG help content, type “.hh” and it will open the WinDBG CHM file. You can also type “.hh <command>” and it will take you to that section
For SOS, you can type “!help” to see all available commands, or “!help <command>” to see a specific one

You can use the shell command to run the output of a WinDBG command to an external process and the results are displayed back into the UI. For example, to quickly find the shared domain, you can run ‘.shell –ci “!dumpdomain” find /i “shared domain”’
You can also take advantage of this to do integrations with PowerShell

When viewing stacks, you can use !clrstack to view the managed stack. You can also use the command “k” to view the native stack
You can also output the stack for all threads by combining the “~” command with a star (for all threads) and the command. For example “~*k” or “~*e!clrstack” (which means execute the command !clrstack)

You can do a lot with automation of WinDBG. For example, you can have WinDBG execute commands when certain breakpoints are hit
You can also automate what happens when CLR Exceptions are hit by using the –c option with sxe. In this example, every time we hit an exception we would spit out the exception and CLR stack trace.

You can also loop through various elements. For example, if we wanted to dump out all strings greater than 5k that were on the heap, we could do the following with the “.foreach” command
Finally, if you want to adjust the colors, you can do that in View->Options

Agenda
.NET Fundamentals
Debugging Crashes
Debugging Hangs
Other Tools

Additional Tools
DebugDiag
Used to automate the capture of exceptions for IIS processes
ADPlus
Used to capture dumps from command line
Managed Debuggers (CorDbg, MDbg)
Managed Debugging Assistants (Visual Studio)
Profilers

More Information
http://blogs.msdn.com/tess
microsoft.com/whdc/DevTools/Debugging
http://windbg.info
http://www.coryfoy.com
foyc at coryfoy dot com
@cory_foy on Twitter
Slides will be posted on CoryFoy.com

http://blog.coryfoy.com	1062
http://www.slideshare.net	77
http://webcache.googleusercontent.com	6
http://static.slidesharecdn.com	2
https://p.yammer.com	1
http://translate.googleusercontent.com	1

Debugging NET Applications With WinDBG

by Cory Foy on Feb 12, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

6 Embeds 1,149

Statistics

Debugging NET Applications With WinDBG — Presentation Transcript