This document provides an overview of analyzing SQL Server crash dumps using debugging tools like WinDbg. It discusses the tools available in Debugging Tools for Windows and how to use them to analyze dump files. The agenda includes a Windows architecture refresher, how to handle and analyze SQL Server dumps, and debugging .NET applications using SOS extensions. It also provides several demonstrations of analyzing real-world issues like a non-yielding scheduler, DBCC CHECKDB errors, and cluster resource issues using these tools.

1. SQL Server CrashDumpAnalysisA brief tour withWinDbg and otheruglytoolsPablo Álvarez DovalDebugging & OptimizationTeam Leadpablod@plainconcepts.com

2. Who am I?

5. SessionObjectivesWhatisthissessionabout?Whatisn’tthissessionabout?

7. Who are you?

8. AgendaTools of theTradeBrief Windows ArchitectureRefresherSQL Server Post-mortem DebuggingHandling SQL Server dumpsAnalyzing SQL Server dumpsDebugging .NET Applicationswith SOS

9. Debugging Tools for WindowsFree download:http://www.microsoft.com/whdc/devtools/debuggingUpdated several times a yearDebuggers, extensions, tools and a great help file:windbg.exe, kd.exe, cdb.exegflags.exe, tlist.exe, etcdebugger.chmCan be installed via xcopy

10. Demo 0: … isitreally so ugly?

11. ThesaurusJust to keep with the forensics analogy:Corpse  Dump fileForensic Lab  WinDbgForensic Scientist  You!Gray’s Anathomy  Windows Internals 5th Ed. We are not going to get into details, but we will do a little refresher of some key concepts

12. Usermode vs. KernelmodeWindows on Windowswowexec.exeUNIXLSA ShellLsass.exeClient/Servercsrss.exeNotepadnotepad.exeVirtual DOS Machinentvdm.exeWin32InterixUser ModeKernel ModeExecutiveServicesI/OIPCMemoryProcessesSecurityWMPNPGraphicsControllerObject ManagerFSDevice DriversMicrokernelHardware AbstractionLayer (HAL)

13. Application, Processes and ThreadsAn application is formed by one or more processesA process is an in-memory executable, which is made up of one or more threads and its resourcesA thread is the basic unit of execution and schedulingin the OS.

14. … isitreallyworthit?

16. Othergoodreasons…

17. Win32 Virtual MemoryAddressing (I)sqlsrv.exeProcess nProcess 1Process 2Thread 1Thread 1Thread 1Thread 1Thread2Thread2Thread2Thread2…::::2 GbThread nThread nThread nThread n4GbKernel2 Gb

18. Win32 Virtual MemoryAddressing(II)

19. Thread Call StacksShows part of the history of the function calls of the threadEach thread has its own Call Stacki.e:ntdll!KiFastSystemCallRetUSER32!NtUserGetMessage+0xcnotepad!WinMain+0xe5notepad!WinMainCRTStartup+0x174kernel32!BaseProcessStart+0x23

20. CallStacks (I)Eachthread of theprocess has itsowncallstack:

21. CallStacks (II)Eachframe has thefollowingstructure:FrameParametersReturnAddressFrame PointerExceptionHandlerLocal VariablesRegistros

22. SymbolsSymbols make the call stack useful:Without Symbols:With Symbols:kernel32!+136aakernel32!CreateFileW+0x35f

23. Symbol formatsCurrent format: .PDBOld Format: .DBGRetail vs. Debug (Free vs. Checked) buildsPrivate symbols vs. public symbols

24. Symbol ServersUses the File System as a Symbol’s database:Organized by name and a unique identifierFolder structure: \\SymSrv\file_name.pdb\unique_number\____i.e:\\Symbols\ntdll.pdb\3B5EDCA52\ntdll.pdb\\Symbols\ntdll.pdb\380FCC4F2\ntdll.pdb

25. Demo 1: Scheduler Non-Yielding

26. ScenarioA customer’s SQL Server 2000 ishanging, showing 17883 errors in SQL Server’sErrorLogWhenthese errores ocurr, SQL Server automaticallytriggersthecreation of a dump…2007-02-12 11:17:14.10 server Error: 17883, Severity: 1, State: 02007-02-12 11:17:14.10 server Process 59:0 (834) UMS Context 0x125ABD80 appears to be non-yielding on Scheduler 1.…

27. Demo 2: DBCC CHECKDB

28. Demo 3: ClusterResources

29. ManagedDebuggingwith .NETWinDbgis a nativedebuggerIn ordertodebug .NET codeweneedto use debuggerextensions:SOS.dll (untilframework .NET 3.5)CLR.dll (framework 4.0)Whyallthis? Isitworthit?

30. Demo 4: ManagedDebuggingwith SOS

31. Somecooltips…Didwereallygettothisslide in time?! Well.. enjoysome free tips! Using SOS from VS.NETMemorydumpanalysisfrominside VS2010

32. Resourcespablod@plainconcepts.com@Plain Conceptshttp://www.geeks.ms/blogs/palvarezhttp://www.geeks.ms/blogs/rcorralhttp://www.geeks.ms/blogs/luisguerrero@MSDN:http://blogs.msdn.com/tess/Books:Microsoft Windows Internals, 5th Ed. [Mark E. Russinovich and David A. Solomon]Microsoft Press.Debugging Applications for Microsoft .NET and Microsoft Windows[John Robbins]Microsoft Press.

33. AnyQuestions?Thanks! 