How to find out production issues? Where to look for errors when application crashes in live environment? How to Visual Studio 2010 for replicating post mortem scenarios in difficult to reproduce errors? Using Source server, PDB symbols in old fashioned way for new age WCF services.

1.  Mario Hewardt  Daniel Pravat
Senior Development Lead Senior Development Lead
Microsoft Corporation Microsoft Corporation
…and we approve this message

3. We Need Reliable Software





5. 

 AWD PDC 2008 Feedback/Questions





6. Open source code
Source: Coverity White Paper

7. with examples







8. 





 http://www.microsoft.com/whdc/devtools/debug
ging/default.mspx




9. 











10. 










11. 







12.  Daniel Pravat

13. 







14. 















15.  Daniel Pravat

16. 
















17.  Daniel Pravat

18.  Daniel Pravat

19. Thread 0 cs_DB1 Acquisition order
cs_DB1
is reversed
The length may be
No progress different
= Deadlock A,B,C = OK
B,C = OK
A,B = OK
A,C = OK
Thread 1 cs_DB2
B,C,A = Deadlock

20. 








21. Remote Test
Test
client Test
SystemTest
SystemTest
System Smart
System
Systems client
Sources Symbols Sources




22.  Daniel Pravat

23. 09async.exe
DisplayError calls  There is another thread smashing
Sleep the stack
Return address is  How to catch this async
saved operation?
Sleep calls  What are the implications of this
SleepEx bug?
Return address is  Can the execution be controlled?
changed to X
saved  Can this execution be prevented?
…  Welcome NX bit
Execution starts
at X

24. 







26. Daniel Pravat

27. 


Build a Extract
Index Publish Use
new public
symbols symbols symbols
version symbols

28. 









29. 









30. 











31. 






32. 



Store the
Extract the information
Build a new Publish Use the
source file required to
version symbols symbols
list retrieve the
file from SC

33. 






34. C:>pdbstr –r –p:%_NT386TREE%sym.priretailexe03sample.pdb –s:srcsrv
SRCSRV: ini ------------------------
VERSION=1
INDEXVERSION=2
VERCTRL=Visual Source Safe
DATETIME=Mon Jan 8 00:04:15 2007
SRCSRV: variables ---------------------
SSDIR=C:AWDVSS
SRCSRVENV=SSDIR=%AWD%
VSSTRGDIR=%targ%%var2%%fnbksl%(%var3%)%var4%
VSS_EXTRACT_CMD=ss.exe get -GL”%vsstrgdir%” -GF- -I-Y -W “$/%var3%” -
V”%var4%”
VSS_EXTRACT_TARGET=%targ%%var2%%fnbksl%(%var3%)%var4%%fnfile%(%var1%
)
AWD=C:AWDVSS
SRCSRVTRG=%VSS_extract_target%
SRCSRVCMD=%VSS_extract_cmd%
SRCSRV: source files --------------------
c:awdchapter3spydbg.cpp*AWD*chapter3/spydbg.cpp*VERSION1
SRCSRV: end ------------------------

35. 








38. 










39. 






40. 






41. 



42. 








43. 





44. 






45. CreateEvent(
…
…);
WIN32 API
User
Kernel
Header
Ref Count Obj Count Object
Event
1 1 <addr>
EPROCESS 1 1 <addr> Header
3 1 <addr>
Mutant

46. HANDLE hFile=CreateFile(
pWorkerData->pszFileName,
FILE_READ_DATA,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
// Use file handle
CloseHandle(hFile);

47. 









48. No Done
Leak?
Type of
resource
Initial
Analysis
Use tools
Avoidance
Strategy

49. Mario Hewardt

50. Mario Hewardt

51. Mario Hewardt

52. 




53. Application
Default
C Runtime
Process Other Heaps
Heap
Heap
Heap Manager
Virtual Memory Manager

54. BYTE* pMem=(BYTE*)
HeapAlloc(GetProcessHeap(), 0, 100);
// Use memory
HeapFree(GetProcessHeap(), 0, pMem);

55. 











56. Mario Hewardt

57. Mario Hewardt

58. 





61. 





62. 






63. 





Web Database
Web client Middle tier
front-end back-end

64. 









65. 



66. The User Identity (principal)








67. 0:000> !token 7bc -n
TS Session ID: 0
User: S-1-5-21-1060284298-2111687655-1957994488-1003 (User: XP-SP2TestAdmin)
Groups:
00 S-1-5-21-1060284298-2111687655-1957994488-513 (Group: XP-SP2None)
Attributes - Mandatory Default Enabled
01 S-1-1-0 (Well Known Group: localhostEveryone)
Attributes - Mandatory Default Enabled
02 S-1-5-32-544 (Alias: BUILTINAdministrators)
Attributes - Mandatory Default Enabled Owner
03 S-1-5-32-545 (Alias: BUILTINUsers)
Attributes - Mandatory Default Enabled
04 S-1-5-4 (Well Known Group: NT AUTHORITYINTERACTIVE)
Attributes - Mandatory Default Enabled
05 S-1-5-11 (Well Known Group: NT AUTHORITYAuthenticated Users)
Attributes - Mandatory Default Enabled
06 S-1-5-5-0-35778 (no name mapped)
Attributes - Mandatory Default Enabled LogonId
07 S-1-2-0 (Well Known Group: localhostLOCAL)
Attributes - Mandatory Default Enabled
Primary Group: S-1-5-21-1060284298-2111687655-1957994488-513 (Group: XP-SP2None)
Privs:
00 0x000000017 SeChangeNotifyPrivilege Attributes - Enabled Default
01 0x000000008 SeSecurityPrivilege Attributes -
...
17 0x000000009 SeTakeOwnershipPrivilege Attributes -
18 0x00000001e SeCreateGlobalPrivilege Attributes - Enabled Default
19 0x00000001d SeImpersonatePrivilege Attributes - Enabled Default
Auth ID: 0:1c3a8
Impersonation Level: Identification
TokenType: Impersonation

68. The Discretionary Access Control List (DACL)








69. 0:000> !acl 000840ac
ACL is:
ACL is: ->AclRevision: 0x2
ACL is: ->Sbz1 : 0x0
ACL is: ->AclSize : 0x1c
ACL is: ->AceCount : 0x1
ACL is: ->Sbz2 : 0x0
ACL is: ->Ace[0]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
ACL is: ->Ace[0]: ->AceFlags: 0x0
ACL is: ->Ace[0]: ->AceSize: 0x14
ACL is: ->Ace[0]: ->Mask : 0x00120089
ACL is: ->Ace[0]: ->SID: S-1-1-0

70. kd> !sd 00084098
->Revision: 0x1
->Sbz1 : 0x0
->Control : 0x8004
SE_DACL_PRESENT
SE_SELF_RELATIVE
->Owner : S-1-5-18
->Group : S-1-5-32-544
->Dacl :
->Dacl : ->AclRevision: 0x2
->Dacl : ->Sbz1 : 0x0
->Dacl : ->AclSize : 0x1c
->Dacl : ->AceCount : 0x1
->Dacl : ->Sbz2 : 0x0
->Dacl : ->Ace[0]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[0]: ->AceFlags: 0x0
->Dacl : ->Ace[0]: ->AceSize: 0x14
->Dacl : ->Ace[0]: ->Mask : 0x00120089
->Dacl : ->Ace[0]: ->SID: S-1-1-0
->Sacl : is NULL

71. The Security Reference Monitor (SRM)










72.  Daniel Pravat

73. void MTAClientCall()
{
COSERVERINFO si;
MULTI_QI mqi;
...
if (SUCCEEDED(CoCreateInstanceEx( CLSID_Calculator,
NULL, CLSCTX_LOCAL_SERVER | CLSCTX_REMOTE_SERVER,
&si, 1, &mqi )))
{
ICalculator * pCalculator = NULL;
pCalculator = (ICalculator*)mqi.pItf;
__int32 result = 0;
wprintf(L"Calling SumSlow ...");
pCalculator->SumSlow(1,2, &result);
pCalculator->Release();
wprintf(L"donen");
}
}

74. 



DCOM
HOST OS SCM
Launch
Remote RPCSS DCOM
client Server server
DCOM
client

75.  Daniel Pravat

76. 







79. 








80. 

81. 




82. 






83. Application
Default
C Runtime
Process Other Heaps
Heap
Heap
Heap Manager
Virtual Memory Manager

84. Look aside Table
0 Unused
1 16
2 24
3 32
… …
127 1024

85. Free Lists Segment List
0 Variable Size 1
2
1 unused …
2 16 X
3 24
… …
127 1016

86. 






87. Free Block Busy Block Busy Block Uncommitted range

88. 





89. Pre-allocation Metadata User accessible part Post-allocation Metadata
Pre-allocation Metadata
Current Previous Seg Tag
Flags Unused User accessible part
Size Size Index Index
Post-allocation Metadata
Suffix Fill Area Heap
User accessible part
Bytes (debug) Extra

90. 



Allocation Size: 16 Allocation Size: 32 Allocation Size: 16
Allocation Size: 64

91. 






92. Mario Hewardt

93. Mario Hewardt

94. Mario Hewardt

97. 






98. 






99. 







100. Windows Error Reporting Service
Crash data over HTTPS
Error
Sent
Dr. Query Fault
Watson Fault response over HTTPS Data
Process
Crash
ISV

101. Mario Hewardt

102. 

 http://www.codeplex.com/wer/






103. 




105. Daniel Pravat

106. Debugging Tools for Windows











107. Debugging Tools for Windows













108. Debugging Tools for Windows










109. Main()
{
...
int operationCount = 0;
for (; ; )
{
operationCount++;
HostInfo hi = HostInfo.GetDefault(hostAddress,
userName, password);
IWSManSession session = GetSession(hi);
string response =
session.Get("wmicimv2/Win32_OperatingSystem", 0);
Console.Write("rNumber of calls: " +
operationCount);
}
...
}

110. static public IWSManSession GetSession(HostInfo hi)
{
IWSManSession session = null;
// Get a cached session
string key = hi.GetKey();
if (!sessionCache.TryGetValue(key, out session))
{
session = CreateSession(hi);
sessionCache[key] = session;
}
return session;
}

111.  Daniel Pravat

112. 





113. Debugging Tools for Windows










114. 







115. 




 Microsoft Advanced Windows Debugging and Troubleshooting
 Crash Dump Analysis
 If broken it is, fix it you should



116. 




118. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.