1. Large Scale Crash
Dump Analysis
with SuperDump
Christoph Neumüller
Team Lead, .NET Agent @ Dynatrace
@discostu105

2. Goals of this talk
Show how the combination of interesting libraries and tools
evolved into a really useful tool.
Why SuperDump might be useful for you!
Make you aware which tools exist for crash & memory dumps.
1
2
3

3. What’s a “crash dump”?
a.k.a “memory dump”
Snapshot of a process
and the memory it
addresses at the time of
an unhandled exception
“Full Dump” & “Minidump”
MINIDUMP_TYPE -> what‘s included?
*.dmp
Linux: coredump

4. How to create a “crash dump”?
Windows Error Reporting (automatic, if enabled)
ProcDump (SysInternals, can dump on crash!)
dbghelp.dll API (MiniDumpWriteDump, it’s on you!)
Task Manager (manual, be aware of bitness!)
Process Explorer (SysInternals, manual)
dumpproc.exe –ma –e myapp.exe
Linux: /proc/sys/kernel/core_pattern
DebugDiag (automatic, if enabled)

5. How to analyze a “crash dump”?
WinDbg
DebugDiagGDB
IDA
Visual
Studio

6. Demo WinDbg & DebugDiag

7. Analyzing a crash dump requires
A proper setup (tools installed, symbol paths set)
Repetitive tasks (download crashdump, open <tool-of-choice>, run
setup-commands (load SOS))
Get relevant stacktraces and exception messages & copy over to JIRA
If crash-reason is not obvious, deeper analysis is needed!
-needs expertise!

8. Goal: reduce time for crash dump analysis

9. Situation @ our company
Our product is a mixture of C++, .NET, Java, NodeJs code. It runs
embedded in customer’s applications.
It runs mostly on Windows and Linux(but also AIX, Solaris, Mac, zLinux, zOS, ...)
Sometimes a monitored application would crash. Was it the agent’s fault?
Crash dumps are delivered by Tests, QA/Acceptance, Customers/Support,
automated crash detection
Our developers use Windows, Linux & Mac

10. So, how can we
automate things?
https://github.com/Microsoft/clrmd

11. ClrMD
https://github.com/Microsoft/clrmd

12. SuperDump.Analysis.exe
Text Output
Architecture
CLRMD

13. Cool. But does it
help productivity
yet?

14. SuperDump.Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result.json.dmp
Web-Frontend
Developers
Hangfire

18. It also helps non-Windows developers to
quick-assess crash-dumps more easily!
Nice! Non-experienced people can analyze
dumps without special tools and knowhow.

19. SuperDump.Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result.json.dmp
Web-Frontend
JIRA
Support
REST API
curl -X POST --header 'Content-Type: application/json' --header
'Accept: application/json' -d '{
"url": "https://dumps.local/mydump.dmp",
}' 'http://superdump.local/api/Dumps'
Response:
{
"location":
"http://superdump.local/Home/BundleCreated?bundleId=czs6140",
"date": "Fri, 05 May 2017 20:13:04 GMT",
}
Developers
Hangfire

20. Dump-analysis is already finished when a dev
looks at a bug in JIRA. We’re speeding up!
But, sometimes DebugDiag
has better results!

21. SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST API
cdb.exe SuperDump.
DebugDiag.exe
WinDbg DebugDiag
WinDbg
.log
DebugDiag
.mht
Developers
Hangfire

22. Awesome.
But still not enough. What if I want to
investigate a very special case. I want all the
power of WinDbg. But in the browser...

23. SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST API
cdb.exe
(WinDbg)
SuperDump.
DebugDiag.exe
DebugDiag
WinDbg
.log
DebugDiag
.mht
cdb.exe
(WinDbg)
Websockets
I/O
Redirect
Browser
jquery.
console
Developers
Hangfire

25. Wow. Now even deep investigations can be
made in the browser.
This is a game changer for non-Windows
developers.

26. SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST API
cdb.exe
(WinDbg)
SuperDump.
DebugDiag.exe
DebugDiag
WinDbg
.log
DebugDiag
.mht
cdb.exe
(WinDbg)
Websockets
I/O
Redirect
Browser
jquery.
console
Docker For Windows
Hyper-V
result
.json
GDB based .sh
scripts
Developers
Hangfire

27. Neat. No more Linux VM’s necessary for
Windows developers to debug Linux
coredumps.

28. SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST API
cdb.exe
(WinDbg)
SuperDump.
DebugDiag.exe
DebugDiag
WinDbg
.log
DebugDiag
.mht
cdb.exe
(WinDbg)
Websockets
I/O
Redirect
Browser
jquery.
console
Docker For Windows
Hyper-V
result
.json
GDB based .sh
scripts
GDB
I/O
Redirect
Developers
Hangfire

30. What SuperDump changed for us
Speed: Much faster triaging
Enabling: People not knowledgeable on crash dump
analysis can do basic analysis
Compatibility: Linux people can debug Windows dumps
and vice-versa
Communication: It’s easier to talk about “a crash”
because it can be referred to by a URI

31. Open-sourced in February 2017 : https://github.com/Dynatrace/superdump
Actively used and developed at Dynatrace
Future stuff:
* Better Linux analysis, based on .NET Core & libunwind (and open source)
* Duplication detection and clustering
* Search and data-mining capabilities (elasticsearch?)
* Extensible analyzers
Techs used: .NET Core, Websockets,
ClrMD, Hangfire, swagger,
swashbuckle, Humanizer, DebugDiag,
Cdb, Gdb, docker, Hyper-V, libunwind

32. The end.
Happy Debugging!

33. Resources
• SuperDump: https://github.com/Dynatrace/superdump
• WinDbg: https://developer.microsoft.com/en-
us/windows/hardware/windows-driver-kit
• DebugDiag: https://www.microsoft.com/en-
us/download/details.aspx?id=49924
• procdump: https://technet.microsoft.com/en-
us/sysinternals/dd996900.aspx
• WinDbg resources:
• Tess Ferrandez: https://blogs.msdn.microsoft.com/tess/tag/debugging/
• Cheat Sheet: http://windbg.info/doc/1-common-cmds.html