Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Large Scale Crash
Dump Analysis
with SuperDump
Christoph Neumüller
Team Lead, .NET Agent @ Dynatrace
@discostu105
Goals of this talk
Show how the combination of interesting libraries and tools
evolved into a really useful tool.
Why Supe...
What’s a “crash dump”?
a.k.a “memory dump”
Snapshot of a process
and the memory it
addresses at the time of
an unhandled e...
How to create a “crash dump”?
Windows Error Reporting (automatic, if enabled)
ProcDump (SysInternals, can dump on crash!)
...
How to analyze a “crash dump”?
WinDbg
DebugDiagGDB
IDA
Visual
Studio
Demo WinDbg & DebugDiag
Analyzing a crash dump requires
A proper setup (tools installed, symbol paths set)
Repetitive tasks (download crashdump, o...
Goal: reduce time for crash dump analysis
Situation @ our company
Our product is a mixture of C++, .NET, Java, NodeJs code. It runs
embedded in customer’s applicati...
So, how can we
automate things?
https://github.com/Microsoft/clrmd
ClrMD
https://github.com/Microsoft/clrmd
SuperDump.Analysis.exe
Text Output
Architecture
CLRMD
Cool. But does it
help productivity
yet?
SuperDump.Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result.json.dmp
Web-Frontend
Developers
Hangf...
It also helps non-Windows developers to
quick-assess crash-dumps more easily!
Nice! Non-experienced people can analyze
dum...
SuperDump.Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result.json.dmp
Web-Frontend
JIRA
Support
RES...
Dump-analysis is already finished when a dev
looks at a bug in JIRA. We’re speeding up!
But, sometimes DebugDiag
has bette...
SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST ...
Awesome.
But still not enough. What if I want to
investigate a very special case. I want all the
power of WinDbg. But in t...
SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST ...
Wow. Now even deep investigations can be
made in the browser.
This is a game changer for non-Windows
developers.
SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST ...
Neat. No more Linux VM’s necessary for
Windows developers to debug Linux
coredumps.
SuperDump.
Analysis.exe
Architecture
SuperDump.Service.exe
CLRMD
ASP.NET Core
result
.json
Web-Frontend
JIRA
Support
REST ...
What SuperDump changed for us
Speed: Much faster triaging
Enabling: People not knowledgeable on crash dump
analysis can do...
Open-sourced in February 2017 : https://github.com/Dynatrace/superdump
Actively used and developed at Dynatrace
Future stu...
The end.
Happy Debugging!
Resources
• SuperDump: https://github.com/Dynatrace/superdump
• WinDbg: https://developer.microsoft.com/en-
us/windows/har...
Large Scale Crash Dump Analysis with SuperDump
Large Scale Crash Dump Analysis with SuperDump
Large Scale Crash Dump Analysis with SuperDump
Large Scale Crash Dump Analysis with SuperDump
Large Scale Crash Dump Analysis with SuperDump
Upcoming SlideShare
Loading in …5
×

Large Scale Crash Dump Analysis with SuperDump

3,688 views

Published on

.NET Memory Dump analysis is a science on it's own. The tools (e.g. WinDbg) require special knowledge. SuperDump is a web-based open-source tool which makes memory dump analysis easy for everyone. And it can be fully automated. This talk shows how SuperDump, which is based on .NET Core, evolved and can now even be used for Linux coredump analysis.

https://github.com/Dynatrace/superdump

Published in: Software
  • Be the first to comment

Large Scale Crash Dump Analysis with SuperDump

  1. 1. Large Scale Crash Dump Analysis with SuperDump Christoph Neumüller Team Lead, .NET Agent @ Dynatrace @discostu105
  2. 2. Goals of this talk Show how the combination of interesting libraries and tools evolved into a really useful tool. Why SuperDump might be useful for you! Make you aware which tools exist for crash & memory dumps. 1 2 3
  3. 3. What’s a “crash dump”? a.k.a “memory dump” Snapshot of a process and the memory it addresses at the time of an unhandled exception “Full Dump” & “Minidump” MINIDUMP_TYPE -> what‘s included? *.dmp Linux: coredump
  4. 4. How to create a “crash dump”? Windows Error Reporting (automatic, if enabled) ProcDump (SysInternals, can dump on crash!) dbghelp.dll API (MiniDumpWriteDump, it’s on you!) Task Manager (manual, be aware of bitness!) Process Explorer (SysInternals, manual) dumpproc.exe –ma –e myapp.exe Linux: /proc/sys/kernel/core_pattern DebugDiag (automatic, if enabled)
  5. 5. How to analyze a “crash dump”? WinDbg DebugDiagGDB IDA Visual Studio
  6. 6. Demo WinDbg & DebugDiag
  7. 7. Analyzing a crash dump requires A proper setup (tools installed, symbol paths set) Repetitive tasks (download crashdump, open <tool-of-choice>, run setup-commands (load SOS)) Get relevant stacktraces and exception messages & copy over to JIRA If crash-reason is not obvious, deeper analysis is needed! -needs expertise!
  8. 8. Goal: reduce time for crash dump analysis
  9. 9. Situation @ our company Our product is a mixture of C++, .NET, Java, NodeJs code. It runs embedded in customer’s applications. It runs mostly on Windows and Linux(but also AIX, Solaris, Mac, zLinux, zOS, ...) Sometimes a monitored application would crash. Was it the agent’s fault? Crash dumps are delivered by Tests, QA/Acceptance, Customers/Support, automated crash detection Our developers use Windows, Linux & Mac
  10. 10. So, how can we automate things? https://github.com/Microsoft/clrmd
  11. 11. ClrMD https://github.com/Microsoft/clrmd
  12. 12. SuperDump.Analysis.exe Text Output Architecture CLRMD
  13. 13. Cool. But does it help productivity yet?
  14. 14. SuperDump.Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result.json.dmp Web-Frontend Developers Hangfire
  15. 15. It also helps non-Windows developers to quick-assess crash-dumps more easily! Nice! Non-experienced people can analyze dumps without special tools and knowhow.
  16. 16. SuperDump.Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result.json.dmp Web-Frontend JIRA Support REST API curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ "url": "https://dumps.local/mydump.dmp", }' 'http://superdump.local/api/Dumps' Response: { "location": "http://superdump.local/Home/BundleCreated?bundleId=czs6140", "date": "Fri, 05 May 2017 20:13:04 GMT", } Developers Hangfire
  17. 17. Dump-analysis is already finished when a dev looks at a bug in JIRA. We’re speeding up! But, sometimes DebugDiag has better results!
  18. 18. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe SuperDump. DebugDiag.exe WinDbg DebugDiag WinDbg .log DebugDiag .mht Developers Hangfire
  19. 19. Awesome. But still not enough. What if I want to investigate a very special case. I want all the power of WinDbg. But in the browser...
  20. 20. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Developers Hangfire
  21. 21. Wow. Now even deep investigations can be made in the browser. This is a game changer for non-Windows developers.
  22. 22. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Docker For Windows Hyper-V result .json GDB based .sh scripts Developers Hangfire
  23. 23. Neat. No more Linux VM’s necessary for Windows developers to debug Linux coredumps.
  24. 24. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Docker For Windows Hyper-V result .json GDB based .sh scripts GDB I/O Redirect Developers Hangfire
  25. 25. What SuperDump changed for us Speed: Much faster triaging Enabling: People not knowledgeable on crash dump analysis can do basic analysis Compatibility: Linux people can debug Windows dumps and vice-versa Communication: It’s easier to talk about “a crash” because it can be referred to by a URI
  26. 26. Open-sourced in February 2017 : https://github.com/Dynatrace/superdump Actively used and developed at Dynatrace Future stuff: * Better Linux analysis, based on .NET Core & libunwind (and open source) * Duplication detection and clustering * Search and data-mining capabilities (elasticsearch?) * Extensible analyzers Techs used: .NET Core, Websockets, ClrMD, Hangfire, swagger, swashbuckle, Humanizer, DebugDiag, Cdb, Gdb, docker, Hyper-V, libunwind
  27. 27. The end. Happy Debugging!
  28. 28. Resources • SuperDump: https://github.com/Dynatrace/superdump • WinDbg: https://developer.microsoft.com/en- us/windows/hardware/windows-driver-kit • DebugDiag: https://www.microsoft.com/en- us/download/details.aspx?id=49924 • procdump: https://technet.microsoft.com/en- us/sysinternals/dd996900.aspx • WinDbg resources: • Tess Ferrandez: https://blogs.msdn.microsoft.com/tess/tag/debugging/ • Cheat Sheet: http://windbg.info/doc/1-common-cmds.html

×