Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Large Scale Crash Dump Analysis with SuperDump

2,863 views

Published on

.NET Memory Dump analysis is a science on it's own. The tools (e.g. WinDbg) require special knowledge. SuperDump is a web-based open-source tool which makes memory dump analysis easy for everyone. And it can be fully automated. This talk shows how SuperDump, which is based on .NET Core, evolved and can now even be used for Linux coredump analysis.

https://github.com/Dynatrace/superdump

Published in: Software
  • Be the first to comment

Large Scale Crash Dump Analysis with SuperDump

  1. 1. Large Scale Crash Dump Analysis with SuperDump Christoph Neumüller Team Lead, .NET Agent @ Dynatrace @discostu105
  2. 2. Goals of this talk Show how the combination of interesting libraries and tools evolved into a really useful tool. Why SuperDump might be useful for you! Make you aware which tools exist for crash & memory dumps. 1 2 3
  3. 3. What’s a “crash dump”? a.k.a “memory dump” Snapshot of a process and the memory it addresses at the time of an unhandled exception “Full Dump” & “Minidump” MINIDUMP_TYPE -> what‘s included? *.dmp Linux: coredump
  4. 4. How to create a “crash dump”? Windows Error Reporting (automatic, if enabled) ProcDump (SysInternals, can dump on crash!) dbghelp.dll API (MiniDumpWriteDump, it’s on you!) Task Manager (manual, be aware of bitness!) Process Explorer (SysInternals, manual) dumpproc.exe –ma –e myapp.exe Linux: /proc/sys/kernel/core_pattern DebugDiag (automatic, if enabled)
  5. 5. How to analyze a “crash dump”? WinDbg DebugDiagGDB IDA Visual Studio
  6. 6. Demo WinDbg & DebugDiag
  7. 7. Analyzing a crash dump requires A proper setup (tools installed, symbol paths set) Repetitive tasks (download crashdump, open <tool-of-choice>, run setup-commands (load SOS)) Get relevant stacktraces and exception messages & copy over to JIRA If crash-reason is not obvious, deeper analysis is needed! -needs expertise!
  8. 8. Goal: reduce time for crash dump analysis
  9. 9. Situation @ our company Our product is a mixture of C++, .NET, Java, NodeJs code. It runs embedded in customer’s applications. It runs mostly on Windows and Linux(but also AIX, Solaris, Mac, zLinux, zOS, ...) Sometimes a monitored application would crash. Was it the agent’s fault? Crash dumps are delivered by Tests, QA/Acceptance, Customers/Support, automated crash detection Our developers use Windows, Linux & Mac
  10. 10. So, how can we automate things? https://github.com/Microsoft/clrmd
  11. 11. ClrMD https://github.com/Microsoft/clrmd
  12. 12. SuperDump.Analysis.exe Text Output Architecture CLRMD
  13. 13. Cool. But does it help productivity yet?
  14. 14. SuperDump.Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result.json.dmp Web-Frontend Developers Hangfire
  15. 15. It also helps non-Windows developers to quick-assess crash-dumps more easily! Nice! Non-experienced people can analyze dumps without special tools and knowhow.
  16. 16. SuperDump.Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result.json.dmp Web-Frontend JIRA Support REST API curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ "url": "https://dumps.local/mydump.dmp", }' 'http://superdump.local/api/Dumps' Response: { "location": "http://superdump.local/Home/BundleCreated?bundleId=czs6140", "date": "Fri, 05 May 2017 20:13:04 GMT", } Developers Hangfire
  17. 17. Dump-analysis is already finished when a dev looks at a bug in JIRA. We’re speeding up! But, sometimes DebugDiag has better results!
  18. 18. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe SuperDump. DebugDiag.exe WinDbg DebugDiag WinDbg .log DebugDiag .mht Developers Hangfire
  19. 19. Awesome. But still not enough. What if I want to investigate a very special case. I want all the power of WinDbg. But in the browser...
  20. 20. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Developers Hangfire
  21. 21. Wow. Now even deep investigations can be made in the browser. This is a game changer for non-Windows developers.
  22. 22. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Docker For Windows Hyper-V result .json GDB based .sh scripts Developers Hangfire
  23. 23. Neat. No more Linux VM’s necessary for Windows developers to debug Linux coredumps.
  24. 24. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Docker For Windows Hyper-V result .json GDB based .sh scripts GDB I/O Redirect Developers Hangfire
  25. 25. What SuperDump changed for us Speed: Much faster triaging Enabling: People not knowledgeable on crash dump analysis can do basic analysis Compatibility: Linux people can debug Windows dumps and vice-versa Communication: It’s easier to talk about “a crash” because it can be referred to by a URI
  26. 26. Open-sourced in February 2017 : https://github.com/Dynatrace/superdump Actively used and developed at Dynatrace Future stuff: * Better Linux analysis, based on .NET Core & libunwind (and open source) * Duplication detection and clustering * Search and data-mining capabilities (elasticsearch?) * Extensible analyzers Techs used: .NET Core, Websockets, ClrMD, Hangfire, swagger, swashbuckle, Humanizer, DebugDiag, Cdb, Gdb, docker, Hyper-V, libunwind
  27. 27. The end. Happy Debugging!
  28. 28. Resources • SuperDump: https://github.com/Dynatrace/superdump • WinDbg: https://developer.microsoft.com/en- us/windows/hardware/windows-driver-kit • DebugDiag: https://www.microsoft.com/en- us/download/details.aspx?id=49924 • procdump: https://technet.microsoft.com/en- us/sysinternals/dd996900.aspx • WinDbg resources: • Tess Ferrandez: https://blogs.msdn.microsoft.com/tess/tag/debugging/ • Cheat Sheet: http://windbg.info/doc/1-common-cmds.html

×