More Related Content
What's hot
What's hot (20)
Similar to Large Scale Crash Dump Analysis with SuperDump
Similar to Large Scale Crash Dump Analysis with SuperDump (20)
Recently uploaded
Recently uploaded (20)
Large Scale Crash Dump Analysis with SuperDump
- 1. Large Scale Crash Dump Analysis with SuperDump Christoph Neumüller Team Lead, .NET Agent @ Dynatrace @discostu105
- 2. Goals of this talk Show how the combination of interesting libraries and tools evolved into a really useful tool. Why SuperDump might be useful for you! Make you aware which tools exist for crash & memory dumps. 1 2 3
- 3. What’s a “crash dump”? a.k.a “memory dump” Snapshot of a process and the memory it addresses at the time of an unhandled exception “Full Dump” & “Minidump” MINIDUMP_TYPE -> what‘s included? *.dmp Linux: coredump
- 4. How to create a “crash dump”? Windows Error Reporting (automatic, if enabled) ProcDump (SysInternals, can dump on crash!) dbghelp.dll API (MiniDumpWriteDump, it’s on you!) Task Manager (manual, be aware of bitness!) Process Explorer (SysInternals, manual) dumpproc.exe –ma –e myapp.exe Linux: /proc/sys/kernel/core_pattern DebugDiag (automatic, if enabled)
- 5. How to analyze a “crash dump”? WinDbg DebugDiagGDB IDA Visual Studio
- 6. Demo WinDbg & DebugDiag
- 7. Analyzing a crash dump requires A proper setup (tools installed, symbol paths set) Repetitive tasks (download crashdump, open <tool-of-choice>, run setup-commands (load SOS)) Get relevant stacktraces and exception messages & copy over to JIRA If crash-reason is not obvious, deeper analysis is needed! -needs expertise!
- 8. Goal: reduce time for crash dump analysis
- 9. Situation @ our company Our product is a mixture of C++, .NET, Java, NodeJs code. It runs embedded in customer’s applications. It runs mostly on Windows and Linux(but also AIX, Solaris, Mac, zLinux, zOS, ...) Sometimes a monitored application would crash. Was it the agent’s fault? Crash dumps are delivered by Tests, QA/Acceptance, Customers/Support, automated crash detection Our developers use Windows, Linux & Mac
- 10. So, how can we automate things? https://github.com/Microsoft/clrmd
- 13. Cool. But does it help productivity yet?
- 18. It also helps non-Windows developers to quick-assess crash-dumps more easily! Nice! Non-experienced people can analyze dumps without special tools and knowhow.
- 19. SuperDump.Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result.json.dmp Web-Frontend JIRA Support REST API curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ "url": "https://dumps.local/mydump.dmp", }' 'http://superdump.local/api/Dumps' Response: { "location": "http://superdump.local/Home/BundleCreated?bundleId=czs6140", "date": "Fri, 05 May 2017 20:13:04 GMT", } Developers Hangfire
- 20. Dump-analysis is already finished when a dev looks at a bug in JIRA. We’re speeding up! But, sometimes DebugDiag has better results!
- 21. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe SuperDump. DebugDiag.exe WinDbg DebugDiag WinDbg .log DebugDiag .mht Developers Hangfire
- 22. Awesome. But still not enough. What if I want to investigate a very special case. I want all the power of WinDbg. But in the browser...
- 25. Wow. Now even deep investigations can be made in the browser. This is a game changer for non-Windows developers.
- 26. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Docker For Windows Hyper-V result .json GDB based .sh scripts Developers Hangfire
- 27. Neat. No more Linux VM’s necessary for Windows developers to debug Linux coredumps.
- 28. SuperDump. Analysis.exe Architecture SuperDump.Service.exe CLRMD ASP.NET Core result .json Web-Frontend JIRA Support REST API cdb.exe (WinDbg) SuperDump. DebugDiag.exe DebugDiag WinDbg .log DebugDiag .mht cdb.exe (WinDbg) Websockets I/O Redirect Browser jquery. console Docker For Windows Hyper-V result .json GDB based .sh scripts GDB I/O Redirect Developers Hangfire
- 30. What SuperDump changed for us Speed: Much faster triaging Enabling: People not knowledgeable on crash dump analysis can do basic analysis Compatibility: Linux people can debug Windows dumps and vice-versa Communication: It’s easier to talk about “a crash” because it can be referred to by a URI
- 31. Open-sourced in February 2017 : https://github.com/Dynatrace/superdump Actively used and developed at Dynatrace Future stuff: * Better Linux analysis, based on .NET Core & libunwind (and open source) * Duplication detection and clustering * Search and data-mining capabilities (elasticsearch?) * Extensible analyzers Techs used: .NET Core, Websockets, ClrMD, Hangfire, swagger, swashbuckle, Humanizer, DebugDiag, Cdb, Gdb, docker, Hyper-V, libunwind
- 33. Resources • SuperDump: https://github.com/Dynatrace/superdump • WinDbg: https://developer.microsoft.com/en- us/windows/hardware/windows-driver-kit • DebugDiag: https://www.microsoft.com/en- us/download/details.aspx?id=49924 • procdump: https://technet.microsoft.com/en- us/sysinternals/dd996900.aspx • WinDbg resources: • Tess Ferrandez: https://blogs.msdn.microsoft.com/tess/tag/debugging/ • Cheat Sheet: http://windbg.info/doc/1-common-cmds.html
Editor's Notes
- MINIDUMP_TYPE : https://msdn.microsoft.com/en-us/library/windows/desktop/ms680519(v=vs.85).aspx
- WER: https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx
- WER: https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx
- https://github.com/Microsoft/clrmd