Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Advanced Debugging Techniques
Similar to Advanced Debugging Techniques (20)
Recently uploaded
Recently uploaded (20)
Advanced Debugging Techniques
- 2. whoami David Szöke Application Engineer @ Unic, Switzerland 2
- 3. Cases 3
- 5. IIS Crash Loop ▸ PRD Environment ▸ Authoring environment suddenly stopped working, 503 ▸ Application Pool disabled ▸ IIS Rapid fail protection kicked in ▸ Nothing in Sitecore logs and Windows Event logs ▸ No solution updates 5
- 6. Gather a crash dump ▸ Download and install Debugging Tools for Windows: https://bit.ly/2WePj2D ▸ Run adplus.exe in crash mode 6 .adplus.exe -pmn "w3wp.exe" -o c:tempcrash-analysis -crash -FullOnFirst
- 7. Automated analysis using DebugDiag ▸ Download and install DebugDiag: https://bit.ly/2IkVWXx 7
- 8. DebugDiag Report ▸ DebugDiag will generate a report: 8
- 9. Automated analysis using PostMortem ▸ PostMortem: https://bit.ly/2JOjUgT ▸ Will also generate a report 9
- 11. Fixing PRD ▸ What item is breaking PRD? ▸ Why doesn’t it let Sitecore start up? ▸ WinDbg: Also part of Debugging Tools for Windows ▸ Can load dump files ▸ Supports managed applications (SOS) ▸ Common commands, sequence and SOS initialization: https://bit.ly/2EGIiN2 11
- 12. Finding the conflicting item 12 0:115>!ClrStack -p […] 0000007763ec6a80 00007ffcc6038f54 Sitecore.Crash.Foundation.Indexing.ComputedFields.ReferencedItemComputedField.GetRootId(Sitecore.D ata.Items.Item) PARAMETERS: this (0x0000007763ec6b20) = 0x0000018da58644b8 item (0x0000007763ec6b28) = 0x0000018aa8779160 0:115> !DumpObj /d 0000018aa8779160 [...] MT Field Offset Type VT Attr Value Name 00007ffcc305a1d8 4001f75 8 Sitecore.Data.ID 0 instance 0000018a26f57f38 _itemID 00007ffd1dad7aa0 4001f76 10 System.Object 0 instance 0000018aa8779230 _lock 00007ffcc4f14178 4001f77 18 ...ms.ItemAppearance 0 instance 0000000000000000 _appearance 00007ffcc4b852f0 4001f78 20 ...ta.Items.ItemAxes 0 instance 0000000000000000 _axes 0:115> !DumpObj /d 0000018a26f57f38 [...] MT Field Offset Type VT Attr Value Name 00007ffd1dac2510 4001c89 18 System.Guid 1 instance 0000018a26f57f50 _guid 00007ffd1dad9e10 4001c8a 10 System.Int32 1 instance -520826525 _hashCode 0:115> dt nt!_GUID 0000018a26f57f50 ntdll!_GUID {d5729bfe-e486-4a37-8e0d-d1af8738a6aa}
- 13. Event Queue 13 ▸ Event not marked as done until processing completed (on web) select * from dbo.EventQueue where InstanceData like '%d5729bfe-e486-4a37-8e0d-d1af8738a6aa%' delete from dbo.EventQueue where InstanceData like '%d5729bfe-e486-4a37-8e0d-d1af8738a6aa%'
- 15. Summary ▸ Data structure expected as tree, but received graph ▸ Recursion not handled in code ▸ StackOverflowException ▸ Event not marked as completed 15
- 16. A Memory Leak 16
- 17. Memory Leak ▸ Noticed by Operations / Monitoring ▸ Related to Index Rebuilds ▸ Got worse over time (more items) 17
- 18. Gathering hang dumps ▸ Using adplus in hang mode ▸ Two separate dumps: before and after 18 .adplus.exe -p 11452 -o c:tempcrash-analysismemory-leak -hang
- 20. Getting an overview ▸ 2GB retained by Byte[] / ServiceProvider ▸ Compare to vanilla instance 20
- 24. Further Research & Sitecore Support ▸ Service resolved in computed index fields ▸ https://github.com/aspnet/DependencyInjection /issues/456 ▸ Answer from Sitecore Support: 24 The thing is that some of the Sitecore functionality depends on the current version of the "Microsoft.Extensions.DependencyInjection.dll" file. Unfortunately, replacing the mentioned file with a newer version was not deeply tested, so it could cause some breaking changes and unexpected behavior. So, I kindly ask you consider upgrading your Sitecore instance to Sitecore 9.1.
- 26. RabbitMQ processing stuck ▸ Customer noticed that items are missing ▸ Middleware logs looked fine ▸ Messages were still in RabbitMq ▸ Nothing in Sitecore logs 26
- 27. Gathering some crash dumps ▸ At least 2, some minutes apart 27 .adplus.exe -p 34084 -hang -o C:Tempcrash-analysisimport-stuck
- 30. Back to DebugDiag 30 SolrNet.Impl.SolrConnection.Get(System.String, System.Collections.Generic.IEnumerable`1>)+1a7 SolrNet.Impl.SolrQueryExecuter`1[[System.__Canon, mscorlib]].Execute(<..>)+71 Sitecore.ContentSearch.SolrProvider.LinqToSolrIndex`1[<..>GetResult(<..>)+297 Sitecore.ContentSearch.SolrProvider.LinqToSolrIndex`1[<..>].Execute[<..>](<..>)+306 Sitecore.Crash.Feature.EventHandlers.ItemSaved.EnsureUniqueTitle.EnsureUnique(<..>)+3f1 Sitecore.Crash.Feature.EventHandlers.ItemSaved.EnsureUniqueTitle.OnItemSaving(<..>)+217 Sitecore.Events.Event+EventSubscribers.RaiseEvent(<..>)+31f Sitecore.Events.Event.RaiseEvent(System.String, System.Object[])+186
- 33. Google «text to ascii» 33
- 34. Summary ▸ Query did not use exact match ▹ Tab interpreted as space by SolR ▸ Input data not sanitized ▸ Method stuck in endless loop ▸ Implemented sanitization in Middleware and Sitecore ▸ Re-coded unique lookup logic 34
- 36. dotMemory ▸ Memory Profiler ▸ Visualizes Memory ▸ Automated analysis ▸ Shows Retentions and dependencies ▸ Compare snapshots ▸ YouTube playlist: https://bit.ly/2IdeDw9 36
- 37. dotTrace ▸ Performance Profiler ▸ Analyze Locks and Hangs ▸ Performance analysis ▸ Compare snapshots ▸ Different data collection modes ▸ YouTube playlist: https://bit.ly/2Kjtfg5 37
- 38. WinDbg ▸ Analyze all the data! ▸ Scriptable + APIs (clrmd) ▸ Steep learning curve, but rewarding ▹ CLR internals ▸ Also a debugger ▸ Recently reworked (Windbg Preview, MS Store) ▸ Doug Rathbone: Investigating ASP.Net Memory Dumps for Idiots (like Me): https://bit.ly/2Wae76W 38
- 39. DebugDiag Analysis ▸ Automated analysis ▸ Extendable ▸ Great for comparing hang dumps ▸ F1 39
- 40. Thank you! 40
Editor's Notes
- David Szöke Application Engineer at Unic in Switzerland This talk is about debugging techniques Because there is always the possibility that you missed some constellations while developing, testing or in the automated tests which could break your enviroment I’ve prepared 3 real world cases which I though might be interesting to you We are going to look at how we approached these issues, what tools we used and how we tracked down the root cause
- Let’s start with the first one
- After executing the command, go and grab a coffee. Depending on the size of your application and type of error, this might take some minutes -pmn: defines process name to be monitored. ADPlus will keep monitoring if a process with this name starts and attach We are also using the –FullOnFirst argument, which will generate a full dump, even if only a first chance exception occurred. -crash: Since the application is in fact crashing, we provide the –crash argument. Another option would be –hang, which we will cover in another case After the dump has been generated, you will end up with something like in the screenshot
- From here, we have several options: We can either directly go to hardcore mode and spin up windbg, or we can first run some automated analysis on the dump. Let’s start with the automated analysis first. Microsoft provides a set of diagnostics tools, called DebugDiag. Next to the analyser, there is also a utility for gathering crash dumps. The nice thing there is, that it also provides a GUI. So if you didn’t like the previous CLI approach, you could also use DebugDiag for that. So let’s spin up DebugDiag Analysis and select our freshly generated crash dumps
- DebugDiag will generate a report which can be displayed in the browser In the .NET Threads Summary we can see that there is StackOverflowException in Thread 114
- An alternative to DebugDiag is PostMortem. It’s a simple tool I wrote a few months ago in order to facilitate and make crash dump analysis somewhat easier. You can clone it from my Github account. Like DebugDiag, it will generate a report which you can display in the browser It will not only tell us, that there is a StackOverflowException, but also display the stack trace of the Thread the exception happened in Looking at the stack trace, we can see that there is clearly something going on in the ReferencedItemComputedField.GetRootId method
- There was in fact a recursive call User requirements: Items imported Reference to each other We expected tree, but got a graph Fixed by implementing a circuit breaker, list that would be passed
- So we were able to fix the bug, but the system was still down and had to go up as soon as possible. Therefore, we had to find the conflicting item. Due to the fact that we had over 100’000 content items in 42 different language versions, we couldn’t just click through each item, but had to come up with a different solution. Let’s enter windbg It’s a multipurpose debugger by Microsoft. It was originally made for unmanaged applications, but Microsoft also provides a debugging extension called «Son Of Strike» allowing you to debug managed applications as well. It’s more or less a CLI and has a pretty steep learning curve, but the more you use it, the more rewarding it’s going to get. Especially when you are able to track down an proof an issue. So for starters, I created a short markdown file which lists some common commands and a way to initialize the SOS extensions.
- With WinDbg open, we opened the dump of the StackOverflowException WinDbg will automatically set you to the current threads where the exception occured We can now run the !ClrStack –p command to display the stack trace, including the parameters Input is highlighted in blue, important addresses are in yellow
- Remote event only removed when processing complete
- We were able to correlate the memory leaks to index rebuilds through monitoring tools and Sitecore logs
- <Explain difference to crash dumps>
- Switch to second dump <explain difference between «Largest Size» and «Largest Retained Size»> It usually helps to compare these graphs with a dump of a vanilla Sitecore instance. Usually, the
- <Explain similar retention>
- Talk about _transientDisposables
- We were able to confirm that the SampleService was in fact registered as transient And indeed, the class also implements Idisposable. And surely enough, it also allocates a byte array We’ve also checked the referenced to the class and were able to confirm, that it was resolved in a computed field
- The original problem was happening in an import job. I’ve ported the problem to an event handler, just for demo purposes and easier implementation
- Again in hang mode Note that there is also a –r parameter which can be used to take a dump in a defined interval
- Run with multiple dumps in «CrashHangMode»
- So if these were really SolR queries, they should show up in the search logs, right? Next thing we did was to enable search logs. By default, we had them disabled on the Solution, as they were generating just too much data
- Requirements Ensure title is unique across all items
- Use while developing Not much time investment needed to get a good understanding of the tool