Troubleshooting Sitecore Performance


Published on

Monitoring & Analyzing Sitecore Performance Issues - August 12, 2013

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Troubleshooting Sitecore Performance

  1. 1. Troubleshooting Sitecore Performance Monitoring & Analyzing Performance Issues August 12, 2013 1
  2. 2. Please review before viewing this presentation 2
  3. 3. Sitecore Enterprise Web CMS • Popular .NET Web Content Management System for mid-to-large organizations • Provides friendly tools for non-technical Content Authors and Marketers • Can be customized and extended to meet almost any conceivable business need 3
  4. 4. Troubleshooting Performance Steps to troubleshoot performance issues • Create baseline logs of performance metrics • Monitor system when it is offline or off hours • Monitor system during normal operation • Regularly review performance logs • Become familiar with normal metrics for your system • Investigate performance metrics that are not normal • Scrutinize counters in areas of unusual performance • Try to narrow the scope of performance problems • Take action to remove and or resolve bottlenecks 4
  5. 5. Performance Monitoring Monitoring helps locate root causes of issues • Inadequate resources or resource misallocation • Non-optimum coding or use of a legacy API • Missing configuration settings or misconfiguration Monitoring helps identify resource bottlenecks • Server hardware & operating systems • IIS & ASP.NET • Databases • Sitecore Monitoring is a recurring process, not an event • Familiarity with the system is a key success factor 5
  6. 6. Why Monitoring is Important Building business trust • Ensuring performance meets user expectations • Averting business-impacting performance issues Accomplishing cost-effective site management • Efficiently diagnosing root causes of issues • Perceiving the corrective actions needed to fix issues • Addressing issues instead of upgrading hardware • Accurately budgeting for future growth needs 6
  7. 7. Windows Performance Monitor Windows Performance Monitor (PerfMon) • Use it to display graphs for all performance logs • System, IIS, ASP.NET, SQL, Sitecore counters, etc. • Computer Mgmt > Performance > Monitoring Tools • Use <Ctrl-h> to toggle highlight of selected counter 7
  8. 8. Understanding Bottlenecks Performance bottlenecks • Performance of system is limited by a single resource • Bottleneck points are limiting components of system Bottleneck indicators • One or more resources are over-consumed • Resource requests exceed available resources • Resource requests must be queued until available • One or more resources are under-consumed • More subtle type of bottleneck that is harder to locate • Resource requests are not at their expected levels • The bottleneck blocking requests is elsewhere 8
  9. 9. Troubleshooting Bottlenecks Identifying bottlenecks • Create a baseline log when system is offline or at rest • Monitor the system to log the current behavior • Compare the current behavior with the baseline log Investigating bottlenecks • Review system counters before Sitecore counters • Look for sustained behaviors, not peaks or spikes • Over-consumption or under-consumption of resources • Use broad counters first; % Disk Time (Reads & Writes) • Use detailed counters to investigate suspicious values; Avg. Disk Bytes Read & Avg. Disk Read Queue Length 9
  10. 10. Viewing Server Performance Logs Inspect server performance logs for issues • Check for signs of memory leaks • Creeping increases in consumption indicate a leak • Check for signs of bottlenecks • High resource consumption for prolonged periods • Related queue length above 2 • Check for non-optimum metrics • Errors are signs of code issues or misconfiguration • Cache misses are signs of configuration issues 10
  11. 11. Server Bottlenecks Server Bottlenecks • One or more server resources are over consumed • The server contains one or more bottleneck points 11 Server Bottleneck Types Memory Processor Disk Network
  12. 12. Server Performance Troubleshooting server performance issues • Log performance objects that contain key counters • Set logging interval to 15 minutes • Log data over multiple weeks • Periodically review logged data for key counters • Use detailed counters to investigate suspect counters • Key counters that show signs of a potential problem • Periodically review Sitecore logs for nested exceptions • Rectify issues found from monitoring & Sitecore logs • Periodically review caching statistics & tune caches 12
  13. 13. Key Server Objects & Counters Memory • Available Bytes • Committed Bytes Processor • % Processor Time System • Processor Queue Length 13 Physical Disk • % Disk Time • Avg. Disk Queue Length Network Interface • Bytes Total/sec • Current Bandwidth • Output Queue Length
  14. 14. Warning Signs for Server Counters Use performance counters to check warning signs • Memory: Committed Bytes is creeping up long term • Processor: % Processor Time is above 75% long term and System: Processor Queue Length is above 10 • Physical Disk: % Disk Time is above 50% long term and Avg. Disk Queue Length is above 2 • Network Interface: Bytes Total/sec shows utilization is above 25% and Output Queue Length is above 2 14
  15. 15. Memory Performance Counters Key memory performance counters • Available Bytes - memory available for system use or apps • Value should be at least 10% of physical memory • Committed Bytes - memory with page file space reserved • Value should be less than 75% of physical memory • A slow rise could indicate a memory leak • Pages/sec - rate paging is used to resolve hard page faults • Preferred range is 0 to 20. Investigate if over 80 • Pool Nonpaged Bytes - memory for objects not paged • A slow rise could indicate a memory leak • Pool Paged Bytes - memory for objects paged to disk 15
  16. 16. Processor Performance Counters Key processor performance counters • % Processor Time – ratio CPU ran non-idle threads • Value should be less than 75% • Check for high value & Processor Queue Length > 10 Key system performance counters • Context Switches/sec - rate CPUs switch threads • A thread is preempted by a higher priority ready thread • CPU switches between user-mode and kernel-mode • High number can indicate high contention in locks • Processor Queue Length - # threads in the CPU queue • Divide this value by the number of processors • A sustained queue > 10 threads per CPU is a problem 16
  17. 17. Disk Performance Counters Key physical disk performance counters • % Disk Time - ratio that disk was busy with requests • Value should be less than 50% • Avg. Disk Queue Length - requests that were queued • Disk delays and bottlenecks are likely when value > 2 Key paging file performance counters* • % Usage - ratio of Page File in use • % Usage Peak - ratio of peak usage of Page File * Paging File counters are also related to memory counters 17
  18. 18. Network Performance Counters Key network interface performance counters • Bytes Total/sec - rate of bytes over NICs • Network utilization should be less than 25% • Output Queue Length – packet queue length • Value should be less than 2 18
  19. 19. Process Performance Counters Key Sitecore process performance counters • Private Bytes - amount of memory used by process • Memory that cannot be shared with other processes • Virtual Bytes - virtual address space used by process • Page File Bytes - space process reserved in page file • IO Data Operations/sec - rate process is doing I/O • IO Other Operations/sec - rate of control functions • % Processor Time – ratio process handled instructions 19
  20. 20. ASP.NET Performance Counters Key ASP.NET performance counters • Application Restarts - # restarts during server uptime • High number indicates problems are causing recycles • Requests Queued - # requests waiting for service • Increase shows server is at limit of concurrent requests • Raise default of 5,000 in the machine.config file • Worker Process Restarts - # worker process restarts • Shows unexpected failures and intentional recycles 20
  21. 21. ASP.NET Application Counters Key ASP.NET Applications performance counters • Errors Total - # of errors executing HTTP requests • Investigate parser, compilation and run-time errors • Pipeline Instance Count – max # concurrent requests • Shows requests that are being processed for the app • High number signifies the CPU is not well utilized by app • Requests/sec - current throughput of application • Values should remain within range under constant load 21
  22. 22. .NET Performance Counters Key .NET CLR Exceptions performance counters • # of Exceps Thrown - # of exceptions thrown • A high value indicates a code or configuration problem • Check Sitecore logs for related application errors Key .NET CLR Memory performance counters • # Induced GC - # calls to the Garbage Collector • A high value indicates a code or configuration problem • Remove calls to GC.Collect and GC.GetTotalMemory • % Time in GC - time spent since end of the last cycle • Investigate managed heap when value is 50% or higher 22
  23. 23. IIS Performance Counters Key Web Service performance counters • The following values should be as high as possible • Bytes Total/sec - # bytes transferred by the service • Total Method Requests/sec - HTTP requests received • Current Connections - # connections to web service Key Web Service Cache performance counters • The following values should be as high as possible • File Cache Hits % - ratio successful lookups since startup • Kernel:URI Cache Hits % - ratio of cache hits to requests • The following values should be as low as possible • Kernel:URI Cache Flushes - # files removed from cache • Kernel:URI Cache Misses - # unsuccessful cache lookups 23
  24. 24. SQL Performance Counters Key SQL Buffer Manager performance counter • Buffer Cache Hit Ratio - % of pages found in buffer • Page did not have to incur a read from disk • Value should be as high as possible Key SQL Statistics performance counters • Batch Requests/Sec - # received SQL batch requests • Over 1000 batch requests/sec may indicate a problem • SQL Compilations/Sec - # Transact-SQL compilations • Over 100 compilations/sec may indicate a problem 24
  25. 25. SQL Access Performance Counters Key SQL Access Methods performance counter • Page Splits/sec – splits from overflowing index pages • An index or data page can split when it becomes full • Page splitting can cause excessive I/O • Increase fill factor to allocate more space 25
  26. 26. Improving SQL Performance Configuring SQL Server for best performance • Set compatibility level to SQL Server 2008 (100) • Set Auto Close and Auto Shrink properties to false • Set Recovery Model to simple • Check for percentage of fragmentation on indexes • Run the Index Physical Statistics Standard Report • Run report against each of the Sitecore databases • Rebuild indexes based on recommended operations • Verify a Maintenance Plan exists and runs regularly • Clean up History, PublishQueue & EventQueue tables • Use SQL script in CMS Performance Tuning Guide 26
  27. 27. Sitecore Performance Counters Key Sitecore Caching performance counters • The following value should be as high as possible • Cache Hits - # of cache hits for all caches • The following values should be as low as possible • Cache Clearings - times that a cache has been cleared • Cache Misses - # of cache misses for all caches Key Sitecore Data performance counters • Data.PhysicalReads - # of read operations from DB • Data.PhysicalWrites - # of write operations to DB 27
  28. 28. Sitecore Counters Configuration Sitecore Performance Counters Configuration • Counters.Enabled – enable/disable counters • Enable performance counters when monitoring • Disable performance counters when not monitoring • Counters.InstanceName – counter names • Change setting when you monitor multiple web apps • Counters.ResetOnStartup – resets values to zero (0) 28
  29. 29. The Sitecore Health Monitor The Sitecore Health Monitor background task • Periodically writes counter values to Sitecore logs • Supports Windows, CLR and Sitecore counters • Web.config contains healthMonitor pipeline settings • Interval specified by HealthMonitorInterval setting • Users can add counters for LogCounterStatus method • instanceType must be specified for added counters 29
  30. 30. The Counter Dump Agent The Sitecore Counter Dump Agent • Periodically collects data on all Sitecore counters • Writes data to files in XML format • Files contain Sitecore counter values for time ranges • The frequency & filename can be set in web.config <agent type="Sitecore.Tasks.CounterDumpAgent" method="Run" interval="01:00:00"> <DumpFile> $(dataFolder)/diagnostics/counters.{date}.{time}.{processid}.txt </DumpFile> </agent> 30
  31. 31. Sitecore TaskManager Module Sitecore Task Manager and Performance Monitor • Shared source module available from Marketplace • A web-based tool to monitor background processes • Indexing, scheduled tasks, publishing, etc. 31
  32. 32. Load Testing Load Testing helps find slow running code paths • Most efficient way to identify code bottlenecks • Run load tests without caching during development • Use ANTS to profile your solution while stress testing Load Testing helps ensure a system will scale • Creates simulated server loads expected on live site • Anticipates performance with increased traffic loads • A series of test cases are used during the testing • Best to monitor on both the server and client sides 32
  33. 33. Load Testing Tools Server-based load testing • JMeter is a free stress testing/monitoring tool • Useful for identifying bottlenecks during development • Available at • Web Capacity Analysis Tool (WCAT) is a free tool • Web-based load testing • Load Tester LITE is provided at no cost • Load Tester PRO can be rented or permanent license • Amazon has rental fees for cloud-based load engines • Available at 33
  34. 34. In Summary • Performance monitoring is a process, not an event • Use PerfMon to log system & Sitecore counters • Log counters for system, IIS, ASP.NET, db, Sitecore, etc. • Monitor performance objects containing key counters • Investigate additional detailed counters as needed • Use Health Monitor to log counters to Sitecore logs • Use Task Manager to monitor background processes • Use load testing to find slow running code paths 34
  35. 35. Caveat Emptor There are many ways to monitor Sitecore performance • Skilled administrators can provide other solutions • Review your monitoring needs with your Sitecore partner • Review proposed changes with all stakeholders • Implement new configurations on a test system first This presentation is intended to be academic • To enhance public knowledge of performance monitoring • To promote better understanding of monitoring Sitecore Author cannot guarantee accuracy or fitness for your site • Author makes no representations or warranties • Author disclaims any liability if you act on this information 35
  36. 36. Acknowledgements Wikimedia Commons Image of ASP.NET Architecture Stack Wikimedia Commons Image of Network Topology Wikimedia Commons Image of Warning Sign Download slide presentation to view URLs 36
  37. 37. More Topics from TIC All Sitecore-related SlideShares from TIC How to Assure Success for a New Sitecore Site (full slide deck) How to Avoid Shooting Yourself in the SEO Implementing Routine Sitecore Maintenance Performance Tips for Sitecore Mobile Users Sitecore 7 – New Features for Users Sitecore Performance Tips for Visitor Engagement 37
  38. 38. The Inside Corner, Inc. • The Inside Corner, Inc. (TIC) is owned by Craig Nelson • TIC is a Sitecore partner & web consultancy • A U.S. point of contact for teams of Sitecore specialists • We create new Sitecore sites & enhance existing ones 38
  39. 39. Primary Developer - Craig Nelson • Sitecore Specialist certified for Sitecore 7 • 10 years of system administration experience • 10 years as a Microsoft Certified Trainer • 5 years of ASP.NET web development experience • 4 years of Sitecore development experience • 3 years as a Sitecore technical instructor • Sitecore architecture & configuration expert • Performance & SEO optimization specialist • Workflow configuration specialist 39
  40. 40. Get Help with your Sitecore Site Follow TIC on LinkedIn • Add TIC to your Circles on Google Plus • Follow TIC on Twitter • Contact Craig Directly • Craig Nelson - Certified 7.0 Developer and Sitecore Partner • cwnelson[at] • Mobile - 707.319.7446 Visit the TIC website • 40
  41. 41. Bibliography All about Sitecore performance counters — A real hero Optimizing Performance in Sitecore SDN - Working with Sitecore Performance Counters Sitecore CMS 6.0 -6.5 Diagnostics Guide Sitecore CMS 6.0 -6.6 Performance Tuning Guide Troubleshooting Performance Problems in SQL 2008 Download slide presentation to view URLs 41
  42. 42. Thank you for taking the time to view this presentation on Troubleshooting Sitecore Performance Please share with others and add a comment if you think it is useful Thanks August 12, 2013 42