• Like

Troubleshooting Sitecore Performance

  • 3,096 views
Uploaded on

Monitoring & Analyzing Sitecore Performance Issues - August 12, 2013

Monitoring & Analyzing Sitecore Performance Issues - August 12, 2013

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,096
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
73
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Troubleshooting Sitecore Performance Monitoring & Analyzing Performance Issues August 12, 2013 1
  • 2. Please review before viewing this presentation http://www.slideshare.net/cwnelson/sitecore-performance-tips http://www.slideshare.net/cwnelson/sitecore-mobile-performance-tips 2
  • 3. Sitecore Enterprise Web CMS • Popular .NET Web Content Management System for mid-to-large organizations • Provides friendly tools for non-technical Content Authors and Marketers • Can be customized and extended to meet almost any conceivable business need 3
  • 4. Troubleshooting Performance Steps to troubleshoot performance issues • Create baseline logs of performance metrics • Monitor system when it is offline or off hours • Monitor system during normal operation • Regularly review performance logs • Become familiar with normal metrics for your system • Investigate performance metrics that are not normal • Scrutinize counters in areas of unusual performance • Try to narrow the scope of performance problems • Take action to remove and or resolve bottlenecks 4
  • 5. Performance Monitoring Monitoring helps locate root causes of issues • Inadequate resources or resource misallocation • Non-optimum coding or use of a legacy API • Missing configuration settings or misconfiguration Monitoring helps identify resource bottlenecks • Server hardware & operating systems • IIS & ASP.NET • Databases • Sitecore Monitoring is a recurring process, not an event • Familiarity with the system is a key success factor 5
  • 6. Why Monitoring is Important Building business trust • Ensuring performance meets user expectations • Averting business-impacting performance issues Accomplishing cost-effective site management • Efficiently diagnosing root causes of issues • Perceiving the corrective actions needed to fix issues • Addressing issues instead of upgrading hardware • Accurately budgeting for future growth needs 6
  • 7. Windows Performance Monitor Windows Performance Monitor (PerfMon) • Use it to display graphs for all performance logs • System, IIS, ASP.NET, SQL, Sitecore counters, etc. • Computer Mgmt > Performance > Monitoring Tools • Use <Ctrl-h> to toggle highlight of selected counter 7
  • 8. Understanding Bottlenecks Performance bottlenecks • Performance of system is limited by a single resource • Bottleneck points are limiting components of system Bottleneck indicators • One or more resources are over-consumed • Resource requests exceed available resources • Resource requests must be queued until available • One or more resources are under-consumed • More subtle type of bottleneck that is harder to locate • Resource requests are not at their expected levels • The bottleneck blocking requests is elsewhere 8
  • 9. Troubleshooting Bottlenecks Identifying bottlenecks • Create a baseline log when system is offline or at rest • Monitor the system to log the current behavior • Compare the current behavior with the baseline log Investigating bottlenecks • Review system counters before Sitecore counters • Look for sustained behaviors, not peaks or spikes • Over-consumption or under-consumption of resources • Use broad counters first; % Disk Time (Reads & Writes) • Use detailed counters to investigate suspicious values; Avg. Disk Bytes Read & Avg. Disk Read Queue Length 9
  • 10. Viewing Server Performance Logs Inspect server performance logs for issues • Check for signs of memory leaks • Creeping increases in consumption indicate a leak • Check for signs of bottlenecks • High resource consumption for prolonged periods • Related queue length above 2 • Check for non-optimum metrics • Errors are signs of code issues or misconfiguration • Cache misses are signs of configuration issues 10
  • 11. Server Bottlenecks Server Bottlenecks • One or more server resources are over consumed • The server contains one or more bottleneck points 11 Server Bottleneck Types Memory Processor Disk Network
  • 12. Server Performance Troubleshooting server performance issues • Log performance objects that contain key counters • Set logging interval to 15 minutes • Log data over multiple weeks • Periodically review logged data for key counters • Use detailed counters to investigate suspect counters • Key counters that show signs of a potential problem • Periodically review Sitecore logs for nested exceptions • Rectify issues found from monitoring & Sitecore logs • Periodically review caching statistics & tune caches 12
  • 13. Key Server Objects & Counters Memory • Available Bytes • Committed Bytes Processor • % Processor Time System • Processor Queue Length 13 Physical Disk • % Disk Time • Avg. Disk Queue Length Network Interface • Bytes Total/sec • Current Bandwidth • Output Queue Length
  • 14. Warning Signs for Server Counters Use performance counters to check warning signs • Memory: Committed Bytes is creeping up long term • Processor: % Processor Time is above 75% long term and System: Processor Queue Length is above 10 • Physical Disk: % Disk Time is above 50% long term and Avg. Disk Queue Length is above 2 • Network Interface: Bytes Total/sec shows utilization is above 25% and Output Queue Length is above 2 14
  • 15. Memory Performance Counters Key memory performance counters • Available Bytes - memory available for system use or apps • Value should be at least 10% of physical memory • Committed Bytes - memory with page file space reserved • Value should be less than 75% of physical memory • A slow rise could indicate a memory leak • Pages/sec - rate paging is used to resolve hard page faults • Preferred range is 0 to 20. Investigate if over 80 • Pool Nonpaged Bytes - memory for objects not paged • A slow rise could indicate a memory leak • Pool Paged Bytes - memory for objects paged to disk 15
  • 16. Processor Performance Counters Key processor performance counters • % Processor Time – ratio CPU ran non-idle threads • Value should be less than 75% • Check for high value & Processor Queue Length > 10 Key system performance counters • Context Switches/sec - rate CPUs switch threads • A thread is preempted by a higher priority ready thread • CPU switches between user-mode and kernel-mode • High number can indicate high contention in locks • Processor Queue Length - # threads in the CPU queue • Divide this value by the number of processors • A sustained queue > 10 threads per CPU is a problem 16
  • 17. Disk Performance Counters Key physical disk performance counters • % Disk Time - ratio that disk was busy with requests • Value should be less than 50% • Avg. Disk Queue Length - requests that were queued • Disk delays and bottlenecks are likely when value > 2 Key paging file performance counters* • % Usage - ratio of Page File in use • % Usage Peak - ratio of peak usage of Page File * Paging File counters are also related to memory counters 17
  • 18. Network Performance Counters Key network interface performance counters • Bytes Total/sec - rate of bytes over NICs • Network utilization should be less than 25% • Output Queue Length – packet queue length • Value should be less than 2 18
  • 19. Process Performance Counters Key Sitecore process performance counters • Private Bytes - amount of memory used by process • Memory that cannot be shared with other processes • Virtual Bytes - virtual address space used by process • Page File Bytes - space process reserved in page file • IO Data Operations/sec - rate process is doing I/O • IO Other Operations/sec - rate of control functions • % Processor Time – ratio process handled instructions 19
  • 20. ASP.NET Performance Counters Key ASP.NET performance counters • Application Restarts - # restarts during server uptime • High number indicates problems are causing recycles • Requests Queued - # requests waiting for service • Increase shows server is at limit of concurrent requests • Raise default of 5,000 in the machine.config file • Worker Process Restarts - # worker process restarts • Shows unexpected failures and intentional recycles 20
  • 21. ASP.NET Application Counters Key ASP.NET Applications performance counters • Errors Total - # of errors executing HTTP requests • Investigate parser, compilation and run-time errors • Pipeline Instance Count – max # concurrent requests • Shows requests that are being processed for the app • High number signifies the CPU is not well utilized by app • Requests/sec - current throughput of application • Values should remain within range under constant load 21
  • 22. .NET Performance Counters Key .NET CLR Exceptions performance counters • # of Exceps Thrown - # of exceptions thrown • A high value indicates a code or configuration problem • Check Sitecore logs for related application errors Key .NET CLR Memory performance counters • # Induced GC - # calls to the Garbage Collector • A high value indicates a code or configuration problem • Remove calls to GC.Collect and GC.GetTotalMemory • % Time in GC - time spent since end of the last cycle • Investigate managed heap when value is 50% or higher 22
  • 23. IIS Performance Counters Key Web Service performance counters • The following values should be as high as possible • Bytes Total/sec - # bytes transferred by the service • Total Method Requests/sec - HTTP requests received • Current Connections - # connections to web service Key Web Service Cache performance counters • The following values should be as high as possible • File Cache Hits % - ratio successful lookups since startup • Kernel:URI Cache Hits % - ratio of cache hits to requests • The following values should be as low as possible • Kernel:URI Cache Flushes - # files removed from cache • Kernel:URI Cache Misses - # unsuccessful cache lookups 23
  • 24. SQL Performance Counters Key SQL Buffer Manager performance counter • Buffer Cache Hit Ratio - % of pages found in buffer • Page did not have to incur a read from disk • Value should be as high as possible Key SQL Statistics performance counters • Batch Requests/Sec - # received SQL batch requests • Over 1000 batch requests/sec may indicate a problem • SQL Compilations/Sec - # Transact-SQL compilations • Over 100 compilations/sec may indicate a problem 24
  • 25. SQL Access Performance Counters Key SQL Access Methods performance counter • Page Splits/sec – splits from overflowing index pages • An index or data page can split when it becomes full • Page splitting can cause excessive I/O • Increase fill factor to allocate more space 25
  • 26. Improving SQL Performance Configuring SQL Server for best performance • Set compatibility level to SQL Server 2008 (100) • Set Auto Close and Auto Shrink properties to false • Set Recovery Model to simple • Check for percentage of fragmentation on indexes • Run the Index Physical Statistics Standard Report • Run report against each of the Sitecore databases • Rebuild indexes based on recommended operations • Verify a Maintenance Plan exists and runs regularly • Clean up History, PublishQueue & EventQueue tables • Use SQL script in CMS Performance Tuning Guide 26
  • 27. Sitecore Performance Counters Key Sitecore Caching performance counters • The following value should be as high as possible • Cache Hits - # of cache hits for all caches • The following values should be as low as possible • Cache Clearings - times that a cache has been cleared • Cache Misses - # of cache misses for all caches Key Sitecore Data performance counters • Data.PhysicalReads - # of read operations from DB • Data.PhysicalWrites - # of write operations to DB 27
  • 28. Sitecore Counters Configuration Sitecore Performance Counters Configuration • Counters.Enabled – enable/disable counters • Enable performance counters when monitoring • Disable performance counters when not monitoring • Counters.InstanceName – counter names • Change setting when you monitor multiple web apps • Counters.ResetOnStartup – resets values to zero (0) 28
  • 29. The Sitecore Health Monitor The Sitecore Health Monitor background task • Periodically writes counter values to Sitecore logs • Supports Windows, CLR and Sitecore counters • Web.config contains healthMonitor pipeline settings • Interval specified by HealthMonitorInterval setting • Users can add counters for LogCounterStatus method • instanceType must be specified for added counters 29
  • 30. The Counter Dump Agent The Sitecore Counter Dump Agent • Periodically collects data on all Sitecore counters • Writes data to files in XML format • Files contain Sitecore counter values for time ranges • The frequency & filename can be set in web.config <agent type="Sitecore.Tasks.CounterDumpAgent" method="Run" interval="01:00:00"> <DumpFile> $(dataFolder)/diagnostics/counters.{date}.{time}.{processid}.txt </DumpFile> </agent> 30
  • 31. Sitecore TaskManager Module Sitecore Task Manager and Performance Monitor • Shared source module available from Marketplace • A web-based tool to monitor background processes • Indexing, scheduled tasks, publishing, etc. http://marketplace.sitecore.net/Modules/TaskManager 31
  • 32. Load Testing Load Testing helps find slow running code paths • Most efficient way to identify code bottlenecks • Run load tests without caching during development • Use ANTS to profile your solution while stress testing Load Testing helps ensure a system will scale • Creates simulated server loads expected on live site • Anticipates performance with increased traffic loads • A series of test cases are used during the testing • Best to monitor on both the server and client sides 32
  • 33. Load Testing Tools Server-based load testing • JMeter is a free stress testing/monitoring tool • Useful for identifying bottlenecks during development • Available at http://jmeter.apache.org/ • Web Capacity Analysis Tool (WCAT) is a free tool • http://www.iis.net/1466/ItemPermalink.ashx Web-based load testing • Load Tester LITE is provided at no cost • Load Tester PRO can be rented or permanent license • Amazon has rental fees for cloud-based load engines • Available at http://www.webperformance.com/ 33
  • 34. In Summary • Performance monitoring is a process, not an event • Use PerfMon to log system & Sitecore counters • Log counters for system, IIS, ASP.NET, db, Sitecore, etc. • Monitor performance objects containing key counters • Investigate additional detailed counters as needed • Use Health Monitor to log counters to Sitecore logs • Use Task Manager to monitor background processes • Use load testing to find slow running code paths 34
  • 35. Caveat Emptor There are many ways to monitor Sitecore performance • Skilled administrators can provide other solutions • Review your monitoring needs with your Sitecore partner • Review proposed changes with all stakeholders • Implement new configurations on a test system first This presentation is intended to be academic • To enhance public knowledge of performance monitoring • To promote better understanding of monitoring Sitecore Author cannot guarantee accuracy or fitness for your site • Author makes no representations or warranties • Author disclaims any liability if you act on this information 35
  • 36. Acknowledgements Wikimedia Commons Image of ASP.NET Architecture Stack Wikimedia Commons Image of Network Topology Wikimedia Commons Image of Warning Sign Download slide presentation to view URLs 36
  • 37. More Topics from TIC All Sitecore-related SlideShares from TIC http://www.slideshare.net/cwnelson/presentations How to Assure Success for a New Sitecore Site (full slide deck) http://www.slideshare.net/cwnelson/creating-new-sitecore-sites-full How to Avoid Shooting Yourself in the SEO http://www.slideshare.net/cwnelson/sitecore-seo-tips Implementing Routine Sitecore Maintenance http://www.slideshare.net/cwnelson/sitecore-maintenance-tips Performance Tips for Sitecore Mobile Users http://www.slideshare.net/cwnelson/sitecore-mobile-performance-tips Sitecore 7 – New Features for Users http://www.slideshare.net/cwnelson/sitecore-7-new-features Sitecore Performance Tips for Visitor Engagement http://www.slideshare.net/cwnelson/sitecore-performance-tips 37
  • 38. The Inside Corner, Inc. • The Inside Corner, Inc. (TIC) is owned by Craig Nelson • TIC is a Sitecore partner & web consultancy • A U.S. point of contact for teams of Sitecore specialists • We create new Sitecore sites & enhance existing ones 38
  • 39. Primary Developer - Craig Nelson • Sitecore Specialist certified for Sitecore 7 • 10 years of system administration experience • 10 years as a Microsoft Certified Trainer • 5 years of ASP.NET web development experience • 4 years of Sitecore development experience • 3 years as a Sitecore technical instructor • Sitecore architecture & configuration expert • Performance & SEO optimization specialist • Workflow configuration specialist 39
  • 40. Get Help with your Sitecore Site Follow TIC on LinkedIn • http://www.linkedin.com/company/the-inside-corner-inc. Add TIC to your Circles on Google Plus • http://google.com/+Theinsidecorner Follow TIC on Twitter • https://twitter.com/TheInsideCorner Contact Craig Directly • Craig Nelson - Certified 7.0 Developer and Sitecore Partner • cwnelson[at]ticdevs.com • Mobile - 707.319.7446 Visit the TIC website • http://ticdevs.com 40
  • 41. Bibliography All about Sitecore performance counters — A real hero Optimizing Performance in Sitecore SDN - Working with Sitecore Performance Counters Sitecore CMS 6.0 -6.5 Diagnostics Guide Sitecore CMS 6.0 -6.6 Performance Tuning Guide Troubleshooting Performance Problems in SQL 2008 Download slide presentation to view URLs 41
  • 42. Thank you for taking the time to view this presentation on Troubleshooting Sitecore Performance Please share with others and add a comment if you think it is useful Thanks August 12, 2013 42