Lifting the Blinds: Monitoring Windows Server 2012
•Download as PPTX, PDF•
2 likes•14,119 views
Operating systems monitor resources continuously in order to effectively schedule processes. In this webinar, Evan Mouzakitis (Datadog) discusses how to get operational data from Windows Server 2012 using a variety of native tools.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Lifting the Blinds: Monitoring Windows Server 2012
Similar to Lifting the Blinds: Monitoring Windows Server 2012 (20)
More from Datadog
More from Datadog (20)
Recently uploaded
Recently uploaded (20)
Lifting the Blinds: Monitoring Windows Server 2012
- 1. Read the full guide at: http://www.datadoghq.com/blog/monitoring-windows-server/ g the Blinds: Monitoring Windows Server
- 2. • SaaS based infrastructure and app monitoring • Open Source Agent • Time series data (metrics and events) • Processing nearly a trillion data points per day • Intelligent Alerting and Insightful Dashboards Datadog Overview
- 3. Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores, Caches, Queues and more... Monitor Everything
- 4. Agenda - Why should I monitor Windows Server? - What are some indicators of performance issues? - How can I collect performance metrics for analysis?
- 8. CPU metrics - PercentProcessorTime - ContextSwitchesPersec - ProcessorQueueLength - DPCsQueuedPersec - PercentPrivilegedTime - PercentDPCTime - PercentInterruptTime
- 9. CPU: ContextSwitchesPersec What it tracks: Number of times the processor switched to a new thread Correlate with: Memory: PageFaultsPersec Disk: DiskTransfersPersec Network: BytesSentPersec/BytesReceivedPersec Issue resolution: Adding processors, thread partitioning, DPC partitioning, hardware interrupt partitioning, disable I/O counters
- 10. CPU: PercentProcessorTime What it tracks: Percentage of time spent performing work (not idle) Correlate with: ProcessorQueueLength Issue resolution: More processors, bigger instance, optimize offending application,
- 11. CPU: ProcessorQueueLength What it tracks: Size of processor queue Correlate with: CPU: PercentProcessorTime, PercentPrivilegedTime, PercentDPCTime, PercentInterruptTime Issue resolution: Adding processors, thread partitioning, DPC partitioning, hardware interrupt partitioning, disable I/O counters
- 12. CPU:DPCsQueuedPersec What it tracks: Deferred procedure call (DPC) enqueue rate Correlate with: CPU: PercentDPCTime Disk: DiskTransfersPersec Network: BytesSentPersec/BytesReceivedPersec Issue resolution: Remove buggy device, rollback driver
- 13. CPU: PercentPrivilegedTime/PercentDPCTime PercentInterruptTime What they track: Percentage of time CPU spent in privileged mode/deferred procedure calls/interrupts Correlate with: ContextSwitchesPersec/PercentPrivilegedTime/PercentDPCTime PercentInterruptTime Issue resolution: Adding processors, thread partitioning, DPC partitioning, hardware interrupt partitioning, disable I/O counters
- 14. Memory metrics - PoolNonpagedBytes - PageFaultsPersec - PagesInputPersec
- 15. Memory: PoolNonpagedBytes What it tracks: Amount of non-paged memory in use Correlate with: Windows Event 2019 “Nonpaged Memory Pool Empty” Issue resolution: Identify troublesome driver/roll back to known good state
- 16. What it tracks: Rate of page faults Correlate with: PagesInputPersec Issue resolution: Increase system memory Memory: PageFaultsPersec
- 17. What it tracks: Rate pages are read (from disk) into memory Correlate with: PageFaultsPersec/ DiskTransfersPersec Issue resolution: Increase system memory, move page file to separate physical disk Memory: PagesInputPersec
- 18. - AvgDiskQueueLength - DiskTransfersPersec - PercentIdleTime Disk Metrics
- 19. Disk: AvgDiskQueueLength What it tracks: Running average of I/O ops in queue Correlate with: DiskTransfersPersec Issue resolution: Move data for I/O-intensive applications to separate disk; add disks to syste
- 20. Disk: DiskTransfersPersec What it tracks: Aggregate I/O rate Correlate with: AvgDiskQueueLength Issue resolution: Move data for I/O-intensive applications to separate disk; add disks to system; increase disk cache
- 21. Disk: PercentIdleTime What it tracks: Percent of time disk is idle Correlate with: AvgDiskQueueLength Issue resolution: Move page file to separate disk; add disks to system; use SSDs
- 22. Tooling
- 23. Word of Warning
- 24. Powershell - Windows’ scripting language (no more batch files!) - Powerful language with deep OS support - Integrates with C# natively - Output is typed (unlike *NIX)
- 25. Powershell
- 26. Powershell
- 27. Perfmon
- 28. Windows Performance Toolkit Requires Windows Assessment and Deployment Kit (formerly Windows Performance Toolkit) https://www.microsoft.com /en- US/download/details.aspx ?id=39982
- 30. Questions? Evan Mouzakitis Research Engineer Twitter: @vagelim Email: evan@datadoghq.com Read the full guide at: http://www.datadoghq.com/blog/monitoring-windows-server/
Editor's Notes
- Our goal is to help you monitor everything from all levels of your stack so that you can make intelligent data based decisions about your applications and infrastructure.
- Why monitor Windows in the first place? Monitoring the performance of the applications that run your business is critical; but applications don’t live in a vacuum. Applications interact with the underlying operating system often to, request resources, preempt the execution of other processes, access hardware devices, and more. Being aware of the health and performance of the operating system gives you more information when troubleshooting issues anywhere higher up in the stack (not to mention that monitoring the operating system is critical for insight into hardware issues). For example, is a SQL Server database query slow because of the query itself, or because the SQL Server is also hosted alongside Exchange and they are competing for disk access? These kinds of issues can only be surfaced when you monitor both the application in question and the underlying operating system.
- A monitoring plan typically tries to cover Work metrics, Resource metrics, and non-metric data like events or code changes. As the broker between applications and hardware resources, when monitoring Windows server we are primarily focused on resource metrics, because that is what the operating system is managing. Work metrics are usually more applicable to application-level monitoring, but as you will see there are a few work metrics related to disk access that we’ll cover here too.
- What kind of resources are we interested in monitoring? What kinds of metrics can we surface from those resources? Generally speaking, the most useful resources to monitor are CPU, RAM, disk, and network. Things like power consumption, thermal monitoring, noise and data of a similar nature, while useful, don’t usually add meaningful context to application or operating system performance issues.
- At the highest level, the following metrics are useful in assessing CPU performance, and can shed light on performance bottlenecks depending on what the kind of work the CPU spends most of its time performing.
- ContextSwitchesPersec tracks the number of times the processor switched to a new execution context. Context switches are computationally expensive; before the processor can enter the execution context of another thread, it must first save the current context, push the old context to the bottom of its priority queue, find the highest priority queue containing an executable thread, pop it from its queue, load its context, and finally execute the thread. In a multi-core machine (common today), context switching add significant overhead. By default, the Windows Task manager measures I/O per-process, and attributing I/O to a particular process in a multi-core multithreaded environment can have a drastic performance impact under heavy I/O loads. If that’s the case, you would benefit from disabling global and per-process I/O counters by adding a CountOperations entry as a REG_DWORD with a value of 0 to the registry under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\I/O System\
- PercentProcessorTime is a metric most everyone is familiar with, even if they don’t know the name. It tracks the percentage of time the CPU was doing something. In and of itself, this metric isn’t all that useful. For example, if I’m analyzing data on a single core machine, I’d expect the CPU to in use 100% of the time. However, when correlated with ProcessorQueueLength, which tracks the number of pending threads, you have enough information to determine whether or not the system is suffering a CPU bottleneck. A queue length greater than 2 * the number of processors, coupled with prolonged periods of maxed out CPU utilization very clearly indicate that the system does not have enough processor resources to perform all of its tasks.
- The processor queue length is a value which reflects the number of threads that are ready to run, but are not able to use the processor. A healthy measure of processor queue length is about 2 * the number of processors on the system. Even on multicore machines, there is only one processorqueuelength performance counter. High values for this counter very clearly indicate CPU contention. You can correlate this metric with other CPU metrics like PercentProcessorTime, PercentPrivilegedTime, PercentDPCTime, PercentInterruptTime to determine where the CPU is spending its time, and to narrow down if the CPU is the bottleneck causing backed up queue.
- Hardware requirements demand real-time, unfettered access to the CPU in order to ensure that high-priority work (like accepting keyboard input) is performed when it is needed. Interrupts provide a means by which devices can interrupt the processor and force it to perform the requested operation (triggering the processor to perform a context switch). Some work from devices may be put off until later, but still must be accomplished in a timely manner. Enter DPCs. Through DPCs, real-time processes like device drivers can schedule lower-priority tasks to be completed after higher-priority interrupts are handled. DPCs are created by the kernel, and can only be called by kernel mode programs. A large or near-constant number of DPCs could point to issues with low-level system software. An unused but buggy sound driver could be the culprit, for example.
- This trio of metrics, taken together, help to shed light on where the CPU is spending its time. In particular, privileged time reflects the time spent executing instructions for kernel-mode programs. Code executing in privileged mode have unrestricted access to the system’s hardware. This includes device drivers, core operating system functions, etc. If you observe a system spending 30 percent or more of its time processing privileged instructions, check the values of PercentDPCTime and PercentInterruptTime. If either of those two metrics report values greater than 20%, it is likely that a poorly written device driver, or very busy peripheral is the culprit.
- As with CPU metrics, Windows exposes a wealth of performance counters tracking memory statistics. We’ve omitted AvailableMemory and similar metrics from this webinar because they are pretty self-explanatory. The three listed here, PageFaultsPersec, PoolNonpagedBytes, and PagesInputPersec provide insight into the nature of issues which may be impacting performance. We’ll touch on each in turn, but at a high level, PageFaultsPersec tracks the rate of page faults, PoolNonpagedBytes describes the current size of non-pageable memory, and the last, PagesInputPersec, describes the rate of pages read from disk (which is distinct from the number of page reads from disk).
- Windows maintains two general pools of memory: a paged pool and non paged pool. The paged pool is for general use and is the pool used by all user space applications for memory allocation. Because user space applications are more tolerant to latency, or, to put it another way, because user space applications don’t generally have real-time requirements, they can get by if the requested memory needs to be read in (or paged in) from disk. Because kernel-level software has real-time execution requirements, device drivers and the like make use of the non paged pool. The non paged pool is guaranteed to reside in physical memory at all times, with no possibility of being paged to disk (hence the name “non paged”). This significantly reduces latency by preventing the possibility of page faults. No memory pool is infinite, and poorly written device drivers could end up exhausting the entire non paged pool if left unchecked. If you are seeing reports of Event 2019, it’s already too late. But keeping an eye on the size of this pool and its growth over time are necessary to identify and deal with any troublesome drivers or hardware.
- Page faults occur when a thread references a page that is not in the current set of memory-resident pages. Because the thread can’t perform its work without the requested memory, a hardware interrupt occurs, the processor enters into kernel-mode (resulting in a context switch—both upon entering and exiting kernel-mode), and attempts to locate the page in memory. If the page is found somewhere else in memory, it is that address which is returned to the requesting thread. This is called a “soft” page fault. If the page is not elsewhere in memory the kernel will look in the page file and read it into memory. This is called a “hard” page fault. Because this operation requires accessing the disk, it is more computationally expensive to perform this type of lookup. Page faults occur under normal operating conditions, but a spike in page faults could result in serious performance degradation, depending on the “hardness” of the fault. By tracking the page fault rate alongside the page input rate, you can differentiate between hard and soft page faults. High values of both metrics unequivocally indicate hard page faults. There’s not much you can do to prevent soft page faults from occurring, but increasing the amount of RAM available on the system is a straightforward way of alleviating hard page faults. It is worth mentioning that when a hard page fault does occur, Windows attempts to retrieve multiple, contiguous pages into memory, to maximize the work performed by each read. This, in turn, can potentially increase a page fault’s performance impact, as more disk bandwidth is consumed reading in potentially unneeded pages. All of this can potentially be avoided by putting your page file (see next section) on a separate physical (not logical) disk, or increasing the amount of RAM available to your system.
- As I mentioned, there are two types of page faults, and tracking PagesInputPersec alongside PageFaultsPersec gives you the information you need to determine the type of page fault occurring. If you are seeing high values of both metrics, the page faults are hard. The effects of hard page faults can be exacerbated if disk is a contentious resource. To give a simplified example, if your have a system with one disk and it’s running an I/O intensive application, page faults will hit this system harder (and performance will degrade in the application) because Windows is competing with the application for disk access (and Windows always wins). This goes to show that an excessive number of page faults can be responsible for system wide effects, completely unrelated to the application experiencing performance degradation.
- Though there are many disk metrics worth tracking, I’ve distilled the list to the most essential, while omitting the obvious, like PercentFreeSpace.
- The AvgDiskQueueLength counter gives an estimated average of the number of I/O operations currently awaiting execution. Generally speaking, this counter should not exceed 2 * the number of drives on the system. If you are seeing greater values than that, it means the system cannot service the number of I/O requests it’s receiving in a timely manner, which can lead to processing delays, degraded application performance, and more.
- DiskTransfersPersec is an aggregate measure of both disk reads and writes. It is useful for shedding light on the cause of bottlenecks. High values for this metric do not always indicate issues; for example if you are running I/O intensive applications on your server you are definitely going to observe high values for this metric (and most likely for PercentIdleTime as well). However, if I/O ops are not being enqueued (per the AvgDiskQueueLength metric) and applications are not hurting for memory (and thus paging to disk), there should be no observable performance impact.
- PercentIdleTime is a pretty intuitive metric that tracks the percent of time disks are idle. Depending on the role of the system under investigation, low idle times may be expected, especially for when running I/O intensive applications like SQL Server or Exchange. If that’s not the case, low values should be investigated. If you don’t already have your page file stored on a separate drive, you should do so. Otherwise, consider either adding disks to the system to increase performance, or swap out HDDs for SSDs if possible.
- Windows offers numerous methods by which you can collect, store, and visualize system performance data. Because the methods are so varied, I will only go through a couple of the tools that I have experience with. All of the tools mentioned are native to Windows Server 2012 R2 so you can get up and running quickly.
- Reading performance counters does not generally appear to have much of an impact on system performance. In my tests, collecting 2631 counters with 1-second sample rate caused a 4 percent increase in user CPU usage (by perfmon). There are a few things to keep in mind, though: depending on the data collected and the duration of the collection, the collected data could be very large. To give you an idea about the size of the data collected, in a test collecting handle and kernel base events, pagefaults, cpu, I/O and memory samples, the data grew at a rate approaching 100 MB/min. Additionally, if you are collecting data from your local machine, you may see occasional spikes in I/O latency; in my tests I observed response times for some user space applications in excess of 2000 ms! Also, I did not attempt to collect performance counters from user applications which may have an impact on the application’s performance. And as I mentioned earlier in the CPU section, if you are sampling I/O with processor-specific information, you most certainly will observe degradation in performance.
- Powershell is great for collecting performance counters programmatically. You can query the event log from powershell as well. You can use powershell to collect metrics from local and remote machines.
- Here are some example powershell commands for retrieving CPU-related performance counters. As you can see, there is a regular pattern. For a full list of commands to retrieve performance counters for CPU, memory, disk, network, and events, check out my “How to collect Windows Server 2012 metrics” article on the datadog blog. https://www.datadoghq.com/blog/collect-windows-server-2012-metrics/#toc-powershell
- Last thing about powershell, if you want to do something in powershell and there’s no pre-packaged cmdlet to get you what you want, you can always interface with WMI to get what you’re looking for.
- In my honest opinion, perfmon is not nearly as useful as xperf or Windows Performance Recorder when it comes to investigating performance issues. It is a good tool to help spot issues, but not so good for getting into the nitty gritty. Here’s a screenshot of perfmon collecting “System Performance counters” a counter set provided out of the box. As you can see, there is a lot going on. My investigation was focusing on the cause of excessive memory use, visualized as the black bar nearly pinned to the 100 mark. From this image it’s clear that something is going on, but since I was only collecting the Total memory usage (as opposed to collection per-process), it isn’t clear which process is exhausting RAM. To determine the underlying cause in this case requires me to re-run perfmon, this time collecting per-process counters in addition to the total, and hoping that my issue arises again. As you’re about to see, we can do better.
- The Windows performance toolkit contains the Windows Performance Recorder & Windows Performance Analyzer (WPA). Though technically not strictly “native” since it requires a download, it is a useful, graphical tool for collecting and analyzing windows performance data and is made by Microsoft.
- Windows performance recorder is a modern replacement for xperf. It features both graphical and command line interfaces. Here you can see the available collection profiles. Collecting data with the Windows Performance Recorder is as easy as clicking “Start”. Technically, Windows Performance Recorder (and xperf) do not merely collect performance counters; they are a tracing mechanism for collecting fine-grained performance data. As you will see, traces are superior to performance counters when investigating performance issues.