Build 2017 - P4099 - Production tracing with Event Tracing for Windows (ETW)
•Download as PPTX, PDF•
1 like•598 views
An overview of ETW and the frameworks built around it, including WPP, manifests, Event Log, and the new TraceLogging framework.
Recommended
More Related Content
More from Windows Developer
More from Windows Developer (20)
Recently uploaded
Recently uploaded (20)
Build 2017 - P4099 - Production tracing with Event Tracing for Windows (ETW)
- 11. • • • •
Editor's Notes
- ETW is a facility provided by the Windows OS for use by all developers. APIs for generating events are available at all levels of the system, from kernel-mode drivers to Windows Store apps. The main API for this is EventWrite (user-mode), EtwWrite (kernel-mode). No special privileges are required to generate an event. Usually, an ETW framework (TraceLogging, manifests, or WPP) is used to manage the interactions with ETW instead of calling these APIs directly. APIs for collecting events are only available in user-mode. The main APIs for this are StartTrace, ControlTrace, EnableTraceEx2. There are two modes of trace collection: process (can only collect events that originated in the same process) and system (can collect events from anywhere on the system, including events from the OS kernel). Process-mode collection does not require any special privileges. System-mode collection requires elevated privileges. Microsoft provides several tools that can control trace collection, including traceview, tracelog, xperf, wpr. ETW collects events and routes them to a destination. The destination can be a log file (circular or sequential), a fixed-size memory buffer (circular, i.e. new events overwrite older events), or a real-time consumer (events delivered to a user-mode program via a ProcessTrace callback). When events are collected in a memory buffer, they can be written to disk on-demand (i.e. if something interesting happens, you can flush the buffer to disk to find out what happened). APIs for extracting events from a trace file or from a real-time consumer are available in user-mode. The main APIs for this are OpenTrace and ProcessTrace. Processing a trace file requires no special privileges. Connecting to a real-time trace requires elevated privileges. Microsoft provides several tools for decoding and analyzing events, including traceview, tracefmt, xperf, wpa. ETW is used within the Windows OS: Boot tracing. OS kernel tracing. Drivers. System services. Shell. Applications. ETW supports several kinds of advanced filtering, but the filtering is most efficient if the event can be filtered based on the provider GUID, the event level, or the event keyword. This is because filtering based on these characteristics is performed very early and very efficiently. ETW performs filtering in stages: The code generating the events can do its own filtering. The EventWrite API will perform filtering in-process. It will return immediately if it determines that the event does not need to be written. The ETW runtime will perform filtering in the kernel. Filtering at stage 1 is most efficient. This filtering can often be done in one or two CPU instructions (a quick check of a Boolean variable). It can also be done before the event data is collected and packed. The ETW runtime provides notification callbacks to support this kind of filtering. The Microsoft-provided ETW frameworks register for these callbacks to handle this kind of filtering for you. If you are using a Microsoft-provided ETW framework, it will perform filtering based on provider GUID, event level, and event keyword automatically before it calls any ETW API. In C/C++, the Microsoft-provided ETW frameworks use macros so that this filtering occurs even before the data is collected. As an example, for something like: WriteMyEvent(GatherData()); the Microsoft-provided ETW framework will generate a macro WriteMyEvent(…) that expands to something like this: if (MyEventEnabled != 0) { data = GatherData(); EventWrite(ProviderHandle, MyEventInfo, data); } This way, your GatherData() function isn’t called unless somebody is actually recording your event. The event generation APIs (e.g. EventWrite) are always non-blocking, even if the event is being recorded. The EventWrite implementation just copies the event data into a buffer. A separate background thread is responsible for delivering the data to its destination (e.g. to write the buffer to disk or to invoke a callback on a real-time listener). As a consequence of this design, ETW provides best-effort delivery – if the buffer is full when your code calls EventWrite, ETW will drop the event instead of waiting for the buffer to be flushed. (ETW does keep track of whether any events have been dropped, so the log analysis tool can let you know if you are viewing an incomplete log.) Suitable for release builds: By default, the tracing does nothing and has essentially no impact on system behavior or performance. If needed (debugging, field diagnosis), you can run a tool (xperf, traceview, tracelog) to enable tracing at any time. The settings will take effect immediately (no need to restart your application). Even when tracing is enabled, the tracing APIs are non-blocking and have a relatively low impact on system performance. (Trace data is flushed to disk by a background thread; if data is logged too quickly for the disk to keep up, events will be dropped.) Separation of concerns: Event producer is responsible for generating good events. The producer should ensure that: Events are useful for diagnosis. (Don’t blindly annotate every function entry/exit – only put events in places where interesting things happen.) Events are properly annotated with severity level to allow for filtering. Events are properly annotated with keywords (aka categories, flags) to allow for filtering. Events include useful correct information, including correlation IDs. Event producer doesn’t need to worry about how the events will be captured. Event consumer decides: Where the log file should go. When the logging is started and stopped. Which producer(s) are included in the log. Can include more than 1! For each producer, what severity level of event should be included (e.g. whether the log is error-only, or includes warnings, or includes informational events, or includes verbose events). For each producer, what keywords (event categories) are included in the log. Can also enable other options, e.g. can collect stack traces for certain events.
- Demo: Start a GUI app that can generate ETW events. In this case, the app was written in C# and uses the EventSource class to generate ETW events. Start the traceview tool. https://msdn.microsoft.com/en-us/windows/hardware/drivers/devtest/traceview This tool has received several important improvements in the Windows 10 Creators Update. Be sure to use the latest version. Start a real-time trace that receives events from the app. To start a trace, we need at least the control GUID(s) of the event provider(s) from which we want to capture events. In the case of EventSource, the GUID is based on a hash of the provider’s name, so it will be the same as long as the provider name doesn’t change. You can access the GUID for your EventSource object by getting the EventSource.Guid property, or you can use a command-line tool to perform the same hash using the source code linked here: https://blogs.msdn.microsoft.com/dcook/2015/09/08/etw-provider-names-and-guids/ Note that we can adjust the trace level on-the-fly without restarting the app. Note that in addition to information that our app included, the events automatically include a timestamp, a process ID, and a thread ID. This kind of tracing works for all kinds of software: drivers, services, applications, and even Windows Store apps.
- Debugging: Sometimes your program doesn’t have an easy way to output debugging data. For example, you can’t easily access printf data from a system service or a web application. Sometimes you want to run through a long scenario and then analyze the log afterwards to determine when something went wrong. e.g. reference counting. Field diagnostics: Tracing present in production builds but disabled. If something is going wrong in production: Start a trace in the production environment (use tracelog or xperf). Reproduce the issue. Stop the trace. Copy the trace to the developer’s machine. Analyze. Flight recorder: Note that starting a trace that captures events from within the same process (in-process-private trace) requires no special permissions. Capturing events from other processes or from the kernel requires elevated permissions. Windows Framework class LoggingSession is designed to support the “flight recorder” scenario. It creates an in-memory circular buffer and captures events, then allows you to flush the buffer to disk on-demand. You might include data from such a trace in bug reports or crash reports. Log file: Let ETW handle the file management, timestamps, PID stamps, and filtering for your log file. Note that starting a trace that captures events from within the same process (in-process-private trace) requires no special permissions. Capturing events from other processes or from the kernel requires elevated permissions. Windows Framework class FileLoggingSession is designed to support the “log file” scenario. It creates an on-disk log, and rotates to a new file when the current file reaches a specified maximum size.
- #include <TraceLoggingProvider.h> The <TraceLoggingProvider.h> header was added in the Windows 10 SDK. However, it can be used for programs that run on Vista or later. TraceLoggingProvider.h is sensitive to WINVER – it will take advantage of a newer API (EventSetInformation) based on the value of WINVER. If your program needs to run on Vista, Windows 7 RTM, or on unpatched Windows 7 SP1, be sure that WINVER is set appropriately for your target OS version. For example, you could add the following before including <windows.h>: #define _WIN32_WINNT _WIN32_WINNT_VISTA The TraceLoggingProvider.h header contains lots of comments. Read them for details. Add a TRACELOGGING_DEFINE_PROVIDER macro to define the provider handle. (Use this macro in a .c or .cpp file.) To share the provider handle between multiple .c or .cpp files, add a TRACELOGGING_DECLARE_PROVIDER macro to a .h file. In your component’s initialization code, add TraceLoggingRegister. In your component’s shutdown code, add TraceLoggingUnregister. For a command-line program, this would probably be added at the start and end of main or wmain. For a GUI program this would probably be added in WinMain or equivalent. For a DLL, add TraceLoggingRegister in the PROCESS_ATTACH section of DllMain, and add TraceLoggingUnregister in the PROCESS_DETACH section of DllMain. For a driver, add TraceLoggingRegister in DriverEntry, and put TraceLoggingUnregister in DriverUnload. TraceLoggingRegister/TraceLoggingUnregister are not reference-counted. Calling TraceLoggingRegister twice in a row (without a call to TraceLoggingUnregister between them) will cause a leak and might lead to a crash. You should call Register once during startup, and call Unregister once during shutdown. TraceLoggingRegister/TraceLoggingUnregister are not thread-safe – it’s not safe to have two threads calling TraceLoggingRegister at the same time, to have two threads calling TraceLoggingUnregister at the same time. It is safe to call TraceLoggingUnregister even if TraceLoggingRegister returned an error. Calling Unregister on an unregistered handle is a no-op. For DLLs and drivers, TraceLoggingUnregister is NOT OPTIONAL. If TraceLoggingRegister is called, you MUST call TraceLoggingUnregister before the DLL or driver unloads. Otherwise, ETW might try to make a callback into your DLL/driver after it has unloaded, causing a crash. For programs (EXEs), it’s not a real problem to exit without calling TraceLoggingUnregister. The system will automatically clean up any handles that weren’t unregistered. Each time you want to write an event, use TraceLoggingWrite. It is safe to call TraceLoggingWrite even if TraceLoggingRegister returned an error. Calling TraceLoggingWrite with an unregistered provider handle is a no-op. TraceLoggingWrite is thread-safe. It is safe to have any number of threads calling TraceLoggingWrite at the same time.
- Demo: Simple C# program that performs 1000 iterations of InterlockedIncrement. Run program for benchmarking purposes. Without any tracing, each iteration seems to take about 8 nanoseconds. Add tracing to the program using the TraceLogging framework (System.Diagnostics.Tracing.EventSource). Add a static EventSource object that we use for logging. Call the Write method to log an event. The Write method requires an event name. The Write method optionally accepts event options (level, keywords). The Write method optionally accepts an event data parameter. It will write one event field for every public gettable property in the object. Run program for benchmarking purposes. When the events are not being recorded, the added tracing has a small impact on runtime. Each iteration is now about 500 nanoseconds. This is the time required to initialize the parameters and call EventSource.Write. The EventSource.Write function notices that tracing has been disabled and returns immediately. To improve performance, you can create a struct instead of using an anonymous class. To improve performance, you could use EventSource.IsEnabled to avoid initializing the parameters when the event is disabled. Use traceview to start a trace. Run the program again. When the events are being recorded, the tracing has a larger impact on runtime. Each iteration is now about 2000 nanoseconds.
- C/C++ TraceLoggingProvider.h: https://msdn.microsoft.com/en-us/library/windows/desktop/dn904637(v=vs.85).aspx Requires Windows 10 SDK. Can be used on Vista or later. .NET EventSource: https://msdn.microsoft.com/en-us/library/system.diagnostics.tracing.eventsource(v=vs.110).aspx Supported in Universal Windows Platform. TraceLogging support on desktop requires .NET 4.6. To use EventSource for TraceLogging on earlier versions of .NET, use the NuGet package “EventSource Redist”. Windows Framework LoggingChannel: https://docs.microsoft.com/en-us/uwp/api/windows.foundation.diagnostics.loggingchannel TraceLogging support requires Windows 10. Note that traceview and tracefmt received substantial improvements in the Windows 10 Creators Update SDK. Be sure to use the latest versions.
- It’s possible to use the EventRegister/EventUnregister/EventWrite APIs directly, but it’s much more common to use a framework. The frameworks take care of a lot under the covers: Set up a callback to get notifications for when providers are enabled, disabled, or adjusted. The framework uses these notifications to update global variables. Perform automatic filtering based on the event’s control GUID, level, and keywords. This filtering uses only global variables, so it only takes a few CPU cycles to skip a disabled event. Without this filtering, you have to call into the EventEnabled, EventProviderEnabled, or EventWrite API to determine whether the event is enabled. These APIs have to look up the provider handle in a handle table to determine whether the event is enabled, so they require several hundred CPU cycles to complete. Efficiently pack your data into the event payload. Help you track the event decoding information – the event’s name, the field names, the field types, etc. Provide a standard format for the information that the event decoder needs to decode the event.
- WPP core characteristics: Events authored directly in C/C++ code (printf-style messages). Event metadata encoded into PDB. (TMF file can be extracted from PDB.) WPP advantages: Events in the trace are small (on-disk, in-memory). Events authored directly in the source code (easy to edit). Supports named enum values (maps). Events include filenames and line numbers of the source code. WPP disadvantages: Only works for C/C++. Requires a preprocessor (tracewpp) to generate code. Uses the “Classic ETW” APIs, so a provider can send events to only 1 consumer at a time. (If another consumer asks for the events, the previous consumer stops receiving events.) Decoding requires finding the correct PDB or TMF file. Very difficult to keep track of the metadata unless the PDB is indexed on a symbol server. Manifest core characteristics: Events authored in an XML manifest. Event metadata is distributed via the XML manifest or as resources in a binary. Manifest advantages: Events in the trace are small (on-disk, in-memory). Events are all defined in one file, can be managed centrally (easy to review). Supports named enum values (maps). Supported in C/C++ (via mc), .NET (via EventSource and EventRegister). Event manifests can be backwards-compatible, i.e. the newest manifest can decode events from earlier versions of a program. Optional format string (localizable). Events can be used with Event Log. Binary containing decoding resources can be registered on the system to allow for decoding by the end user. (Required for interoperability with Event Log.) Uses the “Crimson ETW” APIs, so a provider can send events to up to 8 consumers at a time. Manifest disadvantages: XML manifest can be harder to maintain, less convenient to edit than source code. Requires a preprocessor (mc) to compile the XML manifest and generate code. (.NET EventSource manifests can work in limited scenarios without any special build tools, but if you need the decoding resources in the binary, you have to use EventRegister, mc, and special build steps to embed the resources.) Decoding requires finding the manifest or the correct binary with decoding resources. TraceLogging core characteristics: Events authored directly in C/C++ code. Event metadata is encoded into the event itself. TraceLogging advantages: No preprocessor or special build tool required. Events authored directly in the source code (easy to edit). Supported in C/C++ (TraceLoggingProvider.h), .NET (EventSource 4.6 or later), and Windows Framework (LoggingChannel). Decoding always works (only requires a TraceLogging-aware decoding tool; no metadata file required). TraceLogging disadvantages: Event is larger on disk – each event contains utf-8 strings for the provider name, event name, and field names. Does not support named enum values (maps). Enums are decoded as integers. Does not support format strings. Formatting typically shows each field in name-value pairs. Does not support event obfuscation – customers can decode events.
- Only manifest-based ETW is fully-supported for use with Event Log. For high-volume events, or when log size is a major concern, use manifests (or WPP). TraceLogging events are somewhat larger because each event contains a utf-8 encoded strings for the provider name, event name, and the field names. In most cases, this isn’t a significant concern, but for high-volume events, it can become an issue. Events can be somewhat “obfuscated” with manifests or WPP by not distributing the decoding information. This is not possible with TraceLogging because the decoding information is always present. TraceLogging is by far the easiest framework for getting started. Events are authored directly in code, no special tools (no code generator) are needed during the build process, and the metadata is managed automatically (no need to index PDBs, distribute manifests, or register binaries).