ETW - Monitor Anything, Anytime, Anywhere (Velocity NYC 2017)

Velocity NYC 2017 #VelocityConf | @dinagozil
ETW - Monitor Anything,
Anytime, Anywhere
Dina Goldshtein,
1

Agenda
• Traditional performance tooling
• Introduction to ETW
• Usage scenarios
• ETW tools
2

Introduction
3

Challenges with Traditional Profilers
Invasive
•Recompilation
•Performance overhead
http://imgur.com/gallery/qV4hPqk
4

Live monitoring
•Restart the stock server?
•Restart the aerial radar?
https://commons.wikimedia.org/wiki/File:Ekg_bigeminus_bionerd.jpg
5

Licensing
•Redistribution
•Per-machine license
https://pics.onsizzle.com/california-driver-license-di-mreinsteinthecat-class-c-exp-06-29-2020-end-2949841.png
6

Event Tracing for Windows
• High-speed logging framework supporting more than 100K structured
messages per second
7

messages per second
• Does not require recompilation
• Can be turned on on-demand while running
8

messages per second
• Very small overhead
9

messages per second
• Very small overhead
• User-mode and kernel-mode
• .NET, drivers, services, third party components
10

Architecture Overview
• Generate ETW eventsProviders
• Start and stop ETW collectionControllers
• Log, analyze, or process eventsConsumers
11

Controller
Producer Consumer
Event Session
Buffer Pool
Log files
Events
Events
Events in Real-Time
Logged Events
12

It’s Everywhere
13

Let the Monitoring Begin
14

PerfView – Your ETW Aid
• Completely free from Microsoft’s GitHub
• Allows fine-grained configuration of ETW collection
• Light-weight (but not very visually appealing…)
• Built-in analysis and statistical views
15

DEMO
Profile Visual Studio startup
16

Collect Data
17

CPU Hot Paths
18

Let’s Drill Down
19

JIT
20

DEMO
GC gone mad
21

No Leak but GC-ing Like Crazy
22

PerfView Is Your Friend
23

Evidence Start Piling Up
24

What is Allocated?
25

Where Is It Allocated?
26

(CPU) (Self-)Profiling
• Use a programmatic API to detect high-CPU and record a trace log
• Microsoft TraceEvent Library on NuGet
• Can be used from external process or from inside the process
• Can use with any limited resource, not only CPU
27

Collecting Data
28

Collecting Data
using (var s = new TraceEventSession(name, file)) {
using (var ks = new TraceEventSession("NT Kernel Logger", kFile)) {
}
}
A separate session for the
kernel data is required
29

Collecting Data
ks.EnableKernelProvider(kernelKeywords,
KernelTraceEventParser.Keywords.Profile);
s.EnableProvider(ClrTraceEventParser.ProviderGuid,
TraceEventLevel.Verbose,
(ulong)(ClrTraceEventParser.Keywords.Default));
}
}
Specify the data we’re
interested in
30

Collecting Data
m_collectionToken.WaitHandle.WaitOne();
}
}
Session is closed on Dispose,
so wait for something to stop
tracing…
31

Collecting Data
m_collectionToken.WaitHandle.WaitOne();
generateJitRundownEvents(file, name);
}
}
Generate JIT info to be able
to analyze .NET stacks
32

Managed Stacks
private void generateJitRundownEvents(string file, string name) {
using (var session = new TraceEventSession(name + "Rundown", file)) {
session.EnableProvider(ClrRundownTraceEventParser.ProviderGuid,
(ulong)(ClrRundownTraceEventParser.Keywords.Default));
}
}
The ClrRundown provider
holds all needed information
33

Managed Stacks
private void generateJitRundownEvents(string file, string name) {
using (var session = new TraceEventSession(name + "Rundown", file)) {
session.EnableProvider(ClrRundownTraceEventParser.ProviderGuid,
(ulong)(ClrRundownTraceEventParser.Keywords.Default));
// Poll until 2 second goes by without growth.
for (var prevLength = new FileInfo(file).Length; ;) {
Thread.Sleep(TimeSpan.FromSeconds(2));
var newLength = new FileInfo(file).Length;
if (newLength == prevLength) break;
prevLength = newLength;
}
}
}
This means the file stopped
writing
34

DEMO
Self-profile high CPU
35

Self-Profiling Application
36

CPU Hot Paths
37

EventParsers
https://github.com/Microsoft/dotnetsamples/blob/master/Microsoft.Diagnostics.Tracing/TraceEvent/docs/TraceEvent.md
38

Real-Time Analysis with TraceEvent
39

// create a real time user mode session
using (var session = new TraceEventSession("ObserveProcs")) {
// Set up Ctrl-C to stop the session
Console.CancelKeyPress += (s, args) => session.Stop();
}
Create our own session
Stop collecting on ^C
40

// Turn on the process events (includes starts and stops).
session.EnableKernelProvider(KernelTraceEventParser.Keywords.Process);
}
We’re interested in data
about processes
41

// Subscribe to a callback that prints the information we wish
session.Source.Kernel.ProcessStart += data => {
Console.WriteLine("Process {0} Command Line {1}",
data.ProcessName, data.CommandLine);
};
}
Register on the event and
analyze
42

// Subscribe to a callback that prints the information we wish
session.Source.Kernel.ProcessStart += data => {
Console.WriteLine("Process {0} Command Line {1}",
data.ProcessName, data.CommandLine);
};
session.Source.Process(); // Listen (forever) for events
}
Start and process forever,
until stopped
43

Monitoring a Production System
• TraceEvent allows us to write monitoring systems which can be used
ad-hoc without recompilation or rebooting
44

Monitoring a Production System
• TraceEvent allows us to write monitoring systems which can be used
ad-hoc without recompilation or rebooting
• Someone has already done the job for us
• etrace – a free command-line tool for ETW live tracing
• By my dear husband Sasha
45

DEMO
Boot performance with WPR and WPA
46

Windows Performance Recorder
47

48

49

Something Is Taking a Long Time
50

What’s Running During Boot?
51

Drill Down with Good Old PerfView
52

Drill Down with Good Old PerfView
53

Summary
Omnipresent Performant
Free
54

Thank You!
Dina Goldshtein,
@dinagozil
dinazil@gmail.com
55

ETW - Monitor Anything, Anytime, Anywhere (Velocity NYC 2017)

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to ETW - Monitor Anything, Anytime, Anywhere (Velocity NYC 2017)

Similar to ETW - Monitor Anything, Anytime, Anywhere (Velocity NYC 2017) (20)

More from Dina Goldshtein

More from Dina Goldshtein (16)

Recently uploaded

Recently uploaded (20)

ETW - Monitor Anything, Anytime, Anywhere (Velocity NYC 2017)