The road to zen winds through self-monitoring applications, which can reduce the time and effort in diagnosing and correcting production issues. In this talk, we will see how modern Windows applications can self-monitor, self-diagnose, and potentially self-recover without needing an external monitoring agent or a brute-force restarting watchdog. By harnessing the power of ETW for low-level accurate monitoring, Windows performance counters for zero-overhead statistics, and the CLRMD library for inspecting your own threads, heap objects, and locks, you can take your applications one step closer to self-awareness. This will be illustrated through a series of demos: automatic CPU profiling and pinpointing the busy threads and stacks; automatic GC monitoring, including object allocations; automatic heap analysis to reveal unraveling memory leaks; and more. At the end of the talk, you will be equipped with tools and techniques for implementing self-monitoring in your own applications.

3. • Motivation
• Hierarchy of self-monitoring
• CPU profiling
• GC monitoring
• Heap analysis
• Deadlock detection

4. • Why monitor?

5. • Why monitor?

6. • Why monitor?
• Why our own?

7. • Why monitor?
• Why our own?

8. • Why monitor?
• Why our own?

9. • Why monitor?
• Why our own?

10. • Use a hierarchical monitoring strategy

11. • Use a hierarchical monitoring strategy
• Lightweight for continuous and frequent monitoring
of all basics
Performance counters

12. • Use a hierarchical monitoring strategy
• Lightweight for continuous and frequent monitoring
of all basics
Performance counters
• Medium for less frequency events
ETW ClrMD

13. • Use a hierarchical monitoring strategy
• Lightweight for continuous and frequent monitoring
of all basics
Performance counters
• Medium for less frequency events
ETW ClrMD
• Invasive for deep-dive and concrete diagnostics
CLR Profiling API CLR Debugging API

14. Let’s Get to Business

15. • Monitor CPU using performance counters
• Are we above a certain threshold for a certain
amount of time?
LiveStacks

16. • AuthenticationController takes 95% CPU,
maybe we're being DDoS'ed
• Image processing component takes 100% CPU, need
to auto-scale the app
• Encoding this 30 second video takes 3 minutes at
100% CPU, tell the user she can send us a bug report

17. CPU Spikes
Demo

18. • Monitor GC performance using performance
counters
• Register on ETW’s GC events such as
GCAllocationTick
• Attach ClrMD to get heap breakdown

19. Allocation
Spikes
Demo

20. • Monitor memory usage using performance counters
• Has memory increased above a certain threshold for
a certain time?

21. Find a Leak
Demo

22. • Monitor for potential deadlock
• Attach ClrMD to create wait chains and detect
deadlocks
• Report, try to break, pray for a miracle…

23. Detect a
Deadlock
Demo

24. • Many more scenarios are possible

25. • Many more scenarios are possible
• Monitor heap fragmentation and compact large
objects if needed

26. • Many more scenarios are possible
• Monitor heap fragmentation and compact large
objects if needed
• Native memory leak analysis using ETW

27. • The pros are obvious (visibility, easy scaling…)

28. • The pros are obvious (visibility, easy scaling…)
• But there are some cons as well…

29. • The pros are obvious (visibility, easy scaling…)
• But there are some cons as well…

30. • The pros are obvious (visibility, easy scaling…)
• But there are some cons as well…

31. • The pros are obvious (visibility, easy scaling…)
• But there are some cons as well…

32. • Self-monitoring is important for all kinds of software
• Best to create a hierarchy of monitoring (and
overhead and complexity)
• Lots of scenarios: CPU, GC, memory, deadlocks
• Demos: https://github.com/dinazil/self-aware-
applications