What is going on?
Application Diagnostics on Azure
Maarten Balliauw
@maartenballiauw

Who am I?
Maarten Balliauw
Antwerp, Belgium
Developer Advocate, JetBrains
Founder, MyGet
AZUG
Focus on web
ASP.NET MVC, Azure, SignalR, ...
Former MVP Azure & ASPInsider
Big passion: Azure
http://blog.maartenballiauw.be
@maartenballiauw

Agenda
Logging
Why?
Logging sucks!
The need for semantic/structured logging
Application Insights
SDK, Azure portal
Application Insights Analytics
Find the needle in the haystack!
Build, measure, improve

Logging

Why logging?
Troubleshooting – Did a problem occur? Where? Our code? The machine?
Performance / Cost – Did we make the application slower?
Improvement – Can a problem be detected or avoided?
Trends – Need more storage? Need more servers?
Customer Experience – Are people happy? Or seeing error after error?
Business Decisions – Can we increase sales based on log data / telemetry?

So here’s what we typically do...
System.Diagnostics.Trace.TraceInformation(
"Something happened");
System.Diagnostics.Trace.TraceWarning(
"Error! " + ex.Message);
Does this help troubleshooting? Improve the application? Analyze trends?

And here’s a typical log...
App.exe [12:13:03:985] Information: 0 : Customer address updated.
App.exe [12:13:04:011] Error: 8 : System.NullReferenceException occurred. Value
can not be null.
App.exe [12:13:04:567] Information: 0 : Machine policy value 'Debug' is 0
App.exe [12:13:04:569] Verbose: 9 : ******* RunEngine
******* Action:
******* CommandLine: **********
App.exe [12:13:04:578] Information: 9 : Entered CheckResult()
App.exe [12:13:04:689] Debug: 0 : Created. Req=0, Ret=8, Font: Req=null
Does this help troubleshooting? Improve the application? Analyze trends?

No.

Log files suck.
Stupid string data
Unless using ETW / proper System.Diagnostics listener
Typical log file has no “context”
Typical log file has no correlation
How to get data off our machines?
How to report/analyze?

Log files suck.
Stupid string data
Typical log file has no “context”
Process/thread
Machine name
User performing the action
Data specific to the event being logged
Typical log file has no correlation
How to get data off our machines?
How to report/analyze?

Log files suck.
Stupid string data
Typical log file has no “context”
Typical log file has no correlation, e.g.:
What was the server load when this was logged?
How much memory was consumed at the time?
Which web request invoked this action?
How to get data off our machines?
How to report/analyze?

Log files suck.
Stupid string data
Typical log file has no “context”
Typical log file has no correlation
How to get data off our machines?
How to report/analyze?

Log files suck.
Stupid string data
Typical log file has no “context”
Typical log file has no correlation
How to get data off our machines?
How to report/analyze?
Log processing and analysis – log, store, process, query, proactive analysis
Logstash, ElasticSearch, ... – cumbersome to setup

What if we had (better)
contextual information?

Structured/Semantic logging
“Of or relating to meaning, especially meaning in language”
– the dictionary
Windows Event Log
Event Tracing for Windows (ETW)
Semantic Logging Application Blog (PnP SLAB)
Serilog
…

Event Source
XML representation containing
contextual information
application-specific data elements
Can be read by humans (just a message, like any log)
Can be read by machines (XML, baby!)
add “meaning” – we know what the data points can be
allow automated aggregaton / trend analysis
Fast!
(not very developer friendly)

Using EventSource
DEMO

Event Source
Super awesome
It has structured logging and context
Can collect all kinds of data combined (e.g. HTTP + own event source)
Hard to maintain
Versioning
Ceremony
One method for every single thing we want to log (or at least an event id)
Requires thinking about the events to log
Lots of data

Serilog - https://serilog.net/
Logging library like any other, mostly
Sinks https://github.com/serilog/serilog/wiki/Provided-Sinks
Console, file, event log
Database
ElasticSearch
AppInsights
Structured data as a first-class concept
Enrichers, filters, …

Structured data in Serilog
Simple scalar values (bool, string, int, …)
Collections and dictionaries
Full objects
Rendered by a sink to, for example, a string
Can be stored by a sink to provide additional dimensions on data

Serilog - Basics
DEMO

Enrichment of data
Enrich log entries with additional context
Machine name
User name
Process / thread information
ASP.NET client IP / hostname
ASP.NET user agent
…
Using Serilog enrichers (when applicable)
In custom ASP.NET middleware

But still...
How to get the data off our machine(s)?
How to analyze/aggregate/predict?
Which customer had this Exception and why?
Did our sales go up after we made this page faster?
What button on the front-end crashes this microservice?
Are # Exceptions increasing?
Do I need special infrastructure to store and process logs?

The cloud to the rescue? (Azure)
Azure Insights
Azure platform’s core monitoring and alerting services (standard Metrics, Alerts,
Autoscale, Email Notifications and Audit Logs)
Application Insights
Application monitoring & logging for your applications (web, mobile, desktop, …) in
cloud or other server
Log Analytics (OMS)
Log Analytics, part of the Operations Management Suite (OMS) ingests data from
Azure servers and provides rich ‘global’ view & advanced log search based alerting
capabilities

Application Insights

Application Insights
Azure Service + library/SDK
Solves “where to store”, “how to ship”, “how to analyze”
Enriches data
Correlates data (e.g. client + server + dependency/DB/...)
Telemetry!
Allows structured logging
Allows rich querying, alerting
Works for many, many platforms and languages
Web, Windows, Xamarin, any application type really
.NET, Java, JavaScript, Objective-C, PHP, Python, Ruby, ...

Getting Started
DEMO

Do I have to run on Azure?
No. REST API to send data to
Various tools and SDK’s to collect data specific to language/platform
Status Monitor on non-Azure machines

Data collected
Performance Counters
Requests (both server/client side)
Traces (more later)
Exceptions
Dependencies
Custom Metrics & Events
That is a lot of data that can be correlated!

Different Views

Application Insights portal
Dashboard
Basic metrics (# Alerts, Response times, Exceptions, Failed Requests, ...)
Gateway to all of the below
Application map
Smart Detection / Alerts
Live Metrics Stream
Search, Availability, Failures, Performance, ...
Metrics (and events)
Analytics

Application Map
DEMO

Smart Detection / Alerts
DEMO

Live Metrics Stream
DEMO

Search, Availability,
Failures, Performance, ...
DEMO

Metrics and Events
Send custom events and metrics
Name of event/metric + a value
Events – help find how the application is used and can be optimized
User resized column in a grid
User logged in
User clicked “Share”
No more stock
...
Metrics – help alerting / finding missing functionality
Purchase occurred
Search returned zero results
# Support calls
...

Metrics and Events
DEMO

So what about logging...
Configure Serilog to use Application Insights as a sink
Custom data becomes available (*)
So we can correlate our own data with performance, requests, traces,
exceptions, depenencies, ...
But how to do this? And how to consume?
(*) Make sure the entire object is in the log message, no tracking of non-printed data

Logging
DEMO

Application Insights
Analytics

Application Insights Analytics
Former MS-internal tool “Kusto”
Near-realtime log ingestion and analysis
Lets you run custom queries
requests
| where timestamp > ago(24h)
| summarize count() by client_CountryOrRegion
| top 10 by count_
| render piechart

Queries over logs!
Input dataset
traces
customEvents
pageViews
requests
dependencies
exceptions
availabilityResults
customMetrics
performanceCounters
browserTimings
| Operators
where
count
project
join
limit
order
| Renderer
table
chart (bar, pie, time, area, scatter)
https://docs.microsoft.com/en-us/azure/application-insights/app-insights-analytics-reference

Application Insights
Analytics
DEMO

Not covered in this talk
VS plugin for Application Insights
Run some analysis inside VS
Continuous export
Export data from Application Insights continuously if you want to analyze in
other tools
Retain data longer than 7 days
Join with other data (preview)
https://docs.microsoft.com/en-us/azure/application-insights/app-insights-
analytics-using#import-data
Join logging and metrics with other data sources, query with AI Analytics

Conclusion

But... Why?
Seriously, why are you in this session...

But... Why?
Troubleshooting – Did a problem occur? Where? Our code? The machine?
Performance / Cost – Did we make the application slower?
Improvement – Can a problem be detected or avoided?
Trends – Need more storage? Need more servers?
Customer Experience – Are people happy? Or seeing error after error?
Business Decisions – Can we increase sales based on log data/telemetry?

Structured logging
Enrich technical logs with business data
Allows correlating when things go wrong
Allows making business decisions when using a good data analysis tool on top

Application Insights
Is that good analysis tool! (for web, mobile, desktop!)
Solves
Collecting various sources
Ingesting various sources
Enrichment with structured logging (e.g. Serilog)
Helps analyzing all that data
Default views, metrics, alerts, monitoring
Custom querying – Application Insights Analytics
Build, measure, improve!
Both technical and business – there isn’t always a divide between both...

Thank you!
Need training, coaching, mentoring, performance analysis?
Hire me! https://blog.maartenballiauw.be/hire-me.html
http://blog.maartenballiauw.be
@maartenballiauw