Session for Kommunity KTalks on 17-03-2021

1. Logging, tracing and metrics
Overview of instrumentation
in .NET 5 and Azure
Alex Thissen
@alexthissen
Cloud architect at Xpirit, The Netherlands

2. DevOps needs monitoring
https://docs.microsoft.com/en-us/azure/architecture/checklist/dev-ops#monitoring

3. Loops in DevOps practices

4. Application performance monitoring
Cloud applications are complex
Mostly distributed applications
Using multiple cloud resources
Collect and correlate
instrumentation data
Events, usage, metrics, performance, telemetry, logs, traces, health
Sentry.io
Raygun.io Runscope
NewRelic AlertSite DataDog
AppMetrics Azure Monitor

5. Cloud Application
Azure Resources
Azure Monitor
Azure Subscription
Azure Tenant
Azure Monitor Logs
Applications in Azure
Azure Monitor
Metrics

6. Collect Monitor
Application instrumentation
Logging
Tracing
Metrics
Health
Dashboards
Alerting
Analytics
Profiling
Metrics
streaming

7. Choosing correct type of instrumentation
Log
• What
happened?
• Errors and
warnings and
state changes
• Important
state changes
worth
registering
• Always logged
Trace
• How did it
happen?
• Circumstantial
data for
following flow
• Available on
request
• Publish/
subscribe
model mostly
• High volume
Metric
• How much is
happening?
• Numerical
information of
events
• Suitable for
aggregation
and trends
Health
• How is it
doing?
• Information
indicating
health status
• Internally
determined by
component
Audit
• Who made it
happen?
• Data
regarding
actions
performed by
someone
• Always
• Security and
identity
related
• Proof for non-
repudiability

8. Logging in .NET 5
Leveraging CoreFX libraries
for logging

9. Logging
Persisting important events in a log
Built-in since .NET Core 1.0
Used internally by CoreFX libraries
Bootstrapping changed from .NET Core 3.0 forward

10. Logging providers
Host.CreateDefaultBuilder(args)
.ConfigureLogging((context, builder) =>
{
// Log providers
builder.AddApplicationInsights(options =>
{
options.IncludeScopes = true;
options.TrackExceptionsAsExceptionTelemetry = true;
});
builder.AddConsole(options => {
options.Format = ConsoleLoggerFormat.Systemd;
});
builder.AddDebug();
builder.AddTraceSource(source.Switch,
new ApplicationInsightsTraceListener());
.NET 5 * Added by default
NullLoggerProvider
BatchingLoggerProvider
ConsoleLoggerProvider *
DebugLoggerProvider *
EventLogLoggerProvider
EventSourceLoggerProvider *
TraceSourceLoggerProvider
ASP
.NET Core
ApplicationInsightsLoggerProvider
AzureAppServicesFile
AzureAppServicesBlob
Third party
NLogLogger
Seq
Log4Net
Loggr

11. LoggerFactory and logger instances
LoggerFactory
LogCritical
LogError
LogWarning
LogDebug
LogTrace
LogInformation
ILogger<T>

12. Log severity levels and categories
Hierarchical structure
Much like namespaces in C#, separated by dots
Microsoft
Microsoft.Hosting.Lifetime
Determined from T in ILogger<T>
Except when creating ILogger directly with LoggerFactory.CreateLogger
public enum LogLevel
{
Trace = 0,
Debug = 1,
Information = 2,
Warning = 3,
Error = 4,
Critical = 5,
None = 6
}
ILogger<LeaderboardController>
namespace LeaderboardWebApi.Controllers {
public class LeaderboardController : Controller { … }
}
"LeaderboardWebApi.Controllers.LeaderboardController"

13. Separate configuration
for specific providers
General configuration
for all providers
Log filters
Filter log messages
By provider and per log category based on severity
Can read filters from configuration (appsettings.json)
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
},
"Console": {
"LogLevel": { … }
"IncludeScopes": true
}
}
ILogger<T>

14. Demo
Logging 101

15. Quantum physics applied to logging
Duality of log messages

16. Semantic logging (aka structured logging)
Message templates
Strings with named placeholders for semantics
Avoid string interpolation in messages
Popular log providers supporting semantic logging
// Placeholders in template become queryable custom data
logger.LogInformation("Searching with {SearchLimit} results.", limit);

17. Demo
Structured logging using Seq

18. Scopes
Groups a set of logical operations
Requires setting options to include scope for provider
logging.AddConsole(options => options.IncludeScopes = true);
using (logger.BeginScope("Message attached to logs created in the using block"))
{
logger.LogInformation(LoggingEvents.NewHighScore, “New high score {score}", highScore);
var game = gameRepository.Find(gameId);
if (game == null)
{
logger.LogWarning(LoggingEvents.GameNotFound, "GetById({game}) not found", gameId);
return NotFound();
}
}

19. Logging guidance
Separation of concerns
“A twelve-factor app never concerns itself with routing or storage of its output stream."
Choose your log severity level carefully
You might get paged during the night
Use event IDs when possible
Log messages are not for UI
No need to localize your messages
When in doubt, be generous on logging
Tooling will help you filter and aggregate

20. Alternative logging solutions
Serilog
Popular open source logging framework
Many log providers available
Replaces built-in logging system
Can also log to .NET log providers
More functionality
Log appenders (aka enrichers) for additional data
Large set of providers
Serilog

21. Tracing
Familiar concepts and API
from .NET Framework

22. Diagnostics Trace
High volume noise to follow flow
Not a log of things that happened
Publish/subscribe
Requires a listener to receive information
Information not persisted unless subscribed
System.Diagnostics namespace
Trace and TraceSource entrypoints
Needs compilation constant #TRACE
Uses activities under the cover

23. Tracing infrastructure
Trace
“Singleton” with static methods
Traditional tracing
TraceData
TraceEvent
TraceInformation
TraceTransfer
TraceError
TraceWarning
TraceInformation
Write(If)
WriteLine(If)
TraceSource

24. Activities
Ambient contextual data
Accessible from Activity.Current
Thread-safe and async aware
Used for correlating events
Even across network calls
New W3C tracing format
Application Insights uses activities
Parent/Child relationship
var activity = new Activity("SearchEngine.Run");
activity.SetStartTime(DateTime.Now);
activity.Start();
activity.Track...();

25. Available trace listeners
Namespace Class .NET version
System.Diagnostics DefaultTraceListener Core 1.0+
TextWriterTraceListener Core 1.0+
EventLogTraceListener Core 3.0
ConsoleTraceListener Core 3.0
DelimitedListTraceListener Core 1.0+
XmlWriterTraceListener Core 3.0
EventSchemaTraceListener .NET FX 3.5+
System.Diagnostics.Eventing EventProviderTraceListener .NET FX 3.5+
System.Web IisTraceListener .NET FX 2.0+
WebPageTraceListener .NET FX 2.0+
Microsoft.VisualBasic.Logging FileLogTraceListener .NET FX 2.0+
From .NET Framework 2.0 to .NET Core 3.0 to .NET 5

26. Demo
Tracing 101

27. Health checks
Indicating health status
from .NET 5

28. Health monitoring
services
.AddHealthChecks()
.AddCheck("sync", () => … )
.AddAsyncCheck("async", async () => … )
.AddCheck<SqlConnectionHealthCheck>("SQL")
.AddCheck<UrlHealthCheck>("URL");
ASP
.NET Core application
/health
DefaultHealthCheckService

29. Health check publishers
Pushes out health
info periodically
Options
ASP
.NET Core application
DefaultHealthCheckService
HealthCheckPublisher
HostedService
IEnumerable<IHealthCheckPublisher>
services.AddHealthChecks()
.AddApplicationInsightsPublisher()
.AddPrometheusGatewayPublisher(
"http://pushgateway:9091/metrics",
"pushgateway") IHealthCheckPublisher

30. Probing containers to check for availability and health
Readiness and liveness
Kubernetes node
Kubernetes node
Kubernetes nodes
Containers
Readiness
Liveliness

31. Implementing readiness and liveliness
1. Add health checks with tags
2. Register multiple endpoints
with filter using
Options predicate
/api/v1/…
/health
/health/ready
/health/lively
app.UseHealthChecks("/health/ready",
new HealthCheckOptions() {
Predicate = reg => reg.Tags.Contains("ready")
});
services.AddHealthChecks()
.AddCheck<CircuitBreakerHealthCheck>(
"circuitbreakers",
tags: new string[] { "ready" });
app.UseHealthChecks("/health/lively",
new HealthCheckOptions() {
Predicate = _ => true
});

32. Demo
Health checks and monitoring in ASP
.NET Core

33. Application Insights
Metrics, monitoring, querying
and analyzing your
application

34. DevOps and loops

35. Azure Application Insights
Extensible Application Performance Monitor

36. Application Insights in .NET 5
Logging and tracing
Regular logging provider
Special TraceListener
builder.AddApplicationInsights(options => {
options.TrackExceptionsAsExceptionTelemetry = true;
options.IncludeScopes = true;
});
Telemetry
TelemetryClient StartOperation TrackEvent
Trace.Listeners.Add(new ApplicationInsightsTraceListener());
services.AddApplicationInsightsTelemetry(options =>
{
options.DeveloperMode = true;
});

37. Demo
Application Insights

38. DevOps needs monitoring
https://docs.microsoft.com/en-us/azure/architecture/checklist/dev-ops#monitoring

39. Questions and
answers

40. Resources
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-5.0
https://docs.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks
https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks
https://dev.applicationinsights.io/apiexplorer/metrics
https://github.com/alexthissen/instrumentation