Conference: dotnetdays 2020 Location: Iaşi, Romania Abstract: Ever had production problems and found that you cannot debug to find the problem? Or that you need to find out where potential issues are coming from in your Azure cloud solution and you have no idea what is happening? Your applications need to be instrumented with logging, tracing and metrics, so you can see what is going on where. In .NET Core logging and tracing are built into the framework. We will have a look at the differences between logging, tracing and instrumentation in general. You will learn how to use .NET Core to implement logging and tracing with best practices, do semantic logging, work with logging factories, trace providers. Also, you will learn how to instrument using Application Insights and add W3C compliant tracing for correlation across cloud resources in a distributed application. Finally, we will put everything together to see how your logs and traces can give a rich way to get insights into your applications and services running in the Azure cloud or container based solutions.

2. DevOps needs monitoring
https://docs.microsoft.com/en-us/azure/architecture/checklist/dev-ops#monitoring

3. Loops in DevOps practices

4. Application performance monitoring
Cloud applications are complex
Collect and correlate
instrumentation data
Sentry.ioRaygun.io RunscopeNewRelic AlertSite DataDogAppMetrics Azure Monitor

5. Cloud Application
Azure Resources
Azure Monitor
Azure Subscription
Azure Tenant
Azure Monitor Logs
Applications in Azure
Azure Monitor
Metrics

6. Collect Monitor
Application instrumentation
Logging
Tracing
Metrics
Health
Dashboards
Alerting
Analytics
Profiling
Metrics
streaming

7. Choosing correct type of instrumentation
Log
• What
happened?
• Errors and
warnings and
state changes
• Important
state changes
worth
registering
• Always logged
Trace
• How did it
happen?
• Circumstantial
data for
following flow
• Available on
request
• Publish/
subscribe
model mostly
• High volume
Metric
• How much is
happening?
• Numerical
information of
events
• Suitable for
aggregation
and trends
Health
• How is it
doing?
• Information
indicating
health status
• Internally
determined by
component
Audit
• Who made it
happen?
• Data
regarding
actions
performed by
someone
• Always
• Security and
identity
related
• Proof for non-
repudiability

8. Logging in .NET Core
Leveraging CoreFX libraries
for logging

9. Logging
Persisting important events in a log
Built-in since .NET Core 1.0

10. Logging providers
Host.CreateDefaultBuilder(args)
.ConfigureLogging((context, builder) =>
{
// Log providers
builder.AddApplicationInsights(options =>
{
options.IncludeScopes = true;
options.TrackExceptionsAsExceptionTelemetry = true;
});
builder.AddConsole(options => {
options.Format = ConsoleLoggerFormat.Systemd;
});
builder.AddDebug();
builder.AddTraceSource(source.Switch,
new ApplicationInsightsTraceListener());
.NET Core * Added by default
NullLoggerProvider
BatchingLoggerProvider
ConsoleLoggerProvider *
DebugLoggerProvider *
EventLogLoggerProvider *
EventSourceLoggerProvider *
TraceSourceLoggerProvider *
ASP.NET Core
ApplicationInsightsLoggerProvider
AzureAppServicesFile
AzureAppServicesBlob
Third party
NLogLogger
Seq
Log4Net
Loggr

11. LoggerFactory and logger instances
LoggerFactory
LogCritical
LogError
LogWarning
LogDebug
LogTrace
LogInformation
ILogger<T>

12. Log severity levels and categories
Hierarchical structure
Microsoft
Microsoft.Hosting.Lifetime
Determined from T in ILogger<T>
public enum LogLevel
{
Trace = 0,
Debug = 1,
Information = 2,
Warning = 3,
Error = 4,
Critical = 5,
None = 6
}
ILogger<LeaderboardController>
namespace LeaderboardWebApi.Controllers {
public class LeaderboardController : Controller { … }
}
"LeaderboardWebApi.Controllers.LeaderboardController"

13. Separate configuration
for specific providers
General configuration
for all providers
Log filters
Filter log messages
appsettings.json
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
},
"Console": {
"LogLevel": { … }
"IncludeScopes": true
}
}
ILogger<T>

14. Demo
Logging 101

15. Quantum physics applied to logging
Duality of log messages

16. Semantic logging (aka structured logging)
Message templates
Popular log providers supporting semantic logging
// Placeholders in template become queryable custom data
logger.LogInformation("Searching with {SearchLimit} results.", limit);

17. Demo
Structured logging using Seq

18. Scopes
Groups a set of logical operations
Requires setting options to include scope for provider
logging.AddConsole(options => options.IncludeScopes = true);
using (logger.BeginScope("Message attached to logs created in the using block"))
{
logger.LogInformation(LoggingEvents.NewHighScore, “New high score {score}", highScore);
var game = gameRepository.Find(gameId);
if (game == null)
{
logger.LogWarning(LoggingEvents.GameNotFound, "GetById({game}) not found", gameId);
return NotFound();
}
}

19. Logging guidance
Separation of concerns
Choose your log severity level carefully
Use event IDs when possible
Log messages are not for UI
When in doubt, be generous on logging

20. Alternative logging solutions
Serilog
Replaces built-in logging system
More functionality
Serilog

21. Tracing
Familiar concepts and API
from .NET Framework

22. Diagnostics Trace
High volume noise to follow flow
Publish/subscribe
System.Diagnostics namespace
Trace and TraceSource entrypoints
Uses activities under the cover

23. Tracing infrastructure
Trace
TraceData
TraceEvent
TraceInformation
TraceTransfer
TraceError
TraceWarning
TraceInformation
Write(If)
WriteLine(If)
TraceSource

24. Activities
Ambient contextual data
Activity.Current
Used for correlating events
Parent/Child relationship
var activity = new Activity("SearchEngine.Run");
activity.SetStartTime(DateTime.Now);
activity.Start();
activity.Track...();

25. Available trace listeners
Namespace Class .NET version
System.Diagnostics DefaultTraceListener Core 1.0+
TextWriterTraceListener Core 1.0+
EventLogTraceListener Core 3.0
ConsoleTraceListener Core 3.0
DelimitedListTraceListener Core 1.0+
XmlWriterTraceListener Core 3.0
EventSchemaTraceListener .NET FX 3.5+
System.Diagnostics.Eventing EventProviderTraceListener .NET FX 3.5+
System.Web IisTraceListener .NET FX 2.0+
WebPageTraceListener .NET FX 2.0+
Microsoft.VisualBasic.Logging FileLogTraceListener .NET FX 2.0+
From .NET Framework 2.0 to .NET Core 3.0

26. Demo
Tracing 101

27. Health checks
Indicating health status
from .NET Core

28. Health monitoring
services
.AddHealthChecks()
.AddCheck("sync", () => … )
.AddAsyncCheck("async", async () => … )
.AddCheck<SqlConnectionHealthCheck>("SQL")
.AddCheck<UrlHealthCheck>("URL");
ASP.NET Core application
/health
DefaultHealthCheckService

29. Health check publishers
Pushes out health
info periodically
Options
ASP.NET Core application
DefaultHealthCheckService
HealthCheckPublisher
HostedService
IEnumerable<IHealthCheckPublisher>
services.AddHealthChecks()
.AddApplicationInsightsPublisher()
.AddPrometheusGatewayPublisher(
"http://pushgateway:9091/metrics",
"pushgateway") IHealthCheckPublisher

30. Probing containers to check for availability and health
Readiness and liveness
Kubernetes node
Kubernetes node
Kubernetes nodes
Containers
Readiness
Liveliness

31. Implementing readiness and liveliness
1. Add health checks with tags
2. Register multiple endpoints
with filter using
Options predicate
/api/v1/…
/health
/health/ready
/health/lively
app.UseHealthChecks("/health/ready",
new HealthCheckOptions() {
Predicate = reg => reg.Tags.Contains("ready")
});
services.AddHealthChecks()
.AddCheck<CircuitBreakerHealthCheck>(
"circuitbreakers",
tags: new string[] { "ready" });
app.UseHealthChecks("/health/lively",
new HealthCheckOptions() {
Predicate = _ => true
});

32. Demo
Health checks and monitoring in .NET Core

33. Application Insights
Metrics, monitoring, querying
and analyzing your
application

34. DevOps and loops

35. Azure Application Insights
Extensible Application Performance Monitor

36. Application Insights in .NET Core
Logging and tracing
builder.AddApplicationInsights(options => {
options.TrackExceptionsAsExceptionTelemetry = true;
options.IncludeScopes = true;
});
Telemetry
TelemetryClient StartOperation TrackEvent
Trace.Listeners.Add(new ApplicationInsightsTraceListener());
services.AddApplicationInsightsTelemetry(options =>
{
options.DeveloperMode = true;
});

37. Demo

38. DevOps needs monitoring
https://docs.microsoft.com/en-us/azure/architecture/checklist/dev-ops#monitoring

39. Questions and Answers

40. Resources
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-3.1
https://docs.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks
https://dev.applicationinsights.io/apiexplorer/metrics