Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

I dont feel so well. Integrating health checks in your .NET Core solutions

186 views

Published on

Do you have any idea how your ASP.NET Web Apps and APIs are functioning? Are they behaving healthily or in a degraded state? You might be able to tell from log information and telemetry data, but why not have them tell you how healthy they are themselves? ASP.NET Core 2.2 introduces health endpoints that let your apps and APIs do just that. In this session you will learn how to make health checks an integral part of your solution. We will cover various types of health checks ranging from internal status, such as memory thresholds, to health based on external dependencies, such as databases and HTTP endpoints. Finally, you are going to see how this all can be used in a container cluster to allow the orchestrator to check for liveliness and readiness based on your health endpoints.

  • Be the first to comment

  • Be the first to like this

I dont feel so well. Integrating health checks in your .NET Core solutions

  1. 1. Keeping entire system running Determine state of entire system and intervene How to know health status of individual services? Collecting/correlating performance and health data Sentry.ioRaygun.io RunscopeNewRelic AlertSite DataDogAppMetrics Azure Monitor
  2. 2. Centralized Challenging Doctor, am I sick? 12:34 Let's look at your vitals we measured earlier: - Pulse 60 per minute - Blood pressure 130/80 12:34 Looks fine to me. It seems you are healthy. 12:35 Thanks, doctor! 12:36 Let's look at your vitals we measured: - Pulse 180 per minute - Blood pressure 150/110 12:34 Does not look good. It seems you are unhealthy. 12:35
  3. 3. Modern medicine and health Self-assessment Context matters You know best Let's see. My vitals say: - Pulse 180 per minute - Blood pressure 150/110 12:34 How are you doing today? 12:36 Does not look good. It seems I am unhealthy. 12:34 Oops, we need to do something about that! 3:24 Good to know. Stay healthy! 12:36 It's okay, as I am working out now My back does not hurt. So, I'm healthy! 12:34
  4. 4. Difference between metrics and health info Metrics Many individual measured values and counts of events Watch performance and trends Useful for diagnostics and troubleshooting Logic external to origin Health Intrinsic knowledge of implementation required DevOps mindset: Logic to determine health is part of origin Deployed together, good for autonomy
  5. 5. Availability Latency Internals Simple Advanced External dependencies Readiness & liveliness Preventive Predicting Examples
  6. 6. Healthy • 200 OK • "Everything is fine" Degraded • 200 OK • "Could be doing better or about to become unhealthy" Unhealthy • 503 Service Unavailable • "Not able to perform"
  7. 7. ASP.NET Core application /api/v1/… Middle ware New in .NET Core 2.2 Bootstrap health checks in ASP.NET Core app services.AddHealthChecks(); endpoints.MapHealthChecks("/health); /health DefaultHealthCheckService Microsoft.Extensions.Diagnostics.HealthChecks .Abstractions .EntityFramework Microsoft.AspNetCore.Diagnostics.HealthChecks
  8. 8. What? When? How? When a service is unhealthy, how can you trust its health status?public interface IHealthCheck { Task<HealthCheckResult> CheckHealthAsync( HealthCheckContext context, CancellationToken cancellationToken = default); } From: https://github.com/aspnet/Diagnostics/blob/master/src/ Microsoft.Extensions.Diagnostics.HealthChecks.Abstractions/HealthReport.cs
  9. 9. services .AddHealthChecks() .AddCheck("sync", () => … ) .AddAsyncCheck("async", async () => … ) .AddCheck<SqlConnectionHealthCheck>("SQL") .AddCheck<UrlHealthCheck>("URL"); ASP.NET Core application /health DefaultHealthCheckService
  10. 10. Demo ASP.NET Core 2.2 Health object model Health checks Endpoints
  11. 11. Only 1 out-of-box check DbContext Build your own IHealthCheck Community packages Xabaril/BeatPulse Yours?AspNetCore.Diagnostics.HealthChecks.* Microsoft.Extensions.Diagnostics. HealthChecks.EntityFrameworkCore services.AddHealthChecks() .AddDbContextCheck<GamingDbContext>("EF")
  12. 12. Register multiple health endpoints Middleware options Register custom health check as singleton /api/v1/… /health /ping services.AddSingleton<KafkaHealthCheck>()); services.AddSingleton(new SqlConnectionHealthCheck( new SqlConnection(Configuration.GetConnectionString("TestDB"))));
  13. 13. 1. Customize health endpoint output for more details 2. Query endpoint(s) 3. Build user interface
  14. 14. Demo A bit more advanced
  15. 15. Endpoints Frequency Locations Alerts
  16. 16. Pushes out health info periodically Options ASP.NET Core application DefaultHealthCheckService HealthCheckPublisher HostedService IEnumerable<IHealthCheckPublisher> services.AddHealthChecks() .AddApplicationInsightsPublisher() .AddPrometheusGatewayPublisher( "http://pushgateway:9091/metrics", "pushgateway") IHealthCheckPublisher
  17. 17. Registering a HealthCheckPublisher pre-.NET Core 3.0 // The following workaround permits adding an IHealthCheckPublisher // instance to the service container when one or more other hosted // services have already been added to the app. This workaround // won't be required with the release of ASP.NET Core 3.0. For more // information, see: https://github.com/aspnet/Extensions/issues/639. services.TryAddEnumerable( ServiceDescriptor.Singleton(typeof(IHostedService), typeof(HealthCheckPublisherOptions).Assembly .GetType(HealthCheckServiceAssembly)));
  18. 18. PrometheusGrafana .NET Core app Push gateway HealthCheckPublisher HostedServiceNotification channel  Slack  Telegram  VictorOps  PagerDuty  …
  19. 19. Demo Publishers Prometheus and Grafana
  20. 20. Performance MonitoringAvailability Resiliency
  21. 21. Probing containers to check for availability and health Kubernetes node Kubernetes node Kubernetes nodes Containers Readiness Liveliness
  22. 22. 1. Add health checks with tags 2. Register multiple endpoints with filter using Options predicate /api/v1/… /health /health/ready /health/lively app.UseHealthChecks("/health/ready", new HealthCheckOptions() { Predicate = reg => reg.Tags.Contains("ready") }); services.AddHealthChecks() .AddCheck<CircuitBreakerHealthCheck>( "circuitbreakers", tags: new string[] { "ready" }); app.UseHealthChecks("/health/lively", new HealthCheckOptions() { Predicate = _ => true });
  23. 23. Original pods only taken offline after new healthy one is up Allows roll forward upgrades: Never roll back to previous version
  24. 24. Demo Readiness and liveness probes Docker containers Kubernetes
  25. 25. Expose as little detail as possible Use different internal port Add authentication using middleware Publish instead of endpoint app.UseWhen( ctx => ctx.User.Identity.IsAuthenticated, a => a.UseHealthChecks("/securehealth") );
  26. 26. 1. Assume degraded state 2. Set short timeouts on checks 3. Avoid complicated health checks 4. Register health check as singletons in DI
  27. 27. /api/v1/… /health /ping
  28. 28. Questions and Answers
  29. 29. Resources https://docs.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks https://github.com/aspnet/Diagnostics/tree/master/src https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks https://github.com/alexthissen/healthmonitoring

×