Custom Distributed Tracing in Azure Functions (2021-02-27)
•
0 likes•472 views
Presented at the Global Integration Bootcamp 2021 As developers, we tend to focus on building and shipping our services and apps to production, but it’s quite common that we forget to think about what happens after go-live. Once we reach production, the solution becomes someone else’s problem. But, even if we could build bug-free services, distributed systems will fail. And if we don’t design and build our services with traceability and observability in mind, we won’t give the means to the operations team to troubleshoot problems when they arise. While Azure Functions provide very rich telemetry with Application Insights; in this session we will see how to implement custom tracing and some observability practices in Azure Functions adding business related metadata, leveraging the structured logging capabilities, and going beyond the out-of-the-box features. In the session, I will demo a distributed tracing approach for Azure Functions that works well in integration solutions following the Publish-Subscribe integration pattern.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Custom Distributed Tracing in Azure Functions (2021-02-27)
Similar to Custom Distributed Tracing in Azure Functions (2021-02-27) (20)
More from Paco de la Cruz
More from Paco de la Cruz (12)
Recently uploaded
Recently uploaded (20)
Custom Distributed Tracing in Azure Functions (2021-02-27)
- 1. @pacodelacruz Paco de la Cruz @ Melbourne AU Custom Distributed Tracing & Observability Practices in Azure Functions Global Integration Bootcamp 2021 @pacodelacruz
- 3. @pacodelacruz Introduction Scenario Common operation requirements Tracing design: spans, key-value pairs Solution design: sequence diagram, components Demo Q & A Agenda
- 4. @pacodelacruz Devs tend to focus on building and shipping services The solution in production becomes someone else’s problem Distributed systems will fail We need to give Ops teams the means to troubleshooting Particularly relevant with backend services running in the background Introduction – Why are we talking about this?
- 5. @pacodelacruz Scenario - Participants Webhook pushing user update events (batch) Publisher Azure Function Subscriber Azure Function Service Bus Queue Target System Publish–Subscribe solution of user update events from a HR system via a webhook (batch of events) to a bespoke target system that can’t receive webhook calls Integration layer with temporal decoupling via a queue
- 6. @pacodelacruz Opinionated approach Based on many years of experience Publish-Subscribe pattern Splitter pattern Approach could be tailored Scenario - Notes
- 7. @pacodelacruz Scenario – Sequence Diagram @pacodelacruz
- 8. @pacodelacruz Scenario – Sequence Diagram @pacodelacruz
- 9. @pacodelacruz Scenario – Sequence Diagram @pacodelacruz
- 10. @pacodelacruz Common Operations Requirements Requirement Potential Solution Ability to inspect incoming request payloads for troubleshooting purposes. Archive all request payloads as received. Ability to correlate individual messages to the original batch and understand whether all messages in a batch were processed successfully. Capture a batch record count and a batch identifier in all tracing events for individual messages. Ability to correlate all tracing events for every individual message. Capture a tracing correlation identifier in all tracing events for every message. Being able to find tracing events for a particular message. Capture business-related metadata in all relevant tracing logs, including an interface identifier, message type, and an entity identifier. Receive alerts when certain failures occur. Capture granular tracing event identifiers and tracing event statuses. Ability to differentiate transient failures. Capture a delivery count when messages are being retried to identify whether additional attempts are expected. Ability to query, filter, and troubleshoot what happens to individual messages. Enable correlation of tracing events with standard tracing checkpoints, granular event identifiers, status, and business-related metadata.
- 11. @pacodelacruz Tracing Spans “The span is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. Each component of the distributed system contributes a span - a named, timed operation representing a piece of the workflow.” - opentracing.io
- 12. @pacodelacruz Structured Logging Key-Value Pairs Key Description Scope BatchId Batch identifier to correlate individual messages to the original batch. It is highly recommended when using the splitter pattern. Cross-span CorrelationId Tracing correlation identifier of an individual message. Cross-span EntityType Business identifier of the message type being processed. This allows to filter or query tracing events for a particular entity type. E.g. UserEvent, PurchaseOrder, Invoice, etc. Cross-span EntityId Business identifier of the entity in the message. This together with the EntityType key-value pair allow to filter or query tracing events for messages related to a particular entity. E.g. UserId, PurchaseOrderNumber, InvoiceNumber, etc. Cross-span InterfaceId Business identifier of the interface. This allows to filter or query tracing events for a particular interface. Useful when an organisation defines identifiers for their integration interfaces. Span RecordCount Optional. Only applicable to batch events. Captures the number of individual messages or records that are present in the batch. Span DeliveryCount Optional. Only applicable to subscriber events of individual messages. Captures the number of times the message has been attempted to be delivered. It relies on the Service Bus message DeliveryCount property. Span LogLevel LogLevel as defined by Microsoft.Extensions.Logging Log SpanCheckpoint Defines the tracing span and whether it is the start or finish of it, e.g. PublisherStart or PublisherFinish. Having standard checkpoints allows correlating tracing events in a standard way. Log EventId Captures a specific tracing event that helps to query, analyse, and troubleshoot the solution with granularity. Log Status Stores the status of the tracing event, e.g., succeeded or failed Log
- 13. @pacodelacruz Solution Components Application Insights Tracing Logs Blob Storage Archive Webhook pushing user update events (batch) Publisher Azure Function Subscriber Azure Function Service Bus Queue Target System
- 25. @pacodelacruz Code walkthrough Components overview Call the publisher function Monitor the solution Query the traces Demo github.com/pacodelacruz/ observability-pubsub-functions
- 26. @pacodelacruz Costs associated with Application Insights (ingestion & retention) Tracing data can be client-side sampled or filtered Traces can be server-side sampled or throttled (max req/s & daily cap) Telemetry channel is async & best-efforts (Traces could be lost: app crashes, network issues, etc.) This approach must not be used for auditing or billing purposes Considerations
- 28. @pacodelacruz Additional Resources Intro https://platform.deloitte.com.au/articles/ distributed-tracing-and-observability-with-azure-functions-1-intro Design https://platform.deloitte.com.au/articles/ distributed-tracing-and-observability-with-azure-functions-2-design Implementation https://platform.deloitte.com.au/articles/ distributed-tracing-and-observability-with-azure-functions-3- implementation Repo github.com/pacodelacruz/observability-pubsub-functions