The Durable Functions extension to Azure Functions allows developers to build workflows using higher level abstractions. But what is really going on under the hood? It can be useful to know how all the pieces connect in case you are every trying to solve a trickier issue.
In this presentation, we will take a deep dive into the internals of the Durable Functions extension and the Durable Task framework. In addition to the default Azure Storage durability provider, we will look at the other officially supported providers and how they differ in their implementations.
3. Key takeaways from this presentation
• Orchestrator and activity communication
• State storage using the Azure Storage provider
• Differences in other durability providers
• Goal: improve your ability to debug Durable Functions
4. Introduction/refresher
• Durable Functions = extension for Azure Functions that allows
execution of Durable Task orchestrations
• Durable Task framework allows developers to build long-running
workflows with persistent state, using well known async-await
constructs in .NET
• Orchestrator functions define the workflow, activity functions do
non-deterministic work
5. Durable Task workers
• Workers execute orchestrators and activities by requesting work
items from the durability provider
• Multiple workers run in parallel = increased throughput
• Activity work items can run on any worker, but orchestrator work
items cannot
• Danger of state corruption if more than 1 worker takes a work item for
same orchestration simultaneously
7. Storage durability provider
• Convenient choice for Azure Functions as it has a Storage account
already
• Cheap option, consumption-based pricing in Storage
• Tip: using a v1 Storage account is cheaper (not in new regions)
• Can run emulator locally for Storage
9. Orchestrator and activity communication
• Orchestrator gets executed several times; each time a set of
outbound messages is generated for activities to be triggered
• These are sent through the durability provider and workers
receive them
• Once the worker is done with the activity, it sends a message back
to the orchestrator
12. Durable Entities
• We looked at orchestrators and activities but where are Durable
Entities?
• Entities are just orchestrators
• At high level, you can think of entities as an orchestrator that:
1. Gets state from input
2. Waits for an external event (operation)
3. Computes new state based on event
4. Restarts itself with new state as input
13. Other things provided by Durable Functions
extension
• Durable HTTP
• [Deterministic]
• Replay-safe logger
15. Netherite (public preview)
• Higher performance + higher cost
• In high throughput scenarios, cheaper than high scale of other providers
• Supported by Durable Functions
• Not Consumption plan though
• Uses Event Hubs for orchestrator/activity messaging
• Even though Event Hubs do persist data, it is replicated to Azure Storage
after receive
• Max 32 partitions + 20 throughput units = 20 MB/s
• Uses Azure Storage blobs for storing partition state
• Harder to inspect current state
17. SQL (public preview)
• Uses SQL tables for everything
• Supported by Durable Functions
• Not Consumption plan though
• Stored procedures contain most logic
• Portable, no Azure connection required
• Can do data encryption, backups etc.
• Multi-tenant scenarios with schema per tenant
20. Service Bus (+Storage)
• Mature and transactionally consistent
• Not supported by Durable Functions
• Uses 3 Service Bus queues
• Orchestrator = messages for orchestrators
• Worker = messages for activities
• Tracking = orchestration state tracking
• Requires an instance store + blob store, comes with Storage
implementations for them
• State and history tracking
• Oversized messages and sessions
21. Redis
• Uses a Redis database
• Not supported by Durable Functions
• Workers notified of new messages through Redis channels
• Actual messages sent through Redis lists
• Portable since you can run Redis pretty much anywhere
22. Service Fabric
• Supported only within a Service Fabric cluster
• Uses SF reliable collections for state storage
• Some features limited, e.g. cannot query instance state if it has
completed over an hour ago
23. Emulator
• Used for testing DTfx
• Fully in-memory
• Can use for integration testing your own orchestrations if you use
raw DTfx
• Not designed for production use
Regions that came online after Oct 1, 2020 have similar price for v1 and v2 storage
Control queues trigger orchestrators
# of partitions = # of control queues
Max 1 worker listens to each queue, controlled by blob leases
Work item queue triggers activities
Only one queue, many listeners
Two tables used to store state, one for current orchestration statuses and one for orchestration history