SlideShare a Scribd company logo
Deep Dive into
Durable Functions
CloudBurst
Joonas Westlin
Speaker Intro
• Joonas Westlin
• Developer / Architect @ Zure
@JoonasWestlin
joonasw.net
Key takeaways from this presentation
• Orchestrator and activity communication
• State storage using the Azure Storage provider
• Differences in other durability providers
• Goal: improve your ability to debug Durable Functions
Introduction/refresher
• Durable Functions = extension for Azure Functions that allows
execution of Durable Task orchestrations
• Durable Task framework allows developers to build long-running
workflows with persistent state, using well known async-await
constructs in .NET
• Orchestrator functions define the workflow, activity functions do
non-deterministic work
Durable Task workers
• Workers execute orchestrators and activities by requesting work
items from the durability provider
• Multiple workers run in parallel = increased throughput
• Activity work items can run on any worker, but orchestrator work
items cannot
• Danger of state corruption if more than 1 worker takes a work item for
same orchestration simultaneously
Durability providers
Storage
Service
Bus
Netherite SQL
Redis
Service
Fabric
Emulator
Storage durability provider
• Convenient choice for Azure Functions as it has a Storage account
already
• Cheap option, consumption-based pricing in Storage
• Tip: using a v1 Storage account is cheaper (not in new regions)
• Can run emulator locally for Storage
Storage task hub
Orchestrator and activity communication
• Orchestrator gets executed several times; each time a set of
outbound messages is generated for activities to be triggered
• These are sent through the durability provider and workers
receive them
• Once the worker is done with the activity, it sends a message back
to the orchestrator
Control queue 0
Control queue 1
Control queue 2
Control queue 3
Work item queue
Demo: Durable Function
execution
Durable Entities
• We looked at orchestrators and activities but where are Durable
Entities?
• Entities are just orchestrators
• At high level, you can think of entities as an orchestrator that:
1. Gets state from input
2. Waits for an external event (operation)
3. Computes new state based on event
4. Restarts itself with new state as input
Other things provided by Durable Functions
extension
• Durable HTTP
• [Deterministic]
• Replay-safe logger
What about those other
durability providers?
Netherite (public preview)
• Higher performance + higher cost
• In high throughput scenarios, cheaper than high scale of other providers
• Supported by Durable Functions
• Not Consumption plan though
• Uses Event Hubs for orchestrator/activity messaging
• Even though Event Hubs do persist data, it is replicated to Azure Storage
after receive
• Max 32 partitions + 20 throughput units = 20 MB/s
• Uses Azure Storage blobs for storing partition state
• Harder to inspect current state
Netherite task hub
SQL (public preview)
• Uses SQL tables for everything
• Supported by Durable Functions
• Not Consumption plan though
• Stored procedures contain most logic
• Portable, no Azure connection required
• Can do data encryption, backups etc.
• Multi-tenant scenarios with schema per tenant
SQL task hub
SQL table contents
Service Bus (+Storage)
• Mature and transactionally consistent
• Not supported by Durable Functions
• Uses 3 Service Bus queues
• Orchestrator = messages for orchestrators
• Worker = messages for activities
• Tracking = orchestration state tracking
• Requires an instance store + blob store, comes with Storage
implementations for them
• State and history tracking
• Oversized messages and sessions
Redis
• Uses a Redis database
• Not supported by Durable Functions
• Workers notified of new messages through Redis channels
• Actual messages sent through Redis lists
• Portable since you can run Redis pretty much anywhere
Service Fabric
• Supported only within a Service Fabric cluster
• Uses SF reliable collections for state storage
• Some features limited, e.g. cannot query instance state if it has
completed over an hour ago
Emulator
• Used for testing DTfx
• Fully in-memory
• Can use for integration testing your own orchestrations if you use
raw DTfx
• Not designed for production use
Demo: Durability providers
with raw Durable Task
Links
• https://westl.in/slides
• https://learn.microsoft.com/en-us/azure/azure-functions/durable/
• https://github.com/Azure/azure-functions-durable-extension
• https://github.com/Azure/durabletask
Deep Dive into Durable Functions

More Related Content

Similar to Deep Dive into Durable Functions

Running & Monitoring Docker at Scale
Running & Monitoring Docker at ScaleRunning & Monitoring Docker at Scale
Running & Monitoring Docker at Scale
Datadog
 
Building large scale, job processing systems with Scala Akka Actor framework
Building large scale, job processing systems with Scala Akka Actor frameworkBuilding large scale, job processing systems with Scala Akka Actor framework
Building large scale, job processing systems with Scala Akka Actor framework
Vignesh Sukumar
 
Deep Dive into Azure Data Factory v2
Deep Dive into Azure Data Factory v2Deep Dive into Azure Data Factory v2
Deep Dive into Azure Data Factory v2
Eric Bragas
 
Insider operating system
Insider   operating systemInsider   operating system
Insider operating system
Aditi Saxena
 
Database as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesDatabase as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on Kubernetes
ObjectRocket
 
Tupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FBTupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FB
Docker, Inc.
 
Real Time Operating Systems, Dynamic Precision: Exploring the Realm of Real-...
Real Time Operating Systems,  Dynamic Precision: Exploring the Realm of Real-...Real Time Operating Systems,  Dynamic Precision: Exploring the Realm of Real-...
Real Time Operating Systems, Dynamic Precision: Exploring the Realm of Real-...
Adobe2801
 
Scott Schnoll - Exchange server 2013 high availability and site resilience
Scott Schnoll - Exchange server 2013 high availability and site resilienceScott Schnoll - Exchange server 2013 high availability and site resilience
Scott Schnoll - Exchange server 2013 high availability and site resilience
Nordic Infrastructure Conference
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolith
Markus Eisele
 
Azure servicefabric
Azure servicefabricAzure servicefabric
Azure servicefabric
Abhishek Sur
 
What's new in JBoss ON 3.2
What's new in JBoss ON 3.2What's new in JBoss ON 3.2
What's new in JBoss ON 3.2
Thomas Segismont
 
Azure Messaging Crossroads
Azure Messaging CrossroadsAzure Messaging Crossroads
Azure Messaging Crossroads
Sean Feldman
 
Scheduling Thread
Scheduling  ThreadScheduling  Thread
Scheduling Thread
MuhammadBilal187526
 
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio, Inc.
 
Azure Function Workflow
Azure Function WorkflowAzure Function Workflow
Azure Function Workflow
Andrea Tosato
 
Microservices deck
Microservices deckMicroservices deck
Microservices deck
Raja Chattopadhyay
 
Intro to cluster scheduler for Linux containers
Intro to cluster scheduler for Linux containersIntro to cluster scheduler for Linux containers
Intro to cluster scheduler for Linux containers
Kumar Gaurav
 
Service fabric overview
Service fabric overviewService fabric overview
Service fabric overview
Himanshu Desai
 
(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...
(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...
(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...
BIOVIA
 
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Dylan Butler
 

Similar to Deep Dive into Durable Functions (20)

Running & Monitoring Docker at Scale
Running & Monitoring Docker at ScaleRunning & Monitoring Docker at Scale
Running & Monitoring Docker at Scale
 
Building large scale, job processing systems with Scala Akka Actor framework
Building large scale, job processing systems with Scala Akka Actor frameworkBuilding large scale, job processing systems with Scala Akka Actor framework
Building large scale, job processing systems with Scala Akka Actor framework
 
Deep Dive into Azure Data Factory v2
Deep Dive into Azure Data Factory v2Deep Dive into Azure Data Factory v2
Deep Dive into Azure Data Factory v2
 
Insider operating system
Insider   operating systemInsider   operating system
Insider operating system
 
Database as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesDatabase as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on Kubernetes
 
Tupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FBTupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FB
 
Real Time Operating Systems, Dynamic Precision: Exploring the Realm of Real-...
Real Time Operating Systems,  Dynamic Precision: Exploring the Realm of Real-...Real Time Operating Systems,  Dynamic Precision: Exploring the Realm of Real-...
Real Time Operating Systems, Dynamic Precision: Exploring the Realm of Real-...
 
Scott Schnoll - Exchange server 2013 high availability and site resilience
Scott Schnoll - Exchange server 2013 high availability and site resilienceScott Schnoll - Exchange server 2013 high availability and site resilience
Scott Schnoll - Exchange server 2013 high availability and site resilience
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolith
 
Azure servicefabric
Azure servicefabricAzure servicefabric
Azure servicefabric
 
What's new in JBoss ON 3.2
What's new in JBoss ON 3.2What's new in JBoss ON 3.2
What's new in JBoss ON 3.2
 
Azure Messaging Crossroads
Azure Messaging CrossroadsAzure Messaging Crossroads
Azure Messaging Crossroads
 
Scheduling Thread
Scheduling  ThreadScheduling  Thread
Scheduling Thread
 
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
 
Azure Function Workflow
Azure Function WorkflowAzure Function Workflow
Azure Function Workflow
 
Microservices deck
Microservices deckMicroservices deck
Microservices deck
 
Intro to cluster scheduler for Linux containers
Intro to cluster scheduler for Linux containersIntro to cluster scheduler for Linux containers
Intro to cluster scheduler for Linux containers
 
Service fabric overview
Service fabric overviewService fabric overview
Service fabric overview
 
(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...
(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...
(ATS3-APP08) Top 10 things every Symyx Notebook by Accelrys Administrator sho...
 
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
 

More from Joonas Westlin

Using feature flags in an ASP.NET Core app on Azure
Using feature flags in an ASP.NET Core app on AzureUsing feature flags in an ASP.NET Core app on Azure
Using feature flags in an ASP.NET Core app on Azure
Joonas Westlin
 
Deep Dive into Durable Functions
Deep Dive into Durable FunctionsDeep Dive into Durable Functions
Deep Dive into Durable Functions
Joonas Westlin
 
Zero Credential Development with Managed Identities
Zero Credential Development with Managed IdentitiesZero Credential Development with Managed Identities
Zero Credential Development with Managed Identities
Joonas Westlin
 
Zero Credential Development with Managed Identities
Zero Credential Development with Managed IdentitiesZero Credential Development with Managed Identities
Zero Credential Development with Managed Identities
Joonas Westlin
 
Zero credential development with managed identities
Zero credential development with managed identitiesZero credential development with managed identities
Zero credential development with managed identities
Joonas Westlin
 
Zero credential development with managed identities
Zero credential development with managed identitiesZero credential development with managed identities
Zero credential development with managed identities
Joonas Westlin
 
Zero Credential Development with Managed Identities for Azure resources
Zero Credential Development with Managed Identities for Azure resourcesZero Credential Development with Managed Identities for Azure resources
Zero Credential Development with Managed Identities for Azure resources
Joonas Westlin
 
Zero Credential Development with Managed Identities
Zero Credential Development with Managed IdentitiesZero Credential Development with Managed Identities
Zero Credential Development with Managed Identities
Joonas Westlin
 
7 Deadly Sins in Azure AD App Development
7 Deadly Sins in Azure AD App Development7 Deadly Sins in Azure AD App Development
7 Deadly Sins in Azure AD App Development
Joonas Westlin
 
Introduction to Azure AD and Azure AD B2C
Introduction to Azure AD and Azure AD B2CIntroduction to Azure AD and Azure AD B2C
Introduction to Azure AD and Azure AD B2C
Joonas Westlin
 
Get rid of credentials from your code: Using Managed identities for Azure res...
Get rid of credentials from your code: Using Managed identities for Azure res...Get rid of credentials from your code: Using Managed identities for Azure res...
Get rid of credentials from your code: Using Managed identities for Azure res...
Joonas Westlin
 
Programming with Azure Active Directory
Programming with Azure Active DirectoryProgramming with Azure Active Directory
Programming with Azure Active Directory
Joonas Westlin
 

More from Joonas Westlin (12)

Using feature flags in an ASP.NET Core app on Azure
Using feature flags in an ASP.NET Core app on AzureUsing feature flags in an ASP.NET Core app on Azure
Using feature flags in an ASP.NET Core app on Azure
 
Deep Dive into Durable Functions
Deep Dive into Durable FunctionsDeep Dive into Durable Functions
Deep Dive into Durable Functions
 
Zero Credential Development with Managed Identities
Zero Credential Development with Managed IdentitiesZero Credential Development with Managed Identities
Zero Credential Development with Managed Identities
 
Zero Credential Development with Managed Identities
Zero Credential Development with Managed IdentitiesZero Credential Development with Managed Identities
Zero Credential Development with Managed Identities
 
Zero credential development with managed identities
Zero credential development with managed identitiesZero credential development with managed identities
Zero credential development with managed identities
 
Zero credential development with managed identities
Zero credential development with managed identitiesZero credential development with managed identities
Zero credential development with managed identities
 
Zero Credential Development with Managed Identities for Azure resources
Zero Credential Development with Managed Identities for Azure resourcesZero Credential Development with Managed Identities for Azure resources
Zero Credential Development with Managed Identities for Azure resources
 
Zero Credential Development with Managed Identities
Zero Credential Development with Managed IdentitiesZero Credential Development with Managed Identities
Zero Credential Development with Managed Identities
 
7 Deadly Sins in Azure AD App Development
7 Deadly Sins in Azure AD App Development7 Deadly Sins in Azure AD App Development
7 Deadly Sins in Azure AD App Development
 
Introduction to Azure AD and Azure AD B2C
Introduction to Azure AD and Azure AD B2CIntroduction to Azure AD and Azure AD B2C
Introduction to Azure AD and Azure AD B2C
 
Get rid of credentials from your code: Using Managed identities for Azure res...
Get rid of credentials from your code: Using Managed identities for Azure res...Get rid of credentials from your code: Using Managed identities for Azure res...
Get rid of credentials from your code: Using Managed identities for Azure res...
 
Programming with Azure Active Directory
Programming with Azure Active DirectoryProgramming with Azure Active Directory
Programming with Azure Active Directory
 

Recently uploaded

A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
kalichargn70th171
 
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and MoreManyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
narinav14
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
The Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdfThe Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdf
mohitd6
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
KrishnaveniMohan1
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureBuilding API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
confluent
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
OnePlan Solutions
 
ppt on the brain chip neuralink.pptx
ppt  on   the brain  chip neuralink.pptxppt  on   the brain  chip neuralink.pptx
ppt on the brain chip neuralink.pptx
Reetu63
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
widenerjobeyrl638
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
aeeva
 
Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...
Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...
Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...
Luigi Fugaro
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
The Third Creative Media
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
Zycus
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
Maitrey Patel
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 

Recently uploaded (20)

A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
 
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and MoreManyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
The Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdfThe Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdf
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
 
Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructureBuilding API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
 
ppt on the brain chip neuralink.pptx
ppt  on   the brain  chip neuralink.pptxppt  on   the brain  chip neuralink.pptx
ppt on the brain chip neuralink.pptx
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
 
Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...
Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...
Voxxed Days Trieste 2024 - Unleashing the Power of Vector Search and Semantic...
 
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 

Deep Dive into Durable Functions

  • 1. Deep Dive into Durable Functions CloudBurst Joonas Westlin
  • 2. Speaker Intro • Joonas Westlin • Developer / Architect @ Zure @JoonasWestlin joonasw.net
  • 3. Key takeaways from this presentation • Orchestrator and activity communication • State storage using the Azure Storage provider • Differences in other durability providers • Goal: improve your ability to debug Durable Functions
  • 4. Introduction/refresher • Durable Functions = extension for Azure Functions that allows execution of Durable Task orchestrations • Durable Task framework allows developers to build long-running workflows with persistent state, using well known async-await constructs in .NET • Orchestrator functions define the workflow, activity functions do non-deterministic work
  • 5. Durable Task workers • Workers execute orchestrators and activities by requesting work items from the durability provider • Multiple workers run in parallel = increased throughput • Activity work items can run on any worker, but orchestrator work items cannot • Danger of state corruption if more than 1 worker takes a work item for same orchestration simultaneously
  • 7. Storage durability provider • Convenient choice for Azure Functions as it has a Storage account already • Cheap option, consumption-based pricing in Storage • Tip: using a v1 Storage account is cheaper (not in new regions) • Can run emulator locally for Storage
  • 9. Orchestrator and activity communication • Orchestrator gets executed several times; each time a set of outbound messages is generated for activities to be triggered • These are sent through the durability provider and workers receive them • Once the worker is done with the activity, it sends a message back to the orchestrator
  • 10. Control queue 0 Control queue 1 Control queue 2 Control queue 3 Work item queue
  • 12. Durable Entities • We looked at orchestrators and activities but where are Durable Entities? • Entities are just orchestrators • At high level, you can think of entities as an orchestrator that: 1. Gets state from input 2. Waits for an external event (operation) 3. Computes new state based on event 4. Restarts itself with new state as input
  • 13. Other things provided by Durable Functions extension • Durable HTTP • [Deterministic] • Replay-safe logger
  • 14. What about those other durability providers?
  • 15. Netherite (public preview) • Higher performance + higher cost • In high throughput scenarios, cheaper than high scale of other providers • Supported by Durable Functions • Not Consumption plan though • Uses Event Hubs for orchestrator/activity messaging • Even though Event Hubs do persist data, it is replicated to Azure Storage after receive • Max 32 partitions + 20 throughput units = 20 MB/s • Uses Azure Storage blobs for storing partition state • Harder to inspect current state
  • 17. SQL (public preview) • Uses SQL tables for everything • Supported by Durable Functions • Not Consumption plan though • Stored procedures contain most logic • Portable, no Azure connection required • Can do data encryption, backups etc. • Multi-tenant scenarios with schema per tenant
  • 20. Service Bus (+Storage) • Mature and transactionally consistent • Not supported by Durable Functions • Uses 3 Service Bus queues • Orchestrator = messages for orchestrators • Worker = messages for activities • Tracking = orchestration state tracking • Requires an instance store + blob store, comes with Storage implementations for them • State and history tracking • Oversized messages and sessions
  • 21. Redis • Uses a Redis database • Not supported by Durable Functions • Workers notified of new messages through Redis channels • Actual messages sent through Redis lists • Portable since you can run Redis pretty much anywhere
  • 22. Service Fabric • Supported only within a Service Fabric cluster • Uses SF reliable collections for state storage • Some features limited, e.g. cannot query instance state if it has completed over an hour ago
  • 23. Emulator • Used for testing DTfx • Fully in-memory • Can use for integration testing your own orchestrations if you use raw DTfx • Not designed for production use
  • 25. Links • https://westl.in/slides • https://learn.microsoft.com/en-us/azure/azure-functions/durable/ • https://github.com/Azure/azure-functions-durable-extension • https://github.com/Azure/durabletask

Editor's Notes

  1. Regions that came online after Oct 1, 2020 have similar price for v1 and v2 storage
  2. Control queues trigger orchestrators # of partitions = # of control queues Max 1 worker listens to each queue, controlled by blob leases Work item queue triggers activities Only one queue, many listeners Two tables used to store state, one for current orchestration statuses and one for orchestration history
  3. new: orch start exec start past: orch start exec start task scheduled orch complete new: orch start task complete past: orch start exec start task scheduled orch complete orch start task complete task scheduled orch complete new: orch start task complete past: orch start exec start task scheduled orch complete orch start task complete task scheduled orch complete orch start task complete task scheduled orch complete new: orch start task complete