This document discusses building distributed computing systems using event sourcing on Azure functions. It provides an overview of event sourcing and shows an example code implementation. It then discusses Azure storage options and Event Grid that can be used to build such systems. The document proposes an architecture using durable functions for command and query orchestration with event streams and projections. It notes lessons learned around design, measurement and reading writes. Cost estimates are provided showing such systems can cost less than £1 per day to run on Azure.
Event sourcing escalável e sustentávelErick_Mattos
Vamos contar um pouco de como implementamos Event Sourcing nos microsserviços da conta do Nubank, o que ganhamos com isso, trade-offs, otimizações que fizemos ao longo do tempo e alguns exemplos práticos.
Using event sourcing (backed by Azure table storage) as a persistence mechanism to store durable state for azure functions and using Event Grid to communicate between them
In this session we'll look at aspects affecting your account management before and after AWS Organizations, how AWS Organizations can programmatically create and manage your AWS accounts and apply organisational controls with familiar policies across these accounts to meet your business needs. We'll also cover best practices and troubleshooting tips to get you started.
Event sourcing escalável e sustentávelErick_Mattos
Vamos contar um pouco de como implementamos Event Sourcing nos microsserviços da conta do Nubank, o que ganhamos com isso, trade-offs, otimizações que fizemos ao longo do tempo e alguns exemplos práticos.
Using event sourcing (backed by Azure table storage) as a persistence mechanism to store durable state for azure functions and using Event Grid to communicate between them
In this session we'll look at aspects affecting your account management before and after AWS Organizations, how AWS Organizations can programmatically create and manage your AWS accounts and apply organisational controls with familiar policies across these accounts to meet your business needs. We'll also cover best practices and troubleshooting tips to get you started.
Event Sourcing, from ancient Mesopotamia to modern software
Patterns are all around us, and in software, the design patterns we choose can give us the power of the ancient kings and queens. Event Sourcing is based on the idea that storing events as they happen is better than storing the current state. Imagine having the power to rewind and replay events, to debug, to generate reporting, to go back to ancient history...
We’ll get up close and personal with Command Query Responsibility Segregation (CQRS), which gives us the bonus of scalability out of the box, we’ll be designing and architecting applications at scale, just like the grand cities of the past, in no time at all!
AWS ReInvent 2020: SEC313 - A security operator’s guide to practical AWS Clou...Brian Andrzejewski
AWS CloudTrail helps you discover and troubleshoot security and operational issues by capturing a comprehensive history of changes that occurred in your AWS account within a specified period of time. In this session, you learn about the AWS CloudTrail service and its value for security operations. The session dives deep into sources of data enrichment and reviews how to leverage AWS CloudTrail as part of your security operations and incident response procedures.
YouTube: https://www.youtube.com/watch?v=Tr78kq-Oa70
Improving application design with a rich domain model (springone 2007)Chris Richardson
A classic from 2007. This is a presentationthat I gave at SpringOne in Antwerp, Belgium. It describes show to improve application design by using a rich domain model
Intended for customers who have (or will have) thousands of instances on AWS, this session is about reducing the complexity of managing costs for these large fleets so they run efficiently. Attendees will learn about common roadblocks that prevent large customers from cost optimizing, tools they can use to efficiently remove those roadblocks, and techniques to monitor their rate of cost optimization. The session will include a case study that will talk in detail about the millions of dollars saved using these techniques. Customers will learn about a range of templates they can use to quickly implement these techniques, and also partners who can help them implement these templates.
Building and deploying microservices with event sourcing, CQRS and Docker (QC...Chris Richardson
In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2.
From the 2009 PayPal Developer's Conference (http://ebay.com/devcon/).
Driving Business with PayPal
Many SMB customers are looking for a higher level of integration out of their ecommerce system to drive stronger end customer relationships. Some of the more common additional pieces of functionality above standard checkout processing includes marketing campaigns connected into their product purchasing (special promos, discount coupons, etc.), tie-ins of transactional information into their online administration, and the acceptance of online invoice payments tied into their financial system. In this session we will look at taking PayPal integration to the next level through technologies such as Payments Pro, Instant Payment Notification, and Recurring Payment. In the process not only will we look at lessons learned and recommendations based on the experience of developing several of these end solutions for small and medium businesses. We will use live site case studies but also walk through the design and code of examples.
Developing functional domain models with event sourcing (oakjug, sfscala)Chris Richardson
Event sourcing persists each entity as a sequence of state changing events. An entity’s current state is derived by replaying those events. Event sourcing is a great way to implement event-driven microservices. When one service updates an entity, the new events are consumed by other services, which then update their own state.
In this talk we describe how to implement business logic using event sourcing. You will learn how to write functional, immutable domain models in Scala. We will compare and contrast a hybrid OO/FP design with a purely functional approach.
“Program to an interface, not an implementation” they[1] say …
But when IMyInterface foo = new IMyInterface() is not valid code … how are you supposed to achieve that ? The answer is Dependency Injection.
In this talk, we’ll talk about Dependency injection, what it is and what it is not. We’ll see how it is a valuable set of practices and patterns that help design maintainable software built on top of the SOLID object-oriented principles.
We’ll see how, when used properly, it delivers many benefits such as extensibility and testability … We’ll also cover some anti-patterns, ways of using Dependency Injection that can lead to code that is painful to understand and maintain
This talk is not about DI/IOC containers per se, but focuses on the core concepts of Dependency Injection. Those concepts are essential to understand how to use those “magic-looking” tools (if they are needed at all …)
This talk is not only for .NET developers. It will contain code examples written in C#, but should be understandable by developers with knowledge in other statically-typed object-oriented languages such as Java, Vb.NET, C++ …
Website Development Company is fastest growing company in the IT market for the website design and development. we are best website development company in India as well as in USA we are based in Noida and Delhi NCR. Website development company is powered by Css Founder.com
Providing more control around how to manage your AWS accounts with our newly launched service - AWS Organizations. In this session we'll look at aspects affecting your account management before and after AWS Organizations, how AWS Organizations can programmatically create and manage your AWS accounts and apply organisational controls with familiar policies across these accounts to meet your business needs. We'll also cover best practices and troubleshooting tips to get you started.
Speaker: Pierre Liddle, Solutions Architect, Amazon Web Services
Reactive Microservices with AWS Lambda (SRV201-R1) - AWS re:Invent 2018Amazon Web Services
In this talk, we cover the concepts of reactive microservices architecture, a set of patterns and best practices for implementing backend systems with very high scaling and availability guarantees. We show you how to implement these patterns and best practices as serverless applications using AWS technologies like AWS Lambda, Amazon API Gateway, Amazon DynamoDB, and Amazon Kinesis. In addition, we demonstrate a serverless microservice and the pitfalls to consider when running these microservices in a production environment.
Event Sourcing, from ancient Mesopotamia to modern software
Patterns are all around us, and in software, the design patterns we choose can give us the power of the ancient kings and queens. Event Sourcing is based on the idea that storing events as they happen is better than storing the current state. Imagine having the power to rewind and replay events, to debug, to generate reporting, to go back to ancient history...
We’ll get up close and personal with Command Query Responsibility Segregation (CQRS), which gives us the bonus of scalability out of the box, we’ll be designing and architecting applications at scale, just like the grand cities of the past, in no time at all!
AWS ReInvent 2020: SEC313 - A security operator’s guide to practical AWS Clou...Brian Andrzejewski
AWS CloudTrail helps you discover and troubleshoot security and operational issues by capturing a comprehensive history of changes that occurred in your AWS account within a specified period of time. In this session, you learn about the AWS CloudTrail service and its value for security operations. The session dives deep into sources of data enrichment and reviews how to leverage AWS CloudTrail as part of your security operations and incident response procedures.
YouTube: https://www.youtube.com/watch?v=Tr78kq-Oa70
Improving application design with a rich domain model (springone 2007)Chris Richardson
A classic from 2007. This is a presentationthat I gave at SpringOne in Antwerp, Belgium. It describes show to improve application design by using a rich domain model
Intended for customers who have (or will have) thousands of instances on AWS, this session is about reducing the complexity of managing costs for these large fleets so they run efficiently. Attendees will learn about common roadblocks that prevent large customers from cost optimizing, tools they can use to efficiently remove those roadblocks, and techniques to monitor their rate of cost optimization. The session will include a case study that will talk in detail about the millions of dollars saved using these techniques. Customers will learn about a range of templates they can use to quickly implement these techniques, and also partners who can help them implement these templates.
Building and deploying microservices with event sourcing, CQRS and Docker (QC...Chris Richardson
In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2.
From the 2009 PayPal Developer's Conference (http://ebay.com/devcon/).
Driving Business with PayPal
Many SMB customers are looking for a higher level of integration out of their ecommerce system to drive stronger end customer relationships. Some of the more common additional pieces of functionality above standard checkout processing includes marketing campaigns connected into their product purchasing (special promos, discount coupons, etc.), tie-ins of transactional information into their online administration, and the acceptance of online invoice payments tied into their financial system. In this session we will look at taking PayPal integration to the next level through technologies such as Payments Pro, Instant Payment Notification, and Recurring Payment. In the process not only will we look at lessons learned and recommendations based on the experience of developing several of these end solutions for small and medium businesses. We will use live site case studies but also walk through the design and code of examples.
Developing functional domain models with event sourcing (oakjug, sfscala)Chris Richardson
Event sourcing persists each entity as a sequence of state changing events. An entity’s current state is derived by replaying those events. Event sourcing is a great way to implement event-driven microservices. When one service updates an entity, the new events are consumed by other services, which then update their own state.
In this talk we describe how to implement business logic using event sourcing. You will learn how to write functional, immutable domain models in Scala. We will compare and contrast a hybrid OO/FP design with a purely functional approach.
“Program to an interface, not an implementation” they[1] say …
But when IMyInterface foo = new IMyInterface() is not valid code … how are you supposed to achieve that ? The answer is Dependency Injection.
In this talk, we’ll talk about Dependency injection, what it is and what it is not. We’ll see how it is a valuable set of practices and patterns that help design maintainable software built on top of the SOLID object-oriented principles.
We’ll see how, when used properly, it delivers many benefits such as extensibility and testability … We’ll also cover some anti-patterns, ways of using Dependency Injection that can lead to code that is painful to understand and maintain
This talk is not about DI/IOC containers per se, but focuses on the core concepts of Dependency Injection. Those concepts are essential to understand how to use those “magic-looking” tools (if they are needed at all …)
This talk is not only for .NET developers. It will contain code examples written in C#, but should be understandable by developers with knowledge in other statically-typed object-oriented languages such as Java, Vb.NET, C++ …
Website Development Company is fastest growing company in the IT market for the website design and development. we are best website development company in India as well as in USA we are based in Noida and Delhi NCR. Website development company is powered by Css Founder.com
Providing more control around how to manage your AWS accounts with our newly launched service - AWS Organizations. In this session we'll look at aspects affecting your account management before and after AWS Organizations, how AWS Organizations can programmatically create and manage your AWS accounts and apply organisational controls with familiar policies across these accounts to meet your business needs. We'll also cover best practices and troubleshooting tips to get you started.
Speaker: Pierre Liddle, Solutions Architect, Amazon Web Services
Reactive Microservices with AWS Lambda (SRV201-R1) - AWS re:Invent 2018Amazon Web Services
In this talk, we cover the concepts of reactive microservices architecture, a set of patterns and best practices for implementing backend systems with very high scaling and availability guarantees. We show you how to implement these patterns and best practices as serverless applications using AWS technologies like AWS Lambda, Amazon API Gateway, Amazon DynamoDB, and Amazon Kinesis. In addition, we demonstrate a serverless microservice and the pitfalls to consider when running these microservices in a production environment.
Slides from our unconference-style first meeting on DDD in which we discussed what it was (and wasn't) and embarked on choosing a domain to use to do a soup-to-nuts process with which to learn all the techniques.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
2. Agenda
• Sources of cost
• Introduction to “event sourcing”
• Example code
• Introduction to Azure storage and Azure Event Grid
• Overview of a proposed system architecture
• Lessons learned
• Q & A
22. Make a withdrawal
Target account number for the withdrawal must exist
Balance must be greater than or equal to the withdrawal amount
Withdrawal amount
Concurrency crocodile!
41. Code and architecture recap
Event Grid function triggers
Query orchestration Command orchestration
Event streams and projections
Fan-out
https://github.com/MerrionComputing/AzureFunctions-TheLongRun-Leagues
https://github.com/MerrionComputing/EventsSourcing-on-Azure-Functions
Editor's Notes
Good morning and thank you very much.
My name is Duncan Jones and my twitter handle (probably the easiest way to contract me) is @Merrion
My thesis is something I am calling “Hitchhiker” systems which is to say systems optimised towards lowering cost
First I need to discuss where cost comes from, in the context of cloud computing.
Then briefly explain event sourcing because the solution I have put together makes extensive use of this (and I will explain why at the same time)
Then the environment in which this is to be hosted – using the Azure Event Grid, Durable Serverless functions and Azure storage.
Then I will walk through the way the architecture hangs together
Finally I’ll wrap up with lessons learned and Q&A
The system I am describing here is best suited to non critical or occasional/burst use scenarios rather than enterprise critical systems and is tuned toward minimising cost rather than maximised performance.
It is also an experiment rather than a definitive textbook example.
To manage expectations, what I am looking for is an equivalent to what Blogs did for micropublishing in applications.
There are two main sources of cost that we need to address with regard to cloud hosted systems:
The first is always-on systems, for example virtual machines, for which a cost arises regardless of use.
The second is overprovisioning, where we have to reserve more processing power than we are using so that we are able to ramp-up to match rapidly increasing demand.
Azure serverless functions address these two sources of cost “out of the box”
Event sourcing is a way of storing data, not as the current state of the entity (as in active state) but rather as a sequential history of all the events that have occurred to an entity.
This history is implemented as an append only event stream. It places the onus on getting “state” onto the system that is reading the data which has to run a projection over the event stream.
This is done by (and is the responsibility of) the process that needs that state, either to report it or to use it in a business decision.
This allows us to eliminate one of the “always on” fixed costs – the state consistency management system (or database to use the more common term).
Design / constraints
An account can only be created once - therefore if an event stream already exists for the account, do not create a new one or append an account opened event.
When the serverless function is triggered the account number is taken from the URL and is used to instantiate a new eventstream variable for that bank account.
1) To enforce the constraint that an account must be new we can first check the event stream does not exist, but also we can pass in a constraint that when writing the first event the event stream must be new.
In addition, the same function is able to append more than one event onto the event stream if that makes business sense
Here I am adding a deposit event if the account is opened with an initial deposit and a beneficiary set event if the beneficiary name is provided to the open new account function
The event stream created is stored in the Azure blob storage under a path made of Domain name -> Entity Type
Each blob also has metadata set for the properties of the event stream
The data for each event are in JSON and wrapped in the event context
(Projections only access the data inside the event instance but the outer wrapper is useful for debugging etc)
Design / constraints
Depositing money is the simplest of all operations - we simply need to be sure that the account exists.
Making a deposit is the simplest function – we simply populate a money deposited event and append it to the event stream for the given bank account.
We check the bank account exists first in case the user has incorrectly entered their account number
The deposit event is appended to the stream.
Importantly the state (current balance etc.) of the bank account is not stored in the event
Design / constraints
To get the balance of the account we need to run a projection over the bank account event stream which handles the money deposited event and the money withdrawn event to give the balance as at a given point.
To run a projection we need to specify the domain, entity type and instance identity of the event stream we are going to run the projection over, and also we need to specify the class of the projection to run.
The projection is a very simple piece of code that says what events it will handle (by implementing the IHandleEventType interface) and then has a method to be called each time an instance of that event type is encountered in the event stream.
In this case a money deposited event increases the account balance and a money withdrawn event decreases it.
The projection is run and when it has processed all the events in the event stream the value is returned.
1) Every projection also has a Current Sequence Number property that tells you the number of the last event in the stream that it read
Resulting event stream
Design / constraints
Withdrawing money requires running the balance projection to make sure that the account has the funds available to withdraw and only if it does, post the withdrawal event. When posting the withdrawal event we pass in the sequence number returned by the projection and if any other events have been written to the event stream since then an error is thrown and our withdrawal event is not appended.
Azure function
1) By passing in the sequence number we got from the projection we can tell the AppendEvent method to throw an exception if the actual top sequence of the event stream is higher than that when it comes to write the event – this prevents a concurrency error if another thread appends a withdrawal while you are so doing.
The result of this is that we are prevented from withdrawing more money than we have in the account.
For a more larger and/or complicated system I would recommend using CosmsosDb.
For smaller scale systems there are three other options. For the more read –biased application I would recommend either Azure Table storage or Azure Files – and for a write-biased system I would recommend the Azure AppendBlob storage.
This matches very well the “event sourcing” architecture discussed earlier.
Event grid is “always listening” – if it receives a notification it can trigger an invocation (“wake up”) of the azure function(s) subscribing to it.
Event grid will retry sending to a subscription if the send does not succeed – if they fail for 24 hours then the message will be put in a “deal letter” location.
Events will be delivered at least once – but possibly more than once, so you should code defensively for that possibility
The demonstration system is for organising an amateur running organisation.
It is split into three domains – the league domain being anything to do with the overall organisation of leagues,
The race domain being to do with an individual race,
The runner domain being relating to the runner and their interaction with the organisation.
Importantly there is no privileged communication between the domains – the interface available to each other is the same interface publicly available outside.
A command is issued to the system through an event grid topic.
The parameters required to perform the command as well as any authorisation tokens and externally supplied unique identifier are supplied in the payload to this event grid topic.
When a message is received by the topic it triggers a serverless function to handle the command.
This uses the durable function framework to call activities that perform each step in turn and, if all is OK, appends 1 or more events to the event streams of the entities updated by the command.
As well as the built in durable function orchestration which is backed by a table the command has its own event stream in which the progress is logged.
The orchestration function has an [OrchestrationTrigger] attribute and durable orchestration through which the details of what kicked off the orchestration can be found
The orchestration calls out to Activity methods which do the actual work of each step
Events are appended to the appropriate EventStream in order to record what the command has done – effectively what has changed
An independent sub orchestration is fired off that does all the work to do with notifying other domains of this command’s action
A query is handled in a similar manner but in addition to the parameters you also pass in a return address to which the results of the query should be returned.
This could be, for instance, a webhook or an azure storage location to drop a file into.
This allows the query to be executed asynchronously.
The query runs projections over the event streams of the entities it is getting data for and then performs any aggregation needed over these before returning the results.
Projection requests are written to the query event stream so we can see what work has been done and what is outstanding in running the query
Projections themselves can then be run in parallel by using durable functions “fan-out” mechanism.
Once all the projections are complete any totting up steps could run.
Projection requests are written to the query event stream so we can see what work has been done and what is outstanding in running the query
Projections themselves can then be run in parallel by using durable functions “fan-out” mechanism.
Once all the projections are complete any totting up steps could run.
Domains communicate directly with each other by these commands and queries
Domains can also raise notifications that other domains or indeed other processes can subscribe to.
(These notifications are tagged with the “as of date” and “as of sequence number” information so that the subscriber can determine if it has already received this notification before.)
Because serverless computing is no less susceptible to the “sunk cost fallacy” it is worth investing time in design before getting started coding. It is a mistake to allow the rapid spin-up time of serverless functions as an excuse not to do some design work. I would recommend looking at things like event storming as a way to do this.
If you are dealing with a system that needs to be able to survive being spun down without the worry of data loss requires being rigorous about reading your won writes – i.e. making sure any state change is persisted before it is used.
Event Grid for the reliable messaging between domains (and with the outside world) – this allows us to stitch together domains with publish-subscribe, which means we can easily extend the system.
Serverless function hosting for dynamic scaling.
Durable Function Orchestration for the long running processes that handle the commands and queries.
Durable functions fan out/fan in for processes that can run in parallel.
Azure AppendBlob storage to store the data at rest in event streams.