SlideShare a Scribd company logo
1 of 47
Download to read offline
@samuelroze
Event Streaming Some things you want to know about.
@samuelroze
@samuelroze
Introduction
• My name is Samuel Rozé, I am VPoE at Birdie Care.
Core Team member of Symfony, for my work on Messenger.
• This is an architecture talk.
• We will briefly discuss the values of using stream processing event
streaming.
• We will see the consequences of living the dream of managing distributed
systems. TL;DR: plenty of things will go wrong.
@samuelroze@samuelroze
1. Why …is event streaming even interesting?
@samuelroze
Your product works… you split your services.
@samuelroze
Your services are talking to each other.
@samuelroze
Now you need to introduce targeted discounts…
@samuelroze
Now you need to introduce targeted discounts…
@samuelroze
How will this service get its data?
1. Pull via an API
• A lot of data will be moved each
time the “discount” service computes
discounts for a customer.
• “Discounts” is able to work only
when the 3 other services are
available (cascading failures).
• “Discounts” needs to know about
where are the other services and
how to talk to them.
2. Using “batch”
• Potentially contains loads of duplicated
information (full load each time or the
period is “over X days”)
• Not real-time. “Wait a few days for your
marketing preferences to be propagated”
• A lot can go wrong with all services
properly creating exports every night.
@samuelroze
Event streaming
• Events are flowing in real-time, from and
to multiple services.
• To receive a specific event, services don’t
have to know who is sending events, just
that they can expect these messages.
• Much higher availability because data
goes to the service that requires it when they
are online.
• (When bus does persistence) New
consumers create their context by going
through all the events that have happened in
the system.
• Writing code that works well with the
nature of the distributed system is hard.
• You need a real governance about how
is the message bus used, they are your
new API contracts.
@samuelroze
Event streaming, as a diagram
@samuelroze
Everything we are going to talk about is true for…
@samuelroze@samuelroze
2. What will go wrong? It’s not “if”.
@samuelroze
Let’s start with a simple use-case.
Here we write on the `Basket` entity for example
@samuelroze
Your message is sent to a queue
@samuelroze@samuelroze
Problem A Are you sure that the message 

was sent to the queue?
@samuelroze
A. Are you sure that your message has been sent?
@samuelroze
A. Are you sure that your message has been sent?
@samuelroze
A. Distributed transactions are not really a thing.
@samuelroze
A. What might happen if we don’t care about that?
• Your local “basket” table might have the new product but no worker receives
the “ProductAddedToBasket” event.
• Your local “basket” table might NOT have the new product but workers have
received the “ProductAddedToBasket” event (most likely if you use
database transactions for the entire request)
• Imagine the event being about “payment successful” or even potentially life-
changing like “fall in the home has been detected”… 😬
@samuelroze
A. The outbox pattern
@samuelroze
A. Publishing messages to bus consistently
• In a nutshell, write your message & side effects to your database as part of
one transaction and then get something else to pull the message from the
database and send it to your queue.
• With Symfony, the simplest is actually to use the Doctrine transport for
Symfony Messenger, with the doctrine transaction middleware.
• Alternatively, you can use a dedicated library for this.
• EventSaucePHP/DoctrineOutboxMessageDispatcher
• italolelis/outboxer
@samuelroze
A. Using the Doctrine transport
@samuelroze@samuelroze
Problem B You will receive duplicated
messages.
@samuelroze
B. “At least once delivery”
@samuelroze
B. What might happen if we don’t care about that?
• You consume twice “ProductAddedToBasket”: the product is added twice
instead of just once (as per the user request).
• Depending on your business logic, it might be very important. For example,
what if it is about “Money added to bank account” or “Medication dose
taken”.
@samuelroze
B. You need some idempotence.
• You will receive the same message multiple times, it’s just a matter of time.
• There isn’t much a framework could do, you own the business logic; you
need to handle it by yourself.
• Use an idempotency key. A key that represents a single message and
allows you to know whether or not it’s been processed already.
• (By the way, this also applies to HTTP requests. Stripe’s API is a good
example.)
@samuelroze
B. Using the idempotency key in the handler
• One option is to have your idempotency key is part of your message. Your
team needs to know why it is useful and how to use it.
@samuelroze
B. How to use your “idempotency key”
@samuelroze
B. How to use your “idempotency key”
@samuelroze@samuelroze
Problem C Processing messages in parallel.
@samuelroze
C. Concurrently processing messages
@samuelroze
C. What might happen if we don’t care about that?
• You will lose some state in whatever you updated based on the events, at
some point.
• The easiest solution: don’t process things concurrently. But, not really
practical when things start to scale.
@samuelroze
C. Locking! Optimistic vs Pessimistic
The optimist…
• Assumes that everything will go
right most of the time.
• It validates that everything has
happened as expected when
writing its state to a consistent
storage.
• a.k.a. HTTP’s If-Match, …
The pessimist…
• Believes that I most cases, this
won’t work.
• Before doing any work, it ensures
nobody else is doing it.
• a.k.a. “mutex”, “advisory locks”,
etc…
@samuelroze
C. Pessimistic locking with Symfony Lock
@samuelroze
C. Optimistic locking with Doctrine’s “versions"
@samuelroze
C. Optimistic locking with Doctrine’s “versions"
What’s happening behind the scene
with optimistic locking:
@samuelroze@samuelroze
Problem D Message ordering
@samuelroze
D. Know when there is no ordering guarantee
@samuelroze
D. What might happen if we don’t care about that?
• Hopefully your business logic doesn’t rely too much on the events being
ordered… make sure this is true.
• For example, we rely on “access_granted” and “access_revoked” events to
configure some permission rules. If they are consumed in the wrong
order… this is a different meaning 💥
@samuelroze
D. There are buses that guarantee order
@samuelroze
D. They scale using partitions
@samuelroze
D. Order guaranteed means blocking messages.
@samuelroze
D. For the infrastructure to guarantee ordering…
• You need a message bus that supports it (Kafka, SQS Fifo, Kinesis, etc…).
• You need to carefully design your partitions (or “shards”) so that you know all
message of a specific aggregate will always go to the same partition (a.k.a.
routing keys).
• You need to carefully manage all the errors. You can’t afford a wrong
message blocking an entire topic. But you can’t really post-pone only one single message…
@samuelroze@samuelroze
To wrap up… A few learnings (hopefully).
@samuelroze
We’ve seen a few ways it can go wrong.
• When publishing a message to a bus.
Outbox pattern FTW.
• When receiving multiple time the same message.
Idempotence FTW.
• When concurrently consuming messages.

You need to use optimistic or pessimistic locking.
• You can request ordering from your infrastructure.
But needs careful partition design & error management.
@samuelroze
Thank you!
@samuelroze
@samuelroze
Want to read more?
• Martin Kleppman’s book.
https://dataintensive.net
• https://multithreaded.stitchfix.com/blog/2017/06/26/patterns-of-soa-
idempotency-key/
• https://microservices.io/patterns/data/transactional-outbox.html

More Related Content

Similar to Event streaming: what will go wrong? (Symfony World 2020)

Making operations visible - devopsdays tokyo 2013
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013Nick Galbreath
 
Developing a Globally Distributed Purging System
Developing a Globally Distributed Purging SystemDeveloping a Globally Distributed Purging System
Developing a Globally Distributed Purging SystemFastly
 
Simple SAP Security Breach !!
Simple SAP Security Breach !!Simple SAP Security Breach !!
Simple SAP Security Breach !!SAPYard
 
Event storage in a distributed system
Event storage in a distributed systemEvent storage in a distributed system
Event storage in a distributed systemSteve Pember
 
10 impact of uncertainty in matching
10 impact of uncertainty in matching10 impact of uncertainty in matching
10 impact of uncertainty in matchingRishi Mathur
 
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmxMoved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmxMilen Dyankov
 
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)Livestorm
 
Sage Switch Guide
Sage Switch GuideSage Switch Guide
Sage Switch Guidecraig1201
 
I Want These * Bugs Off My * Internet
I Want These * Bugs Off My * InternetI Want These * Bugs Off My * Internet
I Want These * Bugs Off My * InternetDan Kaminsky
 
Project Planning and Estimation with User Stories
Project Planning and Estimation with User StoriesProject Planning and Estimation with User Stories
Project Planning and Estimation with User StoriesPolished Geek LLC
 
Grails Worst Practices
Grails Worst PracticesGrails Worst Practices
Grails Worst PracticesBurt Beckwith
 
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
 GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res... GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...James Anderson
 
Responsible Microservices
Responsible MicroservicesResponsible Microservices
Responsible MicroservicesVMware Tanzu
 
Mobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B ExperimentsMobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B Experimentslacyrhoades
 
TLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processingTLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processingalooma
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Brian Brazil
 
Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...WeLoveSEO
 

Similar to Event streaming: what will go wrong? (Symfony World 2020) (20)

Making operations visible - devopsdays tokyo 2013
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013
 
Developing a Globally Distributed Purging System
Developing a Globally Distributed Purging SystemDeveloping a Globally Distributed Purging System
Developing a Globally Distributed Purging System
 
Simple SAP Security Breach !!
Simple SAP Security Breach !!Simple SAP Security Breach !!
Simple SAP Security Breach !!
 
Event storage in a distributed system
Event storage in a distributed systemEvent storage in a distributed system
Event storage in a distributed system
 
How to Pitch Bullshit
How to Pitch BullshitHow to Pitch Bullshit
How to Pitch Bullshit
 
10 impact of uncertainty in matching
10 impact of uncertainty in matching10 impact of uncertainty in matching
10 impact of uncertainty in matching
 
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmxMoved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
 
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
 
Sage Switch Guide
Sage Switch GuideSage Switch Guide
Sage Switch Guide
 
I Want These * Bugs Off My * Internet
I Want These * Bugs Off My * InternetI Want These * Bugs Off My * Internet
I Want These * Bugs Off My * Internet
 
Project Planning and Estimation with User Stories
Project Planning and Estimation with User StoriesProject Planning and Estimation with User Stories
Project Planning and Estimation with User Stories
 
Grails Worst Practices
Grails Worst PracticesGrails Worst Practices
Grails Worst Practices
 
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
 GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res... GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
 
Responsible Microservices
Responsible MicroservicesResponsible Microservices
Responsible Microservices
 
Mobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B ExperimentsMobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B Experiments
 
Cucumber
CucumberCucumber
Cucumber
 
TLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processingTLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processing
 
Fighting Spam at Flickr
Fighting Spam at FlickrFighting Spam at Flickr
Fighting Spam at Flickr
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)
 
Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...
 

More from Samuel ROZE

How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patternsSamuel ROZE
 
Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)Samuel ROZE
 
Micro services may not be the best idea
Micro services may not be the best ideaMicro services may not be the best idea
Micro services may not be the best ideaSamuel ROZE
 
Introduction to CQRS and Event Sourcing
Introduction to CQRS and Event SourcingIntroduction to CQRS and Event Sourcing
Introduction to CQRS and Event SourcingSamuel ROZE
 
CQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony applicationCQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony applicationSamuel ROZE
 
How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patternsSamuel ROZE
 
Take care of our micro services
Take care of our micro servicesTake care of our micro services
Take care of our micro servicesSamuel ROZE
 
(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et ToleranceSamuel ROZE
 
Using continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflowsUsing continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflowsSamuel ROZE
 
Symfony CoP: Form component
Symfony CoP: Form componentSymfony CoP: Form component
Symfony CoP: Form componentSamuel ROZE
 
Behat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than thatBehat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than thatSamuel ROZE
 
Docker orchestration with Kubernetes
Docker orchestration with KubernetesDocker orchestration with Kubernetes
Docker orchestration with KubernetesSamuel ROZE
 
Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer Samuel ROZE
 

More from Samuel ROZE (13)

How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patterns
 
Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)
 
Micro services may not be the best idea
Micro services may not be the best ideaMicro services may not be the best idea
Micro services may not be the best idea
 
Introduction to CQRS and Event Sourcing
Introduction to CQRS and Event SourcingIntroduction to CQRS and Event Sourcing
Introduction to CQRS and Event Sourcing
 
CQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony applicationCQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony application
 
How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patterns
 
Take care of our micro services
Take care of our micro servicesTake care of our micro services
Take care of our micro services
 
(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance
 
Using continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflowsUsing continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflows
 
Symfony CoP: Form component
Symfony CoP: Form componentSymfony CoP: Form component
Symfony CoP: Form component
 
Behat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than thatBehat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than that
 
Docker orchestration with Kubernetes
Docker orchestration with KubernetesDocker orchestration with Kubernetes
Docker orchestration with Kubernetes
 
Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer
 

Recently uploaded

Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxTriangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxRomil Mishra
 
Indian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdfIndian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdfalokitpathak01
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.elesangwon
 
AntColonyOptimizationManetNetworkAODV.pptx
AntColonyOptimizationManetNetworkAODV.pptxAntColonyOptimizationManetNetworkAODV.pptx
AntColonyOptimizationManetNetworkAODV.pptxLina Kadam
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Sumanth A
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organizationchnrketan
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSsandhya757531
 
1- Practice occupational health and safety procedures.pptx
1- Practice occupational health and safety procedures.pptx1- Practice occupational health and safety procedures.pptx
1- Practice occupational health and safety procedures.pptxMel Paras
 
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfalene1
 
Novel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending ActuatorsNovel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending ActuatorsResearcher Researcher
 
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...gerogepatton
 
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTFUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTSneha Padhiar
 
Module-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdfModule-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdfManish Kumar
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Romil Mishra
 
Machine Learning 5G Federated Learning.pdf
Machine Learning 5G Federated Learning.pdfMachine Learning 5G Federated Learning.pdf
Machine Learning 5G Federated Learning.pdfadeyimikaipaye
 
Cost estimation approach: FP to COCOMO scenario based question
Cost estimation approach: FP to COCOMO scenario based questionCost estimation approach: FP to COCOMO scenario based question
Cost estimation approach: FP to COCOMO scenario based questionSneha Padhiar
 
The Satellite applications in telecommunication
The Satellite applications in telecommunicationThe Satellite applications in telecommunication
The Satellite applications in telecommunicationnovrain7111
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical trainingGladiatorsKasper
 
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliStructural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliNimot Muili
 

Recently uploaded (20)

Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxTriangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
 
Indian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdfIndian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdf
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
 
AntColonyOptimizationManetNetworkAODV.pptx
AntColonyOptimizationManetNetworkAODV.pptxAntColonyOptimizationManetNetworkAODV.pptx
AntColonyOptimizationManetNetworkAODV.pptx
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organization
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
 
1- Practice occupational health and safety procedures.pptx
1- Practice occupational health and safety procedures.pptx1- Practice occupational health and safety procedures.pptx
1- Practice occupational health and safety procedures.pptx
 
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
 
Novel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending ActuatorsNovel 3D-Printed Soft Linear and Bending Actuators
Novel 3D-Printed Soft Linear and Bending Actuators
 
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
 
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTFUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
 
Module-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdfModule-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdf
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________
 
Machine Learning 5G Federated Learning.pdf
Machine Learning 5G Federated Learning.pdfMachine Learning 5G Federated Learning.pdf
Machine Learning 5G Federated Learning.pdf
 
Cost estimation approach: FP to COCOMO scenario based question
Cost estimation approach: FP to COCOMO scenario based questionCost estimation approach: FP to COCOMO scenario based question
Cost estimation approach: FP to COCOMO scenario based question
 
The Satellite applications in telecommunication
The Satellite applications in telecommunicationThe Satellite applications in telecommunication
The Satellite applications in telecommunication
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training
 
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliStructural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
 

Event streaming: what will go wrong? (Symfony World 2020)

  • 1. @samuelroze Event Streaming Some things you want to know about. @samuelroze
  • 2. @samuelroze Introduction • My name is Samuel Rozé, I am VPoE at Birdie Care. Core Team member of Symfony, for my work on Messenger. • This is an architecture talk. • We will briefly discuss the values of using stream processing event streaming. • We will see the consequences of living the dream of managing distributed systems. TL;DR: plenty of things will go wrong.
  • 3. @samuelroze@samuelroze 1. Why …is event streaming even interesting?
  • 4. @samuelroze Your product works… you split your services.
  • 5. @samuelroze Your services are talking to each other.
  • 6. @samuelroze Now you need to introduce targeted discounts…
  • 7. @samuelroze Now you need to introduce targeted discounts…
  • 8. @samuelroze How will this service get its data? 1. Pull via an API • A lot of data will be moved each time the “discount” service computes discounts for a customer. • “Discounts” is able to work only when the 3 other services are available (cascading failures). • “Discounts” needs to know about where are the other services and how to talk to them. 2. Using “batch” • Potentially contains loads of duplicated information (full load each time or the period is “over X days”) • Not real-time. “Wait a few days for your marketing preferences to be propagated” • A lot can go wrong with all services properly creating exports every night.
  • 9. @samuelroze Event streaming • Events are flowing in real-time, from and to multiple services. • To receive a specific event, services don’t have to know who is sending events, just that they can expect these messages. • Much higher availability because data goes to the service that requires it when they are online. • (When bus does persistence) New consumers create their context by going through all the events that have happened in the system. • Writing code that works well with the nature of the distributed system is hard. • You need a real governance about how is the message bus used, they are your new API contracts.
  • 11. @samuelroze Everything we are going to talk about is true for…
  • 12. @samuelroze@samuelroze 2. What will go wrong? It’s not “if”.
  • 13. @samuelroze Let’s start with a simple use-case. Here we write on the `Basket` entity for example
  • 14. @samuelroze Your message is sent to a queue
  • 15. @samuelroze@samuelroze Problem A Are you sure that the message 
 was sent to the queue?
  • 16. @samuelroze A. Are you sure that your message has been sent?
  • 17. @samuelroze A. Are you sure that your message has been sent?
  • 18. @samuelroze A. Distributed transactions are not really a thing.
  • 19. @samuelroze A. What might happen if we don’t care about that? • Your local “basket” table might have the new product but no worker receives the “ProductAddedToBasket” event. • Your local “basket” table might NOT have the new product but workers have received the “ProductAddedToBasket” event (most likely if you use database transactions for the entire request) • Imagine the event being about “payment successful” or even potentially life- changing like “fall in the home has been detected”… 😬
  • 21. @samuelroze A. Publishing messages to bus consistently • In a nutshell, write your message & side effects to your database as part of one transaction and then get something else to pull the message from the database and send it to your queue. • With Symfony, the simplest is actually to use the Doctrine transport for Symfony Messenger, with the doctrine transaction middleware. • Alternatively, you can use a dedicated library for this. • EventSaucePHP/DoctrineOutboxMessageDispatcher • italolelis/outboxer
  • 22. @samuelroze A. Using the Doctrine transport
  • 23. @samuelroze@samuelroze Problem B You will receive duplicated messages.
  • 24. @samuelroze B. “At least once delivery”
  • 25. @samuelroze B. What might happen if we don’t care about that? • You consume twice “ProductAddedToBasket”: the product is added twice instead of just once (as per the user request). • Depending on your business logic, it might be very important. For example, what if it is about “Money added to bank account” or “Medication dose taken”.
  • 26. @samuelroze B. You need some idempotence. • You will receive the same message multiple times, it’s just a matter of time. • There isn’t much a framework could do, you own the business logic; you need to handle it by yourself. • Use an idempotency key. A key that represents a single message and allows you to know whether or not it’s been processed already. • (By the way, this also applies to HTTP requests. Stripe’s API is a good example.)
  • 27. @samuelroze B. Using the idempotency key in the handler • One option is to have your idempotency key is part of your message. Your team needs to know why it is useful and how to use it.
  • 28. @samuelroze B. How to use your “idempotency key”
  • 29. @samuelroze B. How to use your “idempotency key”
  • 32. @samuelroze C. What might happen if we don’t care about that? • You will lose some state in whatever you updated based on the events, at some point. • The easiest solution: don’t process things concurrently. But, not really practical when things start to scale.
  • 33. @samuelroze C. Locking! Optimistic vs Pessimistic The optimist… • Assumes that everything will go right most of the time. • It validates that everything has happened as expected when writing its state to a consistent storage. • a.k.a. HTTP’s If-Match, … The pessimist… • Believes that I most cases, this won’t work. • Before doing any work, it ensures nobody else is doing it. • a.k.a. “mutex”, “advisory locks”, etc…
  • 35. @samuelroze C. Optimistic locking with Doctrine’s “versions"
  • 36. @samuelroze C. Optimistic locking with Doctrine’s “versions" What’s happening behind the scene with optimistic locking:
  • 38. @samuelroze D. Know when there is no ordering guarantee
  • 39. @samuelroze D. What might happen if we don’t care about that? • Hopefully your business logic doesn’t rely too much on the events being ordered… make sure this is true. • For example, we rely on “access_granted” and “access_revoked” events to configure some permission rules. If they are consumed in the wrong order… this is a different meaning 💥
  • 40. @samuelroze D. There are buses that guarantee order
  • 41. @samuelroze D. They scale using partitions
  • 42. @samuelroze D. Order guaranteed means blocking messages.
  • 43. @samuelroze D. For the infrastructure to guarantee ordering… • You need a message bus that supports it (Kafka, SQS Fifo, Kinesis, etc…). • You need to carefully design your partitions (or “shards”) so that you know all message of a specific aggregate will always go to the same partition (a.k.a. routing keys). • You need to carefully manage all the errors. You can’t afford a wrong message blocking an entire topic. But you can’t really post-pone only one single message…
  • 44. @samuelroze@samuelroze To wrap up… A few learnings (hopefully).
  • 45. @samuelroze We’ve seen a few ways it can go wrong. • When publishing a message to a bus. Outbox pattern FTW. • When receiving multiple time the same message. Idempotence FTW. • When concurrently consuming messages.
 You need to use optimistic or pessimistic locking. • You can request ordering from your infrastructure. But needs careful partition design & error management.
  • 47. @samuelroze Want to read more? • Martin Kleppman’s book. https://dataintensive.net • https://multithreaded.stitchfix.com/blog/2017/06/26/patterns-of-soa- idempotency-key/ • https://microservices.io/patterns/data/transactional-outbox.html