SlideShare a Scribd company logo
@samuelroze
Event Streaming Some things you want to know about.
@samuelroze
@samuelroze
Introduction
• My name is Samuel Rozé, I am VPoE at Birdie Care.
Core Team member of Symfony, for my work on Messenger.
• This is an architecture talk.
• We will briefly discuss the values of using stream processing event
streaming.
• We will see the consequences of living the dream of managing distributed
systems. TL;DR: plenty of things will go wrong.
@samuelroze@samuelroze
1. Why …is event streaming even interesting?
@samuelroze
Your product works… you split your services.
@samuelroze
Your services are talking to each other.
@samuelroze
Now you need to introduce targeted discounts…
@samuelroze
Now you need to introduce targeted discounts…
@samuelroze
How will this service get its data?
1. Pull via an API
• A lot of data will be moved each
time the “discount” service computes
discounts for a customer.
• “Discounts” is able to work only
when the 3 other services are
available (cascading failures).
• “Discounts” needs to know about
where are the other services and
how to talk to them.
2. Using “batch”
• Potentially contains loads of duplicated
information (full load each time or the
period is “over X days”)
• Not real-time. “Wait a few days for your
marketing preferences to be propagated”
• A lot can go wrong with all services
properly creating exports every night.
@samuelroze
Event streaming
• Events are flowing in real-time, from and
to multiple services.
• To receive a specific event, services don’t
have to know who is sending events, just
that they can expect these messages.
• Much higher availability because data
goes to the service that requires it when they
are online.
• (When bus does persistence) New
consumers create their context by going
through all the events that have happened in
the system.
• Writing code that works well with the
nature of the distributed system is hard.
• You need a real governance about how
is the message bus used, they are your
new API contracts.
@samuelroze
Event streaming, as a diagram
@samuelroze
Everything we are going to talk about is true for…
@samuelroze@samuelroze
2. What will go wrong? It’s not “if”.
@samuelroze
Let’s start with a simple use-case.
Here we write on the `Basket` entity for example
@samuelroze
Your message is sent to a queue
@samuelroze@samuelroze
Problem A Are you sure that the message 

was sent to the queue?
@samuelroze
A. Are you sure that your message has been sent?
@samuelroze
A. Are you sure that your message has been sent?
@samuelroze
A. Distributed transactions are not really a thing.
@samuelroze
A. What might happen if we don’t care about that?
• Your local “basket” table might have the new product but no worker receives
the “ProductAddedToBasket” event.
• Your local “basket” table might NOT have the new product but workers have
received the “ProductAddedToBasket” event (most likely if you use
database transactions for the entire request)
• Imagine the event being about “payment successful” or even potentially life-
changing like “fall in the home has been detected”… 😬
@samuelroze
A. The outbox pattern
@samuelroze
A. Publishing messages to bus consistently
• In a nutshell, write your message & side effects to your database as part of
one transaction and then get something else to pull the message from the
database and send it to your queue.
• With Symfony, the simplest is actually to use the Doctrine transport for
Symfony Messenger, with the doctrine transaction middleware.
• Alternatively, you can use a dedicated library for this.
• EventSaucePHP/DoctrineOutboxMessageDispatcher
• italolelis/outboxer
@samuelroze
A. Using the Doctrine transport
@samuelroze@samuelroze
Problem B You will receive duplicated
messages.
@samuelroze
B. “At least once delivery”
@samuelroze
B. What might happen if we don’t care about that?
• You consume twice “ProductAddedToBasket”: the product is added twice
instead of just once (as per the user request).
• Depending on your business logic, it might be very important. For example,
what if it is about “Money added to bank account” or “Medication dose
taken”.
@samuelroze
B. You need some idempotence.
• You will receive the same message multiple times, it’s just a matter of time.
• There isn’t much a framework could do, you own the business logic; you
need to handle it by yourself.
• Use an idempotency key. A key that represents a single message and
allows you to know whether or not it’s been processed already.
• (By the way, this also applies to HTTP requests. Stripe’s API is a good
example.)
@samuelroze
B. Using the idempotency key in the handler
• One option is to have your idempotency key is part of your message. Your
team needs to know why it is useful and how to use it.
@samuelroze
B. How to use your “idempotency key”
@samuelroze
B. How to use your “idempotency key”
@samuelroze@samuelroze
Problem C Processing messages in parallel.
@samuelroze
C. Concurrently processing messages
@samuelroze
C. What might happen if we don’t care about that?
• You will lose some state in whatever you updated based on the events, at
some point.
• The easiest solution: don’t process things concurrently. But, not really
practical when things start to scale.
@samuelroze
C. Locking! Optimistic vs Pessimistic
The optimist…
• Assumes that everything will go
right most of the time.
• It validates that everything has
happened as expected when
writing its state to a consistent
storage.
• a.k.a. HTTP’s If-Match, …
The pessimist…
• Believes that I most cases, this
won’t work.
• Before doing any work, it ensures
nobody else is doing it.
• a.k.a. “mutex”, “advisory locks”,
etc…
@samuelroze
C. Pessimistic locking with Symfony Lock
@samuelroze
C. Optimistic locking with Doctrine’s “versions"
@samuelroze
C. Optimistic locking with Doctrine’s “versions"
What’s happening behind the scene
with optimistic locking:
@samuelroze@samuelroze
Problem D Message ordering
@samuelroze
D. Know when there is no ordering guarantee
@samuelroze
D. What might happen if we don’t care about that?
• Hopefully your business logic doesn’t rely too much on the events being
ordered… make sure this is true.
• For example, we rely on “access_granted” and “access_revoked” events to
configure some permission rules. If they are consumed in the wrong
order… this is a different meaning 💥
@samuelroze
D. There are buses that guarantee order
@samuelroze
D. They scale using partitions
@samuelroze
D. Order guaranteed means blocking messages.
@samuelroze
D. For the infrastructure to guarantee ordering…
• You need a message bus that supports it (Kafka, SQS Fifo, Kinesis, etc…).
• You need to carefully design your partitions (or “shards”) so that you know all
message of a specific aggregate will always go to the same partition (a.k.a.
routing keys).
• You need to carefully manage all the errors. You can’t afford a wrong
message blocking an entire topic. But you can’t really post-pone only one single message…
@samuelroze@samuelroze
To wrap up… A few learnings (hopefully).
@samuelroze
We’ve seen a few ways it can go wrong.
• When publishing a message to a bus.
Outbox pattern FTW.
• When receiving multiple time the same message.
Idempotence FTW.
• When concurrently consuming messages.

You need to use optimistic or pessimistic locking.
• You can request ordering from your infrastructure.
But needs careful partition design & error management.
@samuelroze
Thank you!
@samuelroze
@samuelroze
Want to read more?
• Martin Kleppman’s book.
https://dataintensive.net
• https://multithreaded.stitchfix.com/blog/2017/06/26/patterns-of-soa-
idempotency-key/
• https://microservices.io/patterns/data/transactional-outbox.html

More Related Content

Similar to Event streaming: what will go wrong? (Symfony World 2020)

Making operations visible - devopsdays tokyo 2013
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013
Nick Galbreath
 
Developing a Globally Distributed Purging System
Developing a Globally Distributed Purging SystemDeveloping a Globally Distributed Purging System
Developing a Globally Distributed Purging System
Fastly
 
Simple SAP Security Breach !!
Simple SAP Security Breach !!Simple SAP Security Breach !!
Simple SAP Security Breach !!
SAPYard
 
Event storage in a distributed system
Event storage in a distributed systemEvent storage in a distributed system
Event storage in a distributed system
Steve Pember
 
How to Pitch Bullshit
How to Pitch BullshitHow to Pitch Bullshit
How to Pitch Bullshit
Moritz Haarmann
 
10 impact of uncertainty in matching
10 impact of uncertainty in matching10 impact of uncertainty in matching
10 impact of uncertainty in matching
Rishi Mathur
 
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmxMoved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Milen Dyankov
 
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm
 
Sage Switch Guide
Sage Switch GuideSage Switch Guide
Sage Switch Guide
craig1201
 
I Want These * Bugs Off My * Internet
I Want These * Bugs Off My * InternetI Want These * Bugs Off My * Internet
I Want These * Bugs Off My * Internet
Dan Kaminsky
 
Project Planning and Estimation with User Stories
Project Planning and Estimation with User StoriesProject Planning and Estimation with User Stories
Project Planning and Estimation with User Stories
Polished Geek LLC
 
Grails Worst Practices
Grails Worst PracticesGrails Worst Practices
Grails Worst Practices
Burt Beckwith
 
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
 GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res... GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
James Anderson
 
Responsible Microservices
Responsible MicroservicesResponsible Microservices
Responsible Microservices
VMware Tanzu
 
Mobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B ExperimentsMobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B Experiments
lacyrhoades
 
Cucumber
CucumberCucumber
Cucumber
markjturner
 
TLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processingTLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processing
alooma
 
Fighting Spam at Flickr
Fighting Spam at FlickrFighting Spam at Flickr
Fighting Spam at Flickr
Mikhail Panchenko
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Brian Brazil
 
Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...
WeLoveSEO
 

Similar to Event streaming: what will go wrong? (Symfony World 2020) (20)

Making operations visible - devopsdays tokyo 2013
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013
 
Developing a Globally Distributed Purging System
Developing a Globally Distributed Purging SystemDeveloping a Globally Distributed Purging System
Developing a Globally Distributed Purging System
 
Simple SAP Security Breach !!
Simple SAP Security Breach !!Simple SAP Security Breach !!
Simple SAP Security Breach !!
 
Event storage in a distributed system
Event storage in a distributed systemEvent storage in a distributed system
Event storage in a distributed system
 
How to Pitch Bullshit
How to Pitch BullshitHow to Pitch Bullshit
How to Pitch Bullshit
 
10 impact of uncertainty in matching
10 impact of uncertainty in matching10 impact of uncertainty in matching
10 impact of uncertainty in matching
 
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmxMoved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
Moved to https://slidr.io/azzazzel/web-application-performance-tuning-beyond-xmx
 
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
Livestorm SaaSCast - Admin Automation - Florent Dougs (Dougs)
 
Sage Switch Guide
Sage Switch GuideSage Switch Guide
Sage Switch Guide
 
I Want These * Bugs Off My * Internet
I Want These * Bugs Off My * InternetI Want These * Bugs Off My * Internet
I Want These * Bugs Off My * Internet
 
Project Planning and Estimation with User Stories
Project Planning and Estimation with User StoriesProject Planning and Estimation with User Stories
Project Planning and Estimation with User Stories
 
Grails Worst Practices
Grails Worst PracticesGrails Worst Practices
Grails Worst Practices
 
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
 GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res... GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
GDG Cloud Southlake #6 Tammy Bryant Butow: Chaos Engineering The Road To Res...
 
Responsible Microservices
Responsible MicroservicesResponsible Microservices
Responsible Microservices
 
Mobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B ExperimentsMobile App Feature Configuration and A/B Experiments
Mobile App Feature Configuration and A/B Experiments
 
Cucumber
CucumberCucumber
Cucumber
 
TLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processingTLV Data Plumbers: Exactly once processing
TLV Data Plumbers: Exactly once processing
 
Fighting Spam at Flickr
Fighting Spam at FlickrFighting Spam at Flickr
Fighting Spam at Flickr
 
Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)Systems Monitoring with Prometheus (Devops Ireland April 2015)
Systems Monitoring with Prometheus (Devops Ireland April 2015)
 
Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...Alexis max-Creating a bot experience as good as your user experience - Alexis...
Alexis max-Creating a bot experience as good as your user experience - Alexis...
 

More from Samuel ROZE

How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patterns
Samuel ROZE
 
Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)
Samuel ROZE
 
Micro services may not be the best idea
Micro services may not be the best ideaMicro services may not be the best idea
Micro services may not be the best idea
Samuel ROZE
 
Introduction to CQRS and Event Sourcing
Introduction to CQRS and Event SourcingIntroduction to CQRS and Event Sourcing
Introduction to CQRS and Event Sourcing
Samuel ROZE
 
CQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony applicationCQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony application
Samuel ROZE
 
How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patterns
Samuel ROZE
 
Take care of our micro services
Take care of our micro servicesTake care of our micro services
Take care of our micro services
Samuel ROZE
 
(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance
Samuel ROZE
 
Using continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflowsUsing continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflows
Samuel ROZE
 
Symfony CoP: Form component
Symfony CoP: Form componentSymfony CoP: Form component
Symfony CoP: Form component
Samuel ROZE
 
Behat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than thatBehat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than that
Samuel ROZE
 
Docker orchestration with Kubernetes
Docker orchestration with KubernetesDocker orchestration with Kubernetes
Docker orchestration with Kubernetes
Samuel ROZE
 
Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer
Samuel ROZE
 

More from Samuel ROZE (13)

How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patterns
 
Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)Symfony Messenger (Symfony Live San Francisco)
Symfony Messenger (Symfony Live San Francisco)
 
Micro services may not be the best idea
Micro services may not be the best ideaMicro services may not be the best idea
Micro services may not be the best idea
 
Introduction to CQRS and Event Sourcing
Introduction to CQRS and Event SourcingIntroduction to CQRS and Event Sourcing
Introduction to CQRS and Event Sourcing
 
CQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony applicationCQRS and Event Sourcing in a Symfony application
CQRS and Event Sourcing in a Symfony application
 
How I started to love design patterns
How I started to love design patternsHow I started to love design patterns
How I started to love design patterns
 
Take care of our micro services
Take care of our micro servicesTake care of our micro services
Take care of our micro services
 
(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance(micro)services avec Symfony et Tolerance
(micro)services avec Symfony et Tolerance
 
Using continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflowsUsing continuouspipe to speed up our workflows
Using continuouspipe to speed up our workflows
 
Symfony CoP: Form component
Symfony CoP: Form componentSymfony CoP: Form component
Symfony CoP: Form component
 
Behat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than thatBehat c'est plus que ça | Behat is more than that
Behat c'est plus que ça | Behat is more than that
 
Docker orchestration with Kubernetes
Docker orchestration with KubernetesDocker orchestration with Kubernetes
Docker orchestration with Kubernetes
 
Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer Symfony et serialization avec JMS serializer
Symfony et serialization avec JMS serializer
 

Recently uploaded

Response & Safe AI at Summer School of AI at IIITH
Response & Safe AI at Summer School of AI at IIITHResponse & Safe AI at Summer School of AI at IIITH
Response & Safe AI at Summer School of AI at IIITH
IIIT Hyderabad
 
Ludo system project report management .pdf
Ludo  system project report management .pdfLudo  system project report management .pdf
Ludo system project report management .pdf
Kamal Acharya
 
Unit 1 Information Storage and Retrieval
Unit 1 Information Storage and RetrievalUnit 1 Information Storage and Retrieval
Unit 1 Information Storage and Retrieval
KishorMahale5
 
Unblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen FramesUnblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen Frames
Sinan KOZAK
 
Trends in Computer Aided Design and MFG.
Trends in Computer Aided Design and MFG.Trends in Computer Aided Design and MFG.
Trends in Computer Aided Design and MFG.
Tool and Die Tech
 
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
YanKing2
 
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
PradeepKumarSK3
 
Chlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptxChlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptx
yadavsuyash008
 
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-IDUNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
GOWSIKRAJA PALANISAMY
 
DBMS Commands DDL DML DCL ENTITY RELATIONSHIP.pptx
DBMS Commands  DDL DML DCL ENTITY RELATIONSHIP.pptxDBMS Commands  DDL DML DCL ENTITY RELATIONSHIP.pptx
DBMS Commands DDL DML DCL ENTITY RELATIONSHIP.pptx
Tulasi72
 
Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in City
sunnuchadda
 
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptxPresentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Er. Kushal Ghimire
 
Design and Application of Side Channel Spillways
Design and Application of Side Channel SpillwaysDesign and Application of Side Channel Spillways
Design and Application of Side Channel Spillways
ahmed42488
 
Evento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recapEvento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recap
Rafael Santos
 
libro de modelado de diseño-part-1[160-250].pdf
libro de modelado de diseño-part-1[160-250].pdflibro de modelado de diseño-part-1[160-250].pdf
libro de modelado de diseño-part-1[160-250].pdf
celiosilva66
 
OSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag outOSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag out
Ateeb19
 
Quadcopter Dynamics, Stability and Control
Quadcopter Dynamics, Stability and ControlQuadcopter Dynamics, Stability and Control
Quadcopter Dynamics, Stability and Control
Blesson Easo Varghese
 
Lecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............pptLecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............ppt
RujanTimsina1
 
Synthetic Test Collections for Retrieval Evaluation (Poster)
Synthetic Test Collections for Retrieval Evaluation (Poster)Synthetic Test Collections for Retrieval Evaluation (Poster)
Synthetic Test Collections for Retrieval Evaluation (Poster)
Hossein A. (Saeed) Rahmani
 
Conservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptxConservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptx
AdarshaMR1
 

Recently uploaded (20)

Response & Safe AI at Summer School of AI at IIITH
Response & Safe AI at Summer School of AI at IIITHResponse & Safe AI at Summer School of AI at IIITH
Response & Safe AI at Summer School of AI at IIITH
 
Ludo system project report management .pdf
Ludo  system project report management .pdfLudo  system project report management .pdf
Ludo system project report management .pdf
 
Unit 1 Information Storage and Retrieval
Unit 1 Information Storage and RetrievalUnit 1 Information Storage and Retrieval
Unit 1 Information Storage and Retrieval
 
Unblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen FramesUnblocking The Main Thread - Solving ANRs and Frozen Frames
Unblocking The Main Thread - Solving ANRs and Frozen Frames
 
Trends in Computer Aided Design and MFG.
Trends in Computer Aided Design and MFG.Trends in Computer Aided Design and MFG.
Trends in Computer Aided Design and MFG.
 
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large...
 
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
21EC63_Module1B.pptx VLSI design 21ec63 MOS TRANSISTOR THEORY
 
Chlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptxChlorine and Nitric Acid application, properties, impacts.pptx
Chlorine and Nitric Acid application, properties, impacts.pptx
 
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-IDUNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
UNIT I INCEPTION OF INFORMATION DESIGN 20CDE09-ID
 
DBMS Commands DDL DML DCL ENTITY RELATIONSHIP.pptx
DBMS Commands  DDL DML DCL ENTITY RELATIONSHIP.pptxDBMS Commands  DDL DML DCL ENTITY RELATIONSHIP.pptx
DBMS Commands DDL DML DCL ENTITY RELATIONSHIP.pptx
 
Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 in City
 
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptxPresentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
Presentation slide on DESIGN AND FABRICATION OF MOBILE CONTROLLED DRAINAGE.pptx
 
Design and Application of Side Channel Spillways
Design and Application of Side Channel SpillwaysDesign and Application of Side Channel Spillways
Design and Application of Side Channel Spillways
 
Evento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recapEvento anual Splunk .conf24 Highlights recap
Evento anual Splunk .conf24 Highlights recap
 
libro de modelado de diseño-part-1[160-250].pdf
libro de modelado de diseño-part-1[160-250].pdflibro de modelado de diseño-part-1[160-250].pdf
libro de modelado de diseño-part-1[160-250].pdf
 
OSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag outOSHA LOTO training, LOTO, lock out tag out
OSHA LOTO training, LOTO, lock out tag out
 
Quadcopter Dynamics, Stability and Control
Quadcopter Dynamics, Stability and ControlQuadcopter Dynamics, Stability and Control
Quadcopter Dynamics, Stability and Control
 
Lecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............pptLecture 3 Biomass energy...............ppt
Lecture 3 Biomass energy...............ppt
 
Synthetic Test Collections for Retrieval Evaluation (Poster)
Synthetic Test Collections for Retrieval Evaluation (Poster)Synthetic Test Collections for Retrieval Evaluation (Poster)
Synthetic Test Collections for Retrieval Evaluation (Poster)
 
Conservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptxConservation of Natural Resources Biodiversity.pptx
Conservation of Natural Resources Biodiversity.pptx
 

Event streaming: what will go wrong? (Symfony World 2020)

  • 1. @samuelroze Event Streaming Some things you want to know about. @samuelroze
  • 2. @samuelroze Introduction • My name is Samuel Rozé, I am VPoE at Birdie Care. Core Team member of Symfony, for my work on Messenger. • This is an architecture talk. • We will briefly discuss the values of using stream processing event streaming. • We will see the consequences of living the dream of managing distributed systems. TL;DR: plenty of things will go wrong.
  • 3. @samuelroze@samuelroze 1. Why …is event streaming even interesting?
  • 4. @samuelroze Your product works… you split your services.
  • 5. @samuelroze Your services are talking to each other.
  • 6. @samuelroze Now you need to introduce targeted discounts…
  • 7. @samuelroze Now you need to introduce targeted discounts…
  • 8. @samuelroze How will this service get its data? 1. Pull via an API • A lot of data will be moved each time the “discount” service computes discounts for a customer. • “Discounts” is able to work only when the 3 other services are available (cascading failures). • “Discounts” needs to know about where are the other services and how to talk to them. 2. Using “batch” • Potentially contains loads of duplicated information (full load each time or the period is “over X days”) • Not real-time. “Wait a few days for your marketing preferences to be propagated” • A lot can go wrong with all services properly creating exports every night.
  • 9. @samuelroze Event streaming • Events are flowing in real-time, from and to multiple services. • To receive a specific event, services don’t have to know who is sending events, just that they can expect these messages. • Much higher availability because data goes to the service that requires it when they are online. • (When bus does persistence) New consumers create their context by going through all the events that have happened in the system. • Writing code that works well with the nature of the distributed system is hard. • You need a real governance about how is the message bus used, they are your new API contracts.
  • 11. @samuelroze Everything we are going to talk about is true for…
  • 12. @samuelroze@samuelroze 2. What will go wrong? It’s not “if”.
  • 13. @samuelroze Let’s start with a simple use-case. Here we write on the `Basket` entity for example
  • 14. @samuelroze Your message is sent to a queue
  • 15. @samuelroze@samuelroze Problem A Are you sure that the message 
 was sent to the queue?
  • 16. @samuelroze A. Are you sure that your message has been sent?
  • 17. @samuelroze A. Are you sure that your message has been sent?
  • 18. @samuelroze A. Distributed transactions are not really a thing.
  • 19. @samuelroze A. What might happen if we don’t care about that? • Your local “basket” table might have the new product but no worker receives the “ProductAddedToBasket” event. • Your local “basket” table might NOT have the new product but workers have received the “ProductAddedToBasket” event (most likely if you use database transactions for the entire request) • Imagine the event being about “payment successful” or even potentially life- changing like “fall in the home has been detected”… 😬
  • 21. @samuelroze A. Publishing messages to bus consistently • In a nutshell, write your message & side effects to your database as part of one transaction and then get something else to pull the message from the database and send it to your queue. • With Symfony, the simplest is actually to use the Doctrine transport for Symfony Messenger, with the doctrine transaction middleware. • Alternatively, you can use a dedicated library for this. • EventSaucePHP/DoctrineOutboxMessageDispatcher • italolelis/outboxer
  • 22. @samuelroze A. Using the Doctrine transport
  • 23. @samuelroze@samuelroze Problem B You will receive duplicated messages.
  • 24. @samuelroze B. “At least once delivery”
  • 25. @samuelroze B. What might happen if we don’t care about that? • You consume twice “ProductAddedToBasket”: the product is added twice instead of just once (as per the user request). • Depending on your business logic, it might be very important. For example, what if it is about “Money added to bank account” or “Medication dose taken”.
  • 26. @samuelroze B. You need some idempotence. • You will receive the same message multiple times, it’s just a matter of time. • There isn’t much a framework could do, you own the business logic; you need to handle it by yourself. • Use an idempotency key. A key that represents a single message and allows you to know whether or not it’s been processed already. • (By the way, this also applies to HTTP requests. Stripe’s API is a good example.)
  • 27. @samuelroze B. Using the idempotency key in the handler • One option is to have your idempotency key is part of your message. Your team needs to know why it is useful and how to use it.
  • 28. @samuelroze B. How to use your “idempotency key”
  • 29. @samuelroze B. How to use your “idempotency key”
  • 32. @samuelroze C. What might happen if we don’t care about that? • You will lose some state in whatever you updated based on the events, at some point. • The easiest solution: don’t process things concurrently. But, not really practical when things start to scale.
  • 33. @samuelroze C. Locking! Optimistic vs Pessimistic The optimist… • Assumes that everything will go right most of the time. • It validates that everything has happened as expected when writing its state to a consistent storage. • a.k.a. HTTP’s If-Match, … The pessimist… • Believes that I most cases, this won’t work. • Before doing any work, it ensures nobody else is doing it. • a.k.a. “mutex”, “advisory locks”, etc…
  • 35. @samuelroze C. Optimistic locking with Doctrine’s “versions"
  • 36. @samuelroze C. Optimistic locking with Doctrine’s “versions" What’s happening behind the scene with optimistic locking:
  • 38. @samuelroze D. Know when there is no ordering guarantee
  • 39. @samuelroze D. What might happen if we don’t care about that? • Hopefully your business logic doesn’t rely too much on the events being ordered… make sure this is true. • For example, we rely on “access_granted” and “access_revoked” events to configure some permission rules. If they are consumed in the wrong order… this is a different meaning 💥
  • 40. @samuelroze D. There are buses that guarantee order
  • 41. @samuelroze D. They scale using partitions
  • 42. @samuelroze D. Order guaranteed means blocking messages.
  • 43. @samuelroze D. For the infrastructure to guarantee ordering… • You need a message bus that supports it (Kafka, SQS Fifo, Kinesis, etc…). • You need to carefully design your partitions (or “shards”) so that you know all message of a specific aggregate will always go to the same partition (a.k.a. routing keys). • You need to carefully manage all the errors. You can’t afford a wrong message blocking an entire topic. But you can’t really post-pone only one single message…
  • 44. @samuelroze@samuelroze To wrap up… A few learnings (hopefully).
  • 45. @samuelroze We’ve seen a few ways it can go wrong. • When publishing a message to a bus. Outbox pattern FTW. • When receiving multiple time the same message. Idempotence FTW. • When concurrently consuming messages.
 You need to use optimistic or pessimistic locking. • You can request ordering from your infrastructure. But needs careful partition design & error management.
  • 47. @samuelroze Want to read more? • Martin Kleppman’s book. https://dataintensive.net • https://multithreaded.stitchfix.com/blog/2017/06/26/patterns-of-soa- idempotency-key/ • https://microservices.io/patterns/data/transactional-outbox.html