SlideShare a Scribd company logo
Event-Sourcing 

in an Eventually Consistent world 
Building Reactive Microservices 

for the Unreliable Universe of IoT and Mobile
Lorenzo Nicora
muCon London 2017
Lorenzo Nicora - @nicusXmuCon London 2017
Lorenzo Nicora
Platform Engineer
2
@nicusX
lorenzo.nicora@wipro.com
•Microservices
•Cloud-native
•Event-driven, Reactive
•Java, Spring, Akka; Node.js, Serverless
Lorenzo Nicora - @nicusXmuCon London 2017
Event Sourcing?
3
Lorenzo Nicora - @nicusXmuCon London 2017
Legacy persistence
4
Request/Response <—> Synchronous
Monolith
(single node)
Lorenzo Nicora - @nicusXmuCon London 2017
More client
request!
5
Lorenzo Nicora - @nicusXmuCon London 2017
Scale-out application
6
Clustered

Monolith
Lorenzo Nicora - @nicusXmuCon London 2017 7
ACID Tx —> Locks —> Contention
Get worse scaling out!
Signal propagation —> more latency
<— Req/Resp
- throughput

+ latency
Blocking

clients
<—
Lorenzo Nicora - @nicusXmuCon London 2017 8
Reactive
https://www.reactivemanifesto.org
Lorenzo Nicora - @nicusXmuCon London 2017
Let’s go async! Message-driven!
9
Write latency
Lorenzo Nicora - @nicusXmuCon London 2017
Updates order

matters!
10
Lorenzo Nicora - @nicusXmuCon London 2017 11
Lorenzo Nicora - @nicusXmuCon London 2017 12
Updating State
We are losing information
Cannot reconstruct consistency
Lorenzo Nicora - @nicusXmuCon London 2017
“Update-sourcing”
13
Lorenzo Nicora - @nicusXmuCon London 2017 14
- - - - - - - - - - - -
C
Q
Command
Query
C.Q.(R.)S.
Lorenzo Nicora - @nicusXmuCon London 2017
Where & How

we enforce

consistency?
15
Lorenzo Nicora - @nicusXmuCon London 2017
DDD: Aggregate
16
Lorenzo Nicora - @nicusXmuCon London 2017
The DDD approach
17
Event Sourcing
Lorenzo Nicora - @nicusXmuCon London 2017
“Submit Order X!”
• A request
• Will happen, possibly
• May fail, 

be rejected
18
“Order X submitted”
• Has happened,

in the past
• Cannot fail, 

nor be rejected
Command Event
Lorenzo Nicora - @nicusXmuCon London 2017
each Aggregate instance

Stateful

&

Singe-threaded

Processes one Command at a time
19
Actors!
Lorenzo Nicora - @nicusXmuCon London 2017 20
Lorenzo Nicora - @nicusXmuCon London 2017 21
Microservices - Distributed systems
(OptimisticLocking+Retries)
(GlobalPessimisticLocking)
Shared,in-memorydatastore
Write latency
Lorenzo Nicora - @nicusXmuCon London 2017
What about
queries?
22
Lorenzo Nicora - @nicusXmuCon London 2017 23
Lorenzo Nicora - @nicusXmuCon London 2017
Read Models
24
Weakly
Persistent
Source

of

Truth
Lorenzo Nicora - @nicusXmuCon London 2017 25
Lorenzo Nicora - @nicusXmuCon London 2017
Read Models



represent the state of the domain
26
…for supporting users decisions
…for external systems
…for other microservices
Lorenzo Nicora - @nicusXmuCon London 2017
Read Models

are based on Events,

things that have happened
27
Lorenzo Nicora - @nicusXmuCon London 2017
Where 

do things happen?
28
Lorenzo Nicora - @nicusXmuCon London 2017 29
Our

system
The

external 

world
Lorenzo Nicora - @nicusXmuCon London 2017 30
Enforce Invariants
Transactional Consistency
at write time
Lorenzo Nicora - @nicusXmuCon London 2017 31
Lorenzo Nicora - @nicusXmuCon London 2017
When domain is
•Process-oriented
•Transactional
•Strong consistency
32
Bank current account

E-commerce

…
Lorenzo Nicora - @nicusXmuCon London 2017
We are
controlling
the Process
33
Lorenzo Nicora - @nicusXmuCon London 2017
Different domains:
•We collect external events
•Unreliable event sources
•Unreliable transport
34
Lorenzo Nicora - @nicusXmuCon London 2017
•Logistics, Tracking 

(track deliveries, vehicles…)
•IoT
•Mobile apps 

Unreliable network

Temporarily disconnected
35
Lorenzo Nicora - @nicusXmuCon London 2017
Things happen 

in the external world
36
We try to draw 

a picture of the external world 

and make sense of it
Lorenzo Nicora - @nicusXmuCon London 2017 37
Lorenzo Nicora - @nicusXmuCon London 2017 38
Cmd —> enforce

consistency
—> Event
Relies on:
• Command Order
• Receiving all Commands
Lorenzo Nicora - @nicusXmuCon London 2017 39
Write Fast…
Think later
Lorenzo Nicora - @nicusXmuCon London 2017
~ ACID vs BASE
40
Atomic
Consistent
Isolated
Durable
Basic Availability
Soft-state
Eventual consistency
Lorenzo Nicora - @nicusXmuCon London 2017 41
(External)

Event —>
(Internal)

—> Event
validation

…
Stateless
One event at a time
+++ Scalability
Lorenzo Nicora - @nicusXmuCon London 2017 42
Lorenzo Nicora - @nicusXmuCon London 2017
Do not try

to make sense of Events

on write
43
Lorenzo Nicora - @nicusXmuCon London 2017
Try to build

consistent

Read Models
44
You can rebuild Read Models

when you spot an inconsistency
Lorenzo Nicora - @nicusXmuCon London 2017
•Reorder Events
•Guess missing Events
•Rebuild Read Models on late Events
•Delay Read Models
* Give events time to arrive
* (Watermarks)
45
Lorenzo Nicora - @nicusXmuCon London 2017 46
?
Lorenzo Nicora - @nicusXmuCon London 2017
Building Read Models 

is expensive
and takes time
47
Lorenzo Nicora - @nicusXmuCon London 2017 48
< Process Writes
< Serve Queries
< Build Read Models
Separate

M
icro
services
Lorenzo Nicora - @nicusXmuCon London 2017
• Scales independently
• Asynchronous 

(no time-coupling)
49
Lorenzo Nicora - @nicusXmuCon London 2017
Read Models
are always
Delayed
50
Lorenzo Nicora - @nicusXmuCon London 2017
What’s
the right order?
51
When building Read Models
We may reorder Events,
but…
Lorenzo Nicora - @nicusXmuCon London 2017
Order Events

by Timestamp
52
Which timestamp?
but…
Lorenzo Nicora - @nicusXmuCon London 2017 53
Received
timestamp
Processed
timestamp
Event store
timestamp
Event
timestamp
Source
Our system
Unreliable
Lorenzo Nicora - @nicusXmuCon London 2017 54
Event timestamp
by Event Source, Device
Reliable per-Device only
Lorenzo Nicora - @nicusXmuCon London 2017
Device = Aggregate
“local” Event order
55
Device != Aggregate
No easy Event order
No reliable, global 

Event ordering
Lorenzo Nicora - @nicusXmuCon London 2017 56
Reorder Events 

by causal relation
Not Trivial
“We received A after B, 

but A must happen before B”
Causal Consistency
Lorenzo Nicora - @nicusXmuCon London 2017
Wrap Up
57
Lorenzo Nicora - @nicusXmuCon London 2017
> Distributed systems, microservices
> Scalability is critical
> Low-latency concurrent writes
> Domain fits Event Sourcing
58
Lorenzo Nicora - @nicusXmuCon London 2017 59
DDD approach
Command —> —> Event
stateful

Aggregate
Transactional Consistency
at Write time
Event log
Lorenzo Nicora - @nicusXmuCon London 2017
Things happen

inside our system
Our system reacts to

Commands received
60
Lorenzo Nicora - @nicusXmuCon London 2017
+ Process-oriented systems
+ Strong consistency
- External unreliability
- Scale out 

& low-latency concurrent writes
61
We drive the process
Lorenzo Nicora - @nicusXmuCon London 2017 62
Weak write consistency

(Write fast, think later)
1. Lenient & stateless validation
on write

Command —> Event
2. Enforce consistency

when building Read Models
Lorenzo Nicora - @nicusXmuCon London 2017
Things happen

in the outside world
Our system draws 

pictures of the world
63
Lorenzo Nicora - @nicusXmuCon London 2017
+ Scales out well

& low-latency writes
+ External unreliability
- Delayed Read Models
- No Write-time transaction
64
We are not driving the process
Lorenzo Nicora - @nicusXmuCon London 2017
The world is eventually consistent
The world is unreliable
We are always looking at the past
65
“Guess -> Apologise -> Compensate”
- Jonas Bonér
Lorenzo Nicora - @nicusXmuCon London 2017
Thanks!
66
The video of this presentation 

will soon be available on the conference website:

https://skillsmatter.com/conferences/8549-con-2017-the-microservices-conference#skillscasts

More Related Content

Similar to Event Sourcing in an Eventually Consistent World

Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
OpenCredo
 
The Art of Container Monitoring
The Art of Container MonitoringThe Art of Container Monitoring
The Art of Container Monitoring
Derek Chen
 
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud ComplianceDevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon
 
SDN :: Software Defined Networking –2017 Executive Overview
SDN :: Software Defined Networking –2017 Executive OverviewSDN :: Software Defined Networking –2017 Executive Overview
SDN :: Software Defined Networking –2017 Executive Overview
Christian Esteve Rothenberg
 
BISmark : the broadband internet service benchmark
BISmark : the broadband internet service benchmarkBISmark : the broadband internet service benchmark
BISmark : the broadband internet service benchmark
NaLUG
 
ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)
ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)
ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)
Dina Goldshtein
 
Blockchain Technology
Blockchain TechnologyBlockchain Technology
Blockchain Technology
Giuseppe Andreetti
 
Collecting 600M events/day
Collecting 600M events/dayCollecting 600M events/day
Collecting 600M events/day
Lars Marius Garshol
 
Building Big Architectures by Ramit Surana
Building Big Architectures by Ramit SuranaBuilding Big Architectures by Ramit Surana
Building Big Architectures by Ramit Surana
XP Conference India
 
Building Big Architectures XP Conference 2016
Building Big Architectures XP Conference 2016Building Big Architectures XP Conference 2016
Building Big Architectures XP Conference 2016
Ramit Surana
 
Micro Front-End & Microservices - Plansoft
Micro Front-End & Microservices - PlansoftMicro Front-End & Microservices - Plansoft
Micro Front-End & Microservices - Plansoft
Miki Lombardi
 
Towards Recovering the Software Architecture of Microservice-based Systems
Towards Recovering the Software Architecture of Microservice-based SystemsTowards Recovering the Software Architecture of Microservice-based Systems
Towards Recovering the Software Architecture of Microservice-based Systems
Paolo Di Francesco
 
What's New in Hyper-V 2016 - Thomas Maurer
What's New in Hyper-V 2016 - Thomas MaurerWhat's New in Hyper-V 2016 - Thomas Maurer
What's New in Hyper-V 2016 - Thomas Maurer
ITCamp
 
Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)
Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)
Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)
Getting value from IoT, Integration and Data Analytics
 
Cloud Native Applications on Kubernetes: a DevOps Approach
Cloud Native Applications on Kubernetes: a DevOps ApproachCloud Native Applications on Kubernetes: a DevOps Approach
Cloud Native Applications on Kubernetes: a DevOps Approach
Nicola Ferraro
 
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
Salesforce Developers
 
CHIPS Alliance_Object Automation Inc_workshop
CHIPS Alliance_Object Automation Inc_workshopCHIPS Alliance_Object Automation Inc_workshop
CHIPS Alliance_Object Automation Inc_workshop
Object Automation
 
The Pink road – Dorothy’s journey through an all pink wonderland
The Pink road – Dorothy’s journey through an all pink wonderlandThe Pink road – Dorothy’s journey through an all pink wonderland
The Pink road – Dorothy’s journey through an all pink wonderland
LetsConnect
 
SecOps Workshop (Gregory Pickett)
SecOps Workshop (Gregory Pickett)SecOps Workshop (Gregory Pickett)
SecOps Workshop (Gregory Pickett)
Priyanka Aash
 
TCP1P.net Meetup Vision, Objectives and Roadmap
TCP1P.net Meetup Vision, Objectives and RoadmapTCP1P.net Meetup Vision, Objectives and Roadmap
TCP1P.net Meetup Vision, Objectives and Roadmap
Stefan Ianta
 

Similar to Event Sourcing in an Eventually Consistent World (20)

Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
 
The Art of Container Monitoring
The Art of Container MonitoringThe Art of Container Monitoring
The Art of Container Monitoring
 
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud ComplianceDevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
 
SDN :: Software Defined Networking –2017 Executive Overview
SDN :: Software Defined Networking –2017 Executive OverviewSDN :: Software Defined Networking –2017 Executive Overview
SDN :: Software Defined Networking –2017 Executive Overview
 
BISmark : the broadband internet service benchmark
BISmark : the broadband internet service benchmarkBISmark : the broadband internet service benchmark
BISmark : the broadband internet service benchmark
 
ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)
ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)
ETW - Monitor Anything, Anytime, Anywhere (NDC Oslo 2017)
 
Blockchain Technology
Blockchain TechnologyBlockchain Technology
Blockchain Technology
 
Collecting 600M events/day
Collecting 600M events/dayCollecting 600M events/day
Collecting 600M events/day
 
Building Big Architectures by Ramit Surana
Building Big Architectures by Ramit SuranaBuilding Big Architectures by Ramit Surana
Building Big Architectures by Ramit Surana
 
Building Big Architectures XP Conference 2016
Building Big Architectures XP Conference 2016Building Big Architectures XP Conference 2016
Building Big Architectures XP Conference 2016
 
Micro Front-End & Microservices - Plansoft
Micro Front-End & Microservices - PlansoftMicro Front-End & Microservices - Plansoft
Micro Front-End & Microservices - Plansoft
 
Towards Recovering the Software Architecture of Microservice-based Systems
Towards Recovering the Software Architecture of Microservice-based SystemsTowards Recovering the Software Architecture of Microservice-based Systems
Towards Recovering the Software Architecture of Microservice-based Systems
 
What's New in Hyper-V 2016 - Thomas Maurer
What's New in Hyper-V 2016 - Thomas MaurerWhat's New in Hyper-V 2016 - Thomas Maurer
What's New in Hyper-V 2016 - Thomas Maurer
 
Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)
Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)
Oracle OpenWorld 2017 Review (31st October 2017 - 250 slides)
 
Cloud Native Applications on Kubernetes: a DevOps Approach
Cloud Native Applications on Kubernetes: a DevOps ApproachCloud Native Applications on Kubernetes: a DevOps Approach
Cloud Native Applications on Kubernetes: a DevOps Approach
 
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
 
CHIPS Alliance_Object Automation Inc_workshop
CHIPS Alliance_Object Automation Inc_workshopCHIPS Alliance_Object Automation Inc_workshop
CHIPS Alliance_Object Automation Inc_workshop
 
The Pink road – Dorothy’s journey through an all pink wonderland
The Pink road – Dorothy’s journey through an all pink wonderlandThe Pink road – Dorothy’s journey through an all pink wonderland
The Pink road – Dorothy’s journey through an all pink wonderland
 
SecOps Workshop (Gregory Pickett)
SecOps Workshop (Gregory Pickett)SecOps Workshop (Gregory Pickett)
SecOps Workshop (Gregory Pickett)
 
TCP1P.net Meetup Vision, Objectives and Roadmap
TCP1P.net Meetup Vision, Objectives and RoadmapTCP1P.net Meetup Vision, Objectives and Roadmap
TCP1P.net Meetup Vision, Objectives and Roadmap
 

Recently uploaded

June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Webinar: Designing a schema for a Data Warehouse
Webinar: Designing a schema for a Data WarehouseWebinar: Designing a schema for a Data Warehouse
Webinar: Designing a schema for a Data Warehouse
Federico Razzoli
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
jpupo2018
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 

Recently uploaded (20)

June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Webinar: Designing a schema for a Data Warehouse
Webinar: Designing a schema for a Data WarehouseWebinar: Designing a schema for a Data Warehouse
Webinar: Designing a schema for a Data Warehouse
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 

Event Sourcing in an Eventually Consistent World

  • 1. Event-Sourcing 
 in an Eventually Consistent world  Building Reactive Microservices 
 for the Unreliable Universe of IoT and Mobile Lorenzo Nicora muCon London 2017
  • 2. Lorenzo Nicora - @nicusXmuCon London 2017 Lorenzo Nicora Platform Engineer 2 @nicusX lorenzo.nicora@wipro.com •Microservices •Cloud-native •Event-driven, Reactive •Java, Spring, Akka; Node.js, Serverless
  • 3. Lorenzo Nicora - @nicusXmuCon London 2017 Event Sourcing? 3
  • 4. Lorenzo Nicora - @nicusXmuCon London 2017 Legacy persistence 4 Request/Response <—> Synchronous Monolith (single node)
  • 5. Lorenzo Nicora - @nicusXmuCon London 2017 More client request! 5
  • 6. Lorenzo Nicora - @nicusXmuCon London 2017 Scale-out application 6 Clustered
 Monolith
  • 7. Lorenzo Nicora - @nicusXmuCon London 2017 7 ACID Tx —> Locks —> Contention Get worse scaling out! Signal propagation —> more latency <— Req/Resp - throughput
 + latency Blocking
 clients <—
  • 8. Lorenzo Nicora - @nicusXmuCon London 2017 8 Reactive https://www.reactivemanifesto.org
  • 9. Lorenzo Nicora - @nicusXmuCon London 2017 Let’s go async! Message-driven! 9 Write latency
  • 10. Lorenzo Nicora - @nicusXmuCon London 2017 Updates order
 matters! 10
  • 11. Lorenzo Nicora - @nicusXmuCon London 2017 11
  • 12. Lorenzo Nicora - @nicusXmuCon London 2017 12 Updating State We are losing information Cannot reconstruct consistency
  • 13. Lorenzo Nicora - @nicusXmuCon London 2017 “Update-sourcing” 13
  • 14. Lorenzo Nicora - @nicusXmuCon London 2017 14 - - - - - - - - - - - - C Q Command Query C.Q.(R.)S.
  • 15. Lorenzo Nicora - @nicusXmuCon London 2017 Where & How
 we enforce
 consistency? 15
  • 16. Lorenzo Nicora - @nicusXmuCon London 2017 DDD: Aggregate 16
  • 17. Lorenzo Nicora - @nicusXmuCon London 2017 The DDD approach 17 Event Sourcing
  • 18. Lorenzo Nicora - @nicusXmuCon London 2017 “Submit Order X!” • A request • Will happen, possibly • May fail, 
 be rejected 18 “Order X submitted” • Has happened,
 in the past • Cannot fail, 
 nor be rejected Command Event
  • 19. Lorenzo Nicora - @nicusXmuCon London 2017 each Aggregate instance
 Stateful
 &
 Singe-threaded
 Processes one Command at a time 19 Actors!
  • 20. Lorenzo Nicora - @nicusXmuCon London 2017 20
  • 21. Lorenzo Nicora - @nicusXmuCon London 2017 21 Microservices - Distributed systems (OptimisticLocking+Retries) (GlobalPessimisticLocking) Shared,in-memorydatastore Write latency
  • 22. Lorenzo Nicora - @nicusXmuCon London 2017 What about queries? 22
  • 23. Lorenzo Nicora - @nicusXmuCon London 2017 23
  • 24. Lorenzo Nicora - @nicusXmuCon London 2017 Read Models 24 Weakly
Persistent Source
 of
 Truth
  • 25. Lorenzo Nicora - @nicusXmuCon London 2017 25
  • 26. Lorenzo Nicora - @nicusXmuCon London 2017 Read Models
 
 represent the state of the domain 26 …for supporting users decisions …for external systems …for other microservices
  • 27. Lorenzo Nicora - @nicusXmuCon London 2017 Read Models
 are based on Events,
 things that have happened 27
  • 28. Lorenzo Nicora - @nicusXmuCon London 2017 Where 
 do things happen? 28
  • 29. Lorenzo Nicora - @nicusXmuCon London 2017 29 Our
 system The
 external 
 world
  • 30. Lorenzo Nicora - @nicusXmuCon London 2017 30 Enforce Invariants Transactional Consistency at write time
  • 31. Lorenzo Nicora - @nicusXmuCon London 2017 31
  • 32. Lorenzo Nicora - @nicusXmuCon London 2017 When domain is •Process-oriented •Transactional •Strong consistency 32 Bank current account
 E-commerce
 …
  • 33. Lorenzo Nicora - @nicusXmuCon London 2017 We are controlling the Process 33
  • 34. Lorenzo Nicora - @nicusXmuCon London 2017 Different domains: •We collect external events •Unreliable event sources •Unreliable transport 34
  • 35. Lorenzo Nicora - @nicusXmuCon London 2017 •Logistics, Tracking 
 (track deliveries, vehicles…) •IoT •Mobile apps 
 Unreliable network
 Temporarily disconnected 35
  • 36. Lorenzo Nicora - @nicusXmuCon London 2017 Things happen 
 in the external world 36 We try to draw 
 a picture of the external world 
 and make sense of it
  • 37. Lorenzo Nicora - @nicusXmuCon London 2017 37
  • 38. Lorenzo Nicora - @nicusXmuCon London 2017 38 Cmd —> enforce
 consistency —> Event Relies on: • Command Order • Receiving all Commands
  • 39. Lorenzo Nicora - @nicusXmuCon London 2017 39 Write Fast… Think later
  • 40. Lorenzo Nicora - @nicusXmuCon London 2017 ~ ACID vs BASE 40 Atomic Consistent Isolated Durable Basic Availability Soft-state Eventual consistency
  • 41. Lorenzo Nicora - @nicusXmuCon London 2017 41 (External)
 Event —> (Internal)
 —> Event validation
 … Stateless One event at a time +++ Scalability
  • 42. Lorenzo Nicora - @nicusXmuCon London 2017 42
  • 43. Lorenzo Nicora - @nicusXmuCon London 2017 Do not try
 to make sense of Events
 on write 43
  • 44. Lorenzo Nicora - @nicusXmuCon London 2017 Try to build
 consistent
 Read Models 44 You can rebuild Read Models
 when you spot an inconsistency
  • 45. Lorenzo Nicora - @nicusXmuCon London 2017 •Reorder Events •Guess missing Events •Rebuild Read Models on late Events •Delay Read Models * Give events time to arrive * (Watermarks) 45
  • 46. Lorenzo Nicora - @nicusXmuCon London 2017 46 ?
  • 47. Lorenzo Nicora - @nicusXmuCon London 2017 Building Read Models 
 is expensive and takes time 47
  • 48. Lorenzo Nicora - @nicusXmuCon London 2017 48 < Process Writes < Serve Queries < Build Read Models Separate
 M icro services
  • 49. Lorenzo Nicora - @nicusXmuCon London 2017 • Scales independently • Asynchronous 
 (no time-coupling) 49
  • 50. Lorenzo Nicora - @nicusXmuCon London 2017 Read Models are always Delayed 50
  • 51. Lorenzo Nicora - @nicusXmuCon London 2017 What’s the right order? 51 When building Read Models We may reorder Events, but…
  • 52. Lorenzo Nicora - @nicusXmuCon London 2017 Order Events
 by Timestamp 52 Which timestamp? but…
  • 53. Lorenzo Nicora - @nicusXmuCon London 2017 53 Received timestamp Processed timestamp Event store timestamp Event timestamp Source Our system Unreliable
  • 54. Lorenzo Nicora - @nicusXmuCon London 2017 54 Event timestamp by Event Source, Device Reliable per-Device only
  • 55. Lorenzo Nicora - @nicusXmuCon London 2017 Device = Aggregate “local” Event order 55 Device != Aggregate No easy Event order No reliable, global 
 Event ordering
  • 56. Lorenzo Nicora - @nicusXmuCon London 2017 56 Reorder Events 
 by causal relation Not Trivial “We received A after B, 
 but A must happen before B” Causal Consistency
  • 57. Lorenzo Nicora - @nicusXmuCon London 2017 Wrap Up 57
  • 58. Lorenzo Nicora - @nicusXmuCon London 2017 > Distributed systems, microservices > Scalability is critical > Low-latency concurrent writes > Domain fits Event Sourcing 58
  • 59. Lorenzo Nicora - @nicusXmuCon London 2017 59 DDD approach Command —> —> Event stateful
 Aggregate Transactional Consistency at Write time Event log
  • 60. Lorenzo Nicora - @nicusXmuCon London 2017 Things happen
 inside our system Our system reacts to
 Commands received 60
  • 61. Lorenzo Nicora - @nicusXmuCon London 2017 + Process-oriented systems + Strong consistency - External unreliability - Scale out 
 & low-latency concurrent writes 61 We drive the process
  • 62. Lorenzo Nicora - @nicusXmuCon London 2017 62 Weak write consistency
 (Write fast, think later) 1. Lenient & stateless validation on write
 Command —> Event 2. Enforce consistency
 when building Read Models
  • 63. Lorenzo Nicora - @nicusXmuCon London 2017 Things happen
 in the outside world Our system draws 
 pictures of the world 63
  • 64. Lorenzo Nicora - @nicusXmuCon London 2017 + Scales out well
 & low-latency writes + External unreliability - Delayed Read Models - No Write-time transaction 64 We are not driving the process
  • 65. Lorenzo Nicora - @nicusXmuCon London 2017 The world is eventually consistent The world is unreliable We are always looking at the past 65 “Guess -> Apologise -> Compensate” - Jonas Bonér
  • 66. Lorenzo Nicora - @nicusXmuCon London 2017 Thanks! 66 The video of this presentation 
 will soon be available on the conference website:
 https://skillsmatter.com/conferences/8549-con-2017-the-microservices-conference#skillscasts