SlideShare a Scribd company logo
Write fast, Think later 
Event Sourcing for building reliable micro services

in the unreliable universe of IoT and Mobile
Lorenzo Nicora
Microservices Meetup Munich - 01/2018
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Lorenzo Nicora
Platform Engineer lead
2
@nicusX
lorenzo.nicora@wipro.com
•Microservices
•Cloud-native (AWS)
•Event-driven, Reactive
•Java, SpringBoot, Akka; Node.js, Serverless
(London)
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Event Sourcing?
3
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Assumptions
4
Microservices
-> Distributed systems
-> Communication:
• Synchronous (req/resp, http, REST…)
• Asynchronous (messages)
-> Scale horizontally
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Assumptions
5
To Scale
Low-latency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Legacy persistence
6
Request/Response <—> Synchronous
Monolith
(single node)
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
More client
request!
7
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Scale-out application
8
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 9
ACID Tx —> Locks —> Contention
Does not improve scaling out!
Signal propagation —> more latency
<— Req/Resp
- throughput

+ latency
Blocking

clients
<—
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 10
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 11
Reactive
https://reactivemanifesto.org
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Let’s go async! Message-driven!
12
Write latency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Updates order

matters!
13
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 14
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 15
Updating State
We are losing information
Cannot
reconstruct consistency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
“Update-sourcing”
16
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 17
- - - - - - - - - - - -
C
Q
Command
Query
C.Q.(R.)S.
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Where & How

we enforce

consistency?
18
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
DDD: Aggregate
19
Consistent state —> Consistent state
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
The DDD approach
20
Event Sourcing
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
“Submit Order X!”
• A request
• Will happen, possibly
• May fail, be rejected
21
“Order X submitted”
• Has happened,

in the past
• Cannot fail, 

nor be rejected
Command Event
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
each Aggregate instance

Stateful

&

“Single-threaded”

Processes one Command at a time
22
Actors!
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 23
A456
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 24
Microservices - Distributed systems
(OptimisticLocking+Retries)
(GlobalPessimisticLocking)
Shared,in-memorydatastore
Write latency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
What about
queries?
25
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 26
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Read Models
27
Weakly
Persistent
Source

of

Truth
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 28
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Read Models



represent the state of the domain
29
…for supporting users decisions
…for external systems
…for other microservices
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Read Models

are based on Events,

things that have happened
30
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Where 

do things happen?
31
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 32
Our

system
The

external 

world
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 33
Enforce Invariants
Transactional Consistency
at write time
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 34
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
When domain is
•Process-oriented
•Transactional
•Strong consistency
35
Bank current account

E-commerce

…
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
We are 

in control
of the Process
36
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Different domains:
•We collect external events
•Unreliable event sources
•Unreliable transport
37
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
•Logistics, Tracking 

(track deliveries, vehicles…)
•IoT
•Mobile apps 

Unreliable network

Temporarily disconnected
38
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Things happen 

in the external world
39
We try to draw 

a picture of the external world 

and make sense of it
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 40
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 41
Cmd —> enforce

consistency
—> Event
Relies on:
• Command Order
• Receiving all Commands
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 42
Write Fast…
Think later
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
~ ACID vs BASE
43
Atomic
Consistent
Isolated
Durable
Basic Availability
Soft-state
Eventual consistency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 44
—> Eventvalidation

…
Stateless
One event at a time
+++ Scalability
Event —>
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 45
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Do not try

to make sense of Events

on write
46
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Try to build

consistent

Read Models
47
You can rebuild Read Models

when you spot an inconsistency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
•Reorder Events
•Guess missing Events
•Rebuild Read Models on late Events
•Delay Read Models
* Give events time to arrive
* (Watermarks)
48
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 49
?
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Building Read Models 

is expensive
and takes time
50
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 51
< Process Writes
< Serve Queries
< Build Read Models
Separate

M
icro
services
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
• Scales independently
• Asynchronous 

(no time-coupling)
52
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Read Models
are always
Delayed
53
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
What’s
the right order?
54
When building Read Models
We can reorder Events,
but…
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Order Events

by Timestamp
55
Which timestamp?
but…
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 56
Received
timestamp
Processed
timestamp
Event store
timestamp
Event
timestamp
Source
Our system
Unreliable
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 57
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 58
Event timestamp
by Event Source, Device
Reliable per-Device only
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Device = Aggregate
“local” Event order
59
Device != Aggregate
No easy Event order
No reliable, global 

Event ordering
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 60
Reorder Events 

by causal relation
Not Trivial
“We received A after B, 

but A must happen before B”
Causal Consistency
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Wrap Up
61
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
> Distributed systems, microservices
> Scalability is critical
> Low-latency concurrent writes
> Domain fits Event Sourcing
62
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 63
DDD approach
Command —> —> Event
stateful

Aggregate
Transactional Consistency
at Write time
Event log
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Things happen

inside our system
Our system reacts to

Commands received
64
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
+ Process-oriented systems
+ Strong consistency
- External unreliability
- Scale out 

& low-latency concurrent writes
65
We drive the process
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 66
Weak write consistency

(Write fast, think later)
1. Lenient & stateless validation
on write

Command —> Event
2. Enforce consistency

when building Read Models
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Things happen

in the outside world
Our system draws 

pictures of the world
67
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
+ Scales out well

& low-latency writes
+ External unreliability
- Delayed Read Models
- No Write-time transaction
68
We are not driving the process
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
The world is eventually consistent
The world is unreliable
We are always looking at the past
69
“Guess -> Apologise -> Compensate”
- Jonas Bonér
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 70
Credits:
Dominic Fox medium.com/@poetix
Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018
Thanks!
71

More Related Content

What's hot

Ledingkart Meetup #3: Security Basics for Developers
Ledingkart Meetup #3: Security Basics for DevelopersLedingkart Meetup #3: Security Basics for Developers
Ledingkart Meetup #3: Security Basics for Developers
Mukesh Singh
 
Blockchain and Internet of Things
Blockchain and Internet of ThingsBlockchain and Internet of Things
Blockchain and Internet of Things
Valerie Lampkin
 
blockchain and iot: Opportunities and Challanges
blockchain and iot: Opportunities and Challangesblockchain and iot: Opportunities and Challanges
blockchain and iot: Opportunities and Challanges
Chetan Kumar S
 
Blockchain: Background and Data61 Research Overview
Blockchain: Background and Data61 Research OverviewBlockchain: Background and Data61 Research Overview
Blockchain: Background and Data61 Research Overview
Ingo Weber
 
ArcBlock Presents An Introduction to Blockchain
ArcBlock Presents An Introduction to BlockchainArcBlock Presents An Introduction to Blockchain
ArcBlock Presents An Introduction to Blockchain
ArcBlock
 
Sullivan randomness-infiltrate 2014
Sullivan randomness-infiltrate 2014Sullivan randomness-infiltrate 2014
Sullivan randomness-infiltrate 2014
Cloudflare
 
Blockchain for IoT
Blockchain for IoTBlockchain for IoT
Blockchain for IoT
Federico Tenga
 
Microservices architecture presentation
Microservices architecture presentationMicroservices architecture presentation
Microservices architecture presentation
Joseph SHYIRAMBERE
 
Icsa2018 blockchain tutorial
Icsa2018 blockchain tutorialIcsa2018 blockchain tutorial
Icsa2018 blockchain tutorial
Len Bass
 
Sullivan white boxcrypto-baythreat-2013
Sullivan white boxcrypto-baythreat-2013Sullivan white boxcrypto-baythreat-2013
Sullivan white boxcrypto-baythreat-2013
Cloudflare
 
MRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo App
MRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo AppMRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo App
MRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo App
NGINX, Inc.
 
B.Noviansyah - National Public Key Infrastructure: Friend or Foe?
B.Noviansyah - National Public Key Infrastructure: Friend or Foe?B.Noviansyah - National Public Key Infrastructure: Friend or Foe?
B.Noviansyah - National Public Key Infrastructure: Friend or Foe?
idsecconf
 
Blockchain and Services – Exploring the Links
Blockchain and Services – Exploring the LinksBlockchain and Services – Exploring the Links
Blockchain and Services – Exploring the Links
Ingo Weber
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
Binu Ramakrishnan
 

What's hot (14)

Ledingkart Meetup #3: Security Basics for Developers
Ledingkart Meetup #3: Security Basics for DevelopersLedingkart Meetup #3: Security Basics for Developers
Ledingkart Meetup #3: Security Basics for Developers
 
Blockchain and Internet of Things
Blockchain and Internet of ThingsBlockchain and Internet of Things
Blockchain and Internet of Things
 
blockchain and iot: Opportunities and Challanges
blockchain and iot: Opportunities and Challangesblockchain and iot: Opportunities and Challanges
blockchain and iot: Opportunities and Challanges
 
Blockchain: Background and Data61 Research Overview
Blockchain: Background and Data61 Research OverviewBlockchain: Background and Data61 Research Overview
Blockchain: Background and Data61 Research Overview
 
ArcBlock Presents An Introduction to Blockchain
ArcBlock Presents An Introduction to BlockchainArcBlock Presents An Introduction to Blockchain
ArcBlock Presents An Introduction to Blockchain
 
Sullivan randomness-infiltrate 2014
Sullivan randomness-infiltrate 2014Sullivan randomness-infiltrate 2014
Sullivan randomness-infiltrate 2014
 
Blockchain for IoT
Blockchain for IoTBlockchain for IoT
Blockchain for IoT
 
Microservices architecture presentation
Microservices architecture presentationMicroservices architecture presentation
Microservices architecture presentation
 
Icsa2018 blockchain tutorial
Icsa2018 blockchain tutorialIcsa2018 blockchain tutorial
Icsa2018 blockchain tutorial
 
Sullivan white boxcrypto-baythreat-2013
Sullivan white boxcrypto-baythreat-2013Sullivan white boxcrypto-baythreat-2013
Sullivan white boxcrypto-baythreat-2013
 
MRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo App
MRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo AppMRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo App
MRA AMA: Ingenious: The Journey to Service Mesh using a Microservices Demo App
 
B.Noviansyah - National Public Key Infrastructure: Friend or Foe?
B.Noviansyah - National Public Key Infrastructure: Friend or Foe?B.Noviansyah - National Public Key Infrastructure: Friend or Foe?
B.Noviansyah - National Public Key Infrastructure: Friend or Foe?
 
Blockchain and Services – Exploring the Links
Blockchain and Services – Exploring the LinksBlockchain and Services – Exploring the Links
Blockchain and Services – Exploring the Links
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
 

Similar to Write fast, think later - Event sourcing for IoT and Mobile

From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
Lorenzo Nicora
 
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
OpenCredo
 
The Art of Container Monitoring
The Art of Container MonitoringThe Art of Container Monitoring
The Art of Container Monitoring
Derek Chen
 
Meetic back end redesign - Meetup microservices
Meetic back end redesign - Meetup microservicesMeetic back end redesign - Meetup microservices
Meetic back end redesign - Meetup microservices
inovia
 
Orchestrating microservices like a ninja
Orchestrating microservices like a ninjaOrchestrating microservices like a ninja
Orchestrating microservices like a ninja
Apigee | Google Cloud
 
Microevent
MicroeventMicroevent
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
Salesforce Developers
 
Neo4j: What's Under the Hood & How Knowing This Can Help You
Neo4j: What's Under the Hood & How Knowing This Can Help You Neo4j: What's Under the Hood & How Knowing This Can Help You
Neo4j: What's Under the Hood & How Knowing This Can Help You
Neo4j
 
Advanced applications with MongoDB
Advanced applications with MongoDBAdvanced applications with MongoDB
Advanced applications with MongoDB
Norberto Leite
 
Neo4j GraphTalks - Introduction to GraphDatabases and Neo4j
Neo4j GraphTalks - Introduction to GraphDatabases and Neo4jNeo4j GraphTalks - Introduction to GraphDatabases and Neo4j
Neo4j GraphTalks - Introduction to GraphDatabases and Neo4j
Neo4j
 
WTF is a Microservice - Rafael Schloming, Datawire
WTF is a Microservice - Rafael Schloming, DatawireWTF is a Microservice - Rafael Schloming, Datawire
WTF is a Microservice - Rafael Schloming, Datawire
Ambassador Labs
 
AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...
AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...
AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...
Getting value from IoT, Integration and Data Analytics
 
Blockchain, Integration, Serverless, Microservices - OOW / Code One 2018 Review
Blockchain, Integration, Serverless, Microservices - OOW / Code One 2018 ReviewBlockchain, Integration, Serverless, Microservices - OOW / Code One 2018 Review
Blockchain, Integration, Serverless, Microservices - OOW / Code One 2018 Review
Robert van Mölken
 
Microservices
MicroservicesMicroservices
Microservices
darkofabijan
 
The Gib Five - Modern IT Architecture
The Gib Five - Modern IT ArchitectureThe Gib Five - Modern IT Architecture
The Gib Five - Modern IT Architecture
Anatole Tresch
 
Voxxed Days Thesaloniki 2016 - Streaming Engines for Big Data
Voxxed Days Thesaloniki 2016 - Streaming Engines for Big DataVoxxed Days Thesaloniki 2016 - Streaming Engines for Big Data
Voxxed Days Thesaloniki 2016 - Streaming Engines for Big Data
Voxxed Days Thessaloniki
 
Voxxed days thessaloniki 21/10/2016 - Streaming Engines for Big Data
Voxxed days thessaloniki 21/10/2016 - Streaming Engines for Big DataVoxxed days thessaloniki 21/10/2016 - Streaming Engines for Big Data
Voxxed days thessaloniki 21/10/2016 - Streaming Engines for Big Data
Stavros Kontopoulos
 
Microservices at ibotta pitfalls and learnings
Microservices at ibotta pitfalls and learningsMicroservices at ibotta pitfalls and learnings
Microservices at ibotta pitfalls and learnings
Matthew Reynolds
 
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud ComplianceDevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon
 
New Business Models enabled by Blockchain
New Business Models enabled by BlockchainNew Business Models enabled by Blockchain
New Business Models enabled by Blockchain
Slash
 

Similar to Write fast, think later - Event sourcing for IoT and Mobile (20)

From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
From C to Q, one event at the time: Event sourcing illustrated [Voxxed Ticino...
 
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
Voxxed Bristol 2017 - From C to Q, one event at a time: Event Sourcing illust...
 
The Art of Container Monitoring
The Art of Container MonitoringThe Art of Container Monitoring
The Art of Container Monitoring
 
Meetic back end redesign - Meetup microservices
Meetic back end redesign - Meetup microservicesMeetic back end redesign - Meetup microservices
Meetic back end redesign - Meetup microservices
 
Orchestrating microservices like a ninja
Orchestrating microservices like a ninjaOrchestrating microservices like a ninja
Orchestrating microservices like a ninja
 
Microevent
MicroeventMicroevent
Microevent
 
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
How EA Games & Nurun used Node.js on Heroku to reach 350,000 fans during E3 2015
 
Neo4j: What's Under the Hood & How Knowing This Can Help You
Neo4j: What's Under the Hood & How Knowing This Can Help You Neo4j: What's Under the Hood & How Knowing This Can Help You
Neo4j: What's Under the Hood & How Knowing This Can Help You
 
Advanced applications with MongoDB
Advanced applications with MongoDBAdvanced applications with MongoDB
Advanced applications with MongoDB
 
Neo4j GraphTalks - Introduction to GraphDatabases and Neo4j
Neo4j GraphTalks - Introduction to GraphDatabases and Neo4jNeo4j GraphTalks - Introduction to GraphDatabases and Neo4j
Neo4j GraphTalks - Introduction to GraphDatabases and Neo4j
 
WTF is a Microservice - Rafael Schloming, Datawire
WTF is a Microservice - Rafael Schloming, DatawireWTF is a Microservice - Rafael Schloming, Datawire
WTF is a Microservice - Rafael Schloming, Datawire
 
AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...
AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...
AMIS Oracle OpenWorld en Code One Review 2018 - Blockchain, Integration, Serv...
 
Blockchain, Integration, Serverless, Microservices - OOW / Code One 2018 Review
Blockchain, Integration, Serverless, Microservices - OOW / Code One 2018 ReviewBlockchain, Integration, Serverless, Microservices - OOW / Code One 2018 Review
Blockchain, Integration, Serverless, Microservices - OOW / Code One 2018 Review
 
Microservices
MicroservicesMicroservices
Microservices
 
The Gib Five - Modern IT Architecture
The Gib Five - Modern IT ArchitectureThe Gib Five - Modern IT Architecture
The Gib Five - Modern IT Architecture
 
Voxxed Days Thesaloniki 2016 - Streaming Engines for Big Data
Voxxed Days Thesaloniki 2016 - Streaming Engines for Big DataVoxxed Days Thesaloniki 2016 - Streaming Engines for Big Data
Voxxed Days Thesaloniki 2016 - Streaming Engines for Big Data
 
Voxxed days thessaloniki 21/10/2016 - Streaming Engines for Big Data
Voxxed days thessaloniki 21/10/2016 - Streaming Engines for Big DataVoxxed days thessaloniki 21/10/2016 - Streaming Engines for Big Data
Voxxed days thessaloniki 21/10/2016 - Streaming Engines for Big Data
 
Microservices at ibotta pitfalls and learnings
Microservices at ibotta pitfalls and learningsMicroservices at ibotta pitfalls and learnings
Microservices at ibotta pitfalls and learnings
 
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud ComplianceDevSecCon London 2018: A Journey to Continuous Cloud Compliance
DevSecCon London 2018: A Journey to Continuous Cloud Compliance
 
New Business Models enabled by Blockchain
New Business Models enabled by BlockchainNew Business Models enabled by Blockchain
New Business Models enabled by Blockchain
 

Recently uploaded

Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 

Recently uploaded (20)

Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 

Write fast, think later - Event sourcing for IoT and Mobile

  • 1. Write fast, Think later  Event Sourcing for building reliable micro services
 in the unreliable universe of IoT and Mobile Lorenzo Nicora Microservices Meetup Munich - 01/2018
  • 2. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Lorenzo Nicora Platform Engineer lead 2 @nicusX lorenzo.nicora@wipro.com •Microservices •Cloud-native (AWS) •Event-driven, Reactive •Java, SpringBoot, Akka; Node.js, Serverless (London)
  • 3. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Event Sourcing? 3
  • 4. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Assumptions 4 Microservices -> Distributed systems -> Communication: • Synchronous (req/resp, http, REST…) • Asynchronous (messages) -> Scale horizontally
  • 5. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Assumptions 5 To Scale Low-latency
  • 6. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Legacy persistence 6 Request/Response <—> Synchronous Monolith (single node)
  • 7. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 More client request! 7
  • 8. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Scale-out application 8
  • 9. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 9 ACID Tx —> Locks —> Contention Does not improve scaling out! Signal propagation —> more latency <— Req/Resp - throughput
 + latency Blocking
 clients <—
  • 10. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 10
  • 11. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 11 Reactive https://reactivemanifesto.org
  • 12. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Let’s go async! Message-driven! 12 Write latency
  • 13. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Updates order
 matters! 13
  • 14. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 14
  • 15. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 15 Updating State We are losing information Cannot reconstruct consistency
  • 16. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 “Update-sourcing” 16
  • 17. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 17 - - - - - - - - - - - - C Q Command Query C.Q.(R.)S.
  • 18. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Where & How
 we enforce
 consistency? 18
  • 19. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 DDD: Aggregate 19 Consistent state —> Consistent state
  • 20. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 The DDD approach 20 Event Sourcing
  • 21. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 “Submit Order X!” • A request • Will happen, possibly • May fail, be rejected 21 “Order X submitted” • Has happened,
 in the past • Cannot fail, 
 nor be rejected Command Event
  • 22. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 each Aggregate instance
 Stateful
 &
 “Single-threaded”
 Processes one Command at a time 22 Actors!
  • 23. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 23 A456
  • 24. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 24 Microservices - Distributed systems (OptimisticLocking+Retries) (GlobalPessimisticLocking) Shared,in-memorydatastore Write latency
  • 25. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 What about queries? 25
  • 26. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 26
  • 27. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Read Models 27 Weakly
Persistent Source
 of
 Truth
  • 28. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 28
  • 29. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Read Models
 
 represent the state of the domain 29 …for supporting users decisions …for external systems …for other microservices
  • 30. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Read Models
 are based on Events,
 things that have happened 30
  • 31. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Where 
 do things happen? 31
  • 32. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 32 Our
 system The
 external 
 world
  • 33. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 33 Enforce Invariants Transactional Consistency at write time
  • 34. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 34
  • 35. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 When domain is •Process-oriented •Transactional •Strong consistency 35 Bank current account
 E-commerce
 …
  • 36. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 We are 
 in control of the Process 36
  • 37. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Different domains: •We collect external events •Unreliable event sources •Unreliable transport 37
  • 38. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 •Logistics, Tracking 
 (track deliveries, vehicles…) •IoT •Mobile apps 
 Unreliable network
 Temporarily disconnected 38
  • 39. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Things happen 
 in the external world 39 We try to draw 
 a picture of the external world 
 and make sense of it
  • 40. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 40
  • 41. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 41 Cmd —> enforce
 consistency —> Event Relies on: • Command Order • Receiving all Commands
  • 42. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 42 Write Fast… Think later
  • 43. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 ~ ACID vs BASE 43 Atomic Consistent Isolated Durable Basic Availability Soft-state Eventual consistency
  • 44. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 44 —> Eventvalidation
 … Stateless One event at a time +++ Scalability Event —>
  • 45. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 45
  • 46. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Do not try
 to make sense of Events
 on write 46
  • 47. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Try to build
 consistent
 Read Models 47 You can rebuild Read Models
 when you spot an inconsistency
  • 48. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 •Reorder Events •Guess missing Events •Rebuild Read Models on late Events •Delay Read Models * Give events time to arrive * (Watermarks) 48
  • 49. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 49 ?
  • 50. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Building Read Models 
 is expensive and takes time 50
  • 51. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 51 < Process Writes < Serve Queries < Build Read Models Separate
 M icro services
  • 52. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 • Scales independently • Asynchronous 
 (no time-coupling) 52
  • 53. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Read Models are always Delayed 53
  • 54. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 What’s the right order? 54 When building Read Models We can reorder Events, but…
  • 55. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Order Events
 by Timestamp 55 Which timestamp? but…
  • 56. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 56 Received timestamp Processed timestamp Event store timestamp Event timestamp Source Our system Unreliable
  • 57. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 57
  • 58. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 58 Event timestamp by Event Source, Device Reliable per-Device only
  • 59. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Device = Aggregate “local” Event order 59 Device != Aggregate No easy Event order No reliable, global 
 Event ordering
  • 60. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 60 Reorder Events 
 by causal relation Not Trivial “We received A after B, 
 but A must happen before B” Causal Consistency
  • 61. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Wrap Up 61
  • 62. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 > Distributed systems, microservices > Scalability is critical > Low-latency concurrent writes > Domain fits Event Sourcing 62
  • 63. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 63 DDD approach Command —> —> Event stateful
 Aggregate Transactional Consistency at Write time Event log
  • 64. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Things happen
 inside our system Our system reacts to
 Commands received 64
  • 65. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 + Process-oriented systems + Strong consistency - External unreliability - Scale out 
 & low-latency concurrent writes 65 We drive the process
  • 66. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 66 Weak write consistency
 (Write fast, think later) 1. Lenient & stateless validation on write
 Command —> Event 2. Enforce consistency
 when building Read Models
  • 67. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Things happen
 in the outside world Our system draws 
 pictures of the world 67
  • 68. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 + Scales out well
 & low-latency writes + External unreliability - Delayed Read Models - No Write-time transaction 68 We are not driving the process
  • 69. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 The world is eventually consistent The world is unreliable We are always looking at the past 69 “Guess -> Apologise -> Compensate” - Jonas Bonér
  • 70. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 70 Credits: Dominic Fox medium.com/@poetix
  • 71. Lorenzo Nicora - @nicusXMicroservices Meetup Munich - 2018 Thanks! 71