Skip to main content
Eventsourcing 
with%PHP%and%MongoDB 
Data$Analysis$with$events
Hello! 
My#name#is#Jacopo&Nardiello 
• Programmer,*currently*located*in*London 
• I*care*about*code*quality 
• If*it's*not*tested,*it*doesn't*exist. 
If#you#want#to#follow#me#@jnardiello
What%is%this%talk%about
A"li%le"bit"of"context
About&Onebip 
Mobile'payments!pla%orm. 
Start,up!born!in!2005,!acquired!by! 
Neomobile!group!in!2011.! 
Onebip'today: 
.'70'countries 
.'200+'carriers 
.'5'billions'registered'users
It#all#started#with#a#Monolith 
LAMP%stack
To#Distributed#Systems 
self%contained-services-talking-via-REST
Modern'Services 
First&class&modern&NoSQL&distributed&dbs
But$the$Monolith$is$s)ll$there
The$problem 
A"repor'ng"horror"story
We#need#three#new#reports! 
—"Manager 
Sure,&no&problem!
Deal%with%the%legacy%SQL%Schema
Deal%with%MongoDB
A"li%le"bit"of"queries"here, 
a"li%le"bit"of"map/reduce"there
1"month"later...
Reports(are(finally(ready!
un#l...
Your%queries%are%killing%produc2on! 
—"SysAdmin
Heavy&querying&op/miza/on 
New$indexes s"ll$not$enough
Let's&re(use&data&from&other&reports 
(don't'do'that)
DB#is#Ok,#reports#delivered.
but$then...
Huston,(we(have(a(problem.(Reports( 
are(not(consistent((with(other( 
reports) 
—"business"guy
Mistakes 
were 
made
#1 
Avoid&Mul*ple&sources&of&truth 
It's%hard%to%compare%different%data%from%different%in%a%distributed% 
system%spli7ed%across%mul8ple%domains
#2 
Ubiquitous)language 
Same%words,%different%concepts%across%domains
#3 
Fault&tolerance&to&change 
Changing'a'report'shouldn't'have'side'effects
Most%common%solu+ons
#1 
ETL$+$Map/Reduce
#2 
DWH$+$Consultants
#3#Mad#science 
(Yeppa!)
What%we%wanted
Must%have 
No#down'me#in#produc'on 
Consistent(across(domains
Nice%to%have 
A"system"elas*c"enough"to"extract"any"metric 
Real%me'data
In#DDD#we#found#the#light
Eventsourcing-&-CQRS
Eventsourcing 
The$fundamental$idea$of$Event$Sourcing$is$that$of$ensuring(every( 
change(to(the(state(of(an(applica3on(is(captured(in(an(event(object,$ 
and$that$these$event$objects$are$themselves$stored(in(the(sequence( 
they(were(applied 
—(Mar3n(Fowler
Unrolling(a(stream(of(events 
Star%ng(from(the(beginning(of(%me,(you(are( 
literally(unrolling(history(to(reach(the(state(in(a( 
given(%me
Idea%#1 
Every&change&to&the&state&of&your&applica4on&is&captured&in&an& 
event&object 
"User%Logged%In",#"Payment%Sent",#"User%Landed"
Idea%#2 
Events'are'stored'in'the'sequence'they' 
were'applied'inside'an'eventstore
Command'Query'Responsibility' 
Segrega6on 
(CQRS)
Commands 
Anything(that(happens(in(one(of(your(domains(is(triggered(by(a( 
command(and(generate(one(or(more(events. 
Order received --> payment sent --> Items queued 
--> Confirmation email sent
Idea%#3 
Everything+is+an+event.+No+more+state.
Query 
Generate'read'models'from'events'depending'how'data'needs'to' 
be'actually'used'(by'users'and'other'applica9on'internals)
Idea%#4 
One$way$to$store$data/events$but$poten2ally$infinite$ways$to$read$ 
them 
A"prac'cal"example 
Tech%ops,%Business%control,%Monitoring,%Accoun4ng%they%are%all% 
interested%in%reading%data%from%different%views.
Healthy(NoSQL
You$start$with$this 
{ 
"_id": ObjectId("123"), 
"username": "Flash", 
"city": …, 
"phone": …, 
"email": …, 
}
The$more$successful$your$company$is,$the$ 
more$people 
... 
The$more$people,$the$more$views
With%documental%dbs%it's%magically%easy%to%add%new%fields%to%your% 
collec7ons.
Soon$you$might$end$up$ 
with 
{ 
"_id": ObjectId("123"), 
"username": "Flash", 
"city": …, 
"phone": …, 
"email": …, 
"created_at": …, 
"updated_at": …, 
"ever_tried_to_purchase_something": …, 
"canceled_at": …, 
"acquisition_channel": …, 
"terminated_at": …, 
"latest_purchase_date": …, 
… 
}
A"bomb"wai)ng"to"detonate 
It's%impossible%to%keep%adding%state%changes%to%your%documents%and% 
then%expect%to%be%able%to%extract%them%with%a%single%query.
Exploring*Tools
EventStore 
• Engineered)for)event)sourcing 
• Supports)projec4ons 
• By)the)father)of)CQRS 
• Great)performances 
The$bad 
Win$Only.$Run$on$*nix$with$Mono,$s4ll$too$unstable.
LevelWHEN) 
An#eventstore#built#with#Node.js#and#LevelDB 
8#Faster#than#light 
8#Completely#custom,#no#tool#to#handle#aggregates
The$known$path 
• PHP$(any$other$language$would$just$do$ 
fine) 
• MongoDB$2.x
Why$MongoDB 
Events'are'not'rela,onal 
Scales'Well 
Awesome'aggrega+on'framework
Hands&on
Storing(Events
The$write$architecture 
Service | 
 | 
 [event payload] | 
 | 
Service --- Queue System <------------> API -> MongoDB 
/ | 
/ [event payload] | 
/ | 
Service |
Queues
MongoDB'RS 
A"mongod"replica*set"with"two"logical"dbs." 
1."Eventstore"DB"where"we"would"store" 
2."Repor=ng"DB"where"we"would"store"aggregates"and"final"reports
Anatomy(of(an(event 
{ 
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 
'type': 'an-event-type', 
'data': { 
'meta' : { 
… 
}, 
'payload' : { 
… 
} 
} 
} 
Storing(events
Anatomy(of(an(event 
'meta' : { 
'creation_date': ISODate("2014-21-11T00:00:01Z"), 
'saved_date': ISODate("2014-21-11T00:00:02Z"), 
'source': 'some-bounded-context', 
'correlation_id': 'a-correlation-id' 
}, 
'payload' : { 
'user_id': '1234', 
'animal': 'unicorn', 
'colour': 'pink', 
'purchase_date': ISODate("2014-21-11T00:00:00Z"), 
'price': '20fantaueros' 
} 
Storing(events
Don't&trust&the&network:&Idempotence 
{ 
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 
… 
} 
The$_id$field$is$actually$defined$client0side$and$ensures$ 
idempotence$if$an$event$is$received$two$8mes.
Indexes 
• Events(collec,ons(are(HUGE((~100M*N(documents) 
• Use(indexes(wisely(as(they(are(necessary(yet(expensive( 
• With(suggested(events(structure:(type + 
data.meta.created_at
Benchmarking 
How$many$events/second$can$you$store? 
Our$machines$were$able$to$store$roughly$150$events/sec.$This$ 
number$can$be$greatly$increased$with$dedicated$IOPS,$more$ 
aggressive$inser@ng$policies,$etc..
Final&Tips 
• Use%SSDs%on%your%storage%machines 
• Pay%a5en6on%to%write%concerns
From%Events%to%Meaningful%Metrics
The$Event$Processing$Pipeline 
Sequential Projector -> Event Mapper -> Projection -> Aggregation
A"real"life"problem 
What%is%the%conversion%rate%of%our%registered%users?
#1#The#registra-on#event 
{ 
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 
'type': 'user-registered', 
'data': { 
'meta' : { 
'save_date': ISODate("2014-21-11T00:00:09Z"), 
'created_at': ISODate("2014-21-11T00:00:01Z"), 
'source': 'core-domain', 
'correlation_id': 'user-123456' 
}, 
'payload' : { 
'user_id': 123, 
'username': 'flash', 
'email': 'a-dummy-email@gmail.com', 
'country': 'IT' 
} 
} 
}
#2#The#purchase#event 
{ 
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 
'type': 'user-purchased', 
'data': { 
'meta' : { 
'save_date': ISODate("2014-21-11T00:10:09Z"), 
'created_at': ISODate("2014-21-11T00:10:01Z"), 
'source': 'payment-gateway', 
'correlation_id': 'user-123456' 
}, 
'payload' : { 
'user_id': 123, 
'email': 'a-dummy-email@gmail.com', 
'amount': 20, 
'value': EUR, 
'payment': 'credit_card', 
'item': 'fluffy cat' 
} 
} 
}
Sequen&al)projector)(1) 
Divides'the'stream'of'events'into'batches,'filters'events'by'type' 
and'pass'those'of'interest'to'the'mapper' 
[]->[x]->[]->[x]->[]->[]->[]->[] 
|--------------| |------------| 
| 
| 
| 
| 
---> Projector
Sequen&al)projector)(2) 
• It's&a&good&idea&to&select&fixed&sizes&batches&to&avoid&memory& 
problems&when&you&load&your&Cursor&in&memory 
• Could&be&a&long=running&process&selec>ng&events&as&they&arrive& 
in&real%me
Event&Mapper&(1) 
Translates)event)fields)to)the)Read)Model)domain. 
Takes&an&event&as&input,&applies&a&bunch&of&logic&and&will&return&a& 
list&of&Read&Model&fields.
Event&Mapper&(2) 
Input&Event:& 
user-registered 
Output: 
$output = [ 
'user_id' => 123, // simply copied 
'user_name' => 'flash', // simply copied 
'email' => 'a-dummy-email@gmail.com', // simply copied 
'registered_at' => "2014-21-11T00:00:01Z" // From the data.meta.created_at event field 
];
Event&Mapper&(3) 
Input&Event:& 
user-purchased 
Output: 
$output = [ 
'user_id' => 123, // simply copied 
'email' => 'a-dummy-email@gmail.com', // simply copied 
'purchased_at': "2014-21-11T00:10:01Z" // From the data.meta.created_at event field 
];
Projec'on)(1) 
Essen%ally)it)is)your)read)model.)The$data$that$the$business$is$ 
interested$in.
The$Projec*on$,$a.er$event#1$(2) 
db.users_conversion_rate_projection.findOne() 
{ 
'user_id': 123, 
'user_name': 'flash', 
'email': 'a-dummy-email@gmail.com', 
'registered_at': "2014-21-11T00:00:01Z" 
}
The$Projec*on$,$a.er$event#2$(3) 
{ 
'user_id': 123, 
'user_name': 'flash', 
'email': 'a-dummy-email@gmail.com', 
'registered_at': "2014-21-11", 
'purchased_at': "2014-21-11" // Added this field and rewrote others 
}
The$Projec*on$,$collec*on 
{ 
'user_id': 123, 
'user_name': 'flash', 
'email': 'a-dummy-email@gmail.com', 
'registered_at': "2014-21-11", 
'purchased_at': "2014-21-11" // Added this field and rewrote others 
} 
{ 
'user_id': 456, 
'user_name': 'batman', 
'email': 'a-dummy-email@gmail.com', 
'registered_at': "2014-21-11", 
'purchased_at': "2014-21-11" // Added this field and rewrote others 
} 
{ 
'user_id': 789, 
'user_name': 'superman', 
'email': 'a-dummy-email@gmail.com', 
'registered_at': "2014-21-12", 
'purchased_at': "2014-21-12" // Added this field and rewrote others 
}
The$Projec*on$,$A$few$thoughts$(4) 
Note%that%we%didn't%copy%from%events%to%projec6on%all%the%available% 
fields.%Just%relevant%ones.
From%these%two%events%we%could%have%generated%infinite%read% 
models%such%as: 
• List&all&purchased&products&and&related&amounts&for&the&company& 
buyers& 
• Map&all&sales&and&revenues&for&our&accoun8ng&dept 
• List&transac8ons&for&the&financial&department
One$way$to$write,$infinite$ways$to$ 
read!
The$aggrega(on$(1)$.$Total$registered$users 
var registered = db.users_conversion_rate_projection.aggregate([ 
{ 
$match: { 
"registered_at": { $gte: ISODate("2015-11-21"), $lte: ISODate("2015-11-22") } 
} 
}, 
{ 
$group: { 
_id: { }, 
count: { $sum:1 } 
} 
} 
]);
The$aggrega(on$(2)$.$Users$with$a$purchase 
var purchased = db.users_conversion_rate_projection.aggregate([ 
{ 
$match: { 
"registered_at": { $gte: ISODate("2015-11-21"), $lte: ISODate("2015-11-22") }, 
"purchased_at": { $exists: true } 
} 
}, 
{ 
$group: { 
_id: { }, 
count: { $sum:1 } 
} 
} 
]);
The$aggrega(on$(3)$.$Automate$all$the$things 
• You%can%easily%create%the%aggrega2on%framework%statement%by% 
composi2on%abstrac2ng%the%concept%of%Column. 
• This%way%you%can%dynamically%aggregate%your%projec2ons%on%(for% 
example)%an%API%requests. 
• If%your%Projector%is%a%long%running%process,%your%projec2ons%will% 
be%updated%to%the%second%and%you%automagically%get%real%me% 
data.
Another(events(usage:(Business(&( 
Tech(Monitoring
Beware&of&the&beast! 
No#Silver#Bullet
Events'are'expensive 
Require'a'lot'of'TIME'to'be'parsed
Events'are'expensive 
You$will$end$up$with$this$billion$size$collec2on$ 
(and$coun2ng).
Fixing&wrong&events&is&painful
Events'are'complex
Moving'around'events'is'horribly' 
painful
Mongo%won't%help%you. 
Actually(it(will(make(your(life(incredibly( 
difficult(with(hidden(bugs(and(leaking( 
documenta8on.
Thank&you! 
@jnardiello