How to process data using events on top of PHP and MongoDB. Introducing Eventsourcing and CQRS, how to handle events and how to generate Read Models and Aggregates leveraging the MongoDB Aggregation Framework.

1. Eventsourcing
with%PHP%and%MongoDB
Data$Analysis$with$events

2. Hello!
My#name#is#Jacopo&Nardiello
• Programmer,*currently*located*in*London
• I*care*about*code*quality
• If*it's*not*tested,*it*doesn't*exist.
If#you#want#to#follow#me#@jnardiello

3. What%is%this%talk%about

4. A"li%le"bit"of"context

5. About&Onebip
Mobile'payments!pla%orm.
Start,up!born!in!2005,!acquired!by!
Neomobile!group!in!2011.!
Onebip'today:
.'70'countries
.'200+'carriers
.'5'billions'registered'users

6. It#all#started#with#a#Monolith
LAMP%stack

7. To#Distributed#Systems
self%contained-services-talking-via-REST

8. Modern'Services
First&class&modern&NoSQL&distributed&dbs

9. But$the$Monolith$is$s)ll$there

10. The$problem
A"repor'ng"horror"story

11. We#need#three#new#reports!
—"Manager
Sure,&no&problem!

12. Deal%with%the%legacy%SQL%Schema

14. Deal%with%MongoDB

15. A"li%le"bit"of"queries"here,
a"li%le"bit"of"map/reduce"there

16. 1"month"later...

17. Reports(are(finally(ready!

18. un#l...

19. Your%queries%are%killing%produc2on!
—"SysAdmin

20. Heavy&querying&op/miza/on
New$indexes s"ll$not$enough

21. Let's&re(use&data&from&other&reports
(don't'do'that)

22. DB#is#Ok,#reports#delivered.

23. but$then...

24. Huston,(we(have(a(problem.(Reports(
are(not(consistent((with(other(
reports)
—"business"guy

26. Mistakes
were
made

27. #1
Avoid&Mul*ple&sources&of&truth
It's%hard%to%compare%different%data%from%different%in%a%distributed%
system%spli7ed%across%mul8ple%domains

28. #2
Ubiquitous)language
Same%words,%different%concepts%across%domains

29. #3
Fault&tolerance&to&change
Changing'a'report'shouldn't'have'side'effects

30. Most%common%solu+ons

31. #1
ETL$+$Map/Reduce

32. #2
DWH$+$Consultants

33. #3#Mad#science
(Yeppa!)

34. What%we%wanted

35. Must%have
No#down'me#in#produc'on
Consistent(across(domains

36. Nice%to%have
A"system"elas*c"enough"to"extract"any"metric
Real%me'data

37. In#DDD#we#found#the#light

38. Eventsourcing-&-CQRS

39. Eventsourcing
The$fundamental$idea$of$Event$Sourcing$is$that$of$ensuring(every(
change(to(the(state(of(an(applica3on(is(captured(in(an(event(object,$
and$that$these$event$objects$are$themselves$stored(in(the(sequence(
they(were(applied
—(Mar3n(Fowler

40. Unrolling(a(stream(of(events
Star%ng(from(the(beginning(of(%me,(you(are(
literally(unrolling(history(to(reach(the(state(in(a(
given(%me

41. Idea%#1
Every&change&to&the&state&of&your&applica4on&is&captured&in&an&
event&object
"User%Logged%In",#"Payment%Sent",#"User%Landed"

42. Idea%#2
Events'are'stored'in'the'sequence'they'
were'applied'inside'an'eventstore

43. Command'Query'Responsibility'
Segrega6on
(CQRS)

45. Commands
Anything(that(happens(in(one(of(your(domains(is(triggered(by(a(
command(and(generate(one(or(more(events.
Order received --> payment sent --> Items queued
--> Confirmation email sent

46. Idea%#3
Everything+is+an+event.+No+more+state.

47. Query
Generate'read'models'from'events'depending'how'data'needs'to'
be'actually'used'(by'users'and'other'applica9on'internals)

48. Idea%#4
One$way$to$store$data/events$but$poten2ally$infinite$ways$to$read$
them
A"prac'cal"example
Tech%ops,%Business%control,%Monitoring,%Accoun4ng%they%are%all%
interested%in%reading%data%from%different%views.

49. Healthy(NoSQL

50. You$start$with$this
{
"_id": ObjectId("123"),
"username": "Flash",
"city": …,
"phone": …,
"email": …,
}

51. The$more$successful$your$company$is,$the$
more$people
...
The$more$people,$the$more$views

52. With%documental%dbs%it's%magically%easy%to%add%new%fields%to%your%
collec7ons.

53. Soon$you$might$end$up$
with
{
"_id": ObjectId("123"),
"username": "Flash",
"city": …,
"phone": …,
"email": …,
"created_at": …,
"updated_at": …,
"ever_tried_to_purchase_something": …,
"canceled_at": …,
"acquisition_channel": …,
"terminated_at": …,
"latest_purchase_date": …,
…
}

54. A"bomb"wai)ng"to"detonate
It's%impossible%to%keep%adding%state%changes%to%your%documents%and%
then%expect%to%be%able%to%extract%them%with%a%single%query.

55. Exploring*Tools

56. EventStore
• Engineered)for)event)sourcing
• Supports)projec4ons
• By)the)father)of)CQRS
• Great)performances
The$bad
Win$Only.$Run$on$*nix$with$Mono,$s4ll$too$unstable.

57. LevelWHEN)
An#eventstore#built#with#Node.js#and#LevelDB
8#Faster#than#light
8#Completely#custom,#no#tool#to#handle#aggregates

58. The$known$path
• PHP$(any$other$language$would$just$do$
fine)
• MongoDB$2.x

59. Why$MongoDB
Events'are'not'rela,onal
Scales'Well
Awesome'aggrega+on'framework

60. Hands&on

61. Storing(Events

62. The$write$architecture
Service |
|
[event payload] |
|
Service --- Queue System <------------> API -> MongoDB
/ |
/ [event payload] |
/ |
Service |

63. Queues

64. MongoDB'RS
A"mongod"replica*set"with"two"logical"dbs."
1."Eventstore"DB"where"we"would"store"
2."Repor=ng"DB"where"we"would"store"aggregates"and"final"reports

65. Anatomy(of(an(event
{
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9',
'type': 'an-event-type',
'data': {
'meta' : {
…
},
'payload' : {
…
}
}
}
Storing(events

66. Anatomy(of(an(event
'meta' : {
'creation_date': ISODate("2014-21-11T00:00:01Z"),
'saved_date': ISODate("2014-21-11T00:00:02Z"),
'source': 'some-bounded-context',
'correlation_id': 'a-correlation-id'
},
'payload' : {
'user_id': '1234',
'animal': 'unicorn',
'colour': 'pink',
'purchase_date': ISODate("2014-21-11T00:00:00Z"),
'price': '20fantaueros'
}
Storing(events

67. Don't&trust&the&network:&Idempotence
{
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9',
…
}
The$_id$field$is$actually$defined$client0side$and$ensures$
idempotence$if$an$event$is$received$two$8mes.

68. Indexes
• Events(collec,ons(are(HUGE((~100M*N(documents)
• Use(indexes(wisely(as(they(are(necessary(yet(expensive(
• With(suggested(events(structure:(type +
data.meta.created_at

69. Benchmarking
How$many$events/second$can$you$store?
Our$machines$were$able$to$store$roughly$150$events/sec.$This$
number$can$be$greatly$increased$with$dedicated$IOPS,$more$
aggressive$inser@ng$policies,$etc..

70. Final&Tips
• Use%SSDs%on%your%storage%machines
• Pay%a5en6on%to%write%concerns

71. From%Events%to%Meaningful%Metrics

72. The$Event$Processing$Pipeline
Sequential Projector -> Event Mapper -> Projection -> Aggregation

73. A"real"life"problem
What%is%the%conversion%rate%of%our%registered%users?

74. #1#The#registra-on#event
{
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9',
'type': 'user-registered',
'data': {
'meta' : {
'save_date': ISODate("2014-21-11T00:00:09Z"),
'created_at': ISODate("2014-21-11T00:00:01Z"),
'source': 'core-domain',
'correlation_id': 'user-123456'
},
'payload' : {
'user_id': 123,
'username': 'flash',
'email': 'a-dummy-email@gmail.com',
'country': 'IT'
}
}
}

75. #2#The#purchase#event
{
'_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9',
'type': 'user-purchased',
'data': {
'meta' : {
'save_date': ISODate("2014-21-11T00:10:09Z"),
'created_at': ISODate("2014-21-11T00:10:01Z"),
'source': 'payment-gateway',
'correlation_id': 'user-123456'
},
'payload' : {
'user_id': 123,
'email': 'a-dummy-email@gmail.com',
'amount': 20,
'value': EUR,
'payment': 'credit_card',
'item': 'fluffy cat'
}
}
}

76. Sequen&al)projector)(1)
Divides'the'stream'of'events'into'batches,'filters'events'by'type'
and'pass'those'of'interest'to'the'mapper'
[]->[x]->[]->[x]->[]->[]->[]->[]
|--------------| |------------|
|
|
|
|
---> Projector

77. Sequen&al)projector)(2)
• It's&a&good&idea&to&select&fixed&sizes&batches&to&avoid&memory&
problems&when&you&load&your&Cursor&in&memory
• Could&be&a&long=running&process&selec>ng&events&as&they&arrive&
in&real%me

78. Event&Mapper&(1)
Translates)event)fields)to)the)Read)Model)domain.
Takes&an&event&as&input,&applies&a&bunch&of&logic&and&will&return&a&
list&of&Read&Model&fields.

79. Event&Mapper&(2)
Input&Event:&
user-registered
Output:
$output = [
'user_id' => 123, // simply copied
'user_name' => 'flash', // simply copied
'email' => 'a-dummy-email@gmail.com', // simply copied
'registered_at' => "2014-21-11T00:00:01Z" // From the data.meta.created_at event field
];

80. Event&Mapper&(3)
Input&Event:&
user-purchased
Output:
$output = [
'user_id' => 123, // simply copied
'email' => 'a-dummy-email@gmail.com', // simply copied
'purchased_at': "2014-21-11T00:10:01Z" // From the data.meta.created_at event field
];

81. Projec'on)(1)
Essen%ally)it)is)your)read)model.)The$data$that$the$business$is$
interested$in.

82. The$Projec*on$,$a.er$event#1$(2)
db.users_conversion_rate_projection.findOne()
{
'user_id': 123,
'user_name': 'flash',
'email': 'a-dummy-email@gmail.com',
'registered_at': "2014-21-11T00:00:01Z"
}

83. The$Projec*on$,$a.er$event#2$(3)
{
'user_id': 123,
'user_name': 'flash',
'email': 'a-dummy-email@gmail.com',
'registered_at': "2014-21-11",
'purchased_at': "2014-21-11" // Added this field and rewrote others
}

84. The$Projec*on$,$collec*on
{
'user_id': 123,
'user_name': 'flash',
'email': 'a-dummy-email@gmail.com',
'registered_at': "2014-21-11",
'purchased_at': "2014-21-11" // Added this field and rewrote others
}
{
'user_id': 456,
'user_name': 'batman',
'email': 'a-dummy-email@gmail.com',
'registered_at': "2014-21-11",
'purchased_at': "2014-21-11" // Added this field and rewrote others
}
{
'user_id': 789,
'user_name': 'superman',
'email': 'a-dummy-email@gmail.com',
'registered_at': "2014-21-12",
'purchased_at': "2014-21-12" // Added this field and rewrote others
}

85. The$Projec*on$,$A$few$thoughts$(4)
Note%that%we%didn't%copy%from%events%to%projec6on%all%the%available%
fields.%Just%relevant%ones.

86. From%these%two%events%we%could%have%generated%infinite%read%
models%such%as:
• List&all&purchased&products&and&related&amounts&for&the&company&
buyers&
• Map&all&sales&and&revenues&for&our&accoun8ng&dept
• List&transac8ons&for&the&financial&department

87. One$way$to$write,$infinite$ways$to$
read!

88. The$aggrega(on$(1)$.$Total$registered$users
var registered = db.users_conversion_rate_projection.aggregate([
{
$match: {
"registered_at": { $gte: ISODate("2015-11-21"), $lte: ISODate("2015-11-22") }
}
},
{
$group: {
_id: { },
count: { $sum:1 }
}
}
]);

89. The$aggrega(on$(2)$.$Users$with$a$purchase
var purchased = db.users_conversion_rate_projection.aggregate([
{
$match: {
"registered_at": { $gte: ISODate("2015-11-21"), $lte: ISODate("2015-11-22") },
"purchased_at": { $exists: true }
}
},
{
$group: {
_id: { },
count: { $sum:1 }
}
}
]);

90. The$aggrega(on$(3)$.$Automate$all$the$things
• You%can%easily%create%the%aggrega2on%framework%statement%by%
composi2on%abstrac2ng%the%concept%of%Column.
• This%way%you%can%dynamically%aggregate%your%projec2ons%on%(for%
example)%an%API%requests.
• If%your%Projector%is%a%long%running%process,%your%projec2ons%will%
be%updated%to%the%second%and%you%automagically%get%real%me%
data.

91. Another(events(usage:(Business(&(
Tech(Monitoring

92. Beware&of&the&beast!
No#Silver#Bullet

93. Events'are'expensive
Require'a'lot'of'TIME'to'be'parsed

94. Events'are'expensive
You$will$end$up$with$this$billion$size$collec2on$
(and$coun2ng).

95. Fixing&wrong&events&is&painful

96. Events'are'complex

97. Moving'around'events'is'horribly'
painful

98. Mongo%won't%help%you.
Actually(it(will(make(your(life(incredibly(
difficult(with(hidden(bugs(and(leaking(
documenta8on.

99. Thank&you!
@jnardiello