AMQP
Advanced Message Queueing Protocol
Caveats
Event Logging
Legacy Architecture & Problem
Event Logging
Client

Client

Server

MongoDB

Client
Connection Issues
80
Client

110
Client

Server

MongoDB

65

Client
Connection Issues
80
Client

110
Client

255
Server

MongoDB

65

Client
Server MIA
0

Client

0

0

Server

Client

MongoDB

0

Client
Network Segmentation
0

Client

0

Server

Client

MongoDB

0

Client
Total data loss
DATA LOSS?!
Why AMQP?
• Stable and mature	

• Designed for SX markets	

• Widely supported (good interop)	

• Modular, Fast & Flexible
AMQP Overview
AMQP

Service Model
Wire

Advanced Message Queueing Protocol
AMQP Overview
AMQP

AMQ Model

Wire
AMQP Overview
AMQP

AMQ Model

Server-side service model	

defined components	

rules for wiring

Wire
AMQP Overview
AMQP

AMQ Model

Wire

Network wire-level protocol	

command architecture	

connections, channels, etc
AMQ Model

AMQ Model
Exchange
Queues
Binding
AMQ Model
AMQ Model

Exchange

Queues

Binding
AMQ Model
Virtual Host

AMQ Model

Exchange

Queues

Binding
AMQ Model
Virtual Host

AMQ Model

Publisher

Exchange

Queues

Binding

Consumer
AMQ Model
Virtual Host

Exchange

Publisher

Queues

Binding

Consumer
AMQ Model
Virtual Host
Router
Message

Publisher

Exchange

Queues

Binding

Consumer
AMQ Model
Virtual Host
Router
Message

Publisher

Exchange

Queues

Binding

Consumer
AMQ Model
Virtual Host
Router
Message

Publisher

Exchange

?

Binding

Queues

Consumer
AMQ Model
Virtual Host
Router
Message

Publisher

Exchange

?

Binding

Queues

Consumer
AMQ Model
Virtual Host
Router
Message

Publisher

Exchange

Queues

Binding
Queue:
critical_error_queue	

Exchange:
errors_exchange	

Filter:	

*.error.critical

Consumer
AMQ Model
Virtual Host
Router
Message

Publisher

Exchange

Queues

Binding
Queue:
critical_error_queue	

Exchange:
errors_exchange	

Filter:	

*.error.critical

Consumer
Important notes
Publisher

‣ can create

exchanges and
queues

Exchange

‣ routes messages based on

criteria	

‣ doesn’t store messages	

‣ can inspect message
content	

‣ can be created at runtime
consumers

Queues

store messages	

named	

bound-able to exchange	

criteria	

can be created at
runtime by consumers
‣
‣
‣
‣
‣

Binding

‣ creates a relationship between

Consumer

‣ can create

queues and exchanges	

exchanges and
‣ contains criteria and properties	

 queues
‣ can be created at runtime by
consumers
The Exchange
Exchange
The Exchange
Exchange

Types	

Headers
System
Fanout
Topic
(routing algo)
The Exchange
Exchange

Types	

(routing algo)

Direct

Fanout

Topic

Headers

System
The Exchange
Exchange

Types	

(routing algo)

Direct

Fanout

Topic

Headers

System
Direct Type Exchange
Exchange	

‘events’
Direct Type Exchange
Message	


event.user.click

Message	


event.user.view

Message	


event.user.share

Exchange	

‘events’
Direct Type Exchange
Message	


event.user.click
user_click_queue

Message	


event.user.view

Message	


event.user.share

Exchange	

‘events’
Direct Type Exchange
Message	


event.user.click
user_click_queue

Message	


event.user.view

Exchange	

‘events’

Message	


event.user.share

Binding

➡Queue:
‣ user_click_queue
Direct Type Exchange
Message	


event.user.click
user_click_queue

Message	


event.user.view

Exchange	

‘events’

Message	


event.user.share

Binding

➡Queue:
‣ user_click_queue
➡Exchange:
‣ events
Direct Type Exchange
Message	


event.user.click
user_click_queue

Message	


event.user.view

Exchange	

‘events’

Message	


event.user.share

Binding

➡Queue:
‣ user_click_queue
➡Exchange:
‣ events 	

➡Routing Key:	

‣ event.user.click
Direct Type Exchange
Message	


PHP Script

event.user.click
user_click_queue

Message	


event.user.view

Exchange	

‘events’

PHP Script

Message	


PHP Script

event.user.share

Binding

➡Queue:
‣ user_click_queue
➡Exchange:
‣ events 	

➡Routing Key:	

‣ event.user.click
Direct Type Exchange
Message	


event.user.click
user_click_queue

Message	


event.user.view

Message	


Exchange	

‘events’
user_view_queue

event.user.share

user_share_queue
Yeah, so?
What if…
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


event.user.share

Exchange	

‘events’

user_shares_queue
What if…
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


event.user.share

Exchange	

‘events’

user_shares_queue
What if…
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


event.user.share

Exchange	

‘events’

user_shares_queue
Topic Exchange
Message	


event.user.click

Message	


event.user.view

user_interaction_queue

Exchange	

‘events’

Message	


event.user.share

Binding
Topic Exchange
Message	


PHP Script

event.user.click

Message	


event.user.view

user_interaction_queue

Exchange	

‘events’

PHP Script

Message	


PHP Script

event.user.share

Binding

➡Queue:
‣ user_interaction_queue
➡Exchange:
‣ events 	

➡Routing Key:	

‣ event.user.*
Topic Exchange
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


event.user.share

Message
Message

event.user.*

Exchange	

‘events’

event.user.share

user_shares_queue

Message
Can be done at runtime
Topic Exchange
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


Message
Message

event.user.*

Exchange	

‘events’

event.user.share

user_shares_queue

event.user.share

Click vs Share Conversion?

Message
Topic Exchange
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


Message
Message

event.user.*

Exchange	

‘events’

event.user.share

user_shares_queue

Message

event.user.share

event.user.share	

event.user.click
click_share_queue

Click vs Share Conversion?

Message
Message
Topic Exchange
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


Message
Message

event.user.*

Exchange	

‘events’

event.user.share

user_shares_queue

Message

event.user.share

Message	


*.*.view

event.staff.view

live_view_queue

Staff vs user views?

Message
Message
I’m still pissed
New Event Logging
Explorations & Implementation
Revisiting Our Logging
Client

Client

Client

MongoDB
Event Logging Revised
Client
MQ

Client

MongoDB
MQ

Client
MQ
Event Logging Revised
Client
MQ

Client

MongoDB
MQ

Client
MQ
Event Logging Revised
Client
MQ

Client

MongoDB
MQ

Client
MQ
No data loss!
SRSLY?!
Event Logging Revised
80

Client
MQ

110

MQ

Client
MQ

65

Client
MQ

MongoDB
Event Logging Revised
80

Client
MQ

110

MQ

Client
MQ

65

Client
MQ

MongoDB
Event Logging Revised
80

Client
MQ

110

MQ

Client
MQ

65

Client

MQ
MQ

MongoDB
Event Logging Revised
80

Client
MQ

MQ
110

Client

MongoDB
MQ

MQ
65

Client
MQ
Remember Topic Exchange?
user_interaction_queue

Message	


event.user.click

Message	


event.user.view

Message	


Message
Message

event.user.*

Exchange	

‘events’

event.user.share

user_shares_queue

Message

event.user.share

Message	


event.staff.view

*.*.view
live_view_queue

Message
Message
Provides flexibility
80

MongoDB

Client
MQ

110

MQ

Client

Reporting	

SQL

MQ

65

Client

Notification
MQ
Not Covered
•

durable	


•

auto-delete	


•

passive	


•

qos	


•

no-wait	


•

ack	


•

exclusive	


•

message recovery
A quick demo of AMQP
with RabbitMQ
Thank you
weekeat@peazie.com

AMQP for phpMelb