A presentation on the AMQP protocol with a brief live demo of RabbitMQ to illustrate the protocol.

1. Distributed Messaging
AMQP, RabbitMQ (Quick demo)
Tuesday, 10 September 13

2. Event Logging
Current architecture & problems
Tuesday, 10 September 13

3. Event Logging
Server
Client
Client
Client
MongoDB
Tuesday, 10 September 13

4. Problem
Server
Client
Client
Client
80
110
65
MongoDB
Tuesday, 10 September 13

5. Problem
Server
Client
Client
Client
80
110
65
255
MongoDB
Tuesday, 10 September 13

6. Problem
Server
Client
Client
Client
0
0
0
0
MongoDB
Tuesday, 10 September 13

7. Problem
Server
Client
Client
Client
0
0
0
0
MongoDB
Tuesday, 10 September 13

8. Possible Solutions
• Server tweaks:TCP conns, ulimit, etc
• Multi-master model
• Localised queue
• ... and quite a few more
Tuesday, 10 September 13

9. Possible Solutions
• Server tweaks:TCP conns, ulimit, etc
• Multi-master model
•Localised queue
• ... and quite a few more
Tuesday, 10 September 13

10. Localised Queues
• Fast (no TCP latency)
• Network partition tolerant
• Modular & scalable
• Reliable & Durable
Tuesday, 10 September 13

11. Why Localise?
Server
Client
Client
Client
80
110
65
MongoDB
Q
Q
Q
< 10
Tuesday, 10 September 13

12. Why Localise?
Server
Client
Client
Client
80
110
65
MongoDB
Q
Q
Q
0
Tuesday, 10 September 13

13. Why Localise?
Server
Client
Client
Client
80
110
65
MongoDB
Q
Q
Q
0
Tuesday, 10 September 13

14. Why Localise?
Server
Client
Client
Client
80
110
65
MongoDB
Q
Q
Q
< 10
Tuesday, 10 September 13

15. Why Localise?
Server
Client
Client
Client
80
110
65
MongoDB
Q
Q
Q
< 10
Client
Q
Tuesday, 10 September 13

16. Queue Criteria
• Reliable
• Durable
• Scalable
Tuesday, 10 September 13

17. AMQP
Advanced Message Queueing Protocol
Tuesday, 10 September 13

18. Why AMQP?
• Stable and mature
• Designed for SX markets
• Widely supported (good interop)
• Modular, Fast & Flexible
Tuesday, 10 September 13

19. Service ModelWire
AMQP Overview
AMQP
Advanced Message Queueing Protocol
Tuesday, 10 September 13

20. Service Model Wire
AMQP Overview
AMQP
Tuesday, 10 September 13

21. ➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
AMQ Model Wire
AMQP Overview
AMQP
Tuesday, 10 September 13

22. ➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣Transactions, Exchange, Queues, etc
➡Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
Tuesday, 10 September 13

23. AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣Transactions, Exchange, Queues, etc
➡Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
Tuesday, 10 September 13

24. ExchangeQueuesBindingAMQ Model
AMQ Model
Tuesday, 10 September 13

25. QueuesExchange Binding
AMQ Model
AMQ Model
Tuesday, 10 September 13

26. Virtual Host
QueuesExchange Binding
AMQ Model
AMQ Model
Tuesday, 10 September 13

27. Virtual Host
QueuesExchange Binding
AMQ Model
AMQ Model
Publisher Consumer
Tuesday, 10 September 13

28. Virtual Host
QueuesExchange
Binding
AMQ Model
Publisher Consumer
Tuesday, 10 September 13

29. Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Tuesday, 10 September 13

30. Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Tuesday, 10 September 13

31. Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Tuesday, 10 September 13

32. Virtual Host
QueuesExchange
Binding
AMQ Model
Router
?Message
Publisher Consumer
Tuesday, 10 September 13

33. Virtual Host
QueuesExchange
Binding
AMQ Model
Message
Router
?
Publisher Consumer
Tuesday, 10 September 13

34. Virtual Host
QueuesExchange
Binding
AMQ Model
Message
Router
Publisher
➡ Queue:
‣ critical_error_queue
➡ Exchange:
‣ errors_exchange
➡ Filter:
‣ *.error.critical
Consumer
Tuesday, 10 September 13

35. Virtual Host
QueuesExchange
Binding
AMQ Model
➡ Queue:
‣ critical_error_queue
➡ Exchange:
‣ errors_exchange
➡ Filter:
‣ *.error.critical
Message
Router
Publisher Consumer
Tuesday, 10 September 13

36. QueuesExchange Binding
AMQP Service Model
Tuesday, 10 September 13

37. QueuesExchange Binding
AMQP Service Model
‣ routes messages based on
criteria
‣ doesn’t store messages
‣ can inspect message
content
‣ can be created at runtime
consumers
Tuesday, 10 September 13

38. QueuesExchange Binding
AMQP Service Model
‣ store messages
‣ named
‣ bound-able to exchange
‣ criteria
‣ can be created at
runtime by consumers
‣ routes messages based on
criteria
‣ doesn’t store messages
‣ can inspect message
content
‣ can be created at runtime
consumers
Tuesday, 10 September 13

39. QueuesExchange Binding
AMQP Service Model
‣ creates a relationship between
queues and exchanges
‣ contains criteria and properties
‣ can be created at runtime by
consumers
‣ store messages
‣ named
‣ bound-able to exchange
‣ criteria
‣ can be created at
runtime by consumers
‣ routes messages based on
criteria
‣ doesn’t store messages
‣ can inspect message
content
‣ can be created at runtime
consumers
Tuesday, 10 September 13

40. QueuesExchange Binding
AMQP Service Model
‣ creates a relationship between
queues and exchanges
‣ contains criteria and properties
‣ can be created at runtime by
consumers
‣ store messages
‣ named
‣ bound-able to exchange
‣ criteria
‣ can be created at
runtime by consumers
‣ routes messages based on
criteria
‣ doesn’t store messages
‣ can inspect message
content
‣ can be created at runtime
consumers
Tuesday, 10 September 13

41. QueuesExchange Binding
AMQP Service Model
Publisher Consumer
‣ creates a relationship between
queues and exchanges
‣ contains criteria and properties
‣ can be created at runtime by
consumers
‣ store messages
‣ named
‣ bound-able to exchange
‣ criteria
‣ can be created at
runtime by consumers
‣ routes messages based on
criteria
‣ doesn’t store messages
‣ can inspect message
content
‣ can be created at runtime
consumers
Tuesday, 10 September 13

42. QueuesExchange Binding
AMQP Service Model
Publisher Consumer
‣ creates a relationship between
queues and exchanges
‣ contains criteria and properties
‣ can be created at runtime by
consumers
‣ store messages
‣ named
‣ bound-able to exchange
‣ criteria
‣ can be created at
runtime by consumers
‣ routes messages based on
criteria
‣ doesn’t store messages
‣ can inspect message
content
‣ can be created at runtime
consumers
‣ can create
exchanges and
queues
‣ can create
exchanges and
queues
Tuesday, 10 September 13

43. Exchange
AMQP Service Model
Tuesday, 10 September 13

44. FanoutTopicHeadersSystem
Exchange
AMQP Service Model
Types
(routing algo)
Tuesday, 10 September 13

45. Fanout Topic Headers System
Exchange
AMQP Service Model
Types
(routing algo)
Direct
Tuesday, 10 September 13

46. Fanout Topic Headers System
Exchange
AMQP Service Model
Types
(routing algo)
Direct
Tuesday, 10 September 13

47. Exchange
AMQP Service Model
Direct Type
Tuesday, 10 September 13

48. Exchange
‘events’
AMQP Service Model
Direct Type
Tuesday, 10 September 13

49. Exchange
‘events’
AMQP Service Model
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13

50. Exchange
‘events’
AMQP Service Model
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13

51. Exchange
‘events’
Binding
AMQP Service Model
➡Queue:
‣ user_click_queue
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13

52. Exchange
‘events’
Binding
AMQP Service Model
➡Queue:
‣ user_click_queue
➡Exchange:
‣ events
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13

53. Exchange
‘events’
Binding
AMQP Service Model
➡Queue:
‣ user_click_queue
➡Exchange:
‣ events
➡Routing Key:
‣ event.user.click
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13

54. Exchange
‘events’
Binding
AMQP Service Model
➡Queue:
‣ user_click_queue
➡Exchange:
‣ events
➡Routing Key:
‣ event.user.click
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
user_click_queue
Tuesday, 10 September 13

55. Exchange
‘events’
AMQP Service Model
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
user_click_queue
ConsumerPublisher/
Producer
Tuesday, 10 September 13

56. Exchange
‘events’
AMQP Service Model
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
user_view_queue
user_share_queue
Tuesday, 10 September 13

57. Exchange
‘events’
AMQP Service Model
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
user_view_queue
user_share_queue
Message
Message
Message
Tuesday, 10 September 13

58. Exchange
‘events’
AMQP Service Model
What if...
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Tuesday, 10 September 13

59. Exchange
‘events’
AMQP Service Model
What if...
user_interaction_queue
user_shares_queue
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13

60. Exchange
‘events’
AMQP Service Model
What if...
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Tuesday, 10 September 13

61. Exchange
‘events’
AMQP Service Model
Topic Exchange
user_interaction_queue
Binding
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13

62. Exchange
‘events’
AMQP Service Model
Topic Exchange
user_interaction_queue
Binding
➡Queue:
‣ user_interaction_queue
➡Exchange:
‣ events
➡Routing Key:
‣ event.user.*
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13

63. Exchange
‘events’
AMQP Service Model
Topic Exchange
user_interaction_queue
Binding
➡Queue:
‣ user_interaction_queue
➡Exchange:
‣ events
➡Routing Key:
‣ event.user.*
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
Tuesday, 10 September 13

64. Exchange
‘events’
AMQP Service Model
Topic Exchange
user_shares_queue
Binding
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13

65. Exchange
‘events’
AMQP Service Model
Topic Exchange
user_shares_queue
Binding
➡Queue:
‣ user_interaction_queue
➡Exchange:
‣ events
➡Routing Key:
‣ event.user.share
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13

66. Exchange
‘events’
AMQP Service Model
Topic Exchange
user_shares_queue
Binding
➡Queue:
‣ user_interaction_queue
➡Exchange:
‣ events
➡Routing Key:
‣ event.user.share
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
Tuesday, 10 September 13

67. Message
Exchange
‘events’
AMQP Service Model
Topic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
event.user.*
event.user.share
Tuesday, 10 September 13

68. Message
Exchange
‘events’
AMQP Service Model
Topic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Can be done at runtime
event.user.*
event.user.share
Tuesday, 10 September 13

69. Message
Exchange
‘events’
AMQP Service Model
Topic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Click vs Share Conversion?
event.user.*
event.user.share
Tuesday, 10 September 13

70. Message
Exchange
‘events’
AMQP Service Model
Topic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Click vs Share Conversion?
event.user.*
event.user.share
Message
click_share_queue
Message
event.user.share
event.user.click
Tuesday, 10 September 13

71. Message
Exchange
‘events’
AMQP Service Model
Topic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
live_view_queue
event.user.*
event.user.share
*.*.view
Message
Message
event.staff.view
Message
Message
Staff vs user views?
Tuesday, 10 September 13

72. AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣Transactions, Exchange, Queues, etc
➡Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
Tuesday, 10 September 13

73. ➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣Transactions, Exchange, Queues, etc
➡Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
Tuesday, 10 September 13

74. Wire
Wire Protocol
Tuesday, 10 September 13

75. Wire
Wire Protocol
‣ Datagram specification
‣ Protocol header
‣ Payload format
‣ Data fields specification
‣ Content framing
‣ Heartbeat frames
‣ Channel multiplexing
‣ Visibility guarantee
‣ Channel closure
‣ Content synchronisation
‣ Content ordering
Tuesday, 10 September 13

76. Wire Protocol
Method Payloads
Tuesday, 10 September 13

77. Wire Protocol
Method Payloads
Tuesday, 10 September 13

78. Virtual Host
QueuesExchange
Binding
Summary
➡ Queue:
‣ critical_error_queue
➡ Exchange:
‣ errors_exchange
➡ Filter:
‣ *.error.critical
Message
Router
Publisher Consumer
Tuesday, 10 September 13

79. Revisiting event logging
A proposed solution
Tuesday, 10 September 13

80. Event Logging Revisited
Client
Client
Client
MongoDB
Tuesday, 10 September 13

81. Event Logging Revisited
Client
Client
Client
MQ
MQ
MQ
MongoDB
Tuesday, 10 September 13

82. MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
Tuesday, 10 September 13

83. MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
Tuesday, 10 September 13

84. MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
MQ
Tuesday, 10 September 13

85. MongoDB
Event Logging Revisited
MQ
Client
Client
Client
80
110
65
MQ
MQ
MQ
MQ
Tuesday, 10 September 13

86. Message
Exchange
‘events’
AMQP Service Model
Topic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
live_view_queue
event.user.*
event.user.share
*.*.view
Message
Message
event.staff.view
Message
Message
Remember this?
Tuesday, 10 September 13

87. MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
Reporting
SQL
Notification
Tuesday, 10 September 13

88. RabbitMQ
An AMQP Implementation
Tuesday, 10 September 13

89. Why RabbitMQ?
• Mature (born 2006)
• Acquired by SpringSource (now part of
VMWare) in 2010
• Used by companies like Nokia, Google,
Mozilla, vFabric, SecondLife, OpenStack
Tuesday, 10 September 13

90. Overview
• Written in Erlang/OTP
• Supports other protocols too - STOMP,
MQTT
• Many library supports - Python, PHP,
Ruby, .NET
• RESTful API is available
Tuesday, 10 September 13

91. Hands on Demo
Tuesday, 10 September 13

92. Q&A
Tuesday, 10 September 13

93. Coming soon...
RabbitMQ in Depth (sort of)
Tuesday, 10 September 13