Introduction to Message Queuing
raphael.cohn@stormmq.com
Agenda
A Real Application ! 20 min
Nibbles and Hands–On ! 60 min
Introduction to AMQP ! 20 min
Why Message Queuing? ! 25 m...
History of Message Queuing
Manual Telegraphy Machine Assisted Telegraphy
1920s1911 - 192019th Century 1900s
Telegrams sent...
History of Message Queuing
Telcos UseElectronic Telegraphy
1950s1940s 1960s
IBM System/360 with
BTAM & QTAM
Message Switch...
History of Message Queuing
l
FMCG & Utilities
1990s
MQSeries
phereMQ)
92
Corporates Large Websites YOU
Noughties Today
Sun...
The Integration Tag Soup
WS –File Transfer
!"#$%&'(#)#*#+%+
Message Queuing
SOAP
EMail
SMTP
FTP
RSYNC
NTFS
!','-
!).$%&'/$...
The Integration Tag Soup
WS –File Transfer
!"#$%&'(#)#*#+%+
Message Queuing
SOAP
EMail
SMTP
FTP
RSYNC
NTFS
!','-
!).$%&'/$...
The Integration Tag Soup
Message Queuing
Under Examination, though
Deployment
Intimate System Knowledge
Configuration
Admin
File Transfer
!"#$%&'(#)#*#+%+
Remote Pr...
Ideal: Message Queuing
Systems are “loosely-coupled”
Message
Queuing
Systems don’t
know each other
Why Use it: Loose Coupling
Billing
Catalogue
Shipping
S-a-a-S Inventory
How do we connect them, without one outage or syst...
Why Use it: Loose Coupling
Billing
Catalogue
Shipping
S-a-a-S Inventory
How do we connect them, without one outage or syst...
It should be ideal for the cloud
It should be ideal for the cloud
!
!
!
Until today, messaging is …
(ZeroMQ)
(SQS uses HTTP)
(MSMQ uses COM)
(memcached)
(Java JMS
uses
Source)
(MQSeries / ...
!
!
!
Until today, messaging is …
(ZeroMQ)
(SQS uses HTTP)
(MSMQ uses COM)
(memcached)
(Java JMS
uses
Source)
(MQSeries / ...
Why is it Hell?
You need more staff And moneyYou need bridge technology
Why is it Hell?
You need more staff And moneyYou need bridge technology
However, there’s a solution
A-MQPAdvanced Message Queue Protocol
However, there’s a solution
However, AMQP fixes this
A common wire-level binary
format and protocol
An explicit definition of a
server (aka broker)’s
se...
That is good …
“AMQP will be to Messaging what HTTP was to theWeb”
MRG
Clients run on any Platform Vendors are Interoperab...
That is good …
“AMQP will be to Messaging what HTTP was to theWeb”
MRG
Clients run on any Platform Vendors are Interoperab...
From Anywhere to Anywhere
From Anywhere to Anywhere
EddieVelez
Quick Recap
✓
The fifth male member
of the A-Team, Frankie,
was played by
Question
Message Queuing connects syst...
Agenda
Introduction to AMQP ! 20 min
Why Message Queuing? ! 25 min
Classic Architecture ! 10 min
A Real Application ! 20 m...
Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST
StormMQ
Messaging
Billing
!
Fire and Forget
Shipping
“Too many orders.The website is running like a dog.”
SQL Push SQL Pull
StormMQ
Messaging
Shipping
One-To-Many
“There’s a new widget in the Catalogue: Tell all the systems.”
Catalogue
Shipping
S-a-a-S Inventory
File Trans...
One-To-Many
“There’s a new widget in the Catalogue: Tell all the systems.”
Catalogue
Shipping
S-a-a-S Inventory
StormMQ
Me...
StormMQ
Messaging
Round-Robin
Billing
“How do we easily scale a massive batch job like Billing?”
EMail BillGenerate
StormMQ
Messaging
Publish-Subscribe (“Topics”)
“Shipments Sent, Delivered and Received”
Shipping
S-a-a-S Inventory
Billing...
22
Web Sockets Architecture
Copyright © 2010, Kaazing Corporation,. All rights reserved.
Agenda
Introduction to AMQP ! 20 min
Why Message Queuing? ! 25 min
Classic Architecture ! 10 min
A Real Application ! 20 m...
Messaging: Which Jargon?
EMail
(SMTP, POP3, IMAP)
VoIP
(VoiceMail, XMPP)
Texting
(SMS)
Instant Messaging
(ICQ, MSN, Jabber...
Essential Terms
Message Queue What’s a Message?
Envelope
Payload
Header
Essential Terms
*Strictly speaking
a receiver polls for messages
a consumer has messages pushed
≣ ≊Send Publish Enqueue
≣ ...
Connection
AMQP Client AMQP ServerTCP / IP Network
Connection
Virtual Host
Connections and Channels
TLS
“Shielding”
Channe...
Basic AMQP
Basic AMQP: Connections
Open a Connection to aVirtual Host
Open a Channel
Send a Message
Receive a Message
Clos...
Exchanges Route Messages
Exchanges route Messages to Message Queues
You send messages to an exchange, not a message queue
...
How do Exchanges Route?
Exchange
MQ
A
The Exchange finds a Binding
matching the Routing Key
#
A Binding connects a Routing ...
Types of Exchange
More rarely used exchange types include amq.headers and extensions
Message Queue Name
direct
Like a ‘Map...
And there is more!
Transactions
More Message
Properties
Custom Message
Properties
Immediate Delivery
Additional Exchange
T...
Agenda
Introduction to AMQP ! 20 min
Why Message Queuing? ! 25 min
Classic Architecture ! 10 min
A Real Application ! 20 m...
The Problem
Intrusion Detection
Lots of Logs
Firewall Monitoring
Performance & Failure
What’s needed
Live Debugging Live Logging Live Statistics
Audit ComplianceStatistics Mining SMS Alerts EMail Alerts
It must not get in the way
Add Hardware
Config
MaintenanceNo Extra Admins
Change Software
Design of the Clients
intrusion
api.daemon.info.charon api.memory api.daemon.sshd api.warn
checksstatisticslog
Content-Typ...
checks
Handling Log Messages
api.*.*.warn
Queue
#
Queue
Timestamp
No Acks
QoS Prefetch
WebSockets
log
MessageId
Routing Ke...
From Checks to Alerts
#
Queue
emergency
Queue
#
Queue
checks
alerts
alerts
Contact Details
not in message
Message Aggregat...
And debugging ?
A-MQPAdvanced Message Queue Protocol
And debugging ?
Upcoming SlideShare
Loading in …5
×

StormMQ Introduction to AMQP, Dublin

2,464 views

Published on

StormMQ Introduction to AMQP, Dublin, in conjunction with IrishDev.com, 16th November 2010.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,464
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
46
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

StormMQ Introduction to AMQP, Dublin

  1. 1. Introduction to Message Queuing raphael.cohn@stormmq.com
  2. 2. Agenda A Real Application ! 20 min Nibbles and Hands–On ! 60 min Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min
  3. 3. History of Message Queuing Manual Telegraphy Machine Assisted Telegraphy 1920s1911 - 192019th Century 1900s Telegrams sent using “Store and Forward” 1900 1930s
  4. 4. History of Message Queuing Telcos UseElectronic Telegraphy 1950s1940s 1960s IBM System/360 with BTAM & QTAM Message Switching 1964 First Electronic Mail Solutions 1965 Banking Users 1970s IBM TCAM which is the first true solution Retired 1990! 1971
  5. 5. History of Message Queuing l FMCG & Utilities 1990s MQSeries phereMQ) 92 Corporates Large Websites YOU Noughties Today Sun Release Java JMS, Reinvigorating Enterprise Messaging 2001 AMQP Working Group Formed by Investment Banks 2006 Cloud Enables and Drives StormMQ Adoption 2009
  6. 6. The Integration Tag Soup WS –File Transfer !"#$%&'(#)#*#+%+ Message Queuing SOAP EMail SMTP FTP RSYNC NTFS !','- !).$%&'/$.0%&1$%+ 234%5)6!%$7%$ SOA Message Orientated Middleware JMS MQ MQSeries AMQP Amazon SQS Tibco StormMQ RabbitMQ HTTP TCP/IP DCE / RPC CORBA DCOM REST RMI XML-RPC .NET Remoting Remote Procedure Call
  7. 7. The Integration Tag Soup WS –File Transfer !"#$%&'(#)#*#+%+ Message Queuing SOAP EMail SMTP FTP RSYNC NTFS !','- !).$%&'/$.0%&1$%+ 234%5)6!%$7%$ SOA Message Orientated Middleware JMS MQ MQSeries AMQP Amazon SQS Tibco StormMQ RabbitMQ HTTP TCP/IP DCE / RPC CORBA DCOM REST RMI XML-RPC .NET Remoting Remote Procedure Call
  8. 8. The Integration Tag Soup Message Queuing
  9. 9. Under Examination, though Deployment Intimate System Knowledge Configuration Admin File Transfer !"#$%&'(#)#*#+%+ Remote Procedure Call Don’tWork in the Cloud Scaling
  10. 10. Ideal: Message Queuing Systems are “loosely-coupled” Message Queuing Systems don’t know each other
  11. 11. Why Use it: Loose Coupling Billing Catalogue Shipping S-a-a-S Inventory How do we connect them, without one outage or system change taking everything down like a pack of cards?
  12. 12. Why Use it: Loose Coupling Billing Catalogue Shipping S-a-a-S Inventory How do we connect them, without one outage or system change taking everything down like a pack of cards? Message Queuing lets Systems and Components exchange data, events, commands and actions with one another with no explicit knowledge or need for them to be online
  13. 13. It should be ideal for the cloud
  14. 14. It should be ideal for the cloud
  15. 15. ! ! ! Until today, messaging is … (ZeroMQ) (SQS uses HTTP) (MSMQ uses COM) (memcached) (Java JMS uses Source) (MQSeries / WebsphereMQ) (Talarian, Rendezvous, etc) Platform Restricted BespokeProprietary
  16. 16. ! ! ! Until today, messaging is … (ZeroMQ) (SQS uses HTTP) (MSMQ uses COM) (memcached) (Java JMS uses Source) (MQSeries / WebsphereMQ) (Talarian, Rendezvous, etc)
  17. 17. Why is it Hell? You need more staff And moneyYou need bridge technology
  18. 18. Why is it Hell? You need more staff And moneyYou need bridge technology
  19. 19. However, there’s a solution
  20. 20. A-MQPAdvanced Message Queue Protocol However, there’s a solution
  21. 21. However, AMQP fixes this A common wire-level binary format and protocol An explicit definition of a server (aka broker)’s semantics Open Means Interoperable
  22. 22. That is good … “AMQP will be to Messaging what HTTP was to theWeb” MRG Clients run on any Platform Vendors are Interoperable
  23. 23. That is good … “AMQP will be to Messaging what HTTP was to theWeb” MRG Clients run on any Platform Vendors are Interoperable 64K
  24. 24. From Anywhere to Anywhere
  25. 25. From Anywhere to Anywhere
  26. 26. EddieVelez Quick Recap ✓ The fifth male member of the A-Team, Frankie, was played by Question Message Queuing connects systems and components. Is it ideal for the cloud? Does Loose-Coupling make individual systems more likely to suffer outages? Yes No AMQP is Open. This makes it suitable for programming in C and Javascript? ! ✓
  27. 27. Agenda Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min A Real Application ! 20 min Pizza and Hands–On ! 60 min
  28. 28. Store and Forward Billing “Could you send me a duplicate of my bill please” REST StormMQ Messaging Billing !
  29. 29. Fire and Forget Shipping “Too many orders.The website is running like a dog.” SQL Push SQL Pull StormMQ Messaging Shipping
  30. 30. One-To-Many “There’s a new widget in the Catalogue: Tell all the systems.” Catalogue Shipping S-a-a-S Inventory File Transfer ETL
  31. 31. One-To-Many “There’s a new widget in the Catalogue: Tell all the systems.” Catalogue Shipping S-a-a-S Inventory StormMQ Messaging
  32. 32. StormMQ Messaging Round-Robin Billing “How do we easily scale a massive batch job like Billing?” EMail BillGenerate
  33. 33. StormMQ Messaging Publish-Subscribe (“Topics”) “Shipments Sent, Delivered and Received” Shipping S-a-a-S Inventory Billing RIASent Sent or Returned
  34. 34. 22 Web Sockets Architecture Copyright © 2010, Kaazing Corporation,. All rights reserved.
  35. 35. Agenda Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min A Real Application ! 20 min Nibbles and Hands–On ! 60 min
  36. 36. Messaging: Which Jargon? EMail (SMTP, POP3, IMAP) VoIP (VoiceMail, XMPP) Texting (SMS) Instant Messaging (ICQ, MSN, Jabber) Twitter Enterprise Service Bus (ESB) Dynamic OO Languages (eg Ruby) Message Queuing (MQ) Message Queuing (MQ) What do we mean by Messaging?
  37. 37. Essential Terms Message Queue What’s a Message? Envelope Payload Header
  38. 38. Essential Terms *Strictly speaking a receiver polls for messages a consumer has messages pushed ≣ ≊Send Publish Enqueue ≣ ≊ DequeueConsume*Receive
  39. 39. Connection AMQP Client AMQP ServerTCP / IP Network Connection Virtual Host Connections and Channels TLS “Shielding” Channels Each Channel is Independent: Effectively, aVirtual Connection
  40. 40. Basic AMQP Basic AMQP: Connections Open a Connection to aVirtual Host Open a Channel Send a Message Receive a Message Close Channel Close Connection You only need one channel! " # $ % & '
  41. 41. Exchanges Route Messages Exchanges route Messages to Message Queues You send messages to an exchange, not a message queue Exchange MQ A MQ B
  42. 42. How do Exchanges Route? Exchange MQ A The Exchange finds a Binding matching the Routing Key # A Binding connects a Routing Key to one or more Message Queues $ Every sent message has a Routing Key " The Exchange delivers the Message to the Message Queue % A message queue can be bound more than once to one or more exchanges ( “string”
  43. 43. Types of Exchange More rarely used exchange types include amq.headers and extensions Message Queue Name direct Like a ‘Map’:All MQs bound with the routing key receive copies of the message “” (blank) amq.direct fanout Empty String All MQs bound to the Exchange receive copies of the message amq.fanout topic Dotted Bindings use globbing expressions (wildcards) to route messages to MQs amq.topic A message queue can be bound more than once to an exchange; A message queue can be bound to more than one exchange But a message queue will only receive a message sent once Routing Key Routing Behaviour Default Definitions Point – to – Point One – to – Many Publish – SubscribeTypical Use
  44. 44. And there is more! Transactions More Message Properties Custom Message Properties Immediate Delivery Additional Exchange Types Auto Deletion Fine-Grained User Permissions Queue Purging QoS
  45. 45. Agenda Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min A Real Application ! 20 min Nibbles and Hands–On ! 60 min
  46. 46. The Problem Intrusion Detection Lots of Logs Firewall Monitoring Performance & Failure
  47. 47. What’s needed Live Debugging Live Logging Live Statistics Audit ComplianceStatistics Mining SMS Alerts EMail Alerts
  48. 48. It must not get in the way Add Hardware Config MaintenanceNo Extra Admins Change Software
  49. 49. Design of the Clients intrusion api.daemon.info.charon api.memory api.daemon.sshd api.warn checksstatisticslog Content-Type: application/json; charset=utf-8 Timestamp: 2010-11-03T16:45:56.981Z Priority: Low / High Type: statistics_2011.10.01.1203 Expiry: 5 mins api.firewall.warn.in.portscan.tcp
  50. 50. checks Handling Log Messages api.*.*.warn Queue # Queue Timestamp No Acks QoS Prefetch WebSockets log MessageId Routing Key *.warn Queue checks Statistics A rchitecture is the Sam e
  51. 51. From Checks to Alerts # Queue emergency Queue # Queue checks alerts alerts Contact Details not in message Message Aggregation / Suppression by alerts Uses Message Priority
  52. 52. And debugging ?
  53. 53. A-MQPAdvanced Message Queue Protocol And debugging ?

×