Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

StormMQ Introduction to AMQP, Dublin

2,627 views

Published on

StormMQ Introduction to AMQP, Dublin, in conjunction with IrishDev.com, 16th November 2010.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

StormMQ Introduction to AMQP, Dublin

  1. 1. Introduction to Message Queuing raphael.cohn@stormmq.com
  2. 2. Agenda A Real Application ! 20 min Nibbles and Hands–On ! 60 min Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min
  3. 3. History of Message Queuing Manual Telegraphy Machine Assisted Telegraphy 1920s1911 - 192019th Century 1900s Telegrams sent using “Store and Forward” 1900 1930s
  4. 4. History of Message Queuing Telcos UseElectronic Telegraphy 1950s1940s 1960s IBM System/360 with BTAM & QTAM Message Switching 1964 First Electronic Mail Solutions 1965 Banking Users 1970s IBM TCAM which is the first true solution Retired 1990! 1971
  5. 5. History of Message Queuing l FMCG & Utilities 1990s MQSeries phereMQ) 92 Corporates Large Websites YOU Noughties Today Sun Release Java JMS, Reinvigorating Enterprise Messaging 2001 AMQP Working Group Formed by Investment Banks 2006 Cloud Enables and Drives StormMQ Adoption 2009
  6. 6. The Integration Tag Soup WS –File Transfer !"#$%&'(#)#*#+%+ Message Queuing SOAP EMail SMTP FTP RSYNC NTFS !','- !).$%&'/$.0%&1$%+ 234%5)6!%$7%$ SOA Message Orientated Middleware JMS MQ MQSeries AMQP Amazon SQS Tibco StormMQ RabbitMQ HTTP TCP/IP DCE / RPC CORBA DCOM REST RMI XML-RPC .NET Remoting Remote Procedure Call
  7. 7. The Integration Tag Soup WS –File Transfer !"#$%&'(#)#*#+%+ Message Queuing SOAP EMail SMTP FTP RSYNC NTFS !','- !).$%&'/$.0%&1$%+ 234%5)6!%$7%$ SOA Message Orientated Middleware JMS MQ MQSeries AMQP Amazon SQS Tibco StormMQ RabbitMQ HTTP TCP/IP DCE / RPC CORBA DCOM REST RMI XML-RPC .NET Remoting Remote Procedure Call
  8. 8. The Integration Tag Soup Message Queuing
  9. 9. Under Examination, though Deployment Intimate System Knowledge Configuration Admin File Transfer !"#$%&'(#)#*#+%+ Remote Procedure Call Don’tWork in the Cloud Scaling
  10. 10. Ideal: Message Queuing Systems are “loosely-coupled” Message Queuing Systems don’t know each other
  11. 11. Why Use it: Loose Coupling Billing Catalogue Shipping S-a-a-S Inventory How do we connect them, without one outage or system change taking everything down like a pack of cards?
  12. 12. Why Use it: Loose Coupling Billing Catalogue Shipping S-a-a-S Inventory How do we connect them, without one outage or system change taking everything down like a pack of cards? Message Queuing lets Systems and Components exchange data, events, commands and actions with one another with no explicit knowledge or need for them to be online
  13. 13. It should be ideal for the cloud
  14. 14. It should be ideal for the cloud
  15. 15. ! ! ! Until today, messaging is … (ZeroMQ) (SQS uses HTTP) (MSMQ uses COM) (memcached) (Java JMS uses Source) (MQSeries / WebsphereMQ) (Talarian, Rendezvous, etc) Platform Restricted BespokeProprietary
  16. 16. ! ! ! Until today, messaging is … (ZeroMQ) (SQS uses HTTP) (MSMQ uses COM) (memcached) (Java JMS uses Source) (MQSeries / WebsphereMQ) (Talarian, Rendezvous, etc)
  17. 17. Why is it Hell? You need more staff And moneyYou need bridge technology
  18. 18. Why is it Hell? You need more staff And moneyYou need bridge technology
  19. 19. However, there’s a solution
  20. 20. A-MQPAdvanced Message Queue Protocol However, there’s a solution
  21. 21. However, AMQP fixes this A common wire-level binary format and protocol An explicit definition of a server (aka broker)’s semantics Open Means Interoperable
  22. 22. That is good … “AMQP will be to Messaging what HTTP was to theWeb” MRG Clients run on any Platform Vendors are Interoperable
  23. 23. That is good … “AMQP will be to Messaging what HTTP was to theWeb” MRG Clients run on any Platform Vendors are Interoperable 64K
  24. 24. From Anywhere to Anywhere
  25. 25. From Anywhere to Anywhere
  26. 26. EddieVelez Quick Recap ✓ The fifth male member of the A-Team, Frankie, was played by Question Message Queuing connects systems and components. Is it ideal for the cloud? Does Loose-Coupling make individual systems more likely to suffer outages? Yes No AMQP is Open. This makes it suitable for programming in C and Javascript? ! ✓
  27. 27. Agenda Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min A Real Application ! 20 min Pizza and Hands–On ! 60 min
  28. 28. Store and Forward Billing “Could you send me a duplicate of my bill please” REST StormMQ Messaging Billing !
  29. 29. Fire and Forget Shipping “Too many orders.The website is running like a dog.” SQL Push SQL Pull StormMQ Messaging Shipping
  30. 30. One-To-Many “There’s a new widget in the Catalogue: Tell all the systems.” Catalogue Shipping S-a-a-S Inventory File Transfer ETL
  31. 31. One-To-Many “There’s a new widget in the Catalogue: Tell all the systems.” Catalogue Shipping S-a-a-S Inventory StormMQ Messaging
  32. 32. StormMQ Messaging Round-Robin Billing “How do we easily scale a massive batch job like Billing?” EMail BillGenerate
  33. 33. StormMQ Messaging Publish-Subscribe (“Topics”) “Shipments Sent, Delivered and Received” Shipping S-a-a-S Inventory Billing RIASent Sent or Returned
  34. 34. 22 Web Sockets Architecture Copyright © 2010, Kaazing Corporation,. All rights reserved.
  35. 35. Agenda Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min A Real Application ! 20 min Nibbles and Hands–On ! 60 min
  36. 36. Messaging: Which Jargon? EMail (SMTP, POP3, IMAP) VoIP (VoiceMail, XMPP) Texting (SMS) Instant Messaging (ICQ, MSN, Jabber) Twitter Enterprise Service Bus (ESB) Dynamic OO Languages (eg Ruby) Message Queuing (MQ) Message Queuing (MQ) What do we mean by Messaging?
  37. 37. Essential Terms Message Queue What’s a Message? Envelope Payload Header
  38. 38. Essential Terms *Strictly speaking a receiver polls for messages a consumer has messages pushed ≣ ≊Send Publish Enqueue ≣ ≊ DequeueConsume*Receive
  39. 39. Connection AMQP Client AMQP ServerTCP / IP Network Connection Virtual Host Connections and Channels TLS “Shielding” Channels Each Channel is Independent: Effectively, aVirtual Connection
  40. 40. Basic AMQP Basic AMQP: Connections Open a Connection to aVirtual Host Open a Channel Send a Message Receive a Message Close Channel Close Connection You only need one channel! " # $ % & '
  41. 41. Exchanges Route Messages Exchanges route Messages to Message Queues You send messages to an exchange, not a message queue Exchange MQ A MQ B
  42. 42. How do Exchanges Route? Exchange MQ A The Exchange finds a Binding matching the Routing Key # A Binding connects a Routing Key to one or more Message Queues $ Every sent message has a Routing Key " The Exchange delivers the Message to the Message Queue % A message queue can be bound more than once to one or more exchanges ( “string”
  43. 43. Types of Exchange More rarely used exchange types include amq.headers and extensions Message Queue Name direct Like a ‘Map’:All MQs bound with the routing key receive copies of the message “” (blank) amq.direct fanout Empty String All MQs bound to the Exchange receive copies of the message amq.fanout topic Dotted Bindings use globbing expressions (wildcards) to route messages to MQs amq.topic A message queue can be bound more than once to an exchange; A message queue can be bound to more than one exchange But a message queue will only receive a message sent once Routing Key Routing Behaviour Default Definitions Point – to – Point One – to – Many Publish – SubscribeTypical Use
  44. 44. And there is more! Transactions More Message Properties Custom Message Properties Immediate Delivery Additional Exchange Types Auto Deletion Fine-Grained User Permissions Queue Purging QoS
  45. 45. Agenda Introduction to AMQP ! 20 min Why Message Queuing? ! 25 min Classic Architecture ! 10 min A Real Application ! 20 min Nibbles and Hands–On ! 60 min
  46. 46. The Problem Intrusion Detection Lots of Logs Firewall Monitoring Performance & Failure
  47. 47. What’s needed Live Debugging Live Logging Live Statistics Audit ComplianceStatistics Mining SMS Alerts EMail Alerts
  48. 48. It must not get in the way Add Hardware Config MaintenanceNo Extra Admins Change Software
  49. 49. Design of the Clients intrusion api.daemon.info.charon api.memory api.daemon.sshd api.warn checksstatisticslog Content-Type: application/json; charset=utf-8 Timestamp: 2010-11-03T16:45:56.981Z Priority: Low / High Type: statistics_2011.10.01.1203 Expiry: 5 mins api.firewall.warn.in.portscan.tcp
  50. 50. checks Handling Log Messages api.*.*.warn Queue # Queue Timestamp No Acks QoS Prefetch WebSockets log MessageId Routing Key *.warn Queue checks Statistics A rchitecture is the Sam e
  51. 51. From Checks to Alerts # Queue emergency Queue # Queue checks alerts alerts Contact Details not in message Message Aggregation / Suppression by alerts Uses Message Priority
  52. 52. And debugging ?
  53. 53. A-MQPAdvanced Message Queue Protocol And debugging ?

×